I have some large zip files which contain files that are over 4GB, in the current case 16GB, when extracted. I can extract the files no problem using Winzip 8.1 and WinRar 3.60, but when I try using DotNetZip I am getting the follwoing error once 4GB has been extracted:
\r\nCRC error: the file being extracted appears to be corrupted. Expected 0x33384163, Actual 0x7C445495
\r\nI am using a Script Task in SSIS and the code used is as follows:
\r\nPrivate Sub Decompress()
\r\n\r\n
Using m_ZIP As ZipFile = ZipFile.Read(Me.ZIPFileName)
\r\nDim m_ZipEntry As ZipEntry
\r\nFor Each m_ZipEntry In m_ZIP
\r\nTry
\r\nm_ZipEntry.Extract(Path.GetDirectoryName(Me.ZIPFileName), ExtractExistingFileAction.OverwriteSilently)
\r\nCatch ex As Exception
\r\nMessageBox.Show(ex.Message)
\r\nEnd Try
\r\nNext
\r\nEnd Using
\r\nEnd Sub
\r\n\r\n
\r\n
Could you let me know if there is something I am doing wrong or if it is possible to perform this extraction? The zip file is supplied by a 3rd party and having had other dealing with them I doubt I will be able to get information from them as to how it was created.
\r\nThanks in advance
\r\n", "PostedDate": "2009-07-22T05:27:50.117-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "214771", "ThreadId": "63188", "Html": "
Your code looks fine.
\r\nWhat version of DotNetZip are you using? Earlier versions had some problems with very large ZIP files.
\r\nHow do you know the problem happens with 4gb extracted - how can you tell?
\r\n", "PostedDate": "2009-07-22T07:20:28.123-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "214988", "ThreadId": "63188", "Html": "
Thanks for the confirmation of the code.
\r\nThe version I am using is the latest version downloaded from here last week, 1.8 (not sure of the exact release as it is on my work pc and I am currently at home).
\r\nI can see it arrives to 4GB (or thereabouts) as I have the folder being extracted to visible, and updating, during the extraction.
", "PostedDate": "2009-07-22T14:57:49.033-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215022", "ThreadId": "63188", "Html": "Hmmmm, ok. I will run some tests.
", "PostedDate": "2009-07-22T16:05:37.317-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215119", "ThreadId": "63188", "Html": "\r\n\r\nCheeso
\r\n\r\n\r\n\r\n
Just to confirm the version I am using is 1.8.4.5.
\r\n\r\n\r\n\r\n
Also the attached screenshot might help\r\nyou. As you can see the file (Datoscsmo_ufd_20090627.txt) in the zip file shows\r\nas being 4GB, but when extracted it is 16GB. Might this be part of the problem?
\r\n\r\n\r\n\r\n
Regards
\r\n\r\nYes - the size mismatch you mentioned could be part of the problem.
\r\nWhere's the screen shot? is the screen shot produced by DotNetZip? Or WinZip?
\r\nIs the zipfile produced by DotNetZip, or some other tool?
\r\nCan you restate the problem for me. Originally the problem was, DotNetZip was throwing when you were extracting, and when the app had extracted more than 4gb on a single file. Is that still the problem?
\r\n", "PostedDate": "2009-07-23T04:01:13.6-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215227", "ThreadId": "63188", "Html": "
I attached a screen shot in my previous e-mail reply. I don't seem to be able to paste into here so if you can't get it from the previous message let me know how I can get it to you if it is necessary. It was taken from Winzip, although I get the same results from WinRar and stepping through the code DotNetZip only recognizes the file size as 4,048,964 kb in size compared to the 16,536,980 KB of the unzipped file in Windows Explorer.
\r\nI have no idea as to how the zip file is produced as it is supplied by a 3rd party and, as mentioned before, they have been singularly unhelpful in helping us out with this data previously. Saying that I would be pretty certain that the file is created with an application such as WinZip.
\r\nThe original problem is that DotNetZip throws the error "CRC error: the file being extracted appears to be corrupted. Expected 0x33384163, Actual 0x7C445495" on extraction upon reaching 4GB. At present I only have one file of this size and if I catch the error and move on to the next file within the zip file that extracts okay.
\r\nThanks.
\r\n\r\n
\r\n
\r\n
\r\n
ok, clear. There's a gateway that posts the email that you send, to the forums. When I respond to you I am using the forums on codeplex.com, not sending/receiving email. In the forums UI, it is just HTML and you can embed a picture. It needs to be available somewhere. I often use tinypic.com for the purpose. The steps are: take screenshot, save screenshot, upload to tinypic, get URL, edit HTML on codeplex forum, paste URL to picture.
\r\nAre you saying that WinRar and WinZip can successfully extract the file? in the screenshot is it clear whether the 4gb size is referring to uncompressed or compressed size? Which is it?
\r\nWhen reading or extracting a zip archive, DotNetZip first looks in the zipfile for the metadata for the entry. This metadata includes the uncompressed size, the compressed size, the filename, the compression method, the CRC, a comment, and other stuff. If the app then calls ZipEntry.Extract(), dotnetzip will decompress the bytes in the zip file, until reaching the "compressed size" number of bytes. It then compares the CRC in the metadata with the resulting CRC of the uncompressed data.
\r\nNormally this all works nicely. But what appears to be happening in your case is the CRC check is failing. This can happen if
\r\nIf the problem is that there is not enough compressed data in the file, you should get a different error - a length check should fail. Supposing the original compressed size was 4.8gb, and the zip file had only 4.2gb, the decompress will throw when it runs out of data. If there is a problem with corruption of the compressed data in the zip file, you will get yet another different error, a ZLIB protocol error.
\r\nNot all unzip tools verify the CRC of an entry on extract, but I believe WinZip does.
\r\nIt sounds to me that you may have the latter case - the compressed size may not be valid.
\r\n", "PostedDate": "2009-07-23T04:37:18.233-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215245", "ThreadId": "63188", "Html": "\r\n\r\n
Thanks for your quick reply. I will keep\r\nnotes of how to attach an image for future reference
\r\n\r\n\r\n\r\n
As you say it seems to be the metadata compressed\r\nsize, or CRC, that is causing the problem. We download a file from the 3rd\r\nparty on a monthly basis and looking at the old files the compressed size\r\nstored in the metadata never equals the size of the extracted file, so something\r\nin the way that the zip file is produced is not working correctly but although\r\nI will get in touch regarding the problem it is causing us I don’t expect\r\na reply from them.
\r\n\r\n\r\n\r\n
It looks like for this file only (your\r\nlibrary works really well for all other zip files we receive) I’ll have\r\nto look at an alternative method of extraction (probably by directly calling\r\nWinzip from my app)
\r\n\r\n\r\n\r\n
Kind\r\nregards
\r\n\r\n\r\n\r\n
??
\r\nJust one thing - you said the "compressed size stored in the metadata never equals the size of the extracted file" . In a zip, the uncompressed size stored in the metadata should equal the size of the extracted file.
\r\nAs for alternative methods of extraction, if you use DotNetZip and open a read stream on the entry, the CRC check is not done by the library. The app is expected to do that itself. Check out the doc for ZipEntry.OpenReader(). That might work for you if the only problem is a CRC check.
\r\n\r\n
", "PostedDate": "2009-07-23T05:29:14.67-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215673", "ThreadId": "63188", "Html": "\r\n\r\n
Sorry I got my Uncompressed and Compressed\r\nmixed up. The compressed size does show correctly, the uncompressed doesn’t\r\n(this depends a bit on how you class correctly, as it does show the same\r\nuncompressed size as displayed when the zip file is opened in Winzip/Winrar,\r\nbut this is not the actual size of the final extracted file). Either way it\r\nlooks as though it is more a problem with the actual file than your library.
\r\n\r\n\r\n\r\n
I will try the OpenReader method if I get a\r\nchance.
\r\n\r\n\r\n\r\n
Thanks for all you help.
\r\n\r\n\r\n\r\n
if winzip, winrar and DotNetZip all agree on the uncompressed size of an entry, and that size does not match the size of the actual file used to produce the entry, then it seems like there was an error during the creation of that zip file.
\r\n", "PostedDate": "2009-07-24T04:14:11.947-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225337", "ThreadId": "63188", "Html": "
Cheeso
\r\nI have tried using the OpenReader method and have pasted my code below, but I am still unable to fully extract the large files from the zip file. Further below I have added a screen shot which shows the zip file and the extracted file.
\r\nThe file that is causing me problems is Datoscsmo_ufd_20090801.txt. As you can see the uncompressed size of the file within the zp file is 1,142,685,138 bytes, which is the final value of totalBytesRead when the code is completed. However extracting the file using Winzip leaves me with a file of over 13GB (14,027,587,026 bytes according to the file properties).
\r\nThis may be something I am unable to resolve using the DotNetZip library, but I would welcome any comments you may have.
\r\nThanks
\r\n\r\n
\r\n
Using zip As New ZipFile(Me.ZIPFileName)\r\n Dim e1 As ZipEntry\r\n For Each e1 In zip\r\n sNombreFichero = Path.Combine(Path.GetDirectoryName(Me.ZIPFileName), Path.GetFileName(e1.FileName))\r\n Using swWrite As Stream = File.Create(sNombreFichero)\r\n Using s As Ionic.Zlib.CrcCalculatorStream = e1.OpenReader\r\n Dim n As Integer\r\n Dim buffer As Byte() = New Byte(4096) {}\r\n Dim totalBytesRead As Integer = 0\r\n Do\r\n n = s.Read(buffer, 0, buffer.Length)\r\n swWrite.Write(buffer, 0, buffer.Length)\r\n totalBytesRead = (totalBytesRead + n)\r\n Loop While (n > 0)\r\n End Using\r\n End Using\r\n Next\r\n End Using\r\n\r\n
![\"\"](\"http://i27.tinypic.com/23ly2bo.jpg\")
use
\r\nDim totalBytesRead as Long
\r\ni.e. use a 64 bits integer
\r\n", "PostedDate": "2009-08-19T04:36:46.467-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225393", "ThreadId": "63188", "Html": "
Gilles
\r\nGood spot that, but at the moment that is not causing the error. Even using a Dim totalBytesRead as Long, the final extracted file size with DotNetZip is 1,142,685,138 bytes compared to over 13GB when extracted using Winzip or Winrar. Then problem seems to be that DotNetZip is only expecting that amount of data to be extracted and stops at that point, despite the fact there must be more data within the zip file.
\r\n\r\n
\r\n
", "PostedDate": "2009-08-19T05:51:38.597-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225419", "ThreadId": "63188", "Html": "
When the file is uncompressed (with the inflate algorithm), the end of the file can be detected from the data itself (when Read returns 0 bytes). If the library stops when having reading uncompressedsize bytes, the CRC is probably wrong, if there is actually more data.
\r\nIt let me think that .zip files can conversely contain holes with data not described by entries; it is not pleasant from a security point of view. But this is how self extracting archives work, as I understand.
", "PostedDate": "2009-08-19T06:53:27.34-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225467", "ThreadId": "63188", "Html": "Lew, when you do
\r\nswWrite.Write(buffer, 0, buffer.Length)\r\n\r\n
you probably want, instead,
\r\nswWrite.Write(buffer, 0, n)\r\n\r\n
This is because the preceding stream read can be partially filled, and only the first n bytes are valid.
\r\nSecondly, what WinZip is showing and doing seems confusing to me. It reports an uncompressed size of 1gb, but then when you unpack, it generates a file of size 13gb. IS that right? Am I understanding correctly? If so, that indicates a problem in WinZip, to me.
\r\nWhat does DotNetZip tell you the uncompressed size is? You seem to be telling me that when extracting with DotNetZip, you get a file of size X, where X is the UncompressedSize of the ZipEntry. This seems to be correct behavior on the part of DotNetZip, unless I am misunderstanding something.
\r\nThe surprise is that WinZip's reported UncompressedSize does not match the actual Uncompressed size of a file, when extracted by WinZip. This seems to be a problem in WinZip, if I am understanding the situation correctly.
\r\nGilles: yes, the zip file can contain arbitrary data, and there is no requirement that the data for each zip entry immediately follows the prevvious. It is not only self-extracting archives that work this way. And yes, this is a way that viruses have been delivered. Scanning engines must check zip files for malicious payloads, because of this.
", "PostedDate": "2009-08-19T08:26:12.727-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225743", "ThreadId": "63188", "Html": "Cheeso
\r\nThanks for you reply, but I still get the same result with DotNetZip whereby the size of the extracted file is equal to the UncompressedSize of the ZipEntry.
\r\nWith both Winzip and Winrar when viewing the zip file within the respective application shows that the zipped file has a size of 1GB, yet both applications extract a file of size 13GB.
\r\nBasically your understanding of my situation is correct. Unfortunately, as mentioned before the file in question is one we receive on a monthly basis from a 3rd party who have, up to now, been wholly uncooperative when we have requested help. I have been able to find a work around using something other than DotNetZip but for consistency was hoping to use DotNetZip for these files as well.
\r\nThanks for your help
\r\n", "PostedDate": "2009-08-20T00:11:56.7-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "225897", "ThreadId": "63188", "Html": "
Lewshouse, I don't understand the situation I think.
\r\nYou're telling me that DotNetZip is creating a file with THE EXPECTED SIZE. The UncompressedSize IS the expected size of the file after de-compression. And DotNetZip is producing a file of that size. And you're telling me, you don't want that. This is what I don't understand.
\r\nIt seems to be correct behavior on the part of DotNetZip. I'm not trying to be difficult here. I understand that WinZip and WinRar both produce a file with 13gb size, but it does not seem to be correct behavior. Even WinZip shows the UncompressedSize as 1.1gb, so why should it produce a file of size 13gb?
\r\nIf I have a file that is 1000 bytes normally, and I zip it up into 400 bytes, then I get an entry that has UncompressedSize = 1000, and CompressedSize = 400. When I extract that file I expect it to be 1000 bytes in the filesystem. This is what is happening with you, though the numbers are higher. DotNetZip is producing a file upon extraction that has a size equal to the UncompressedSize. This seems like correct behavior.
\r\nI don't know where the 13gb is coming from.
\r\nIf you're telling me that the original file is or was 13gb, and when zipped, the zipfile says the UncompressedSize was around 1.1gb (as I think you mentioned), then it seems like the zip file is broken. And if the zipfile is broken, there needs to be a change by whoever is producing it. And yes, I understand they're not responsive, but ... (shrug).
\r\nI also understand that its frustrating that WinZip produces a 13gb file. But like I said, it seems to be broken behavior. Something isn't right.
\r\nSometimes a zipfile can be internally inconsistent. There are two places to store the metadata for each entry - the name, the uncompressed and compressed sizes, and so on. One is called the "entry header", and the other is called the "Central directory". In some cases the entry header and the central directory get out of sync. There is a CheckZip method on the ZipFile class that can correct these problems, in some cases. You might try running your zipfile through that method. It may help.
\r\n\r\n
\r\n
\r\n
", "PostedDate": "2009-08-20T08:06:15.5-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "226186", "ThreadId": "63188", "Html": "\r\n\r\n
Cheeso
\r\n\r\n\r\n\r\n
Thanks for your replies and attempts to\r\nhelp in this matter. Unfortunately although DotNetZip functions correctly and\r\nextracts a file of the size expected from the UncompressedSize property I need\r\nto extract the whole, in this case, 13GB, and need to be able to do it programmatically\r\nwithout recourse to directly calling an application such as WinZip.
\r\n\r\n\r\n\r\n
Since I have started this job I have\r\nreceived four versions of the zip file from the 3rd party and in\r\neach case the file extracted by WinZip/WinRar is significantly larger than the uncompressed\r\nsize they show, so something is obviously occurring during the creation of the\r\nzip file, but as I cannot get any response from the supplier (a large Spanish\r\nelectricity supplier) am unable to find out what this is.
\r\n\r\n\r\n\r\n
For now I will have to look at alternatives\r\nwhich is a shame as if I change from DotNetZip for this supplier I will have to\r\ndo so for the other 4 we use and therefore re-write code that functions\r\nperfectly.
\r\n\r\n\r\n\r\n
\r\n\r\n
\r\n\r\n
\r\n\r\n
Cheeso,
\r\nwould it make sense to read the deflated stream until its end (read returning 0) , not taking into account the stored uncompressed size?
\r\nIt seems other utilities are doing that way, and it would not modify the behavior for correct files.
", "PostedDate": "2009-08-21T01:13:14.457-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "226207", "ThreadId": "63188", "Html": "GM - cannot really do that. The problem is, if I keep reading past the "uncompressed size" I am reading metadata in the zipfile for the next entry. It will break.
\r\nLH - what about trying the CheckZip method? Have you considered that? It's a simple API call. Worth a try.
", "PostedDate": "2009-08-21T01:44:05.55-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "226225", "ThreadId": "63188", "Html": "\r\n\r\nCheeso
\r\n\r\n\r\n\r\n
I’ve tried the CheckZip method but that returns true.
\r\n\r\n\r\n\r\n
Thanks
\r\n\r\n\r\n\r\n
Lewshouse
\r\n\r\nwell then I'm at a loss for how to help.
\r\nyou can't get me the zip file because of privacy concerns.
\r\nBy your description, DotNetZip appears to be working correctly, although I understand it is not working the same as WinZip or WinRar.
\r\nI don't know how to fix this?
\r\nGood luck.
\r\n", "PostedDate": "2009-08-21T08:52:20.513-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "226944", "ThreadId": "63188", "Html": "
Perhaps the compressedsize is smaller than the actual deflated data in entry, and Winzip continues inflating after compressedsize bytes have been read, until the inflate algorithm returns 0 bytes.
", "PostedDate": "2009-08-24T01:12:05.753-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "227048", "ThreadId": "63188", "Html": "if you can recompile the code from Cheeso, try the following change for OpenReader to see if the problem is acyually here:
\r\n<font size="2">\r\n
private Ionic.Zlib.CrcCalculatorStream InternalOpenReader(string password)
\r\n\r\n{
\r\n///...
\r\n// change\r\n
<font size="2" color="#0000ff"><font size="2" color="#0000ff">\r\n return
new Ionic.Zlib.CrcCalculatorStream((CompressionMethod == 0x08) ? new Ionic.Zlib.DeflateStream(input2, Ionic.Zlib.CompressionMode.Decompress, true): input2, _UncompressedSize);
\r\n// into
\r\n\r\n\r\n
<font size="2" color="#0000ff"><font size="2" color="#0000ff">\r\n return (CompressionMethod == 0x08)? new Ionic.Zlib.CrcCalculatorStream(new Ionic.Zlib.DeflateStream(input2, Ionic.Zlib.CompressionMode.Decompress, true)): new Ionic.Zlib.CrcCalculatorStream(input2,_UncompressedSize);
\r\n\r\n", "PostedDate": "2009-08-24T07:26:13.16-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "259537", "ThreadId": "63188", "Html": "
hi every time i extract any game or some big file it always used to show that the cabinet file require for the installation is crrupt it is problem with this pack or cd/dvd may be corrupt
\r\nand the file get easily be extracted on met friends pc kindly help me pls
", "PostedDate": "2009-11-22T05:56:46.743-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "259544", "ThreadId": "63188", "Html": "Sorry, I can't understand what you're asking.
\r\nAlso, I don't know anything about cabinet files. I think maybe you're in the wrong place.
\r\nIf you really are using DotNetZip, open a new discussion, ..pls.
", "PostedDate": "2009-11-22T06:39:39.877-08:00", "UserRole": null, "MarkedAsAnswerDate": null } ]