Hey,
\r\nI'm a little new to the zip process so bear with me. I had some code that was not compatible with large .zip files so I was looking around for something to handle it. Came across this and decided to try it. Unlike other packages this one would open and start extracting the files so thought problem solved. However, just before it completed, got an "Oversubscribed dynamic bit lengths tree" error.
\r\nThe .zip file was created by WinZip 12 and is slightly over 4.3 GB. The file extracts fine with WinZip but using the DotNetZip library it throws the foremention error raised from the zlib library. It always throws this error on the same file when going through it. I have looked at the file details in WinZip and it has the Zip64 markings so it should be a valid file for this size. However the file that is has issues with is the first file where the "relative offset of local header" exceeds 0xFFFFFFFF (got this for the WinZip details of the file). I tried skipping pass this file but every file after that throws the error as well.
\r\nIs there some setting I'm missing in the DotNetZip for Zip64 (already tried setting Zip64Option.Always)? Could it be the .zip file? Does anyone know the issue and/or solution to this error? Any help would be greatly appreciated. Thanks.
\r\n-Wayne
\r\n\r\n
", "PostedDate": "2009-07-23T18:33:07.477-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215583", "ThreadId": "63392", "Html": "
ah,,, hm, I gotta say, I think you're in trouble. "Oversubscribed dynamic bit lengths tree" is an error I cannot explain. It happens in the ZLIB layer, which I did not write. I also don't understand the error enough to explain it to you.
\r\nIf it happens with every entry in the ZIP file, then maybe it is
\r\nIs it just one file you are trying to decompress? Then why not use the tool you used to compress it? If it is more than one file, then why not use the same thing on both ends - either WinZip or DotNetZip.
\r\nps: The Zip64Option applies only when saving a file. You don't need to set it when reading a file.
", "PostedDate": "2009-07-23T21:02:22.847-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215603", "ThreadId": "63392", "Html": "Thanks for the quick response. How can you tell if it using Deflate64? There is approximately 31,500 files in the .zip. DotNetZip works for the first 31,200 files and then gets the error. When looking at the details in WinZip, the exact file where it first has problems is the first file where the "relative offset of local header" field that WinZip outputs exceeds 0xFFFFFFFF. All the rest of the files from that point on list the "relative offset of local header" as 0xFFFFFFFF as well. I have no idea what the field means but guessing that it is related to the zlib error.
\r\nUnfortunately, we are not creating the .zip files, just receiving them. In this case they gave us a .zip with a password so we had to extract it first with the password and then compress it back without a password so it would go through our system.
", "PostedDate": "2009-07-23T22:37:37.22-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215667", "ThreadId": "63392", "Html": "Ok, well then if you are just receiving them, why not use WinZip to read the zip file? All the other tools seem to fail opening it. So use the thing that created it, to open it. You apparently have WinZip, because you are using it to examine the file. I don't understand why you don't just extract it with WinZip. Or ask your sender to re-pack that file differently.
\r\nDeflate64 is defined in the appnote from PKWare http://www.pkware.com/documents/casestudies/APPNOTE.TXT as compression level 9. Looking at the code it's pretty clear that DotNetZip would throw a specific exception if there was an unsupported compression level. You're not getting that exception so I don't think it's using Deflate64.
\r\nThe "relative offset of local header" is a reference to the location in the zip file where the zipentry data begins. For entries that appear prior to the 4.2gb mark in the zip file, the 32-bit quantity suffices. For entries beyond that point, ROLH would be higher than 0xffffffff. According to the zip64 extension, in that case the archiving tool should put a value of 0xffffffff in the place where ROLH is stored normally, and then store the actual offset elsewhere, in a 64-bit quantity. So what you are seeing is, for the last few files in that archive, the offset within the zip archive itself is beyond 0xffffffff.
\r\nIn looking at the tests for DotNetZip, they cover the case where WinZip12 unzips a ZIP64 archive that DotNetZip produces, but not the converse. That test succeeds - WinZip can extract a ZIP64-archive larger than 4.2gb that is produced by DotNetZip. I'll build another test for the other case, and we'll see if it works.
", "PostedDate": "2009-07-24T03:03:28.32-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215672", "ThreadId": "63392", "Html": "Also, "oversusbcribed dynamic bit lengths tree" generally indicates corruption. Have you got a secure hash on that file? Something that would allow you to verify that the zip file had not been corrupted or damaged between the time it was produced and the time you got it?
\r\nIf you want to learn more about the "oversubscribed dynamic bit lengths tree" error - you can read these resources:
\r\n", "PostedDate": "2009-07-24T03:35:54.84-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215845", "ThreadId": "63392", "Html": "
I think I have seen something similar to this before and I have a theory as to what causes it. I think it is due to the way that DotNetZip parses the Zip64 extra field.
\r\nThe case where I have seen it (which sounds similar to what the original poster here described) is where the Zip file itself is over 4GB but none of the individual files have an uncompressed or compressed size over 4GB. In this case, when creating the Zip file, WinZip just puts the relative offset of local header in the Zip64 field. The field is therefore just 8 bytes in length. However, when reading the file with DotNetZip, this piece of code in ProcessExtraField appears to be assuming that if it expects to find the relative offset in the Zip64 field, then the uncompressed and compressed size will be there too. If the length of the field is less than 24 it won't read the relative offset.
\r\n\r\n
if (this._UncompressedSize == 0xFFFFFFFF && DataSize >= 8)\r\n{\r\n this._UncompressedSize = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\nif (this._CompressedSize == 0xFFFFFFFF && DataSize >= 16)\r\n{\r\n this._CompressedSize = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\nif (this._RelativeOffsetOfLocalHeader == 0xFFFFFFFF && DataSize >= 24)\r\n{\r\n this._RelativeOffsetOfLocalHeader = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\n\r\n
As far as I understand it, the Zip spec says that any of the uncompressed size, compressed size, relative offset and disk number can be present in the Zip64 field but that they will always appear in the same order. So I think DotNetZip needs to cater for the case where just the relative offset is present.
\r\nMy guess as to what is causing the error is that the relative offset gets incorrectly set to 0xFFFFFFFF and it tries to decompress the data at that point in the file. This causes a problem with ZLib because it is not valid deflated data.
\r\n", "PostedDate": "2009-07-24T12:59:44.983-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215870", "ThreadId": "63392", "Html": "
I think you are on to something. Looking at the .zip detail again in WinZip 12, the extact file where the relative offset reports 0xFFFFFFFF, there is a Zip64 tag on the file showing up. The length of the file is only 8 bits but appears to hold the RelativeOffset.
\r\nI copy in the .zip entries of files
\r\n31303 (last file before the relative offset exceeds 0xFFFFFFFF),
\r\n31304 (a directory entry and relative offset set to 0xFFFFFFFF)
\r\n31305 (a file entry with relative offset set to 0xFFFFFFFF)
\r\nNote: the filename were changed to due to the sensitive of the information
\r\nThanks for your help.
\r\n\r\n
\r\n
\r\n
\r\n
\r\n \r\n \r\n Central directory entry PK0102 (4+42): #31303 ====================================== part number in which file begins (0000): 1 relative offset of local header: 4294430697 (0xfff7cfe9) bytes version made by operating system (00): MS-DOS, OS/2, NT FAT version made by zip software (20): 2.0 operat. system version needed to extract (00): MS-DOS, OS/2, NT FAT unzip software version needed to extract (20): 2.0 general purpose bit flag (0x0002) (bit 15..0): 0000.0000 0000.0010 file security status (bit 0): not encrypted extended local header (bit 3): no compression method (08): deflated compression sub-type (deflation): maximum file last modified on (0x0000395b 0x0000baa7): 2008-10-27 23:21:14 32-bit CRC value: 0xaea49b0e compressed size: 725291 bytes uncompressed size: 1001275 bytes length of filename: 104 characters length of extra field: 36 bytes length of file comment: 0 characters internal file attributes: 0x0001 apparent file type: text external file attributes: 0x00000020 non-MSDOS external file attributes: 0x000000 MS-DOS file attributes (0x20): arc Current Location part 1 offset 4338824867 filename:<FILENAME>.txt Current Location part 1 offset 4338824971 extra field 0x000a (PKWARE Win32 Filetimes), 4 header and 32 data bytes: 00 00 00 00 01 00 18 00 00 19 3c 9d b4 38 c9 01 ..........<..8.. 00 19 3c 9d b4 38 c9 01 00 19 3c 9d b4 38 c9 01 ..<..8....<..8.. The Extended Timestamps are: Creation Date: 2008-10-27 23:21:14 Last Modified Date: 2008-10-27 23:21:14 Last Accessed Date: 2008-10-27 23:21:14 Current Location part 1 offset 4338825007 Central directory entry PK0102 (4+42): #31304 ====================================== part number in which file begins (0000): 1 relative offset of local header: 4294967295 (0xffffffff) bytes version made by operating system (00): MS-DOS, OS/2, NT FAT version made by zip software (45): 4.5 operat. system version needed to extract (00): MS-DOS, OS/2, NT FAT unzip software version needed to extract (45): 4.5 general purpose bit flag (0x0000) (bit 15..0): 0000.0000 0000.0000 file security status (bit 0): not encrypted extended local header (bit 3): no compression method (00): none (stored) file last modified on (0x00003ae7 0x00005509): 2009-07-07 10:40:18 32-bit CRC value: 0x00000000 compressed size: 0 bytes uncompressed size: 0 bytes length of filename: 59 characters length of extra field: 48 bytes length of file comment: 0 characters internal file attributes: 0x0000 apparent file type: binary external file attributes: 0x00000010 non-MSDOS external file attributes: 0x000000 MS-DOS file attributes (0x10): dir Current Location part 1 offset 4338825053 filename:<DIRECTORY> Current Location part 1 offset 4338825112 extra field 0x000a (PKWARE Win32 Filetimes), 4 header and 32 data bytes: 00 00 00 00 01 00 18 00 90 65 96 3a 19 ff c9 01 .........e.:.... 50 0f e1 71 19 ff c9 01 00 54 7f 81 42 39 c9 01 P..q.....T.B9.. The Extended Timestamps are: Creation Date: 2008-10-28 16:16:56 Last Modified Date: 2009-07-07 10:40:18 Last Accessed Date: 2009-07-07 10:41:52 extra field 0x0001 (ZIP64 Tag), 4 header and 8 data bytes: 9a e1 02 00 01 00 00 00 ........ ZIP64 Tag Value(s): Value #1: 4295156122 Current Location part 1 offset 4338825160 Central directory entry PK0102 (4+42): #31305 ====================================== part number in which file begins (0000): 1 relative offset of local header: 4294967295 (0xffffffff) bytes version made by operating system (00): MS-DOS, OS/2, NT FAT version made by zip software (45): 4.5 operat. system version needed to extract (00): MS-DOS, OS/2, NT FAT unzip software version needed to extract (45): 4.5 general purpose bit flag (0x0002) (bit 15..0): 0000.0000 0000.0010 file security status (bit 0): not encrypted extended local header (bit 3): no compression method (08): deflated compression sub-type (deflation): maximum file last modified on (0x0000395b 0x0000baab): 2008-10-27 23:21:22 32-bit CRC value: 0xe7ac4931 compressed size: 367 bytes uncompressed size: 635 bytes length of filename: 99 characters length of extra field: 48 bytes length of file comment: 0 characters internal file attributes: 0x0001 apparent file type: text external file attributes: 0x00000020 non-MSDOS external file attributes: 0x000000 MS-DOS file attributes (0x20): arc Current Location part 1 offset 4338825206 filename:<FILENAME>.txt Current Location part 1 offset 4338825305 extra field 0x000a (PKWARE Win32 Filetimes), 4 header and 32 data bytes: 00 00 00 00 01 00 18 00 00 cd 00 a2 b4 38 c9 01 .............8.. 00 cd 00 a2 b4 38 c9 01 00 cd 00 a2 b4 38 c9 01 .....8.......8.. The Extended Timestamps are: Creation Date: 2008-10-27 23:21:22 Last Modified Date: 2008-10-27 23:21:22 Last Accessed Date: 2008-10-27 23:21:22 extra field 0x0001 (ZIP64 Tag), 4 header and 8 data bytes: f3 e1 02 00 01 00 00 00 ........ ZIP64 Tag Value(s): Value #1: 4295156211
\r\n
", "PostedDate": "2009-07-24T14:34:05.623-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215875", "ThreadId": "63392", "Html": "
RMLewin, good suggestion.
\r\n..ProcessExtraField appears to be assuming that if it expects to find the relative offset in the Zip64 field, then the uncompressed and compressed size will be there too. If the length of the field is less than 24 it won't read the relative offset..... As far as I understand it, the Zip spec says that any of the uncompressed size, compressed size, relative offset and disk number can be present in the Zip64 field but that they will always appear in the same order. So I think DotNetZip needs to cater for the case where just the relative offset is present.\r\n
You are correct, the ProcessExtraField assumes that both uncompressed and compressed sizes are in the extra field. I assume RelativeOffset *may* be present. Here's the part of the spec I am relying on, for that:
\r\nValue Size Description\r\n ----- ---- -----------\r\n (ZIP64) 0x0001 2 bytes Tag for this "extra" block type\r\n Size 2 bytes Size of this "extra" block\r\n Original \r\n Size 8 bytes Original uncompressed file size\r\n Compressed\r\n Size 8 bytes Size of compressed data\r\n Relative Header\r\n Offset 8 bytes Offset of local header record\r\n Disk Start\r\n Number 4 bytes Number of the disk on which\r\n this file starts \r\n\r\n This entry in the Local header must include BOTH original\r\n and compressed file size fields. If encrypting the \r\n central directory and bit 13 of the general purpose bit\r\n flag is set indicating masking, the value stored in the\r\n Local Header for the original file size will be zero.\r\n\r\n\r\n
I have not yet examined a ZIP64 file generated by WinZip to see the ZIP64 extra field.
", "PostedDate": "2009-07-24T14:58:23.54-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215877", "ThreadId": "63392", "Html": "I think the key phrase is "The [ZIP64] entry in the Local Header must include both original and compressed file size fields."
\r\nOn the other hand, there is also an entry in the central directory record for the ZIP64 info. I have designed the code with the idea that the entry in the central directory will also include both original and compressed file size fields, but this is not correct.
\r\nThis is a bug.
", "PostedDate": "2009-07-24T15:03:35.82-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215878", "ThreadId": "63392", "Html": "\r\n
Changing the Datasize to just 8 in every condition should fix this. I want to try this to test but am unable to compile the Zip Full DLL project stating error "Unable to copy from the "obj\\Debug\\Ionic.Zip.dll" to "bin\\Debug\\Ionic.Zip.dll""\r\n
if (this._UncompressedSize == 0xFFFFFFFF && DataSize >= 8)\r\n{\r\n this._UncompressedSize = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\nif (this._CompressedSize == 0xFFFFFFFF && DataSize >= 8)\r\n{\r\n this._CompressedSize = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\nif (this._RelativeOffsetOfLocalHeader == 0xFFFFFFFF && DataSize >= 8)\r\n{\r\n this._RelativeOffsetOfLocalHeader = BitConverter.ToInt64(Buffer, j);\r\n j += 8;\r\n}\r\n
| \r\n It's not as simple as that, I think. But you're right that it's a pretty simple fix. \r\nThe problem with your code is that it does not handle the error case where DataSize==8, and more than one of {UncompressedSize, CompressedSize, RelativeOFfset} is 0xFFFFFFFF. In that case, DotNetZip must throw an error. \r\nI'm testing a fix now. But it will take me a while to create the 5gb zip file. \r\n | \r\n
This is the part of the spec that I was referring to:
\r\n"The order of the fields in the zip64 extended information record is fixed, but the fields will only appear if the corresponding Local or Central directory record field is set to 0xFFFF or 0xFFFFFFFF."
", "PostedDate": "2009-07-24T15:16:35.37-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "215883", "ThreadId": "63392", "Html": "| \r\n Yep, and I had that part covered - "will only appear etc etc. " is covered by the test for 0xFFFFFFFF. But the error was assuming that the statement that the original and compressed file sizes MUST be present applied to both the Local Header as well as the central directory header. It clearly applies to the local header. It apparently does not apply to the central directory. \r\n | \r\n
ok I haven't run the tests yet, having some trouble with disk space.
\r\nBut there's a release v1.8.4.11 available, with the fix for the handling of the ZIP64 header. Try it and see if it works for your file. let me know.
", "PostedDate": "2009-07-24T20:53:53.457-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "216096", "ThreadId": "63392", "Html": "All the tests I ran succeeded with these changes.
", "PostedDate": "2009-07-25T21:51:17.63-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "216437", "ThreadId": "63392", "Html": "Thanks Cheeso. That fixed the problem I was having..... Its not easy, being cheesy....
", "PostedDate": "2009-07-27T09:47:22.81-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "216465", "ThreadId": "63392", "Html": "Thanks for reporting it, and thanks for working through it with me.
\r\nps: there was a bug in my original fix, so you definitely want v1.8.4.12, not v1.8.4.11.
", "PostedDate": "2009-07-27T10:43:07.877-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]