I am using DotNetZip to create backups of directories and their contents with VB .Net. For security reasons, I chose to use a zip file obscured inside a second zip file, so that contents aren't as visible, with the encapsulating zip file being uncompressed\r\n & AES-256 encrypted. It contains both the lightly-compressed "Inner" zip file of the backup itself and a small text "label" file with version info and a few other bits of information.
\r\nOccasionally, the resulting zip gets corrupted during the zip operation when it is being encapsulated in the Outer Zip file; I know it's here because I have the program check the file contents list after each zip operation to ensure that there isn't a failure\r\n to generate a proper zip file, and it doesn't throw any errors in between the two operations, run sequentially. It's a bit of a rough way to check but it has worked until now. I haven't been able to replicate the error in testing, and this has only happened\r\n a few times out of a few hundred runs, but is troubling due to my need for consistent backups.
\r\nI have to create the Inner zip file as a second file rather than an output stream because of memory limitations on the systems that we are running it on (a few of our different production computers running XP / Vista / 7, so far no commonality). I\r\n am adding a Threading.Thread.Sleep(2000) to be 100% sure the transition from "DotNetZip-XXXXX.tmp" to resulting "Inner" zip file isn't being caught in mid-stream and that all disk access has completed before the encapsulation begins.
\r\nI was hoping someone could help here:
\r\nIs there a way I can get the contents of the Inner Zip file after the encapsulation, while it is inside the Outer? Is there a way to do this without extracting the files, even temporarily? If so, I could check the contents list to verify it's not corrupt. \r\n Is there a way I can check the CRC values of the inner files a different way? Am I not seeing something else?
\r\nIf anyone could provide any other insights, I'd really appreciate it. Thanks!
\r\nHere is the core of the compressing sub, with a few bits chopped out above and below... not very pretty, but it usually works ;) . It handles the "Outer" zip / "Inner" zip switch with "blnCompressing" - if true, create an "Inner" zip / if false, create\r\n an "Outer":
\r\n\r\n
Using zip1 As ZipFile = New ZipFile\r\n zip1.UseZip64WhenSaving = Zip64Option.AsNecessary\r\n zip1.Name = strZipToCreatePub\r\n If blnCompressingZip Then\r\n zip1.CompressionLevel = Ionic.Zlib.CompressionLevel.BestSpeed\r\n zip1.AddDirectory(strToBeZippedPub, "")\r\n zip1.Comment = strComment & ", " & Now.ToString("hh:mm tt") & ", " & Now.ToString("MM-dd-yy")\r\n Else\r\n zip1.Encryption = EncryptionAlgorithm.WinZipAes256\r\n zip1.Password = strPasswordPub\r\n zip1.CompressionLevel = Ionic.Zlib.CompressionLevel.None\r\n zip1.AddFile(strLabelFilePub, "")\r\n zip1.AddFile(strToBeZippedPub, "")\r\n End If\r\n Me._entriesToZip = intAllFilesAndDirs 'this is the total # of files that are being zipped to display in the progress bar\r\n Me.SetProgressBars()\r\n AddHandler zip1.SaveProgress, New EventHandler(Of SaveProgressEventArgs)(AddressOf Me.zip1_SaveProgress)\r\n zip1.Save(zip1.Name)\r\n End Using\r\n\r\n
\r\n
\r\n", "PostedDate": "2010-10-25T12:29:44.607-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "513073", "ThreadId": "232268", "Html": "\r\n
I understand what you're doing, and it makes sense to me.
\r\nWhat you want to do, I think, is validate the outer zip after it's been written. I think you said you want to validate the contents of the inner zip, but without extracting those files to the filesystem.
\r\nTo do this, open the outer zip for reading with ZipFile.Read(). There's just one ZipEntry in it, if I understand the situation correctly. Call\r\n\r\nZipEntry.OpenReader() on that entry, which gives you a readable stream. You can then\r\n\r\ncreate a ZipInputStream around that read-only stream.
\r\nYou also may be able to wrap the result of ZipEntry.OpenReader() in a BufferedStream, and call ZipFile.Read() on THAT.
\r\nYou cannot succeed by calling ZipFile.Read() using the result of ZipEntry.OpenReader(), in order to read and validate the inner zip. This is because when the ZipFile class is used to read a zip file, it seeks around in\r\n the file. It seeks to the end, reads some things, then seeks backwards, and so on. The result of ZipEntry.OpenReader(), though, is a read-forward-only stream. It doesn't support seeks. Using a BufferedStream basically just caches all\r\n the inner zip data, and so once again Seek() will work. But be careful, this only works if the buffer size is as large as the entire inner zip file. Maybe impractical.
\r\nOn the other hand, the ZipInputStream does support reading from a read-forward-only stream, so it can be used to validate the result of ZipEntry.OpenReader().
\r\nThe interface you want to use on ZipInputStream is \r\nGetNextEntry(); call that and then call ZipInputStream.Read() to read the de-compressed content of the entry in the inner zip. Do GetNextEntry + Read it in a loop to read all entries.
\r\nIf you call ZipInputStream.Read() repeatedly until it returns 0 (implying no more data is available for the ZipEntry), the Read() method will automatically and transparently validate the CRC for the entry, and throw an exception if the CRC does not\r\n match the expected value. If you do not call Read() until it returns 0, then the ZipInputStream.Read() method will not validate the CRC of the entry.
\r\nTherefore to validate the CRC of the entry, just call Read() into a small buffer (say 2k), repeatedly, until it returns 0.
\r\nDoes this answer the question?
\r\nps: I am interested in tracking down any problem with corruption, but to do it, I guess you realize I need a reproducible test case.
\r\nlast thing: if you want to add an entry to a zip, and the entry is a small string, you can call AddEntry() and pass a System.String. For your "label" entry, you may find that option interesting.
\r\n\r\n", "PostedDate": "2010-10-26T17:45:43.817-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "525903", "ThreadId": "232268", "Html": "Well, I got better info from my end users and caught a corruption in the act. A backup utility that was set to backup everything put into the destination zip directory would sometimes step in and try to backup parts of the zip file as it was copying in.\r\n Without the password , it could backup the zip header and file list of the zip, but nothing else... presto! Corrupted backup, good source, good destination.
\r\n\r\n\r\n\r\nFrom: Cheeso
\r\n\r\n\r\nI understand what you're doing, and it makes sense to me.
\r\nWhat you want to do, I think, is validate the outer zip after it's been written. I think you said you want to validate the contents of the inner zip, but without extracting those files to the filesystem.
\r\nTo do this, open the outer zip for reading with ZipFile.Read(). There's just one ZipEntry in it, if I understand the situation correctly. Call\r\n\r\nZipEntry.OpenReader() on that entry, which gives you a readable stream. You can then\r\n\r\ncreate a ZipInputStream around that read-only stream.
\r\nYou also may be able to wrap the result of ZipEntry.OpenReader() in a BufferedStream, and call ZipFile.Read() on THAT.
\r\nYou cannot succeed by calling ZipFile.Read() using the result of ZipEntry.OpenReader(), in order to read and validate the inner zip. This is because when the ZipFile class is used to read a zip file, it seeks around in the file.\r\n It seeks to the end, reads some things, then seeks backwards, and so on. The result of ZipEntry.OpenReader(), though, is a read-forward-only stream. It doesn't support seeks. Using a BufferedStream basically just caches all the inner zip data, and so once\r\n again Seek() will work. But be careful, this only works if the buffer size is as large as the entire inner zip file. Maybe impractical.
\r\nOn the other hand, the ZipInputStream does support reading from a read-forward-only stream, so it can be used to validate the result of ZipEntry.OpenReader().\r\n
\r\nThe interface you want to use on ZipInputStream is \r\nGetNextEntry(); call that and then call ZipInputStream.Read() to read the de-compressed content of the entry in the inner zip. Do GetNextEntry + Read it in a loop to read all entries.\r\n
\r\nIf you call ZipInputStream.Read() repeatedly until it returns 0 (implying no more data is available for the ZipEntry), the Read() method will automatically and transparently validate the CRC for the entry, and throw an exception if the CRC does not match\r\n the expected value. If you do not call Read() until it returns 0, then the ZipInputStream.Read() method will not validate the CRC of the entry.
\r\nTherefore to validate the CRC of the entry, just call Read() into a small buffer (say 2k), repeatedly, until it returns 0.\r\n
\r\nDoes this answer the question?
\r\nps: I am interested in tracking down any problem with corruption, but to do it, I guess you realize I need a reproducible test case.
\r\nlast thing: if you want to add an entry to a zip, and the entry is a small string, you can call AddEntry() and pass a System.String. For your "label" entry, you may find that option interesting.
\r\n\r\n\r\n\r\nRead the \r\nfull discussion online.
\r\nTo add a post to this discussion, reply to this email (DotNetZip@discussions.codeplex.com)
\r\nTo start a new discussion for this project, email \r\nDotNetZip@discussions.codeplex.com
\r\nYou are receiving this email because you subscribed to this discussion on CodePlex. You can\r\n\r\nunsubscribe on CodePlex.com.
\r\nPlease note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
\r\n
Wow, I'm glad you figured this out. I'll think about the possibility of how the library could be modified, to avoid this pitfall.
\r\n\r\n", "PostedDate": "2010-11-23T08:40:55.44-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "526084", "ThreadId": "232268", "Html": "Oh, sorry. It wasn't a problem with DotNetZip after all! I'd only written to see if I could detect the bad CRC's somehow on my end... which you explained very well, and thanks again for that.
ah, ok, I understand.
\r\nI think DotNetZip could suffer a similar problem if you zip into a directory that you are currently zipping. It would be nice to be able to avoid that pitfall but it may be that this is not important enough to enough people, for me to spend time on\r\n solving it.
\r\nRegarding the timestamp thing - yes, that is better covered in a new thread if you please. It's not really for tracking - it's for search. I want people to be able to search on relevant topics, and keeping threads narrowly constrained helps out\r\n there.
\r\nthanks for posting the resolution to your problem!
\r\n\r\n", "PostedDate": "2010-11-24T09:34:43.783-08:00", "UserRole": null, "MarkedAsAnswerDate": null } ]