Hello to everybody!
\r\nI'm writing a class in C# for a backup utility with incremental backup option, but my strategy is terribly slow. How I can optimize it? Am I doing something wrong?
\r\nFor my tests, I used:
\r\nFirst and complete zipping takes 3 minutes and 23 seconds (compressed zip is about 340MB)
\r\nIncremental (and partial) backup takes 2 minutes and 35 seconds (compressed zip of incremental backup is about 160MB): it loses one minutes and 17 seconds to compare the files and its ModifiedTime (so, one minutes and 17 seconds before Save() method is invoked).
\r\nWell, now let me show you my code:
\r\nZipFile OldZip = ZipFile.Read(nameOfZipFile);\r\n
ZipFile NewZip = new ZipFile();
NewZip.UseZip64WhenSaving = Zip64Option.AsNecessary;
NewZip.CompressionLevel = CompressionLevel.Level2;
NewZip.UseUnicodeAsNecessary = true;
foreach (FileInfo currentFileOnDisk in FileList)
{
ZipEntry candidateZipEntry = OldZip[currentFileOnDisk.FullName.Replace(mailclientsClass.thunderbirdProfilePath, "ThunderbirdProfile")];
if (candidateZipEntry != null && currentFileOnDisk.LastWriteTimeUtc > candidateZipEntry.ModifiedTime)
NewZip.AddFile(currentFileOnDisk.FullName, Path.GetDirectoryName(currentFileOnDisk.FullName.Replace(mailclientsClass.thunderbirdProfilePath, "ThunderbirdProfile")));
else if (candidateZipEntry == null)
NewZip.AddFile(currentFileOnDisk.FullName, Path.GetDirectoryName(currentFileOnDisk.FullName.Replace(mailclientsClass.thunderbirdProfilePath, "ThunderbirdProfile")));
}
NewZip.Save(nameOfZipFile+incrementalCounter.ToString());
Yes, I know, this code is not complete for a good incremental backup strategy, because I need more istructions to check removed files & C. but for now I'm in trouble for its slowness. Is it possible to optimize it?
\r\n\r\n
Many thanks!
\r\nBye,
\r\nAlessandro.
", "PostedDate": "2009-11-01T16:58:55.19-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "253150", "ThreadId": "73763", "Html": "| \r\n Ciao Alessandro, \r\nI have a couple of suggestions. \r\n
\r\n | \r\n
Wow! Wow! Wow! Mr. Dino, the 1.9.0.29 is really amazing!
\r\nI repeated my test (my PC is a DELL Inspiron 6400 with a Core 2 Duo processor) and it takes only 1 minutes and 52 seconds with a gain of 80% (Yes, I know: you will not believe me, but the 1.8.4.26 completed the test in 3 minutes and 23 seconds).
\r\nI added only these options:
\r\n\r\n
NewZip.ParallelDeflateThreshold = 2097152; //2MB\r\nNewZip.BufferSize = 65536 * 8;\r\n\r\n\r\n
But I will do more tests, because I think that I can get more performance from your library! For example, I will try ZipOutputStream and your precious advices.
\r\nI'm going to use a catalog in SQLite too, to trace the last modified files (now I open each ZIP and I check the modified time of each zip entry).
\r\n\r\n
P.S: I will keep you informed if I will get more performance with your advices!
\r\nBye,
\r\nand...
\r\nwhile(true)\r\n
{
Console.WriteLine("THANKS SO MUCH!");
}
", "PostedDate": "2009-11-05T01:38:35.103-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "253699", "ThreadId": "73763", "Html": "
| \r\n Wow, that's a nice gain in performance. \r\nListen, Alex, about the ParallelDeflateThreshold - it's a new part of DotNetZip, maybe I haven't documented it clearly enough. I want to make sure you understand what it is doing. \r\nBy setting it to 2mb, you are saying that only files larger than 2mb should be deflated with multiple threads. In my experience, on my machine, any file larger than 512k showed a significant performance advantage when using the parallel deflate. So you may get more of a gain if you lower that number from 2mb to maybe 1mb or 875k. But maybe you already tested it and found that 2mb is the right number. \r\nAlso see my other post about the performance analysis of v1.9.0.29. It shows the effects I measured when modifying the CodecBufferSize and (IO) BufferSize . \r\nGood luck! \r\nps: the Parallel deflate capability is an ON/OFF thing for ZipOutputStream. On the ZipFile, the library knows the size of the files it is zipping, and so it can make a decision on whether to use the parallel deflate based on the size of the file. In the ZipOutputStream, because of the different programming model, there's no way for the library to know how large the file is going to be, before it is written. So, with ZipOutputStream, the only meaningful values for the ParallelDeflateThreshold property are 0 (always use parallel deflate) or -1L (never). Any other value implies (never). \r\nOne last thing: be sure you are using RELEASE dlls for DotNetZip. The difference between a DEBUG dll and a RELEASE dll can be 40-50%. I have both RELEASE and DEBUG dlls on the downloads page. For performance, you want RELEASE. \r\n\r\n | \r\n
Thanks so much for explanations!
\r\nYesterday I tried ParallelDeflateThreshold set to 2MB for a quick test, but after I read your explanations, I tried ParallelDeflateThreshold property set to 512KB.
\r\nI tried also ParallelDeflateThreshold set to 750KB but I didn't observe any difference, instead with it set to 2MB, it seems to be slightly slower (2 seconds)
\r\nI set up BufferSize to 128KB too. Now, with ParallelDeflateThreshold set to 512KB and BufferSize set to 128KB, it seems to have the best performance on my machine.
\r\nThanks to your DotNetZip 1.9, now my program is faster then KLS MailBackup. You have done a really great job with parallel deflate!
\r\nP.S: I downloaded this file: DotNetZipLib-Runtime-v1.9.zip and I use the Reduced DLL. I think that it is the RELEASE version, isn't?
\r\nThanks so much for your job!
", "PostedDate": "2009-11-06T03:22:42.087-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "254201", "ThreadId": "73763", "Html": "I'm glad it's working for you!
\r\nAbout the RELEASE version - yes, you have the right one. In the Devkit download, there is a DEBUG dll and a RELEASE dll . The Runtime download, which is the one you grabbed, includes only the RELEASE version.
\r\n", "PostedDate": "2009-11-06T04:12:46.68-08:00", "UserRole": null, "MarkedAsAnswerDate": null } ]