I haven't a clue what I'm doing wrong. I'm trying to create a zip file of a LOT (65,000+) of small files, so I wanted to multi-thread if possible. The problem is the zip it creates all the files are 0 bytes. I pretty much copied the code\r\n from one of your examples, but it isn't working. A little help please!
\r\n
\r\nI already had a simple .zip option working, but it took quite a long time so I wanted to try to use a multi-threaded option.
Thanks Much!
\r\n\r\n
\r\n
\r\n
' Create List of Files to Zip\r\nDim FilesToZip As New ArrayList\r\nDim FilterList() As String = {"*.txt", "*.properties"}\r\nFilesToZip.AddRange(Directory.GetFiles(ServerWorldPath, "*.*", SearchOption.AllDirectories))\r\nFor Each Filter As String In FilterList\r\n FilesToZip.AddRange(Directory.GetFiles(My.Settings.strHey0ModFolder, Filter, SearchOption.TopDirectoryOnly))\r\nNext\r\n\r\n' Zip the Files\r\nUsing Output As ZipOutputStream = New ZipOutputStream(BackupFilename)\r\n Output.CompressionLevel = Zlib.CompressionLevel.Default\r\n
Output.EnableZip64 = Zip64Option.Always\r\n Output.ParallelDeflateThreshold = 0\r\n For Each inputFile As String In FilesToZip\r\n Console.WriteLine(inputFile)\r\n Output.PutNextEntry(inputFile)\r\n Using Input As FileStream = File.Open(inputFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)\r\n Dim n As Integer\r\n Dim buffer As Byte() = New Byte(2048) {}\r\n Do While (n = Input.Read(buffer, 0, buffer.Length) > 0)\r\n Output.Write(buffer, 0, n)\r\n Loop\r\n End Using\r\n Next\r\nEnd Using
I don't know what the problem is; it could be a bug in the library.
\r\nBut using parallel deflation in your scenario won't speed things up. Parallel deflation will speed things up if you have files that are larger than... say, 32k or 64k at a minimum. This is because this is the default size of the moving window in the DEFLATE\r\n implementation in the DotNetZip library. If you have lots of "small" files, where small means less than 16k, then using parallel deflation will be detrimental to performance. I think this is all explained in the documentation.
\r\nCheck it out.
\r\n\r\n", "PostedDate": "2010-11-30T15:21:40.467-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "529161", "ThreadId": "236291", "Html": "\r\n
These are small files and LOTS of them... The folder I'm compressing contains:
\r\n72,487 files & 4,161 folders -- Yes you read the correctly. The files are ~3-4kb each
As I couldn't get the above code to work, using this until I can find a better solution:
\r\n\r\n
\r\n
\r\n
' Create Zip of World Files\r\n Using ZipF As New ZipFile(ZipFileName)\r\n ZipF.CompressionLevel = Zlib.CompressionLevel.Default\r\n ZipF.UseZip64WhenSaving = Zip64Option.AsNecessary\r\n ZipF.ParallelDeflateThreshold = 0\r\n ZipF.BufferSize = 4096\r\n ZipF.AddDirectory(ServerWorldPath, ServerWorldName)\r\n ZipF.AddSelectedFiles("*.txt", My.Settings.strHey0ModFolder, "", False)\r\n ZipF.AddSelectedFiles("*.properties", My.Settings.strHey0ModFolder, "", False)\r\n ZipF.Save()\r\n End Using\r\n\r\n ' Print File Info\r\n Dim fi As New FileInfo(ZipFileName)\r\n stdIn.WriteLine("say * Zipping Finished - Size: " & Math.Round(fi.Length / 1024 ^ 2, 0, MidpointRounding.ToEven) & " Mb *")\r\n
Now I would love to get more threads working to speed things up, I only see 20-25% CPU usage when it is zipping. Now with 7zip using Deflate64 & 4 threads it uses 100% and finishes very quickly!
DotNetZip (1 Thread): 81 seconds
7zip (4 Threads): 14 seconds
\r\n", "PostedDate": "2010-11-30T18:20:41.637-08:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "530723", "ThreadId": "236291", "Html": "\r\n
yep, there's a big opportunity for efficiency if I parallelize the zipping on a file-by-file basis.
\r\nIn v1.9, I created a parallel stream that can deflate large files much faster. This doesn't help when there are many many files.
\r\nThat's something I had hoped to address in v2.0. For now I don't have any good options for you, to decrease that 81 seconds.
\r\n\r\n", "PostedDate": "2010-12-03T08:28:44.19-08:00", "UserRole": null, "MarkedAsAnswerDate": null } ]