Question, I am attempting to zip large amounts of data automatically on a schedule, the resulting zip file would be too large to store in memory (GBs). Is it possible to stream the resulting zip file out to a stream object as it is generated, and not to store the file contents in memory and then save it all at once? So something like...
\r\nI'm new to this library, so please bare with me...
\r\n using (ZipFile zip = new ZipFile())
{
zip.SetOutputStream(xxx);
zip.AddEntry(xxx,yyy);
zip.AddEntry(xxx,yyy);
zip.AddEntry(xxx,yyy);
zip.Flush();
\r\n
}
", "PostedDate": "2010-04-25T00:32:13.553-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "435463", "ThreadId": "210462", "Html": "
IIRC, the library supports deferred adding of data - if you add an entry with a stream or a WriteDelegate, the library reads the data of the individual items while saving, so memory usage should be moderate.
", "PostedDate": "2010-04-26T05:25:10.14-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "435618", "ThreadId": "210462", "Html": "Yes, you are correct hardcodet, there are options for deferring.
\r\nFirst, there are two main interfaces for creating zip fiels in DotNEtZip. One is the ZipFile class, which you've seen. The other is the ZipOutputStream class. Let's take them in reverse.
\r\nUsing ZipOutputStream, you treat the zip as a writable stream. Your code creates this stream, and then for each entry you want to appear in the zip file, specify the entry name, then write the data for that entry into the stream. As the data is written it is zipped. It is a full streaming model. It is a forward-only write-stream. The code for that looks like this:
\r\nusing (var output= new ZipOutputStream(outputFileName))\r\n{\r\n output.Password = "VerySecret!";\r\n output.Encryption = EncryptionAlgorithm.WinZipAes256;\r\n\r\n foreach (string inputFileName in filesToZip)\r\n {\r\n System.Console.WriteLine("file: {0}", inputFileName);\r\n\r\n output.PutNextEntry(inputFileName);\r\n using (var input = File.Open(inputFileName, FileMode.Open, FileAccess.Read,\r\n FileShare.Read | FileShare.Write ))\r\n {\r\n byte[] buffer= new byte[2048];\r\n int n;\r\n while ((n= input.Read(buffer,0,buffer.Length)) > 0)\r\n {\r\n output.Write(buffer,0,n);\r\n }\r\n }\r\n }\r\n}\r\n
This is a handy model, and may be what you want. It isn't satisfactory for all uses, though, because this model requires the use of bit 3, which is a part of the zip spec that somehow isn't supported on some platforms, or by some tools. It's really not that exotic, so I don't know why, but in any case some 3rd party tools will choke when consuming zips produced in this way. The other attribute of the ZipOutputStream is that it is an output stream only. there's no support for updating a zip file, or for random access, or simply reading a zip file.
\r\nThe alternative when producing a zipfile is to use the ZipFile class. The main verbs you use with ZipFile are AddFile and AddEntry. Contrary to your suggestion, in most cases using these verbs does not cause the entire contents of the entries to be stored in memory at any one time, ever. If you are adding files via AddFile, at the time of the call to ZipFile.AddFile, DotNetZip stores in memory the metadata about the entry - the name, whether it will use encryption or not, where the data will come from when the zipFile is eventually saved, and so on. Regardless of the size of the file you are adding, what's stored in memory is generally less than 256 bytes worth of data. At the time of ZipFile.Save, the source file is read and its data compressed and written to the zip, in a streaming manner. So, the entire contents of the entry is never held in memory.
\r\nThe only exception to that, is when calling AddEntry() with a byte array or a string defining the content of the entry to be written into the zipfile. The overloads accepting these types of inputs are intended to support the insertion of entries in the zip file with dynamically-sourced content - for example a readme.txt file that contains a few lines of text. You can do this with the AddEntry overload that accepts a string. Of course in this case the entire string is in memory at one time.
\r\nStream stream = ObtainStreamFromSomewhere(); \r\nusing (ZipFile zip = new ZipFile())\r\n{\r\n // The content for this entry will be read from a filesystem file,\r\n // at the time of the call to Save(). \r\n // The entire data from the file is never held in memory.\r\n zip.AddFile("C:\\\\whatever\\\\Name-of-Entry1.txt", "files"); \r\n\r\n // The contents for this entry will be read from the provided stream,\r\n // at the time of the call to Save().\r\n // The entire contents of the stream is never held in memory. \r\n zip.AddEntry("files\\\\Name-of-Entry2.bin", stream);\r\n\r\n // This content for this entry will be obtained from a string. \r\n // Obviously, the string is held in memory. \r\n zip.AddEntry("files\\\\Readme.txt","this is the content of the readme entry");\r\n\r\n // Save() will read data from each of the above sources and\r\n // write to the provided zip file, in a streaming fashion.\r\n zip.Save("c:\\\\archive.zip");\r\n}\r\n
In the prior email, hardcodet suggested an option, the WriteDelegate. WriteDelegate is a different source that you can use with the ZipFile class. The use of the WriteDelegate may be interesting to you, but it is really orthogonal to the issue of whether the entry data is stored in memory at one time. As I said above, allowing 2 exceptions, the content for entries is never stored in memory.
\r\nSo what does the WriteDelegate do? It switches from a pull model to a push model. What I mean is this: When calling ZipFile.AddFile or ZipFile.AddEntry, your app provides a source for entry data to DotNetZip. This source might be a filesystem file, a stream, a string, or a byte array. When your app calls ZipFile.Save(), DotNetZip then retrieves the data for each entry, from the source you provided. This is what I might call "Pull". DotNetZip reads the source your app provided.
\r\nThere are some cases where the application does not have a source from which DotNetZip can "pull" content. An example is a .NET DataSet. There's a nice WriteXml() method on the DataSet class, which can write an xml representation of a dataset into a stream. But there's no way to get a stream for the dataset, if you see what I mean. The DataSet can write to a sink, but cannot act as a source of data. The WriteDelegate solves that problem. The way it works: at the time of ZipFile.Save, for each entry that has a WriteDelegate as a source, DotNetZip will invoke your application code, and allow your code to write directly into the zip stream. It's something like the model for ZipOutputStream, if that makes sense, but just for a single entry. The code looks like this:
\r\nprivate void WriteEntry (String filename, Stream output)\r\n{\r\n DataSet ds1 = ObtainDataSet();\r\n ds1.WriteXml(output);\r\n}\r\n\r\nprivate void Run()\r\n{\r\n using (var zip = new ZipFile())\r\n {\r\n zip.AddEntry(zipEntryName, WriteEntry);\r\n zip.Save(zipFileName);\r\n }\r\n}\r\n
All of this is described in various places in the fairly complete reference that is available at http://dotnetzip.codeplex.com/documentation .
\r\nI've been meaning to write some programming guide material, to complement that reference. The workitem for that is http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=9032 This response is the kind of information I would put in that programming-guide document page.
\r\n", "PostedDate": "2010-04-26T11:46:20.69-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]