Hi Cheeso, and thanks for your answer, but the example is no good.
\r\n\r\n
using (ZipFile zip = new ZipFile())\r\n {\r\n foreach (var f in filesToInclude)\r\n {\r\n zip.AddFile(f, "files");\r\n }\r\n zip.AddFileFromString("Readme.txt", "", ReadmeText);\r\n zip.Save(Response.OutputStream);\r\n }\r\n Response.End();\r\n\r\nWhereas I need something like the following (so that while I'm saving the zip, the user is already receiving it):\r\n
I can show you that this is what many of the posts where you advertised DotNetZip with the former code, are actually requesting.
It can be done with ZipSharpLib but not with DotNetZip.
using (ZipFile zip = new ZipFile()) \r\n { \r\n foreach (var f in filesToInclude)\r\n { \r\n zip.AddFile(f, "files");\r\n zip.WriteToOutput(Response.OutputStream); \r\n Response.Flush(); \r\n } \r\n zip.AddFileFromString("Readme.txt", "", ReadmeText);\r\n zip.WriteToOutput(Response.OutputStream); \r\n Response.Flush(); \r\n } \r\n Response.End(); \r\n // In any case, thanks for the great tool!\r\n
\r\n
\r\n
Keywords for this question:
realtime zip, real time zip,
zip created on the fly,
immediate zip download,
streamed zip to output
Yes, in fact you can do what you want, using what I have called the just-in-time stream provisioning mechanism in v1.8 of DotNetZip.
\r\nAlthough, I would suggest that you be careful about insisting that the browser "must receive the file as the server sends it". [EDIT: Of course you want the browser to receive the file as soon as the server sends it - this is what streaming is. And as I've told you, and as I have explained in other places that you seem to have read, DotNetZip uses a streaming approach. By saying the browser "must receive the file as soon as the server sends it," I think you are really implying "the browser must receive the file as soon as my app calls AddFile()." I think you are concluding, incorrectly and without justification, that the compression is happening at the time of the call to AddFile. ] It seems like an implementation detail that typically, you would not need to concern yourself with. Is it performance that concerns you? Which aspect? Time-to-first-byte? Time-to-last-byte? Something else? If performance is your concern, I suggest that you benchmark the easy approach with DotNetZip, before dismissing it as not appropriate or "no good". It actually *does* stream the zip file to the browser.
\r\nOn the other hand I can see situations involving dynamically-created input streams, where the JIT stream provisioning would be ideal. There may also be some benefit when the zip file is huge, with tens of thousands of entries. But these are not mainstream scenarios.
\r\nYou offered to contribute something, and I appreciate the offer. But before we go forward, I need to better understand the problem we're trying to solve here.
\r\n", "PostedDate": "2009-06-29T09:24:49.227-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208014", "ThreadId": "45459", "Html": "
I need zip exactly for this reason, I'm putting together a group of very large files, some of them media files, others part of an application (without compression).
\r\nCall this performance, or call it common sense:
a. Why should the user wait the whole time it takes to create this zipfile in memory, and only then start downloading it?
b. Why should the server's memory have a full copy of this giant file aggragation, when it could be simply streaming it off, a chunk at a time?
How do I use the JIT stream provisioning mechanism? Is it documented?
\r\nMy offer to contribute is thus: (I didn't see you source code yet):
Add four methods: WriteEntry(OutputStream out, ZipEntry entry),
WriteHeader/Footer(OutputStream out, List<string> files);
and CalculateZipfileSize(OutputStream out, List<string> files);)
These methods replace the "Save", which do the following:
a. Throw exception if the method of compression is not STORE (ie UNCOMPRESSED) for all files.
b. Calculates the size of the zipfile, according to the files which will be stored in it
c. Writes an entry to an output (or stream).
d. Writes the zip header or zip footer
Moshe
", "PostedDate": "2009-07-02T09:10:27.76-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208019", "ThreadId": "45459", "Html": "Or maybe instead of WriteEntry have Write(OutputStream out, Array buffer)
and I would have low level control on what is currently in the buffer...
so I could send chunks of information say 100K at a time.
Thanks for your suggestions. I think you have a basic misconception about how the DotNetZip library works.
\r\nWhen you call zip.AddFile(foo...) , the call stores metadata about the entry you have added. It does not produce a zip file in memory at that time. It stores information about the entry that will eventually be part of the zipfile.
\r\nWhat information? Things like: the filename, the timestamp on the file, and whether encryption and compression should be used when the file is eventally put into the Zip archive. No compression or encryption is performed at the time of the call to AddFile() or AddEntry() and their cousins. These operations (AddFile etc) are generally very fast, because they only collect and store metadata.
\r\nIt is when you call Save() that the zipfile is then constructed, using the metadata stored in memory. The compression and encryption is performed at that time. If you call Save() and pass a stream, DotNetZip writes the zip content directly to the output stream. Because it uses a stream approach, the application never has "a full copy of this giant file aggregation" in memory as you put it. I appreciate your concern about that, because it would be silly to create the entire zip file in memory. But it does not happen.
\r\nWhen you call Save() the possibly compressed, possibly encrypted content is sent to the stream, which does whatever it wants with it. In particular, if you call ZipFile.Save(), passing Response.OutputStream as the stream, then DotNetZip will put the zipfile bytes directly into the Response.OutputStream, as they are constructed, and the browser is able to immediately begin downloading the bytes while the compression is happening. Obviously this can and does happen before the zipfile is fully constructed. In the general case the browser will be dowloading bytes for an entry within the zipfile before even that particular entry is fully compressed.
\r\nIn no case is there "a giant file aggregation" stored in memory on the server. Ever. The only time you will have a giant aggregation in memory is if you call Save() to a MemoryStream(), which by definition stores data into memory. If you for example call Save() with a filename, then the zipped data is streamed to the file, in a manner similar to the streaming to the browser. Here again, there is no "giant aggregation."
\r\nAll of this is documented in the helpfile. http://cheeso.members.winisp.net/DotNetZipHelp The JIT streaming mechanism is also fully documented and explained, although now it sounds like you don't need it.
\r\nRegarding CalculateZipfileSize(), that's an interesting idea, and I can see the utility in it. But getting the size is not possible, without actually performing the compression and encryption. There's no way to predict, a-priori, how well a particular data stream will compress. Because you don't know the size until you actually do the compression, you cannot know the size of the zip file until it is created.
\r\nYour efforts to improve DotNetZip are admirable, but I suggest before you devote any more time to improving it, you first spend some time better understanding how it works.
\r\n", "PostedDate": "2009-07-02T11:26:02.13-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208083", "ThreadId": "45459", "Html": "
ps: regarding your desire to manage the chunks that get sent - see the BufferSize property on the ZipFile.
\r\nHowever, keep in mind that the stream is very fast, and there are multiple levels of buffering between a typical webserver and browser. As a result there is no guarantee that there will be a discrete "one chunk at a time" behavior. The chunk size on the client (browser) will not be the same size as the chunksize on the server. The BufferSize property allows you to balance memory size versus compression overhead. For many small files, a smaller buffer is warranted. If you have large files, a buffersize of 1mb might be better. There are no hard and fast rules, you'll have to test it.
\r\nAnd yes, ZipFile.BufferSize is documented, too.
\r\nhttp://cheeso.members.winisp.net/DotNetZipHelp
", "PostedDate": "2009-07-02T11:39:30.587-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208640", "ThreadId": "45459", "Html": "Wow! OK! So if I understand correctly, DotNetZip will do real-time streaming on the fly, by simply saving to OutputStream when done defining,
and no "write()" needs to be done during definition, since no processing is done at that stage anyway.
I'll check to see if it works OK for me, (and starts streaming immediately) and if so will also correct various forums (about sharpZip)
where I followed up on your entry,(I had written to say that your library seemed not to stream on the fly)
\r\n
Thanks! Moshe
", "PostedDate": "2009-07-05T00:40:26.383-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208664", "ThreadId": "45459", "Html": "DotNetZip 1.8 - When trying it with small files, it works fine (seemingly).
\r\nWhen replacing the same files with large ones (same filenames so everything is OK "pathwise"),
in IE8 I do not get the "Download file" dialog, but rather I wait a while, with the IE progress bar advancing and it says the page is downloading,
and then finally it stops and I get: Internet Explorer cannot display the webpage. - Obviously a timeout.
How can I cause it to give me the "download file" dialog first, and only then continue streaming while creating the output?
\r\nBTW the Calculate() is useful for ForceNoCompression - but that usage IS VERY common.
\r\nThanks again, and will get my company to contribute ...
\r\nMoshe
", "PostedDate": "2009-07-05T05:29:12.323-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208671", "ThreadId": "45459", "Html": "I'd have to see your server-side code to give you any insight into why zipping large files doesn't work with IE8. Normally what you describe - getting the download file dialog first - is how it happens. Have you tried on IE7 or Firefox?
\r\n", "PostedDate": "2009-07-05T06:45:38.66-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "208861", "ThreadId": "45459", "Html": "
Thanks for the speedy response!!!
Firefox fails in the same way. Please see remarks at the bottom of this posting.
The server side code is:
protected void Page_Load(object sender, EventArgs e)\r\n {\r\n tryDotNetZip(new ArrayList(){"ErrorPage.htm", "temp1.tmp", "temp2.tmp", "ErrHandler.cs"});\r\n }\r\n\r\n\r\n private void tryDotNetZip(ArrayList filesToInclude)\r\n {\r\n Response.Clear();\r\n string zipName = String.Format("archive-{0}.zip", DateTime.Now.ToString("yyyy-MMM-dd-HHmmss"));\r\n //const string zipName = "moshe.zip";\r\n Response.ContentType = "application/zip";\r\n Response.AddHeader("content-disposition", "filename=" + zipName);\r\n //Response.Flush();\r\n using (ZipFile zip = new ZipFile())\r\n {\r\n //zip.BufferSize = 2000;\r\n zip.ForceNoCompression = true;\r\n\r\n foreach (string f in filesToInclude)\r\n {\r\n string filename = Path.Combine(Server.MapPath("."), f);\r\n zip.AddFile(filename, "files");\r\n }\r\n zip.Save(Response.OutputStream);\r\n //Response.Flush();\r\n }\r\n Response.End();\r\n }\r\n\r\n\r\n
_________________________
Please note that all files exist, and that the files temp1.tmp and temp2.tmp are replaced from small text files (with a rapid and successful zip) to large zipfiles,
using a simple "rename" call in DOS.
I also tried changing the names in the ArrayList of the called files to "temp1Big.tmp, temp2Big.tmp" (with respective files in the directory) but to no avail.
Thanks again, and hope you enjoyed the holiday weekend
(I'm on the other side of the atalantic, reason for the strange hours),
Moshe
Tried content type application/octet-Sream: Did not help.
(In SharpZipLib when buffering, I got a "corrupt zip file", if I used Application/Zip, and only changing to octet helped me...)
Also tried adding the Response.Flush(); after the header setting (and before using zip). Did not help either.
Thanks, Moshe
(I don't know how to edit an entry here, so posting another one with a correction)
\r\nI meant to say:
\r\nPlease note that all files exist, and that the files temp1.tmp and temp2.tmp are replaced from small text files (which result in a rapid and successful zipfile output)
to renamed large zipfiles, using a simple "rename" call in DOS. So now, temp1.tmp and temp2.tmp are very large files (247 MB each)
I also tried changing the names in the ArrayList of the called files to "temp1Big.tmp, temp2Big.tmp" (with respective files in the directory) but to no avail.
I read a little more about HTTP and streamed transfer, in RFC 2616, and also in the doc for HttpResponse. What I learned was that in ASP.NET the http response is, by default, buffered. If you turn off buffering, you should get a chunked transfer-encoding, which means it will send data as soon as it is written in the server. Use this:
\r\nResponse.BufferOutput= false; \r\n\r\n
..directly after Response.Clear() to turn off buffering.
\r\n\r\n
", "PostedDate": "2009-07-06T08:26:40.66-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "210200", "ThreadId": "45459", "Html": "
Works!!! Thanks!!!!!
\r\nMoshe
", "PostedDate": "2009-07-08T16:16:26.407-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "210220", "ThreadId": "45459", "Html": "Glad it helped!
", "PostedDate": "2009-07-08T17:27:45.833-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]