Hello,
\r\nI've read all the articles about memory errors and I'm still not sure why I get an OutOfMemory Exception on the Save() method
I'm just trying to create an in memory zip file, the lpMemStream variable is the MemoryStream of the file I want to compress. This code works fine until I get to about a 99MB file, then I get the OutOfMemory Exception. I am using version 1.6.3.18. Any ideas? Thanks
Dim lobjMem As New System.IO.MemoryStream()
Using lobjZipFile As Ionic.Utils.Zip.ZipFile = New Ionic.Utils.Zip.ZipFile(lobjMem)
lobjZipFile.AddFileStream("temp.txt", "", lpMemStream)
lobjZipFile.Save()
mlngLength = lobjMem.Length
\r\n Return lobjMem
End Using
Stack trace:
\r\n at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Ionic.Utils.Zip.CountingStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Ionic.Utils.Zip.CountingStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Compression.DeflateStream.InternalWrite(Byte[] array, Int32 offset, Int32 count, Boolean isAsync)
at System.IO.Compression.DeflateStream.Write(Byte[] array, Int32 offset, Int32 count)
at Ionic.Utils.Zip.ZipEntry._WriteFileData(ZipCrypto cipher, Stream s)
at Ionic.Utils.Zip.ZipEntry._EmitOne(Stream outstream, ZipCrypto& cipher)
at Ionic.Utils.Zip.ZipEntry.Write(Stream outstream)
at Ionic.Utils.Zip.ZipFile.Save()
yes -
\r\nWhat you are doing - storing a 99mb MemoryStream - is a challenge. It has nothing to do with DotNetZip. Think of a MemoryStream as a section of memory that expands its capacity automatically. It's really nice to use, but it's hungry for memory. In particular, it's hungry for memory when it expands. I don't know exactly how a MemoryStream works, but I can make a pretty good guess. When you initially create the MemoryStream, it allocates a contiguous buffer of some fixed size. Let's say it's 1k: 1024 bytes. Then you write into the buffer. Each time you write, the logic for MemoryStream checks to see if there is enough space. When you fill up 1k, it has to expand. So it allocates a new buffer. As I said I don't know how it works, but typically auto-expanding objects will double in size with each expansion. So then it allocates a 2k buffer, and copies the 1k into it. It frees the original 1k buffer. Then you keep writing, and fill up the 2k buffer. It allocates a 4k buffer, for 6k total. Copy and free. Then you fill up the 4k buffer, and so on. Pretty soon you fill up a 64mb buffer, and it has to allocate a 128mb contiguous buffer. Even if you have a 2gb machine, that might be challenging to do. There may be plenty of memory available, but not necessarily a contiguous chunk of 128mb. So you get a memory error.
\r\nWhy do you want to put a 99mb zip file in memory? 99mb seems like something that is more appropriately stored in a filesystem.
\r\n\r\n
", "PostedDate": "2009-09-02T18:15:06.493-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "230475", "ThreadId": "67797", "Html": "
Also you may want to consider moving to a more recent version of DotNetZip.
\r\n", "PostedDate": "2009-09-02T18:15:48.217-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "230695", "ThreadId": "67797", "Html": "
Thanks Cheeso. I think your reasoning is spot on and I'm not sure what else I can do at this point.
\r\nThe reason I want to put this file in memory is because I am ultimately passing the bytes of the memory stream to a web service and I was hoping to compress the bytes before I pass it. Even with the overhead of compressing the bytes my web service performs better when the bytes are compessed. I was trying to avoid writing out temporary files on the user's machine.
\r\n", "PostedDate": "2009-09-03T07:39:33.537-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "230748", "ThreadId": "67797", "Html": "
If you want to send 99mb, you are probably using a stream to do the transfer, is that right? If not, you should change your design. (And if you're using ASMX, you should change your design).
\r\nIf you ARE using streamed transfer, then you can wrap your Stream with a Ionic.Zip.DeflateStream (or GZipStream, or ZlibStream) on the sending size, and on the receiving side. These classes can compress, and de-compress, as you read or write. They are streams, so they do their work without accumulating data in memory.
\r\nThere is supposed to be some new compression capability in WCF 4.0, but I don't have any information on it.
", "PostedDate": "2009-09-03T09:44:37.457-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "230814", "ThreadId": "67797", "Html": "I am using WCF and passing a stream. (So I guess my design is good ;-)
\r\nI've considered streaming using MTOM but I'm unclear whether I need to write special code on the client and server to handle chunking. Does WCF take care of this for you?
\r\nIt's definitely not a bug in your software, I am having the same memory exception when I try to use an xmlwriter and call writer.WriteBase64(). Even if I try to chunk this method, it errors out because it appears to buffer the data before it actully writes everything out.( Flush doesn't work either)
", "PostedDate": "2009-09-03T12:02:40.19-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "230831", "ThreadId": "67797", "Html": "I don't know about the xmlwriter and WriteBase64.
\r\nGetting back to your use of the DotNetZip library, the exception you showed indicates that you were writing all the data to a memory stream. This isn't what you want.
\r\nYou want to write all the data to an output stream that does not accumulate the data in a single block of memory. The {GZip,Deflate,Zlib}Stream classes will do that.
\r\nIt's not a zip file but it is compressed, which is what you want I think.
\r\nDid you follow the steps here: http://msdn.microsoft.com/en-us/library/ms789010.aspx ? Did you enable streaming at the transport level? <httpTransport transferMode="Streamed" maxReceivedMessageSize="67108864"/> Can you read the stream at the receiver before the sender has completed transmission? That would prove that streaming is actually working.
\r\nI don't see why you'd want to go to MTOM; the idea is to compress the data for transmission, right? That should not mean an attachment. You had the idea to use a zipfile to get the compression you want, but actually it would be easier to just use a compressing stream, which means no attachment, no MTOM.
\r\n", "PostedDate": "2009-09-03T12:37:39.48-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]