Hi,
\r\nI'm trying to zip the output of DataSet.WriteXml via a stream to a zipfile (and of course re-populate a DataSet from a zipped file using DataSet.ReadXml).
\r\nThe following code creates a zip archive with one entry; however, that entry has zero length. Any ideas on what I'm doing wrong?
\r\n\r\n
DataSet1 ds = new DataSet1();\r\nds.Pricing.AddPricingRow(new DateTime(2009, 1, 1), "IBM", 100);\r\nds.Pricing.AddPricingRow(new DateTime(2009, 1, 2), "IBM", 101);\r\nds.Pricing.AddPricingRow(new DateTime(2009, 1, 3), "IBM", 102);\r\n\r\nds.WriteXml(@"c:\\temp\\test.xml"); //this works fine\r\n\r\nIonic.Zip.ZipFile zip = new Ionic.Zip.ZipFile();\r\nMemoryStream ms = new MemoryStream();\r\nIonic.Zip.ZipEntry entry = zip.AddEntry("test.xml", "", ms);\r\n\r\nds.WriteXml(ms);\r\n\r\nzip.Save(@"c:\\temp\\test.xml.zip");\r\nzip.Dispose();\r\n\r\n
Any pointers greatly appreciated. Thank you. WLM
\r\n", "PostedDate": "2009-09-29T18:32:25.42-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "240247", "ThreadId": "70516", "Html": "
Maybe try ms.Seek(0,SeekOrigin.Begin) before calling zip.Save().
\r\nI think you've written to the MemoryStream, but the pointer in the MemoryStream is at the end of the stream, at the time you call zip.Save(). You want it at the beginning.
", "PostedDate": "2009-09-29T20:57:38.137-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "240324", "ThreadId": "70516", "Html": "Beautiful! Worked like a charm. Thanks.
", "PostedDate": "2009-09-30T02:54:14.44-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "240508", "ThreadId": "70516", "Html": "Whoops--I spoke too soon.
\r\nSetting the stream position to zero did indeed fix the problem in my simple example above; however, in the "real world", I need to persist very large datasets. When using the MemoryStream technique above I got an OutOfMemoryException. Looks like I need to write directly to the zip file. Is there a way to accomplish this using streams? I realize I could just call the standard DataSet.WriteXml method, zip the resulting file to an archive, and then delete the original file, but I would prefer to do it in one fell swoop without that workaround. Any thoughts on how to accomplish this are appreciated.
\r\nMany thanks,
WLM
| \r\n When ZipFile.Save() is called, all entries are written out to the output. For any entries added as streams with ZipFile.AddEntry(), zipFile.Save calls Read() on those streams. In other words, ZipFile requires a stream that can be read. On the other hand, DataSet can put its data only on a stream that accepts Write(). DataSet requires a stream that can be written. \r\nWhat you need is two threads and an adapter stream - Something like the BlockingStream Stephen Taub provided in MSDN Magazine . Except I think his class implementation allocates an unlimited number of buffers, which may get you into the out-of-memory condition again. What you want is a BlockingStream with a finite set of buffers. \r\nI previously had a need for something like that so I wrote one. Enclosed below. To use it: \r\n\r\n | \r\n
// thread #1\r\nAdapterStream as = new AdapterStream(8192); // 8k buffer\r\ndataset.WriteXml(as); \r\nas.BlockSync(); // returns when ZipFile.Save() completes\r\n\r\n// thread #2\r\nusing (var zip = new ZipFile())\r\n{\r\n zip.AddEntry("mydata.xml", "", as); \r\n zip.Save(whatever);\r\n}\r\n\r\n\r\n
\r\n
AdapterStream looks like this:
\r\nusing System;\r\nusing System.Collections.Generic;\r\nusing System.IO;\r\nusing System.Threading;\r\n\r\nnamespace Cheeso.IO\r\n{\r\n /// <summary>\r\n /// A Stream for passing data between a producer thread and a consumer thread.\r\n /// </summary>\r\n ///\r\n /// <remarks>\r\n /// Probably want this to use at least 2 blocks, each with a\r\n /// separate lock, to allow more concurrency.\r\n /// </remarks>\r\n internal class AdapterStream : Stream\r\n {\r\n /// <summary>Used as a monitor to protect state accessible to callers of Read.</summary>\r\n private object _lockForAll;\r\n \r\n private byte[] _circularBuffer;\r\n private int _posnWrite;\r\n private int _posnRead;\r\n private int _bytesAvailable;\r\n\r\n \r\n /// <summary>\r\n /// Event used to signal when no more data will be written to the stream.\r\n /// </summary>\r\n private ManualResetEvent _blockSync;\r\n \r\n /// <summary>\r\n /// Event used to signal when the circular buffer has space\r\n /// available for writing.\r\n /// </summary>\r\n private ManualResetEvent _spaceAvailable;\r\n \r\n /// <summary>\r\n /// Event used to signal that data is available for reading.\r\n /// </summary>\r\n private ManualResetEvent _dataAvailable;\r\n \r\n /// <summary>Array of events waited on by readers.</summary>\r\n private WaitHandle[] _readEvents;\r\n \r\n /// <summary>The index into _waitHandles for the _blockSync event.</summary>\r\n private int _blockSyncIndex;\r\n\r\n\r\n /// <summary>Initializes the blocking stream.</summary>\r\n public AdapterStream(int size)\r\n {\r\n _circularBuffer= new byte[size];\r\n _blockSync = new ManualResetEvent(false);\r\n _dataAvailable = new ManualResetEvent(false);\r\n _spaceAvailable = new ManualResetEvent(true);\r\n _readEvents = new WaitHandle[] { _dataAvailable, _blockSync };\r\n _blockSyncIndex= 1;\r\n //_lockForRead = new object();\r\n _lockForAll = new object();\r\n }\r\n\r\n \r\n /// <summary>Determines whether data can be read from the stream.</summary>\r\n public override bool CanRead { get { return true; } }\r\n /// <summary>Determines whether the stream can be seeked.</summary>\r\n public override bool CanSeek { get { return false; } }\r\n /// <summary>Determines whether data can be written to the stream.</summary>\r\n public override bool CanWrite { get { return true; } }\r\n\r\n /// <summary>Flush is a no-op.</summary>\r\n public override void Flush() { }\r\n \r\n /// <summary>Not supported.</summary>\r\n public override long Length { get { throw new NotSupportedException(); } }\r\n \r\n /// <summary>Not supported.</summary>\r\n public override long Position\r\n {\r\n get { throw new NotSupportedException(); }\r\n set { throw new NotSupportedException(); }\r\n }\r\n\r\n /// <summary>Not supported.</summary>\r\n public override long Seek(long offset, SeekOrigin origin)\r\n { throw new NotSupportedException(); }\r\n\r\n \r\n /// <summary>Not supported.</summary>\r\n public override void SetLength(long value) { throw new NotSupportedException(); }\r\n\r\n \r\n /// <summary>\r\n /// Reads a sequence of bytes from the stream. Read will block\r\n /// until at least one byte is available for reading or until\r\n /// the stream has been ended for writing.\r\n /// </summary>\r\n /// <param name="buffer">\r\n /// An array of bytes. When Read returns, the buffer contains the specified\r\n /// byte array with the values between offset and (offset + count - 1) replaced\r\n /// by the bytes read from the current source.\r\n /// </param>\r\n /// <param name="offset">\r\n /// The zero-based byte offset in buffer at which to begin storing the data read\r\n /// from the current stream.\r\n /// </param>\r\n /// <param name="count">The maximum number of bytes to be read from the current stream.</param>\r\n /// <returns>\r\n /// The total number of bytes read into the buffer. This can be less than the\r\n /// number of bytes requested if that many bytes are not currently available,\r\n /// or zero if the end of the stream has been reached.\r\n /// </returns>\r\n /// <remarks>This method is thread-safe.</remarks>\r\n public override int Read(byte[] buffer, int offset, int count)\r\n {\r\n // Validate all arguments and instance state\r\n if (buffer == null) throw new ArgumentNullException("buffer");\r\n if (offset < 0 || offset >= buffer.Length) throw new ArgumentOutOfRangeException("offset");\r\n if (count < 0 || offset + count > buffer.Length) throw new ArgumentOutOfRangeException("count");\r\n if (_dataAvailable == null) throw new ObjectDisposedException(GetType().Name);\r\n\r\n // If no data has been requested, we don't need to do anything\r\n if (count == 0) return 0;\r\n \r\n // Loop until we get data or until writing has completed\r\n while (true)\r\n {\r\n // Wait for either data to be available or for the sequence to be\r\n // completed.\r\n int handleIndex = WaitHandle.WaitAny(_readEvents);\r\n \r\n lock (_lockForAll)\r\n {\r\n // If nothing is available, loop around to wait again.\r\n // This is the blocking part. \r\n if (_bytesAvailable==0)\r\n {\r\n if (handleIndex == _blockSyncIndex)\r\n {\r\n // All done with this item in the sequence,\r\n // nothing more to read.\r\n\r\n // Signal that the writer can begin writing again (new sequence).\r\n _spaceAvailable.Set();\r\n\r\n // Turn off the sequence break. It's been handled. \r\n _blockSync.Reset();\r\n return 0; // EOF\r\n }\r\n\r\n // In this case, there's no data available to Read, but the\r\n // writer hasn't completed the writes on this sequence. So\r\n // loop around, block, and try to read again. This should\r\n // never happen, because _bytesAvailable should be nonzero\r\n // if _dataAvailable is signalled. No harm though.\r\n continue;\r\n }\r\n \r\n // Read as many bytes as possible\r\n int bytesToRead= (_bytesAvailable > count) ? count : _bytesAvailable;\r\n \r\n // Will we need to wrap around in the circ buffer to satisfy the read?\r\n if (_posnRead + bytesToRead > _circularBuffer.Length)\r\n {\r\n // yes - need to wrap around \r\n int thisBatch = _circularBuffer.Length - _posnRead;\r\n Buffer.BlockCopy(_circularBuffer, _posnRead, buffer, offset, thisBatch);\r\n _posnRead = 0;\r\n offset += thisBatch; // matters only if we loop until the read is done.\r\n thisBatch = bytesToRead - thisBatch;\r\n Buffer.BlockCopy(_circularBuffer, _posnRead, buffer, offset, thisBatch);\r\n _posnRead += thisBatch;\r\n }\r\n else\r\n {\r\n // Satisfy the read without wrap.\r\n Buffer.BlockCopy(_circularBuffer, _posnRead, buffer, offset, bytesToRead);\r\n _posnRead += bytesToRead;\r\n }\r\n \r\n _bytesAvailable -= bytesToRead;\r\n if (_bytesAvailable == 0) _dataAvailable.Reset();\r\n \r\n _spaceAvailable.Set();\r\n \r\n return bytesToRead;\r\n }\r\n }\r\n }\r\n \r\n\r\n /// <summary>Writes a sequence of bytes to the stream.</summary>\r\n /// <param name="buffer">An array of bytes. Write copies count bytes from buffer to the stream.</param>\r\n /// <param name="offset">The zero-based byte offset in buffer at which to begin copying bytes to the stream.</param>\r\n /// <param name="count">The number of bytes to be written to the current stream.</param>\r\n /// <remarks>This method is thread-safe, and will block until the write can be completed.</remarks>\r\n public override void Write(byte[] buffer, int offset, int count)\r\n {\r\n // Validate all arguments and instance state\r\n if (buffer == null) throw new ArgumentNullException("buffer");\r\n if (offset < 0 || offset >= buffer.Length) throw new ArgumentOutOfRangeException("offset");\r\n if (count < 0 || offset + count > buffer.Length) throw new ArgumentOutOfRangeException("count");\r\n if (_dataAvailable == null) throw new ObjectDisposedException(GetType().Name);\r\n\r\n // If no data is being written, nothing to do.\r\n if (count == 0) return;\r\n\r\n while (count > 0)\r\n {\r\n // Wait for space to become available.\r\n _spaceAvailable.WaitOne();\r\n \r\n // Copy the written data into the circularBuffer.\r\n // Make sure to signal to any readers that data is available.\r\n lock (_lockForAll)\r\n {\r\n int bufferSpaceAvailable = (_posnRead <= _posnWrite)\r\n ? _circularBuffer.Length - _posnWrite\r\n : _posnRead - _posnWrite - 1;\r\n\r\n int bytesToWrite = (count < bufferSpaceAvailable) ? count : bufferSpaceAvailable;\r\n \r\n Buffer.BlockCopy(buffer, offset, _circularBuffer, _posnWrite, bytesToWrite);\r\n count -= bytesToWrite;\r\n offset += bytesToWrite; \r\n _bytesAvailable += bytesToWrite;\r\n _posnWrite += bytesToWrite;\r\n if (_posnWrite >= _circularBuffer.Length)\r\n _posnWrite = 0;\r\n _dataAvailable.Set();\r\n\r\n if (_posnWrite==_posnRead)\r\n _spaceAvailable.Reset();\r\n }\r\n }\r\n }\r\n\r\n /// <summary>\r\n /// Called by the writer to signal to the reader that a block of\r\n /// data is complete.\r\n /// </summary>\r\n ///\r\n /// <remarks>\r\n /// This should be called by the writer. This method is thread-safe. Subsequent\r\n /// calls to Write will block until the reader completes all reads of the\r\n /// just-completed block.\r\n /// </remarks>\r\n public void BlockSync()\r\n {\r\n if (_dataAvailable == null) throw new ObjectDisposedException(GetType().Name);\r\n lock (_lockForAll)\r\n {\r\n // This will signal the reader that there's a break.\r\n _blockSync.Set();\r\n\r\n // This will block the writer til the reader reaches EOF.\r\n // We don't want the writing filling the buffer again, until\r\n // the reader has finished reading this block.\r\n _spaceAvailable.Reset();\r\n }\r\n }\r\n\r\n\r\n \r\n private void FreeEvent(ref ManualResetEvent e)\r\n {\r\n if (e != null)\r\n {\r\n e.Close();\r\n e = null;\r\n }\r\n }\r\n\r\n\r\n \r\n /// <summary>Closes the stream and releases all resources associated with it.</summary>\r\n /// <remarks>This method is not thread-safe.</remarks>\r\n public override void Close()\r\n {\r\n base.Close();\r\n \r\n // Free the events\r\n FreeEvent(ref _dataAvailable);\r\n FreeEvent(ref _spaceAvailable);\r\n FreeEvent(ref _blockSync);\r\n }\r\n }\r\n}\r\n\r\n\r\n
\r\n
", "PostedDate": "2009-09-30T10:31:13.347-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "240690", "ThreadId": "70516", "Html": "
Hi Cheeso,
\r\nThanks for the reply and code. I understand what you are saying from a high level, but the ability to create something like your AdapterStream is beyond my meager skills. Unfortunately the AdapterStream class as written does not compile. In the read method, the bytesToCopy variable in "return bytesToCopy" is not recognized. Perhaps you meant _bytesAvailable?
\r\nI also think some of the methods that are stubbed out with NotSupportedExceptions are being called behind the scenes by the ZipFile class.
\r\nI attempted to get this running on different threads but I think there are some other things going on in addition to the fact that I probably didn't implement the treading properly.
\r\nBest regards,
WLM
\r\n
using System;\r\nusing System.Collections.Generic;\r\nusing System.Linq;\r\nusing System.Text;\r\nusing Ionic.Zip;\r\nusing System.Threading;\r\n\r\nnamespace CompressionTest\r\n{\r\n\tclass DataStore\r\n\t{\r\n\t\tprivate DataSet1 m_ds;\r\n\t\tprivate AdapterStream m_as;\r\n\r\n\t\tpublic void Write()\r\n\t\t{\r\n\t\t\tm_ds = new DataSet1();\r\n\t\t\tm_ds.Pricing.AddPricingRow(new DateTime(2009, 1, 1), "IBM", 100);\r\n\t\t\tm_ds.Pricing.AddPricingRow(new DateTime(2009, 1, 2), "IBM", 101);\r\n\t\t\tm_ds.Pricing.AddPricingRow(new DateTime(2009, 1, 3), "IBM", 102);\r\n\r\n\t\t\tm_ds.WriteXml(@"c:\\temp\\test.xml"); //this works fine\r\n\r\n\t\t\tThread t = new Thread(new ThreadStart(Thread1));\r\n\t\t\tt.Start();\r\n\r\n\t\t\tt = new Thread(new ThreadStart(Thread2));\r\n\t\t\tt.Start();\r\n\r\n\t\t}\r\n\r\n\t\tprivate void Thread1()\r\n\t\t{\r\n\t\t\tm_as = new AdapterStream(8192); // 8k buffer\r\n\t\t\tm_ds.WriteXml(m_as);\r\n\t\t\tm_as.BlockSync(); // returns when ZipFile.Save() completes\r\n\t\t}\r\n\r\n\t\tprivate void Thread2()\r\n\t\t{\r\n\t\t\tusing (var zip = new ZipFile())\r\n\t\t\t{\r\n\t\t\t\tzip.AddEntry("test.xml", "", m_as);\r\n\t\t\t\tzip.Save(@"c:\\temp\\test.xml.zip");\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n}\r\n\r\n
\r\n
| \r\n Hey Wlm, ah, that code I posted was not quite right. I've fixed it and updated it in the post above. It compiles now :0. Anyway, this is the code I used to verify that the idea works: \r\n | \r\n
private Cheeso.IO.AdapterStream as1;\r\nprivate DataSet ds1;\r\n\r\nprivate void SaveDataset(object ignored)\r\n{\r\n System.Console.WriteLine("writing dataset to as1...");\r\n ds1.WriteXml(as1); \r\n as1.BlockSync(); \r\n}\r\n\r\npublic void Run()\r\n{\r\n System.Data.SqlClient.SqlConnection c1= new System.Data.SqlClient.SqlConnection(connstring1);\r\n\r\n System.Data.SqlClient.SqlDataAdapter da = new System.Data.SqlClient.SqlDataAdapter();\r\n da.SelectCommand= new System.Data.SqlClient.SqlCommand(strSelect);\r\n da.SelectCommand.Connection= c1;\r\n\r\n System.Console.WriteLine("Create dataset...");\r\n\r\n ds1 = new DataSet();\r\n System.Console.WriteLine("Fill dataset...");\r\n da.Fill(ds1, "Invoices");\r\n\r\n System.Console.WriteLine("Save dataset to xml file...");\r\n ds1.WriteXml(xmlOutputFile); //this works fine\r\n\r\n as1 = new Cheeso.IO.AdapterStream(8192);\r\n\r\n ThreadPool.QueueUserWorkItem(new WaitCallback(SaveDataset), null);\r\n\r\n using(Ionic.Zip.ZipFile zip = new Ionic.Zip.ZipFile())\r\n {\r\n Ionic.Zip.ZipEntry entry = zip.AddEntry(zipEntryName, "", as1);\r\n zip.Save(zipFileName);\r\n }\r\n\r\n // compare datasets\r\n DataSet ds2 = new DataSet();\r\n ds2.ReadXml(xmlOutputFile);\r\n\r\n DataSet ds3 = new DataSet();\r\n using(Ionic.Zip.ZipFile zip = Ionic.Zip.ZipFile.Read(zipFileName))\r\n {\r\n // load a DataSet directly from a Stream created for the entry in the zip\r\n var s = zip[zipEntryName].OpenReader();\r\n ds3.ReadXml(s);\r\n }\r\n\r\n CompareDatasets(ds2, ds3);\r\n}\r\n\r\n\r\n
| \r\n The DataSet is written to the AdapterStream within SaveDataset(). Each call to AdapterStream.Write() buffers the data. At the same time, the zip.Save() calls Read() on the Stream and gets the data out of the buffer. \r\n | \r\n
Cheeso,
\r\nWow, thanks for taking the time to straightened that out. I did get one NotSupportedException in the Length property of AdapterStream, but haven't been able to replicate it since. That said, I set a breakpoint and the ZipFile class is definitely calling the Length method and handling the exception. I changed it to return _circularBuffer.Length with no ill effects as far as I can tell. Otherwise it works great. Hopefully these postings will help someone else out down the road. Thanks.
\r\nWLM
", "PostedDate": "2009-10-02T02:57:20.797-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "241387", "ThreadId": "70516", "Html": "| \r\n > return _circularBuffer.Length \r\nThat is not the correct value to return for the stream. \r\nFor some streams, such as non-seekable (forward-only) read-only streams, the Stream.Length property is only knowable after all bytes have actually been read. The AdapterStream is one such stream type. The DotNetZip library is aware that this occurs for some stream types, which is why it handles the exception thrown. \r\nIt isn't necessary, and it is incorrect, for you to change the Length property to return _circularBuffer.Length . \r\nIt could result in a broken zip file - in other words a zipentry will not contain all the bytes from your dataset. \r\ngood luck. \r\n\r\n | \r\n
Understood. I'll quickly change that back to the way it was. Some of my xml files were 100's of megs. They compress down nicely now, usually by around 95%.
", "PostedDate": "2009-10-02T13:42:58.293-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "241506", "ThreadId": "70516", "Html": "XML is very compressible. Particularly DataSet XML.
", "PostedDate": "2009-10-02T14:08:56.843-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "241827", "ThreadId": "70516", "Html": "WLM,
\r\nfollowup. I modified DotNetZip to add a new feature to deal with this situation more cleanly.
\r\nIf you download v1.9.0.12 (available now) you will be able to do something like this to save a dataset into a zip file, without writing a file to the filesystem.
\r\nDataSet ds1 = new DataSet();\r\nda.Fill(ds1, "Invoices");\r\nusing(Ionic.Zip.ZipFile zip = new Ionic.Zip.ZipFile())\r\n{\r\n zip.AddEntry(zipEntryName, (name,stream) => ds1.WriteXml(stream) );\r\n zip.Save(zipFileName);\r\n}\r\n\r\n
Using that programming model requires .NET 3.5, for the anonymous methods.
\r\nI recommend using this new feature over the AdapterStream thing I sent you.
", "PostedDate": "2009-10-03T23:57:45.693-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "241990", "ThreadId": "70516", "Html": "Works perfectly. Very nice to have it all contained within DotNetZip, hiding the complexity from neophytes like me.
\r\nThanks Cheeso.
\r\n", "PostedDate": "2009-10-04T18:55:45.64-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "242047", "ThreadId": "70516", "Html": "
just to clarify, can we close the supplied stream at the end of the write? many xxx.write(stream) methods are closing the stream at the end.
", "PostedDate": "2009-10-05T00:27:38.133-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "242213", "ThreadId": "70516", "Html": "| \r\n GM, \r\nwhat are you asking here? Do you want the application to be responsible for closing the stream? Or do you want DotNetZip to be responsible for closing the stream? Or ... are you simply asking for clarification on the way it works? \r\nThe way it works now, is that DotNetZip opens and configures the stream, before providing it to the application when calling the delegate. DotNetZip also takes responsibility for closing the stream, after the delegate returns. \r\nOn the other hand, the doc for System.IO.Stream says that \r\nIf the stream is already closed, a call to Close throws no exceptions.\r\n Do you have a particular implementation of xxx.Write(stream) that closes the stream, and causes a problem ? \r\n\r\n | \r\n
I wanted to know if it is allowed to close the supplied stream, because there are a lot of existing methods in the framework that can be used, and a lot are closing the stream.
\r\nmy understanding is now: there is no problem.
", "PostedDate": "2009-10-06T06:34:23.053-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]