The following code throws an exception when I write some data in front of the zip file. It works fine for start=0.
\r\nWhat am I doing wrong?
\r\n\r\n
using System;\r\nusing System.Collections.Generic;\r\nusing System.Linq;\r\nusing System.Text;\r\nusing System.IO;\r\nusing Ionic.Zip;\r\nusing System.Diagnostics;\r\n\r\nnamespace ZipTest\r\n{\r\n\tclass Program\r\n\t{\r\n\t\tconst int start = 64;\r\n\t\tstatic long size;\r\n\t\tstatic void Save()\r\n\t\t{\r\n\t\t\tusing (FileStream f = File.Create("test.zip"))\r\n\t\t\t{\r\n\t\t\t\tf.Write(new byte[start], 0, start);\r\n\t\t\t\tZipFile zip = new ZipFile(Encoding.UTF8);\r\n\t\t\t\tzip.AddEntry("a.txt", "abc");\r\n\t\t\t\tzip.Save(f);\r\n\t\t\t\tsize = f.Length;\r\n\t\t\t}\r\n\t\t}\r\n\r\n\t\tstatic void Load()\r\n\t\t{\r\n\t\t\tusing (FileStream f = File.OpenRead("test.zip"))\r\n\t\t\t{\r\n\t\t\t\tf.Seek(start, SeekOrigin.Begin);\r\n\t\t\t\tZipFile zip = ZipFile.Read(f, Encoding.UTF8);//throws\r\n\t\t\t\t/*An unhandled exception of type 'Ionic.Zip.BadReadException' occurred in Ionic.Zip.Reduced.dll\r\n\t\t\t\tAdditional information: ZipEntry::ReadHeader(): Bad signature (0xCADDAD9C) at position 0x00000074*/\r\n\t\t\t}\r\n\t\t}\r\n\r\n\t\tstatic void Main(string[] args)\r\n\t\t{\r\n\t\t\tSave();\r\n\t\t\tLoad();\r\n\t\t}\r\n\t}\r\n}\r\n\r\n\r\n
Edit: Sorry, didn't see that you were also starting to write at that position - my original anwer was wrong. Appears it's either how the stream is written or parsed.
\r\n\r\n
", "PostedDate": "2010-04-19T00:40:42.947-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "432782", "ThreadId": "209615", "Html": "
Hi MasterOfChaos,
\r\nI think the problem is to do with the way ZipFile calculates offsets to data blocks within the zip stream. Some of it is using absolute file positions including the padding, and other parts are relative positions from the start of the zip data - they don't seem to quite marry up between the read and write operations.
\r\nYou could work around this by zipping the data to a temporary file (or memory) and then embedding the output from that into your file stream. This will force all offsets to be relative to the start of the zip data and you can do the reverse later to unzip them again.
\r\nBe aware that if you do this you'll lose the ability to extract data directly from the composite file using WinZip (which currently works with your test.zip), so it depends on what you're trying to achieve. I've put some sample code below - if you're creating large zip files then memory may not be the best place to store the temporary zip data.
\r\nCheers,
\r\nMike
\r\n\r\n
namespace Discussion_209615\r\n{\r\n static class Program\r\n {\r\n\r\n const int start = 64;\r\n const string fileName = "test-pad.zip";\r\n\r\n static long size;\r\n \r\n static void Save()\r\n {\r\n using (FileStream f = File.Create(fileName))\r\n {\r\n f.Write(new byte[start], 0, start);\r\n using (MemoryStream m = new MemoryStream())\r\n {\r\n // zip into a memory stream\r\n ZipFile zip = new ZipFile(Encoding.UTF8);\r\n zip.AddEntry("a.txt", "abc");\r\n zip.Save(m);\r\n m.Flush();\r\n m.Seek(0, SeekOrigin.Begin);\r\n // read the data to a byte array\r\n byte[] data = new byte[m.Length];\r\n if (m.Read(data, 0, data.Length) != data.Length) throw new System.InvalidOperationException();\r\n // write to the file\r\n f.Write(data, 0, data.Length);\r\n }\r\n size = f.Length;\r\n }\r\n }\r\n\r\n static void Load()\r\n {\r\n using (FileStream f = File.OpenRead(fileName))\r\n {\r\n f.Seek(start, SeekOrigin.Begin);\r\n // read from the file\r\n byte[] data = new byte[f.Length - f.Position];\r\n if (f.Read(data, 0, data.Length) != data.Length) throw new System.InvalidOperationException();\r\n using (MemoryStream m = new MemoryStream())\r\n {\r\n // write to a memory stream\r\n m.Write(data, 0, data.Length);\r\n m.Flush();\r\n m.Seek(0, SeekOrigin.Begin);\r\n // read the zip data\r\n ZipFile zip = ZipFile.Read(m, Encoding.UTF8);//throws\r\n zip.ExtractAll("C:\\\\temp\\\\DotNetZip\\\\debug\\\\Discussion_209615\\\\temp");\r\n }\r\n /*An unhandled exception of type 'Ionic.Zip.BadReadException' occurred in Ionic.Zip.Reduced.dll\r\n Additional information: ZipEntry::ReadHeader(): Bad signature (0xCADDAD9C) at position 0x00000074*/\r\n }\r\n }\r\n\r\n static void Main(string[] args)\r\n {\r\n Save();\r\n Load();\r\n }\r\n\r\n }\r\n\r\n}\r\n\r\n
\r\nusing System;\r\nusing System.IO;\r\n\r\n\r\nnamespace Vfs.Transfer.Util\r\n{\r\n /// <summary>\r\n /// A stream that provides streamed access to a given range within\r\n /// another stream, but does not allow reading or writing outside\r\n /// this block's boundaries.\r\n /// </summary>\r\n public class ChunkStream : Stream\r\n {\r\n /// <summary>\r\n /// The stream from which the data is read.\r\n /// </summary>\r\n public Stream DecoratedStream { get; private set; }\r\n\r\n /// <summary>\r\n /// The offset (starting point) of the chunk in the\r\n /// decorated stream.\r\n /// </summary>\r\n public long Offset { get; private set; }\r\n\r\n /// <summary>\r\n /// The size of the chunk. This is this stream's length.\r\n /// </summary>\r\n public long ChunkSize { get; set; }\r\n\r\n private long position;\r\n\r\n /// <summary>\r\n /// The virtual position within the chunk. The current position\r\n /// within the <see cref="DecoratedStream"/> is\r\n /// <see cref="Offset"/> + <see cref="Position"/>. \r\n /// </summary>\r\n public override long Position\r\n {\r\n get { return position; }\r\n set\r\n {\r\n if(position < 0 || position > ChunkSize)\r\n {\r\n string msg = "Cannot set Position property to '{0}': Position cannot be negative or bigger than the chunk size of [{1}] bytes.";\r\n msg = String.Format(msg, value, ChunkSize);\r\n throw new ArgumentOutOfRangeException("value", msg);\r\n }\r\n position = value;\r\n\r\n //not a problem if we set the position above the stream length\r\n //common implementations just correct it\r\n DecoratedStream.Position = Offset + value;\r\n }\r\n }\r\n\r\n /// <summary>\r\n /// Initializes a new instance of the <see cref="T:System.IO.Stream"/> class. \r\n /// </summary>\r\n public ChunkStream(Stream decoratedStream, long chunkSize, long offset, bool initStreamPosition)\r\n {\r\n DecoratedStream = decoratedStream;\r\n ChunkSize = chunkSize;\r\n Offset = offset;\r\n\r\n if (initStreamPosition)\r\n {\r\n decoratedStream.Position = Offset;\r\n }\r\n }\r\n\r\n /// <summary>\r\n /// When overridden in a derived class, reads a sequence of bytes from the current stream and advances the position within the stream by the number of bytes read.\r\n /// </summary>\r\n /// <returns>\r\n /// The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.\r\n /// </returns>\r\n /// <param name="buffer">An array of bytes. When this method returns, the buffer contains the specified byte array with the values between <paramref name="offset"/> and (<paramref name="offset"/> + <paramref name="count"/> - 1) replaced by the bytes read from the current source. \r\n /// </param><param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin storing the data read from the current stream. \r\n /// </param><param name="count">The maximum number of bytes to be read from the current stream. \r\n /// </param><exception cref="T:System.ArgumentException">The sum of <paramref name="offset"/> and <paramref name="count"/> is larger than the buffer length. \r\n /// </exception><exception cref="T:System.ArgumentNullException"><paramref name="buffer"/> is null. \r\n /// </exception><exception cref="T:System.ArgumentOutOfRangeException"><paramref name="offset"/> or <paramref name="count"/> is negative. \r\n /// </exception><exception cref="T:System.IO.IOException">An I/O error occurs. \r\n /// </exception><exception cref="T:System.NotSupportedException">The stream does not support reading. \r\n /// </exception><exception cref="T:System.ObjectDisposedException">Methods were called after the stream was closed. \r\n /// </exception><filterpriority>1</filterpriority>\r\n public override int Read(byte[] buffer, int offset, int count)\r\n {\r\n ValidateReadWriteParams(buffer, offset, count);\r\n\r\n //do not read further than the chunk\r\n int bytesToRead = (int)Math.Min(count, ChunkSize - Position);\r\n\r\n //if we do not have anything to read, don't read at all - this closes underlying stream\r\n //implementations!!!\r\n if(bytesToRead == 0) return 0;\r\n \r\n int receivedBytes = DecoratedStream.Read(buffer, offset, bytesToRead);\r\n \r\n //update received bytes\r\n position += receivedBytes;\r\n\r\n return receivedBytes;\r\n }\r\n\r\n\r\n /// <summary>\r\n /// When overridden in a derived class, writes a sequence of bytes to the current stream and advances the current position within this stream by the number of bytes written.\r\n /// </summary>\r\n /// <param name="buffer">An array of bytes. This method copies <paramref name="count"/> bytes from <paramref name="buffer"/> to the current stream. \r\n /// </param><param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin copying bytes to the current stream. \r\n /// </param><param name="count">The number of bytes to be written to the current stream. \r\n /// </param><exception cref="T:System.ArgumentException">The sum of <paramref name="offset"/> and <paramref name="count"/> is greater than the buffer length. \r\n /// </exception><exception cref="T:System.ArgumentNullException"><paramref name="buffer"/> is null. \r\n /// </exception><exception cref="T:System.ArgumentOutOfRangeException"><paramref name="offset"/> or <paramref name="count"/> is negative. \r\n /// </exception><exception cref="T:System.IO.IOException">An I/O error occurs. Also thrown if an attempt to write\r\n /// beyond the block size is made.\r\n /// </exception><exception cref="T:System.NotSupportedException">The stream does not support writing. \r\n /// </exception><exception cref="T:System.ObjectDisposedException">Methods were called after the stream was closed. \r\n /// </exception><filterpriority>1</filterpriority>\r\n public override void Write(byte[] buffer, int offset, int count)\r\n {\r\n ValidateReadWriteParams(buffer, offset, count);\r\n\r\n long remaining = ChunkSize - Position;\r\n if(count > remaining)\r\n {\r\n string msg = "Blocked attempt to write block of [{0}] bytes - write goes beyond the current chunk. Chunk size is [{1}], stream position is [{2}].";\r\n msg = String.Format(msg, count, ChunkSize, Position);\r\n throw new IOException(msg);\r\n }\r\n\r\n //do not read further than the chunk\r\n int bytesToWrite = (int)Math.Min(count, ChunkSize - Position);\r\n\r\n //if there is nothing to write, really, don't invoke the stream\r\n if (bytesToWrite == 0) return;\r\n\r\n //write data and advance position\r\n DecoratedStream.Write(buffer, offset, bytesToWrite);\r\n position += bytesToWrite;\r\n }\r\n\r\n\r\n private static void ValidateReadWriteParams(byte[] buffer, int offset, int count)\r\n {\r\n if (buffer == null) throw new ArgumentNullException("buffer");\r\n\r\n if (offset < 0)\r\n {\r\n throw new ArgumentOutOfRangeException("offset", "Offset cannot be negative.");\r\n }\r\n if (count < 0)\r\n {\r\n throw new ArgumentOutOfRangeException("count", "Number of bytes to read cannot be negative.");\r\n }\r\n if ((buffer.Length - offset) < count)\r\n {\r\n throw new ArgumentException("Invalid offset length.");\r\n }\r\n }\r\n\r\n\r\n public override void Flush()\r\n {\r\n DecoratedStream.Flush();\r\n }\r\n\r\n /// <summary>\r\n /// When overridden in a derived class, sets the position within the current stream.\r\n /// </summary>\r\n /// <returns>\r\n /// The new position within the current stream.\r\n /// </returns>\r\n /// <param name="offset">A byte offset relative to the <paramref name="origin"/> parameter. \r\n /// </param><param name="origin">A value of type <see cref="T:System.IO.SeekOrigin"/> indicating the reference point used to obtain the new position. \r\n /// </param><exception cref="T:System.IO.IOException">An I/O error occurs. \r\n /// </exception><exception cref="T:System.NotSupportedException">The stream does not support seeking, such as if the stream is constructed from a pipe or console output. \r\n /// </exception><exception cref="T:System.ObjectDisposedException">Methods were called after the stream was closed. \r\n /// </exception><filterpriority>1</filterpriority>\r\n public override long Seek(long offset, SeekOrigin origin)\r\n {\r\n long proposedPosition;\r\n\r\n switch(origin)\r\n {\r\n case SeekOrigin.Begin:\r\n proposedPosition = offset;\r\n break;\r\n case SeekOrigin.Current:\r\n proposedPosition = Position + offset;\r\n break;\r\n case SeekOrigin.End:\r\n proposedPosition = ChunkSize + offset;\r\n break;\r\n default:\r\n throw new ArgumentOutOfRangeException("origin");\r\n }\r\n\r\n if(proposedPosition < 0 || proposedPosition > ChunkSize)\r\n {\r\n throw new IOException("Invalid offset with regards to origin");\r\n }\r\n\r\n Position = proposedPosition;\r\n return proposedPosition;\r\n }\r\n\r\n public override void SetLength(long value)\r\n {\r\n throw new NotSupportedException();\r\n }\r\n\r\n\r\n public override bool CanRead\r\n {\r\n get { return DecoratedStream.CanRead; }\r\n }\r\n\r\n public override bool CanSeek\r\n {\r\n get { return DecoratedStream.CanSeek; }\r\n }\r\n\r\n public override bool CanWrite\r\n {\r\n get { return DecoratedStream.CanWrite; }\r\n }\r\n\r\n public override long Length\r\n {\r\n get { return ChunkSize; }\r\n }\r\n }\r\n}\r\n",
"PostedDate": "2010-04-19T11:36:19.977-07:00",
"UserRole": null,
"MarkedAsAnswerDate": null
},
{
"Id": "432822",
"ThreadId": "209615",
"Html": "@hardcodet - I'm probably missing something subtle, but wouldn't a regular FileStream do the same thing with chunking in this scenario instead of the MemoryStream in my example? The zip library seems to stream the input and output rather than load everything in one go...
", "PostedDate": "2010-04-19T11:52:11.51-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "432829", "ThreadId": "209615", "Html": "Pointy,\r\n\r\nAbsolutely - if I've read the code correctly, the ChunkStream does pretty much the same thing as your implementation (hiding the real stream, and just presenting a chunk of data as a stream that can be addressed with absolute and relative positions). The difference is that the ChunkStream is a bit more generic (you can wrap the original stream with a single line of code), and it might be a bit more resource friendly in terms of memory consumption in case you're dealing with big files (you wouldn't want hundreds of MB in RAM). But nothing wrong with your approach :)", "PostedDate": "2010-04-19T12:15:24.027-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "432938", "ThreadId": "209615", "Html": "Cool - pretty much what I thought. I was just checking there wasn't a gotcha hiding somewhere...
", "PostedDate": "2010-04-19T16:38:04.73-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "433857", "ThreadId": "209615", "Html": "Of course I can work around the issue by making ZipFile believe it is reading/writing at the beginning of the stream.
\r\nBut I think the documentation stated that I should be able to read/write at any position in a stream. And normal archivers being able to open my file is definitely nice, but not necessary.
\r\nZipFile.Write seems to get the current position of the stream and use that as the starting offset for absolute adresses in the zipfile. ZipFile.Read on the other hand seems to use 0 always.
\r\nIMO the Read behaviour is better as default, since Position is not a well defined concept in all streams. Instead I should be able to pass that start offset manually to ZipFile.
Then I can use any position I need in the cases where I want to be compatible with other archivers, and get position indepence(which is a very desirable feature IMO) by default.
I would have prefered if zip only used relative adresses, but without a time-machine it'll be hard to fix that.
", "PostedDate": "2010-04-21T13:39:10.713-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "433875", "ThreadId": "209615", "Html": "MoC,
\r\njust catching up.
\r\nYou need to NOT seek when reading the ZipFile. A zipfile can have arbitrary stuff in the front of the file. It will be readable by winzip and other tools. It is also readable by DotNetZip when structured in this way. You don't need to seek to the "start" of the zip file, to successfully read it with DotNetZip.
\r\nThe way to think about it - the zipfile has offsets stored within it, in the "central directory". These offsets point to particular places in the file. If you write 64 bytes of data, or any amount of data, into a file, then save a zipfile into the same FileStream, the offsets that DotNetZip writes will be correct.
\r\nIf you attempt to read a valid zipfile starting at byte 64, the offsets will be wrong.
\r\nI don't know if this clears things up, but. .. . . Just don't seek when you read.
\r\n\r\n
\r\n
Let me offer some additional description.
\r\nSuppose you open a FileStream and write 100 bytes of data. At that point you also Save a ZipFile to that same stream. The result is a valid zip file with 100 bytes of data in front of it. It should be readable by any tool. It will be readable as a zipfile with DotNetZIp. There is no special seek required.
\r\nSuppose you open a MemoryStream and save a ZipFile into it. Then you open a FileStream and write 100 bytes of data. Then you copy to that FileStream, all the data from the MemoryStream. The result is a bytestream with 100 bytes of something, followed by the contents of a zip file. It will not be readable as a zipfile by any tool. If you want to read it with DotNetZip, you need to open the file, seek forward 100 bytes, then read.
\r\nIf this behavior seems illogical or inconsistent to you, you can open a workitem and suggest changes. So far I don't see the problem, though. I see that Save() works differently than Read(), with respect to the offsets. I don't see that as a huge problem though.
\r\nThe reason it works the way it does, is because of the ZipFile structure itself. Some zips require this format. For example, self-extracting archives. This approach also allows easy creation of self-extracting archives, whereas using a zero-based offset on Save() would not.
\r\nUpdate: ok, I can see a case where you have an open stream, and you'd like to just write the zip file into it, without getting the automagic offset re-calculation.
\r\nThe way you could do this is to define an OffsetStream that does this for you. I can't think of a way to do it directly with DotNetZip currently, without caching the content in a MemoryStream. I think it's a reasonable request.
\r\n\r\n