[ { "Id": "425186", "ThreadId": "207718", "Html": "

Hi,

\r\n

I was having some problems the ZlibStream class where it was raising an exception while decompressing a stream and after looking into it, it looks like there are some cases where it can't decompress a byte array even when the data was previously compresssed using ZlibStream. I don't know whether I'm just doing something wrong, but I've managed to distill the problem out to the code below which creates a random buffer that throws an exception when it compress it and subsequently tries to decompress it. Strangely, some buffers only fail when using UncompressBuffer and some only fail when using ZlibStream.Write with CompressionMode = Decompress.

\r\n

If you run the TestRoundtrips code below you'll get a "Bad state (incorrect data check)" exception in the first call to my custom DecompressViaStream function, but it might just be me misunderstanding what ZlibStream is supposed to do...

\r\n

Mike

\r\n

 

\r\n
\r\n
Module ZlibStreamTest\r\n\r\n#Region " Test Methods "\r\n\r\n    Public Sub TestRoundtrips()\r\n        Dim objOriginal() As Byte\r\n        Dim objCompressed() As Byte\r\n        Dim objDecompressed() As Byte\r\n        ' get an invalid buffer. (size and chars parameters found empirically for a\r\n        ' decent success / fail rate on random buffer generation).\r\n        objOriginal = ZlibStreamTest.GenerateFailBuffer(100000, 232, False, True)\r\n        ' compress by stream, decompress by stream\r\n        objCompressed = ZlibStreamTest.CompressViaStream(objOriginal)\r\n        objDecompressed = ZlibStreamTest.DecompressViaStream(objCompressed)\r\n        If Not ZlibStreamTest.CompareArrays(objDecompressed, objOriginal) Then\r\n            Throw New System.InvalidOperationException\r\n        End If\r\n        ' compress by stream, decompress by array\r\n        objCompressed = ZlibStreamTest.CompressViaStream(objOriginal)\r\n        objDecompressed = ZlibStreamTest.DecompressViaArray(objCompressed)\r\n        If Not ZlibStreamTest.CompareArrays(objDecompressed, objOriginal) Then\r\n            Throw New System.InvalidOperationException\r\n        End If\r\n        ' compress by array, decompress by stream\r\n        objCompressed = ZlibStreamTest.CompressViaArray(objOriginal)\r\n        objDecompressed = ZlibStreamTest.DecompressViaStream(objCompressed)\r\n        If Not ZlibStreamTest.CompareArrays(objDecompressed, objOriginal) Then\r\n            Throw New System.InvalidOperationException\r\n        End If\r\n        ' compress by array, decompress by array\r\n        objCompressed = ZlibStreamTest.CompressViaArray(objOriginal)\r\n        objDecompressed = ZlibStreamTest.DecompressViaArray(objCompressed)\r\n        If Not ZlibStreamTest.CompareArrays(objDecompressed, objOriginal) Then\r\n            Throw New System.InvalidOperationException\r\n        End If\r\n    End Sub\r\n\r\n#End Region\r\n\r\n#Region " Buffer Generation Methods "\r\n\r\n    Public Function GenerateRandomBuffer(ByVal size As Integer, ByVal chars As Integer) As Byte()\r\n        Dim objRng As Random\r\n        Dim objBuffer() As Byte\r\n        Dim strValidChars As String\r\n        ' initialise some working variables\r\n        objRng = New Random\r\n        ReDim objBuffer(size - 1)\r\n        ' buld the list of characters we'll allow in the random buffer\r\n        strValidChars = String.Empty\r\n        For intIndex As Integer = 0 To 255\r\n            If (intIndex < chars) Then\r\n                strValidChars = strValidChars & ChrW(intIndex)\r\n            End If\r\n        Next intIndex\r\n        ' build a random buffer\r\n        For intByteIndex As Integer = 0 To (objBuffer.Length - 1)\r\n            objBuffer(intByteIndex) = Convert.ToByte(objRng.Next(0, strValidChars.Length - 1))\r\n        Next intByteIndex\r\n        ' return the result\r\n        Return objBuffer\r\n    End Function\r\n\r\n    ''' <summary>\r\n    ''' Generates random buffers until one fails zlib decompression via a base stream.\r\n    ''' </summary>\r\n    ''' <param name="size">\r\n    ''' Number of bytes in the returned buffer.\r\n    ''' </param>\r\n    ''' <param name="chars">\r\n    ''' Number of unique characters to use in the returned buffer. A higher value means \r\n    ''' faster matching of a buffer which has the required "fail" properies.\r\n    ''' </param>\r\n    ''' <param name="streamFail">True if the buffer should fail when using a stream to decompress.</param>\r\n    ''' <param name="arrayFail">True if the buffer should fail when using and array to decompress.</param>\r\n    ''' <returns></returns>\r\n    ''' <remarks></remarks>\r\n    Public Function GenerateFailBuffer(ByVal size As Integer, ByVal chars As Integer, ByVal streamFail As Boolean, ByVal arrayFail As Boolean) As Byte()\r\n        Dim intLoopCount As Integer\r\n        Dim objRawBytes() As Byte\r\n        Dim objCompressed() As Byte\r\n        Dim objDecompressed() As Byte\r\n        Dim blnSkipBuffer As Boolean\r\n        ' keep building buffers until we find one we want to keep\r\n        While True\r\n            Debug.WriteLine("running iteration " & intLoopCount)\r\n            intLoopCount = intLoopCount + 1\r\n            ' build a random buffer\r\n            objRawBytes = ZlibStreamTest.GenerateRandomBuffer(size, chars)\r\n            objCompressed = ZlibStreamTest.CompressViaStream(objRawBytes)\r\n            blnSkipBuffer = False\r\n            ' read the whole compressed file into a byte array, then decompress it\r\n            Try\r\n                objDecompressed = ZlibStreamTest.DecompressViaStream(objCompressed)\r\n                If Not ZlibStreamTest.CompareArrays(objRawBytes, objDecompressed) Then\r\n                    Throw New System.InvalidOperationException\r\n                End If\r\n                If streamFail Then blnSkipBuffer = True\r\n            Catch ex As Ionic.Zlib.ZlibException\r\n                ' ignore this error - we're looking for a file which we can read in\r\n                ' wholesale, but that fails streaming\r\n                Debug.WriteLine("failed reading whole file")\r\n                If Not streamFail Then blnSkipBuffer = True\r\n            End Try\r\n            ' read the compressed file on the fly\r\n            If Not blnSkipBuffer Then\r\n                Try\r\n                    objDecompressed = ZlibStreamTest.DecompressViaArray(objCompressed)\r\n                    If Not ZlibStreamTest.CompareArrays(objRawBytes, objDecompressed) Then\r\n                        Throw New System.InvalidOperationException\r\n                    End If\r\n                    If arrayFail Then blnSkipBuffer = True\r\n                Catch ex As Ionic.Zlib.ZlibException\r\n                    Debug.WriteLine("failed reading captive stream")\r\n                    If Not arrayFail Then blnSkipBuffer = True\r\n                End Try\r\n            End If\r\n            If Not blnSkipBuffer Then\r\n                Return objRawBytes\r\n            End If\r\n        End While\r\n        Throw New System.InvalidOperationException\r\n    End Function\r\n\r\n#End Region\r\n\r\n#Region " Compression Methods "\r\n\r\n    Public Function CompressViaArray(ByVal data() As Byte) As Byte()\r\n        Dim objFileBytes() As Byte\r\n        ' decompress the file bytes\r\n        objFileBytes = Ionic.Zlib.ZlibStream.CompressBuffer(data)\r\n        ' return the result\r\n        Return objFileBytes\r\n    End Function\r\n\r\n    Public Function CompressViaStream(ByVal data() As Byte) As Byte()\r\n        Dim objFileBytes() As Byte\r\n        Dim intBytesRead As Integer\r\n        ' compress the raw data via a memory stream\r\n        Using objCompressed As System.IO.MemoryStream = New System.IO.MemoryStream\r\n            ' write the compressed data to the stream\r\n            Using objZlibOutput As Ionic.Zlib.ZlibStream = New Ionic.Zlib.ZlibStream(objCompressed, Ionic.Zlib.CompressionMode.Compress, True)\r\n                objZlibOutput.Write(data, 0, data.Length)\r\n            End Using\r\n            ' reset the stream\r\n            objCompressed.Flush()\r\n            objCompressed.Seek(0, IO.SeekOrigin.Begin)\r\n            ' read the compressed data\r\n            ReDim objFileBytes(Convert.ToInt32(objCompressed.Length) - 1)\r\n            intBytesRead = objCompressed.Read(objFileBytes, 0, objFileBytes.Length)\r\n            ' check we read the full length of the data\r\n            If Not (intBytesRead = objCompressed.Length) Then\r\n                Throw New System.InvalidOperationException\r\n            End If\r\n        End Using\r\n        ' return the result\r\n        Return objFileBytes\r\n    End Function\r\n\r\n    Public Function DecompressViaArray(ByVal data() As Byte) As Byte()\r\n        Dim objFileBytes() As Byte\r\n        ' decompress the file bytes\r\n        objFileBytes = Ionic.Zlib.ZlibStream.UncompressBuffer(data)\r\n        ' return the result\r\n        Return objFileBytes\r\n    End Function\r\n\r\n    Public Function DecompressViaStream(ByVal data() As Byte) As Byte()\r\n        Dim objFileBytes() As Byte\r\n        Dim intBytesRead As Integer\r\n        ' decompress the data via a memory stream\r\n        Using objCompressed As System.IO.MemoryStream = New System.IO.MemoryStream\r\n            ' write the compressed data to the stream\r\n            Using objZlibOutput As Ionic.Zlib.ZlibStream = New Ionic.Zlib.ZlibStream(objCompressed, Ionic.Zlib.CompressionMode.Decompress, True)\r\n                objZlibOutput.Write(data, 0, data.Length)\r\n            End Using\r\n            ' reset the stream\r\n            objCompressed.Flush()\r\n            objCompressed.Seek(0, IO.SeekOrigin.Begin)\r\n            ' read the decompressed data\r\n            ReDim objFileBytes(Convert.ToInt32(objCompressed.Length) - 1)\r\n            intBytesRead = objCompressed.Read(objFileBytes, 0, objFileBytes.Length)\r\n            ' check we read the full length of the data\r\n            If Not (intBytesRead = objCompressed.Length) Then\r\n                Throw New System.InvalidOperationException\r\n            End If\r\n        End Using\r\n        ' return the result\r\n        Return objFileBytes\r\n    End Function\r\n\r\n#End Region\r\n\r\n#Region " Comparison Methods "\r\n\r\n    Public Function CompareArrays(ByVal obj1() As Byte, ByVal obj2() As Byte) As Boolean\r\n        If Not (obj1.Length = obj2.Length) Then Return False\r\n        For intIndex As Integer = 0 To (obj1.Length - 1)\r\n            If Not (obj1(intIndex) = obj2(intIndex)) Then Return False\r\n        Next intIndex\r\n        Return True\r\n    End Function\r\n\r\n#End Region\r\n\r\nEnd Module
\r\n
", "PostedDate": "2010-03-29T18:13:13.773-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "425560", "ThreadId": "207718", "Html": "

I understand you believe that ZlibStream is not decompressing, what it has already compressed.

\r\n

Show me a simpler example, something with fewer lines of code.  You said you distilled it, but there's an awful lot of code there.

\r\n

For "round trip" problems with string data it should be as simple as this:

\r\n
\r\n
String data = @"The good will is not good because of what it effects \r\nor accomplishes or because of its competence to achieve some intended \r\nend: it is good only because of its willing (i.e. it is good in itself).\r\n\r\nEven if it should happen that, by a particularly unfortunate fate or by\r\nthe niggardly provision of a step-motherly nature, this will should be\r\nwholly lacking in power to accomplish its purpose, and if even the\r\ngreatest effort should not avail it to achieve anything of its end, and\r\nif there remained only the good will not as a mere wish, but as the\r\nsummoning of all the means in our power - it will sparkle like a jewel\r\nall by itself, as something that had its full worth in itself.";\r\n\r\nSystem.Console.WriteLine("original:");\r\nSystem.Console.WriteLine(data);\r\nvar c = ZlibStream.CompressString(data);\r\nstring uncompressed = ZlibStream.UncompressString(c);\r\nSystem.Console.WriteLine("uncompressed:");\r\nSystem.Console.WriteLine(uncompressed);\r\nSystem.Console.WriteLine("are equal?: {0}", data.Equals( uncompressed));
\r\n
\r\n

Right? and for byte array data, like this:

\r\n
\r\n
byte[]  bdata = {\r\n0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,\r\n0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,\r\n0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,\r\n0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,\r\n0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,\r\n0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,\r\n0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,\r\n0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,\r\n0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f\r\n};\r\n\r\nFunc<byte[],byte[],bool> check = (a,b) => {\r\n    if (a.Length != b.Length) return false;\r\n    for (int i=0; i < a.Length;  i++)\r\n    {\r\n        if (a[i]!=b[i]) return false;\r\n    }\r\n    return true;};\r\n\r\nbyte[] c = ZlibStream.CompressBuffer(bdata);\r\nbyte[] buncompressed = ZlibStream.UncompressBuffer(c);\r\nSystem.Console.WriteLine("are equal?: {0}", check(bdata, buncompressed));
\r\n
\r\n

I just ran both of those snippets and the roundtrip checks out. So I'm not clear on the problem.

\r\n

You said something about random buffers and streams, but that seems like a different thing. Make it really clear and simple for me and I will look into it.  Minimal code to illustrate the problem.

", "PostedDate": "2010-03-30T09:25:47.523-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "426029", "ThreadId": "207718", "Html": "

Sorry about the length - I was trying to be as complete as possible. Basically, some data doesn't roundtrip properly if you compress then decompress using ZlibStream - you get an exception when you try to decompress.

\r\n

The smallest buffer I could find that shows this problem is about 50k and ran to about 3000+ lines when I formatted it like your second example. I've created a post in the Issue Tracker instead so I can attach the buffer as a file - see http://dotnetzip.codeplex.com/WorkItem/View.aspx?WorkItemId=10562.

\r\n

M

", "PostedDate": "2010-03-31T07:49:27.687-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "426081", "ThreadId": "207718", "Html": "

Great, I'll have a look.

\r\n

 

", "PostedDate": "2010-03-31T09:19:52.883-07:00", "UserRole": null, "MarkedAsAnswerDate": null }, { "Id": "426101", "ThreadId": "207718", "Html": "

Mike, Thanks for the report. I reproduced the problem here.  WIll let you know what I find.

", "PostedDate": "2010-03-31T10:24:51.723-07:00", "UserRole": null, "MarkedAsAnswerDate": null } ]