Re: Bug in lrzip 0.631-1 (32 bit version) with -d -o - options

[David Balažic <xerces9 at gmail dot com>]

Here is a smaller reproducible test case (I did the preparation part
in 64 bit cygwin):

 - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-i386.iso
 - download http://releases.ubuntu.com/16.10/ubuntu-16.10-desktop-amd64.iso

$ md5sum ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso
e9e9a6c6b3c8c265788f4e726af25994 *ubuntu-16.10-desktop-i386.iso
3f50877c05121f7fd8544bef2d722824 *ubuntu-16.10-desktop-amd64.iso

$ cat ubuntu-16.10-desktop-i386.iso ubuntu-16.10-desktop-amd64.iso > test.iso

$ md5sum test.iso
d6e8cb13ce36e2c04961004782eec139 *test.iso

$ lrzip -v test.iso
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 4
Detected 17160601600 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Temporary Directory set as: /tmp/
Compression mode is: LZMA. LZO Compressibility testing enabled
Heuristically Computed Compression Window: 109 = 10900MB
Output filename is: test.iso.lrz
File size: 3204448256
Will take 1 pass
Beginning rzip pre-processing phase
test.iso - Compression Ratio: 1.352. Average Compression Speed: 15.357MB/s.
Total time: 00:03:18.85

$ lrzip -i test.iso.lrz
test.iso.lrz:
lrzip version: 0.6 file
Compression: rzip + lzma
Decompressed file size: 3204448256
Compressed file size: 2370391407
Compression ratio: 1.352
MD5 used for integrity testing
MD5: d6e8cb13ce36e2c04961004782eec139

in 32 bit cygwin:

$ TMP=/cygdrive/d/tmp/  lrzip -v  -d  -o -
/cygdrive/c/cygwin64/home/stein/test.iso.lrz  | tee >(md5sum --tag)
>(sha1sum --tag) > /dev/null
The following options are in effect for this DECOMPRESSION.
Threading is ENABLED. Number of CPUs detected: 4
Detected 17160601600 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Output Filename Specified: -
Temporary Directory set as: /cygdrive/d/tmp/
Outputting to stdout.
Detected lrzip version 0.6 file.
MD5 being used for integrity testing.
Decompressing...
Unable to decompress entirely in ram, will use physical files
Dumping temporary file to control->outFILE.

Average DeCompression Speed: 29.105MB/s
Dumping temporary file to control->outFILE.
[OK] - 3204448256 bytes
Total time: 00:01:45.07
SHA1 (-) = da39a3ee5e6b4b0d3255bfef95601890afd80709
MD5 (-) = d41d8cd98f00b204e9800998ecf8427e

The MD5 sum is wrong. (both md5 and sha1 values are the hashes of an
empty file, so the data is lost somewhere)

I attach the cygcheck -s -v -r output again (with cygwin 2.7.0-0.1 , I
later checked with 2.7.0-0.2 too, got the same results)

Regards,
David


On 1 February 2017 at 22:25, David Balažic <xerces9@gmail.com> wrote:
> I tried the  2.7.0-0.1  test release and now the behavior changed.
>
> Before I got consistently a wrong MD5 sum of 8bd6ad48f2cea6a710af70b434d57673
> With this release I get c43a02c309fa5e0abe778201e9ceec46.
>
> So something changed.
>
> Either the problem is in cygwin or lrzip, I guess.
>
> Regards,
> David
>
>
> On 30 January 2017 at 16:23, David Balažic <xerces9@gmail.com> wrote:
>> I tried in Ubuntu 32 bit (both the packaged lrzip  and a self compiled
>> one) and there the problem does not happen, so it looks like either:
>>  - bad lrzip in cygwin
>>  - cygwin pipe issues?
>>
>> Regards,
>> David
>>
>>
>> On 25 January 2017 at 23:15, David Balažic <xerces9@gmail.com> wrote:
>>> Hi!
>>>
>>> The 32 bit version of lrzip 0.631-1 contains a bug that corrupts the
>>> decompressed dat in some circumstances.
>>>
>>> I reproduced the problem on 2 PCs (the md5sum of the broken output was
>>> the same on both systems).
>>>
>>> I seems to happen when the (de)compressed file size is bigger than the
>>> available RAM (note that the 32 bit version uses max 4GB in any case)
>>> and lrzip resorts to using a temporary file.
>>>
>>> See below for reproducing:
>>>
>>> $ lrzip -i sda.img.lrz2
>>> sda.img.lrz2:
>>> lrzip version: 0.6 file
>>> Compression: rzip + lzma
>>> Decompressed file size: 64017212928
>>> Compressed file size: 7210541304
>>> Compression ratio: 8.878
>>> MD5 used for integrity testing
>>> MD5: 6594f5b0d22efd345003260054165842
>>>
>>> $ date; df -h ; TMP=/cygdrive/i/t/tmp/  lrzip -v  -d  -o -
>>> sda.img.lrz2  | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null   ;
>>> date
>>> Tue Jan 24 21:29:01 CET 2017
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> C:/cygwin       114G   94G   21G  83% /
>>> D:              541G  534G  7.1G  99% /cygdrive/d
>>> I:              391G  279G  113G  72% /cygdrive/i
>>> Q:               60G   57G  2.8G  96% /cygdrive/q
>>> The following options are in effect for this DECOMPRESSION.
>>> Threading is ENABLED. Number of CPUs detected: 4
>>> Detected 17160601600 bytes ram
>>> Compression level 7
>>> Nice Value: 19
>>> Show Progress
>>> Verbose
>>> Output Filename Specified: -
>>> Temporary Directory set as: /cygdrive/i/t/tmp/
>>> Outputting to stdout.
>>> Detected lrzip version 0.6 file.
>>> MD5 being used for integrity testing.
>>> Decompressing...
>>> Unable to decompress entirely in ram, will use physical files
>>> Dumping temporary file to control->outFILE.
>>>
>>> [1]+  Stopped                 TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o -
>>> sda.img.lrz2 | tee >(md5sum --tag) >(sha1sum --tag) > /dev/null
>>> Tue Jan 24 21:31:39 CET 2017
>>>
>>> stein@hofer8 /cygdrive/i/Zotac_bak
>>> $ fg
>>> TMP=/cygdrive/i/t/tmp/ lrzip -v -d -o - sda.img.lrz2 | tee >(md5sum
>>> --tag) >(sha1sum --tag) > /dev/null
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>> Dumping temporary file to control->outFILE.
>>>
>>> Average DeCompression Speed:  0.668MB/s
>>> Dumping temporary file to control->outFILE.
>>> [OK] - 64017212928 bytes
>>> Total time: 25:22:26.25
>>> SHA1 (-) = 6c519210541eb128c03b7c0f803adb2b46ee2a72
>>> MD5 (-) = 8bd6ad48f2cea6a710af70b434d57673
>>>
>>>
>>> The correct md5sum is 6594f5b0d22efd345003260054165842.
>>>
>>>
>>> Simply decompressing the file (lrzip -d -o sda.img sda.img.lrz2) to
>>> filesystem works fine, only when piped to stdout the problem happens.
>>>
>>> The 64 bit version does not have this problem.
>>>
>>>
>>> I will check if the same problem happens with the native linux build
>>> of lrzip (it takes a day...).
>>>
>>>
>>> I tried to reproduce the problem with a smaller file, but there it did
>>> not happen. Maybe my first test file has some corruption that causes
>>> this (unlikely).
>>>
>>> Some version information (complete cygcheck -s -v -r output attached):
>>>
>>> base-cygwin                           3.8-1
>>> cygwin                                2.6.1-1
>>> lrzip                                 0.631-1
>>>
>>> Regards,
>>> David

Attachment: cygcheck.svr
Description: Binary data

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple