Bug #46582 for IO-Compress: Inflation Error when uncompressing a bzip2 file of XML data

A data error usually means the bzip2 file is corrupt. Can you test the integrity of the file with the commandline bzip2 command that should be available on your Linux box. bzip2 -t your-bzip2-file If that shows the file is ok, can you get back to me? Paul

Hi Paul, Thanks for your reply. Sorry if this email is a bit long because I want to give you my findings. I have tested 'bzip2 -t test.xml.bz2' and I got nothing returned (no successful or error message). I guessed this corrupt check was OK since I compressed it using 'bzip2' command by myself. Moreover, when I uncompressed the XML file and parse/read by XML::Parser module, it was possible to manipulate the file until EOF. So, I am not sure if it is the data problem. I think read() sub in IO::Uncompress::Bunzip2 actually works since it can print out uncompressed XML contents as I showed in my first email. However, the read() as in 'while (read() > 0);' stops after some iterations (always at the same position, e.g. line#40 000, in the bzip2 XML file before EOF) and I got the message like 'Error reading from test.xml.bz2: Inflation Error: Data Error'. To make sure that it was not because of data contents, I created another XML file starting from the line, e.g. line#40 000, where the program stopped (deleted previously successfully read() ) and compressed (bzip2) it. The read() sub in IO::Uncompress::Bunzip2 worked like the first case where it stopped at some point in the file before EOF, and an interesting is that it processed exactly the same size of an XML input. So, I think error in data contents can be forgotten. I am very new to perl but I tried to debug and I found where in your code it is actually stopped. The program exited from Base.pm at line 862 when $status returned from uncompr() were as follows: ------------------------------------------------------------------------- STATUS_ERROR = 1 self->{Uncomp}{Error} = Inflation Error: Data Error self->{Uncomp}{ErrorNo} = Data Error ------------------------------------------------------------------------- $status returned from uncompr() is in Adapter/Bzip2.pm at the line 4: $status = $inf->bzinflate($from, $to);. I am wondering if the size of a compressed file causes a problem to bzinflate($from, $to)? I tested with the size of 2MB only (25MB of uncompressed size), but my real data input will be about 150GB... Thank you very much for your kindly help. Nattiya On Mon, Jun 1, 2009 at 5:47 PM, Paul Marquess via RT <bug-IO-Compress@rt.cpan.org> wrote: Show quoted text

Thanks for the file - I can confirm that the file is fine, but my module does think there is an error with it. Will investigate & get back to you. Paul

Nattiya, I have a fix available for the problem you reported. I'll post an update onto CPAN in the next few days. If you want to tryout the fix, here is what you need to change. Run Perl -V to see where your perl library files are installed - they are probably under /usr/lib/perl5. Now find the file IO/Uncompress/Base.pm goto line 857 and change this line $self->pushBack($temp_buf) if $beforeC_len != length $temp_buf; to this $self->pushBack($temp_buf) ; If you have any further problems, please get back to me. Paul

Hi Nattiya, good to hear it is working now. PLease get back to me if you encounter any problems. cheers Paul

Hi Paul, Thank you for the updated version. I have questions related to functionality, not bug. If you could not answer here because it is out of your scope, then I understand :) It is about scalability/efficiency because of my huge amount of data. Basically, I want to use 2-3 computers working in parallel to uncompress the same file and hopefully this will faster the process. So, I need to use seek() into a file at a specified position -- without uncompress along the way. The current seek() in your module sequentially reads data and uncompresses it even the part before the specified position. This might be useful in some cases, but for my case, it just waste CPU time uncompressing unwanted data (data before a seeking position). I do not want to disturb you or harm your code, but I might need to change something to optimize it to my task, In order to better understanding your code, I would like to ask you as follows: 1. What is sub smartSeek() in Base.pm? Is it for this purpose I mention above -- seek() to the position and ignore uncompressing? 2. What is the maximum value of BlockSize (let say I have more than 10GB of memory)? I would like to set it as much as possible to faster uncompress process. As you are an expert on Compression/Uncompression, I hope that you can help me answering so I can work on my own. Thank you for your kindly response/time. Bests, Nattiya

Hi Paul, Show quoted text
I am appreciated your very good suggestion and I will try to work out on it :) Thank you! Nattiya