Bug #76298 for Archive-Zip: Bug with perl module Archive::zip

Wed Apr 04 05:02:31 2012 Vikas.Agrawal [...] rbccm.com - Ticket created

Subject:	Bug with perl module Archive::zip
Date:	Wed, 4 Apr 2012 10:02:18 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

Hi, Archive::zip is unable to compress total number of 84715 files in to single zip file. Resultant zip file is having only around about 19000 files. Thanks & Regards Vikas Agrawal Archive::zip version = 1.30 perl 5, version 12, subversion 3 (v5.12.3) built for MSWin32-x86-multi-thread osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread ActivePerl Build 1204 [294330] Compiled at Feb 9 2011 14:38:22 Show quoted text

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Thu Apr 05 07:28:58 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

Subject:	RE: [rt.cpan.org #76298] AutoReply: Bug with perl module Archive::zip
Date:	Thu, 5 Apr 2012 12:28:46 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

Seems member counter is able to handle just 2^16 elements and starts again after reaching the element hence try to compress 84715 files in to single zip file results in zip file with only 19179 (84715-65536) compressed files in it. Show quoted text

-----Original Message----- From: Bugs in Archive-Zip via RT [mailto:bug-Archive-Zip@rt.cpan.org] Sent: 04 April 2012 10:03 To: Agrawal, Vikas Subject: [rt.cpan.org #76298] AutoReply: Bug with perl module Archive::zip Greetings, This message has been automatically generated in response to the creation of a trouble ticket regarding: "Bug with perl module Archive::zip", a summary of which appears below. There is no need to reply to this message right now. Your ticket has been assigned an ID of [rt.cpan.org #76298]. Your ticket is accessible on the web at: https://rt.cpan.org/Ticket/Display.html?id=76298 Please include the string: [rt.cpan.org #76298] in the subject line of all future correspondence about this issue. To do so, you may reply to this message. Thank you, bug-Archive-Zip@rt.cpan.org ------------------------------------------------------------------------- Hi, Archive::zip is unable to compress total number of 84715 files in to single zip file. Resultant zip file is having only around about 19000 files. Thanks & Regards Vikas Agrawal Archive::zip version = 1.30 perl 5, version 12, subversion 3 (v5.12.3) built for MSWin32-x86-multi-thread osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread ActivePerl Build 1204 [294330] Compiled at Feb 9 2011 14:38:22 _______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Wed Apr 18 05:10:33 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

Subject:	RE: [rt.cpan.org #76298] AutoReply: Bug with perl module Archive::zip
Date:	Wed, 18 Apr 2012 10:10:15 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

Actually all 84715 files have been added into the zip created but name of members are not visible when opened using winzip. Though member names can be seen if Open zipped file using Winrar. It seems adding content is fine but adding member names is going wrong somewhere and counter rolls over when handling more than 2^16 names. Show quoted text

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Wed Apr 18 10:43:53 2012 nedkonz [...] cpan.org - Correspondence added

On Wed Apr 18 05:10:33 2012, Vikas.Agrawal@rbccm.com wrote: Show quoted text

> Actually all 84715 files have been added into the zip created but name > of members are not visible when opened using winzip. Though member names > can be seen if Open zipped file using Winrar. It seems adding content is > fine but adding member names is going wrong somewhere and counter rolls > over when handling more than 2^16 names.

There is no counter rollover in Archive::Zip, because there is no counter. And if there were, it would be much larger than 16 bits. Most likely is a bug in WinZip. Especially since WinRAR is able to see all the members. If you see all the member names when you use another program, or when using the command line tool "unzip -v" (I have found these tools to be the best way to diagnose zip problems), then it is unlikely that Archive::Zip is at fault. You should contact Corel and ask about this WinZip problem.

Wed Apr 18 10:43:54 2012 The RT System itself - Status changed from 'new' to 'open'

Wed Apr 18 10:43:55 2012 nedkonz [...] cpan.org - Status changed from 'open' to 'resolved'

Wed Apr 18 10:48:20 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

Subject:	RE: [rt.cpan.org #76298] Resolved: Bug with perl module Archive::zip
Date:	Wed, 18 Apr 2012 15:48:00 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

I think it is not a problem with Winzip since I opened the created zip file using Winrar and extracted contents in a directory and zipped again directly using Winzip GUI tool. This new zip file does show all 84715 files. Clearly, there is some problem with Archive::Zip storing/attaching member names to it's contents. Show quoted text

-----Original Message----- From: Ned Konz via RT [mailto:bug-Archive-Zip@rt.cpan.org] Sent: 18 April 2012 15:44 To: Agrawal, Vikas Subject: [rt.cpan.org #76298] Resolved: Bug with perl module Archive::zip <URL: https://rt.cpan.org/Ticket/Display.html?id=76298 > According to our records, your request has been resolved. If you have any further questions or concerns, please respond to this message.

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Wed Apr 18 10:48:21 2012 The RT System itself - Status changed from 'resolved' to 'open'

Wed Apr 18 10:58:44 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

Subject:	RE: [rt.cpan.org #76298] Resolved: Bug with perl module Archive::zip
Date:	Wed, 18 Apr 2012 15:58:20 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

I used unzip -v as suggested and that also reported only 19179 files but with an error H:\prod\FOG\messages\tradePublishingServer\to_gat>"c:\Program Files\WinZip\WZUNZ IP.EXE" -v newarchive2010.zip 7212 DeflatN 2034 72% 07/04/2010 13:05 290ac185 559206_PENDING_T RANSACTION_COMMITTED_1_0_0_130542.xml note: didn't find end-of-central-dir signature at end of central dir. Possible cause: file transfer error ------ ------ --- ------- 161184170 42636634 74% 19179 <<Picture (Device Independent Bitmap)>> Show quoted text

-----Original Message----- From: Ned Konz via RT [mailto:bug-Archive-Zip@rt.cpan.org] Sent: 18 April 2012 15:44 To: Agrawal, Vikas Subject: [rt.cpan.org #76298] Resolved: Bug with perl module Archive::zip <URL: https://rt.cpan.org/Ticket/Display.html?id=76298 > According to our records, your request has been resolved. If you have any further questions or concerns, please respond to this message.

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Download ole0.bmp
image/bmp 647.8k

Mon Apr 23 19:14:50 2012 pmqs [...] cpan.org - Correspondence added

What size is newarchive2010.zip ?

Tue Apr 24 04:05:53 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

CC:	<SMPETERS [...] cpan.org>
Subject:	RE: [rt.cpan.org #76298] Bug with perl module Archive::zip
Date:	Tue, 24 Apr 2012 09:05:30 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

218, 352, 932 Bytes Show quoted text

-----Original Message----- From: Paul Marquess via RT [mailto:bug-Archive-Zip@rt.cpan.org] Sent: 24 April 2012 00:15 To: Agrawal, Vikas Cc: SMPETERS@cpan.org Subject: [rt.cpan.org #76298] Bug with perl module Archive::zip <URL: https://rt.cpan.org/Ticket/Display.html?id=76298 > What size is newarchive2010.zip ?

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Tue Apr 24 11:19:52 2012 pmqs [...] cpan.org - Correspondence added

If your archive is only 218352932 bytes long, it isn't blowing the 32- bit limit because of the overall size of the archive. Your problem then is down to a 16-bit counter stored in the end-of- central-directory record at the very end of the zip file. The field in question stores the number of members in the zip file. A Zip file with 84715 entries is larger than the max 16-bit value, so A::Zip stores 19179 in this field. To prove that I created a zip file with 84715 members using the code at the end of this message. Below is a dump of the End-of-central-directory record. Note the size reported for the number of entries in the zip file is 19179. End-of-central-directory record: ------------------------------- Zip archive file size: 8449310 (000000000080ED1Eh) Actual end-cent-dir record offset: 8449288 (000000000080ED08h) Expected end-cent-dir record offset: 8449288 (000000000080ED08h) (based on the length of the central directory and its expected offset) This zipfile constitutes the sole disk of a single-part archive; its central directory contains 19179 entries. The central directory is 4309359 (000000000041C16Fh) bytes long, and its (expected) offset in bytes from the beginning of the zipfile is 4139929 (00000000003F2B99h). No trouble with accessing the zip file with Info-Zip, but the version of WinZip I have does indeed have a problem. It thinks there are only 19179 members. Looks like WinZip assumes the field in the central directory record is accurate. There is an extension to the zip specification, called Zip64, that increases the size of that counter to a 32-bit value. It also allows much larger archives, but that isn't relevant to this discussion. I don't think A::Zip supports Zip64 yet. If you can't work around the number of elemnts in the zip file, Info-Zip supports Zip64 as does IO::Compress::Zip The next problem you will be faced with is whether the program you want to use to uncompress your zip file supports Zip64. Not all do. The version of WinZip I have doesn't, but I believe more recent versions do support it. Newer version of Info-Zip do support it. use Archive::Zip qw( :ERROR_CODES :CONSTANTS ); my $zip = Archive::Zip->new(); for my $i (1 .. 84715) { my $string_member = $zip->addString( 'This is a test', $i ); $string_member->desiredCompressionMethod( COMPRESSION_DEFLATED ); } # Save the Zip file unless ( $zip->writeToFileNamed('someZip.zip') == AZ_OK ) { die 'write error'; }

Tue Apr 24 11:19:52 2012 The RT System itself - System error

Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.

Tue Apr 24 11:19:53 2012 The RT System itself - System error

Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.

Wed Apr 25 11:50:42 2012 Vikas.Agrawal [...] rbccm.com - Correspondence added

CC:	"Paul Marquess" <Paul.Marquess [...] btinternet.com>
Subject:	RE: [rt.cpan.org #76298] AutoReply: Bug with perl module Archive::zip
Date:	Wed, 25 Apr 2012 16:49:39 +0100
To:	<bug-Archive-Zip [...] rt.cpan.org>
From:	"Agrawal, Vikas" <Vikas.Agrawal [...] rbccm.com>

Thanks for your so detailed explanation Paul. I was trying to convey this problem to Archive::zip perl module moderator. Sending your message to cpan.org so that it becomes clear to moderator that problem is in Archive::zip since my WinZip (9.0 SR-1 6224) can create/read 32bit end-of-central-directory record? Also, is there any work around to fix end-of-central-directory record for now until Archive::Zip fix is available? Regards Vikas Show quoted text

________________________________ From: Paul Marquess [mailto:Paul.Marquess@btinternet.com] Sent: 25 April 2012 16:38 To: Agrawal, Vikas Subject: RE: #76298: Bug with perl module Archive::zip Hi Vikas, See inline. Paul From: Agrawal, Vikas [mailto:Vikas.Agrawal@rbccm.com] Sent: 25 April 2012 15:25 To: Paul Marquess Subject: RE: #76298: Bug with perl module Archive::zip Thanks Paul. I had not received your e-mail but have seen it online now. Why A::zip storing it wrong (only 16bit) for end-of-central-directory record? [Paul] The specification for zip files has been around for a long time. There are a huge number of programs that can create zip files. Some conform strictly to the specification, some do not. It looks like Winzip is strict in its implementation. The Linux unzip program recognises the variability of zip implementations and ignores the 16-bit file counter field. I assume Archive::Zip have just forgotten about the issue where the 16-bit counter can overflow. If I zip all these 84715 files manually using WinZip then end-of-central-directory record does not have any 16bit problem. [Paul] That depends on whether the version of WinZIp you have supports the ZIP64 extensions. The table below is from http://kb.winzip.com/kb/entry/294/ - it looks like all versions since 9.0 support ZIP64. If you have a newish version you should be ok. The only issue is what you intend to do with the zip file you create - obviously it will only be usable by something that can read zip files with ZIP64 enabled. Additional Features WinZip Versions 16.5 16.0 15.5 15.0 14.5 14.0 12.1 12.0 11.2 11.0 10.0 9.0 8.1 FTP upload** * * * * * * * * * * Burn to CD/DVD** * * * * * * * * * * Photo Thumbnails * * * * * * * * * * Alternate extension * * * * * * * * * * Extended time stamps * * * * * * * * * * Image viewer (internal)** * * * * * * * * * * WinZip Pro * * * * * * * * * * * New Zip on CD/DVD** * * * * * * * * * * * Select files * * * * * * * * * * * WinZip Jobs** * * * * * * * * * * * Create split Zip files * * * * * * * * * * * Folder view * * * * * * * * * * * Check for Updates * * * * * * * * * * * 64-bit context menu * * * * * * * * * * * Zip64 large file support * * * * * * * * * * * * WinZip Quick Pick * * * * * * * * * * * * * Split Zip files (existing) * * * * * * * * * * * * * HTML Help files * * * * * * * * * WinZip Wizard * * * * * * * * * External ARJ, ARC, LHA * * * * Virus scanner configuration * * * Note: For Wi Regards Vikas

________________________________ From: Paul Marquess [mailto:Paul.Marquess@btinternet.com] Sent: 25 April 2012 15:16 To: Agrawal, Vikas Subject: #76298: Bug with perl module Archive::zip Hi I posted a reply yesterday to your response about the size of your zip file (https://rt.cpan.org/Ticket/Display.html?id=76298), but you may not have received notification. The administrator tells me that the system was under heavy load and a lot of email notifications didn't get out. Anyway - take a look at my response if you haven't already. Email me or continue the dialog on RT if you want to discuss options for working around it. Paul

_______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and RBC Europe Limited. In accordance with English law requirements, details regarding RBC Europe Limited are set out below: RBC EUROPE LIMITED Registered in England and Wales 995939 Registered Address: Riverbank House, 2 Swan Lane, London, EC4R 3BF. Authorised and regulated by the Financial Services Authority. Member of the London Stock Exchange.

Message body is not shown because it is too large.