Bug #91129 for Perl-Metrics-Simple: Seems to be a file size limit of 2^20 bytes (1048576)

Mon Dec 02 11:42:44 2013 steve.michel [...] siemens.com - Ticket created

Subject:	Seems to be a file size limit of 2^20 bytes (1048576)
Date:	Mon, 2 Dec 2013 11:42:26 -0500
To:	<bug-Perl-Metrics-Simple [...] rt.cpan.org>
From:	Steve Michel <steve.michel [...] siemens.com>

*Hi,** * *It seems to work on most files, but I found it failed on a large module when its size exceeded 2^20 bytes. (+1 fails)* *Can this limit be increased somehow ? Also, some interest in reporting comment lines as a percentage of code, but it does not seem to support that yet. **I was hoping to try this out, but the example usage for countperl fails as:** *C:\apps\devop_tools\Perl64>perl c:\apps\devop_tools\perl64\site\bin\countperl u:\DMS\cm\dms\lib\DtCmd.pm Did not supply a document! at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 367. Perl::Metrics::Simple::Analysis::File::_prune_non_code_lines(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metri cs/Simple/Analysis/File.pm line 160 Perl::Metrics::Simple::Analysis::File::_make_pruned_document(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metri cs/Simple/Analysis/File.pm line 109 Perl::Metrics::Simple::Analysis::File::_init('Perl::Metrics::Simple::Analysis::File=HASH(0x326bce8)', 'path', 'u:\DMS\cm\dms \lib\DtCmd.pm') called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 86 Perl::Metrics::Simple::Analysis::File::new('Perl::Metrics::Simple::Analysis::File', 'path', 'u:\DMS\cm\dms\lib\DtCmd.pm') ca lled at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple.pm line 44 Perl::Metrics::Simple::analyze_files('Perl::Metrics::Simple=HASH(0x4ef180)', 'u:\DMS\cm\dms\lib\DtCmd.pm') called at c:\apps \devop_tools\perl64\site\bin\countperl line 97 main::set_analysis() called at c:\apps\devop_tools\perl64\site\bin\countperl line 60 main::main() called at c:\apps\devop_tools\perl64\site\bin\countperl line 53 *It is installed locally as:** ***Module id = Perl::Metrics::Simple CPAN_USERID MATISSE (Matisse Enzer <matisse@cpan.org>) CPAN_VERSION 0.17 CPAN_FILE M/MA/MATISSE/Perl-Metrics-Simple-0.17.tar.gz UPLOAD_DATE 2012-11-15 MANPAGE Perl::Metrics::Simple - Count packages, subs, lines, etc. of many files. INST_FILE C:\apps\devop_tools\Perl64\site\lib\Perl\Metrics\Simple.pm INST_VERSION 0.17 *And passed installation tests as:** * C:\apps\devop_tools\Perl64\bin\perl.exe -MExtUtils::Command -e cp -- bin/countperl blib\script\countperl pl2bat.bat blib\script\countperl C:\apps\devop_tools\Perl64\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', 'blib\arch')" t/*.t t/001_load.t ........... ok t/0020_find_files.t .... ok t/0030_analyze.t ....... ok t/0040_statistics.t .... ok t/0050_file.t .......... ok t/0901_pod.t ........... ok t/0902_pod_coverage.t .. ok t/perlcritic.t ......... skipped: Author test. Set $ENV{TEST_AUTHOR} to a true value to run. All tests successful. Files=8, Tests=90, 3 wallclock secs ( 0.05 usr + 0.03 sys = 0.08 CPU) Result: PASS MATISSE/Perl-Metrics-Simple-0.17.tar.gz C:/apps/mvs9/vc/bin/nmake test -- OK *Any direction or solution would be greatly appreciated.** ** **Thanks** *

Tue Dec 10 10:55:01 2013 matisse [...] spamcop.net - Correspondence added

Steve: countperl doesn't have any built-in limit, so a limit on file size is probably doing from perl or a system library. Can you try on a could of different machines, and/or with different versions of Perl? Also, 2^20 is exactly 1 megabyte ( 2^20 = 1048576 = 1024 * 1024) which is very large as files go these days. Are you sure you are hitting this with any file larger than that size? Can you attach an example of a failing file to this ticket? -M On Mon Dec 02 11:42:44 2013, steve.michel@siemens.com wrote: Show quoted text

> *Hi,** > * > *It seems to work on most files, but I found it failed on a large > module when its size exceeded 2^20 bytes. (+1 fails)* > > *Can this limit be increased somehow ? > > Also, some interest in reporting comment lines as a percentage of > code, but it does not seem to support that yet. > > **I was hoping to try this out, but the example usage for countperl > fails as:** > *C:\apps\devop_tools\Perl64>perl > c:\apps\devop_tools\perl64\site\bin\countperl > u:\DMS\cm\dms\lib\DtCmd.pm > Did not supply a document! at > C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm > line 367. > Perl::Metrics::Simple::Analysis::File::_prune_non_code_lines(undef) > called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metri > cs/Simple/Analysis/File.pm line 160 > Perl::Metrics::Simple::Analysis::File::_make_pruned_document(undef) > called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metri > cs/Simple/Analysis/File.pm line 109 > Perl::Metrics::Simple::Analysis::File::_init('Perl::Metrics::Simple::Analysis::File=HASH(0x326bce8)', > 'path', 'u:\DMS\cm\dms > \lib\DtCmd.pm') called at > C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm > line 86 > Perl::Metrics::Simple::Analysis::File::new('Perl::Metrics::Simple::Analysis::File', > 'path', 'u:\DMS\cm\dms\lib\DtCmd.pm') ca > lled at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple.pm > line 44 > Perl::Metrics::Simple::analyze_files('Perl::Metrics::Simple=HASH(0x4ef180)', > 'u:\DMS\cm\dms\lib\DtCmd.pm') called at c:\apps > \devop_tools\perl64\site\bin\countperl line 97 > main::set_analysis() called at > c:\apps\devop_tools\perl64\site\bin\countperl line 60 > main::main() called at > c:\apps\devop_tools\perl64\site\bin\countperl line 53 > > > *It is installed locally as:** > ***Module id = Perl::Metrics::Simple > CPAN_USERID MATISSE (Matisse Enzer <matisse@cpan.org>) > CPAN_VERSION 0.17 > CPAN_FILE M/MA/MATISSE/Perl-Metrics-Simple-0.17.tar.gz > UPLOAD_DATE 2012-11-15 > MANPAGE Perl::Metrics::Simple - Count packages, subs, lines, > etc. of many files. > INST_FILE > C:\apps\devop_tools\Perl64\site\lib\Perl\Metrics\Simple.pm > INST_VERSION 0.17 > > *And passed installation tests as:** > * C:\apps\devop_tools\Perl64\bin\perl.exe -MExtUtils::Command > -e cp -- bin/countperl blib\script\countperl > pl2bat.bat blib\script\countperl > C:\apps\devop_tools\Perl64\bin\perl.exe "- > MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', > 'blib\arch')" t/*.t > t/001_load.t ........... ok > t/0020_find_files.t .... ok > t/0030_analyze.t ....... ok > t/0040_statistics.t .... ok > t/0050_file.t .......... ok > t/0901_pod.t ........... ok > t/0902_pod_coverage.t .. ok > t/perlcritic.t ......... skipped: Author test. Set $ENV{TEST_AUTHOR} > to a true value to run. > All tests successful. > Files=8, Tests=90, 3 wallclock secs ( 0.05 usr + 0.03 sys = 0.08 > CPU) > Result: PASS > MATISSE/Perl-Metrics-Simple-0.17.tar.gz > C:/apps/mvs9/vc/bin/nmake test -- OK > > > *Any direction or solution would be greatly appreciated.** > ** > **Thanks** > *

Tue Dec 10 10:55:02 2013 The RT System itself - Status changed from 'new' to 'open'

Tue Dec 10 10:55:02 2013 matisse [...] spamcop.net - Taken

Tue Dec 10 10:55:43 2013 matisse [...] spamcop.net - Correspondence added

oops typo, I meant to write that 1 megabyte is NOT a very large file these days. On Tue Dec 10 10:55:01 2013, MATISSE wrote: Show quoted text

> Also, 2^20 is exactly 1 megabyte ( 2^20 = 1048576 = 1024 * 1024) which > is very large as files go these days. Are you sure you are hitting > this with any file larger than that size? > > Can you attach an example of a failing file to this ticket?

Thu Dec 12 09:47:24 2013 steve.michel [...] siemens.com - Correspondence added

Subject:	Re: [rt.cpan.org #91129] Seems to be a file size limit of 2^20 bytes (1048576)
Date:	Thu, 12 Dec 2013 09:46:44 -0500
To:	<bug-Perl-Metrics-Simple [...] rt.cpan.org>
From:	Steve Michel <steve.michel [...] siemens.com>

Hi , Thanks for the reply. I agree, 1Mb is pretty small as files sizes go....although on the large side for perl modules. (btw: don't waste too much time on this...I generated the metrics on the failing file by removing all leading whitespace to drop filesize <1Mb...or works on linux) Still it is a curious threshold, and repeatable on other modules...I've attached an example small, big with the only difference being comments. I also show it failing on Tidy.pm (this is the only >1Mb .pm file in my perl installations. I also installed on a linux box to test it there against the same files, but it ONLY fails on Windows. Note that the perl on the linux box is 5.8.8.120, whereas the windows perl is 5.16.3.1603. Thanks, Steve ============= perl version exhibiting failure on Windows 12-Dec-13 08:31 AM 1,278,966 TestBigWebappUtil.pm - Only difference for this file is the addition of many comment lines to exceed 1Mb file size C:\apps\devop_tools\Perl64\bin>perl.exe -ver This is perl 5, version 16, subversion 3 (v5.16.3) built for MSWin32-x64-multi-thread (with 1 registered patch, see perl -V for more detail) Copyright 1987-2012, Larry Wall Binary build 1603 [296746] provided by ActiveState http://www.ActiveState.com Built Mar 13 2013 13:31:10 C:\apps\devop_tools\Perl64\site\bin>countperl u:\scripts\TestBigWebappUtil.pm Did not supply a document! at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 367. Perl::Metrics::Simple::Analysis::File::_prune_non_code_lines(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 160 Perl::Metrics::Simple::Analysis::File::_make_pruned_document(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 109 Perl::Metrics::Simple::Analysis::File::_init('Perl::Metrics::Simple::Analysis::File=HASH(0x2fb53a8)', 'path', 'u:\scripts\TestBigWebappUtil.pm') called at C:/apps/devop_too ls/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 86 Perl::Metrics::Simple::Analysis::File::new('Perl::Metrics::Simple::Analysis::File', 'path', 'u:\scripts\TestBigWebappUtil.pm') called at C:/apps/devop_tools/Perl64/site/lib /Perl/Metrics/Simple.pm line 44 Perl::Metrics::Simple::analyze_files('Perl::Metrics::Simple=HASH(0x4f2d48)', 'u:\scripts\TestBigWebappUtil.pm') called at countperl line 97 main::set_analysis() called at countperl line 60 main::main() called at countperl line 53 ==================== On smaller file it works fine 26-Sep-13 12:08 PM 2,548 WebappUtil.pm C:\apps\devop_tools\Perl64\site\bin>countperl u:\scripts\WebappUtil.pm Use of uninitialized value $value in concatenation (.) or string at countperl line 132. (this noise could be easily fixed) Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Use of uninitialized value $value in concatenation (.) or string at countperl line 132. Perl files found: 1 . . . List of subroutines, with most complex at top --------------------------------------------- complexity sub path size 1 {code not in named subroutines} u:\scripts\WebappUtil.pm 40 ==================== ==================== perl version succeeding on Linux --- Using same TestBigWebappUtil.pm file that failed on Windows above Show quoted text

> /plm/cinas/chucktestshare2/perl.d/as/perl/lnx64/bin/perl -ver

This is perl, v5.8.8 built for x86_64-linux-thread-multi (with 30 registered patches, see perl -V for more detail) Copyright 1987-2006, Larry Wall Binary build 820 [274679] provided by ActiveState http://www.ActiveState.com Built Jan 22 2007 19:51:55 /plm/cinas/chucktestshare2/perl.d/as/perl/lnx64/site/bin> countperl /plm/cinas/chucktestshare2/perl.d/as/perl/lnx64/site/lib/Perl/Tidy.pm > /tmp/Tidy.pm.out /plm/cinas/chucktestshare2/perl.d/as/perl/lnx64/site/bin> countperl /tmp/WebappUtil.pm > /tmp/WebappUtil.pm.out /plm/cinas/chucktestshare2/perl.d/as/perl/lnx64/site/bin> countperl /tmp/TestBigWebappUtil.pm > /tmp/TestBigWebappUtil.pm.out ====================== Here's an example using a >1Mb CPAN module (only module in my perl installation >1Mb) 21-Aug-13 10:08 AM 1,115,493 Tidy.pm C:\apps\devop_tools\Perl64\site\bin>countperl C:\apps\devop_tools\Perl64\site\lib\Perl\Tidy.pm Did not supply a document! at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 367. Perl::Metrics::Simple::Analysis::File::_prune_non_code_lines(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 160 Perl::Metrics::Simple::Analysis::File::_make_pruned_document(undef) called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 109 Perl::Metrics::Simple::Analysis::File::_init('Perl::Metrics::Simple::Analysis::File=HASH(0x306f238)', 'path', 'C:\apps\devop_tools\Perl64\site\lib\Perl\Tidy.pm') called at C:/apps/devop_tools/Perl64/site/lib/Perl/Metrics/Simple/Analysis/File.pm line 86 Perl::Metrics::Simple::Analysis::File::new('Perl::Metrics::Simple::Analysis::File', 'path', 'C:\apps\devop_tools\Perl64\site\lib\Perl\Tidy.pm') called at C:/apps/devop_tool s/Perl64/site/lib/Perl/Metrics/Simple.pm line 44 Perl::Metrics::Simple::analyze_files('Perl::Metrics::Simple=HASH(0x4f2de8)', 'C:\apps\devop_tools\Perl64\site\lib\Perl\Tidy.pm') called at countperl line 97 main::set_analysis() called at countperl line 60 main::main() called at countperl line 53 ====================== Matisse Enzer via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=91129 > > > oops typo, I meant to write that 1 megabyte is NOT a very large file these days. > > On Tue Dec 10 10:55:01 2013, MATISSE wrote: >

>> Also, 2^20 is exactly 1 megabyte ( 2^20 = 1048576 = 1024 * 1024) which >> is very large as files go these days. Are you sure you are hitting >> this with any file larger than that size? >> >> Can you attach an example of a failing file to this ticket?

Message body is not shown because sender requested not to inline it.

Mon Dec 08 10:52:05 2014 MOREGAN [...] cpan.org - Correspondence added

The 1MB limit was imposed by the PPI library. As of 1.218, PPI no longer has an input document size limit.

Tue Dec 09 00:41:28 2014 matisse [...] spamcop.net - Status changed from 'open' to 'resolved'