Bug #116340 for Config-General: Testing Config::General consumes extreme amounts of memory

Wed Jul 20 09:59:09 2016 sinan [...] unur.com - Ticket created

Subject:	Testing Config::General consumes extreme amounts of memory
Date:	Wed, 20 Jul 2016 09:58:54 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

With Config-General-2.62, running tests causes memory consumption to increase first linearly. My 4 GB laptop became unusable pretty quickly. My 12 GB laptop took a longer time to get there, but did eventually crash with "Out of Memory" after cpanm Config::General exhausted all available physical and virtual memory (total committed is close to 30 GB). After that, the computers were unusable for an extended period of time. http://www.cpantesters.org/cpan/report/9fadc536-6d07-1014-8f72-e9fd0d0b81cc I only observe this on 64-bit Windows 10 machines with perl 5.24 built using MSVC 2013 with Config::General 2.62. I did not observe it on my ArchLinux install with perl 5.24 and Config::General 2.62, so it could be a perl bug that only shows up in a specific set of circumstances, but I decided to first report it here because I did not have any problems with Config::General 2.61. Thank you. -- Sinan

Wed Jul 20 10:38:56 2016 sinan [...] unur.com - Correspondence added

Subject:	Re: [rt.cpan.org #116340] AutoReply: Testing Config::General consumes extreme amounts of memory
Date:	Wed, 20 Jul 2016 10:38:47 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

Interestingly, adding: sub _open { # # open the config file, or expand a directory or glob or include # print STDERR "'$_'\n" for @_; sleep 1; enables the first 52 tests to run without expanding memory use (under 550MB footprint). The test seems to stall at #52 with: t\run.t .. 52/78 'Config::General::Interpolated=HASH(0x179464ee770)' 't/complex.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 'complex/n*.cfg' 't/' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' 'Config::General::Interpolated=HASH(0x179464ee770)' 't\complex\n1.cfg' ....

Wed Jul 20 10:47:16 2016 sinan [...] unur.com - Correspondence added

CC:	bug-Config-General [...] rt.cpan.org
Subject:	Re: [rt.cpan.org #116340] AutoReply: Testing Config::General consumes extreme amounts of memory
Date:	Wed, 20 Jul 2016 10:47:07 -0400
To:	"A. Sinan Unur" <sinan [...] unur.com>
From:	"A. Sinan Unur" <sinan [...] unur.com>

So, we have: C:\...\Config-General-2.62> dir t\complex\n* /b n1.cfg n2.cfg But, it seems, when `'complex/n*.cfg'` are requested, `_open` gets stuck loading n1.cfg over and over and over again.

Mon Jul 25 05:06:46 2016 tlinden [...] cpan.org - Correspondence added

Hi, unfortunately I can' reproduce, because I don't have access to such a system. Attached you'll find 2 patches: glob-bsd.patch: please apply this one first on a fresh copy of General.pm and re-run the unit tests. If it fails (which I suspect), then apply the patch glob-debug.patch (on a fresh copy of General.pm as well), re-run the unit tests and post the output. Thanks, Tom

Subject:

glob-bsd.patch

*** General.pm~ 2016-07-25 10:56:41.704657624 +0200 --- General.pm 2016-07-25 10:57:20.914656375 +0200 *************** *** 19,25 **** use IO::File; use FileHandle; use File::Spec::Functions qw(splitpath file_name_is_absolute catfile catpath); ! use File::Glob qw/:glob/; # on debian with perl > 5.8.4 croak() doesn't work anymore without this. --- 19,25 ---- use IO::File; use FileHandle; use File::Spec::Functions qw(splitpath file_name_is_absolute catfile catpath); ! use File::Glob qw/:bsd_glob/; # on debian with perl > 5.8.4 croak() doesn't work anymore without this.

Subject:

glob-debug.patch

*** General.pm~ 2016-07-25 10:56:41.704657624 +0200 --- General.pm 2016-07-25 11:00:37.018657451 +0200 *************** *** 19,25 **** use IO::File; use FileHandle; use File::Spec::Functions qw(splitpath file_name_is_absolute catfile catpath); ! use File::Glob qw/:glob/; # on debian with perl > 5.8.4 croak() doesn't work anymore without this. --- 19,25 ---- use IO::File; use FileHandle; use File::Spec::Functions qw(splitpath file_name_is_absolute catfile catpath); ! use File::Glob qw/:bsd_glob/; # on debian with perl > 5.8.4 croak() doesn't work anymore without this. *************** *** 470,475 **** --- 470,478 ---- } } + # test + print STDERR "include: <$_>\n" for @include; + die "done\n"; # Multiple results or no expansion results (which is fine, # include foo/* shouldn't fail if there isn't anything matching) # rt.cpan.org#79869: local $this->{IncludeGlob};

Mon Jul 25 05:06:47 2016 The RT System itself - Status changed from 'new' to 'open'

Mon Jul 25 10:06:24 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Mon, 25 Jul 2016 10:06:09 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

Hello and thank you for your response. glob-bsd.patch does not fix the problem, and here is what I with the glob-debug.patch applied to fresh copy: C:\Users\sinan\src\Config-General> nmake test Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. "C:\opt\perl\5.24.0\bin\perl.exe" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib\lib', 'blib\a rch')" t\*.t t\run.t .. 1/78 include: <t/sub1/cfg.sub1c> include: <t/sub1/cfg.sub1d> include: <t/sub1/cfg.sub1e> done # Looks like you planned 78 tests but ran 26. # Looks like your test exited with 255 just after 26. t\run.t .. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 52/78 subtests Test Summary Report ------------------- t\run.t (Wstat: 65280 Tests: 26 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 78 tests but ran 26. Files=1, Tests=26, 1 wallclock secs ( 0.08 usr + 0.01 sys = 0.09 CPU) Result: FAIL Failed 1/1 test programs. 0/26 subtests failed. NMAKE : fatal error U1077: 'C:\opt\perl\5.24.0\bin\perl.exe' : return code '0xff' Stop. On Mon, Jul 25, 2016 at 5:06 AM, T. Linden via RT <bug-Config-General@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=116340 > > > Hi, > > unfortunately I can' reproduce, because I don't have access to such a system. > > Attached you'll find 2 patches: > > glob-bsd.patch: please apply this one first on a fresh copy of General.pm and re-run the unit tests. > > If it fails (which I suspect), then apply the patch glob-debug.patch (on a fresh copy of General.pm as well), re-run the unit tests and post the output. > > > Thanks, > Tom

Mon Jul 25 10:57:23 2016 tlinden [...] cpan.org - Correspondence added

Args, my bad. Of course there are other tests before the failing one. So, please try this: create a script t.pl (or whatever) inside the source directory: #!/usr/bin/perl use lib qw(t blib/lib); use Config::General; my $conf47 = new Config::General( -ConfigFile => "t/complex.cfg", -InterPolateVars => 1, -DefaultConfig => { this => "that", default => "imported" }, -MergeDuplicateBlocks => 1, -MergeDuplicateOptions => 1, -StrictVars => 1, -SplitPolicy => 'custom', -SplitDelimiter => '\s*=\s*', -IncludeGlob => 1, -IncludeAgain => 1, -IncludeRelative => 1, -AutoTrue => 1, -FlagBits => { someflags => { LOCK => 1, RW => 2, TAINT => 3 } }, -StoreDelimiter => ' = ', -SlashIsDirectory => 1, -SaveSorted => 1 ); execute it with the patched General.pm and post the output. And - if you could be so kind, then please do the same with the previous version 1.61, which you said works, with the attached patch applied and also send the output. Thanks, Tom

Subject:

glob-1.61.patch

*** General-2.61~.pm 2016-07-25 16:53:25.645656594 +0200 --- General.pm 2016-07-25 16:53:40.435656425 +0200 *************** *** 442,448 **** } } } ! if (@include == 1) { $configfile = $include[0]; } --- 442,449 ---- } } } ! print STDERR "include: <$_>\n" for @include; ! die "done\n"; if (@include == 1) { $configfile = $include[0]; }

Mon Jul 25 11:34:11 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Mon, 25 Jul 2016 11:33:52 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

That prints, as one would expect: C:\Users\sinan\src\Config-General> perl t.pl include: <t\complex\n1.cfg> include: <t\complex\n2.cfg> done This makes me wonder: Are those filenames eval'ed or interpolated into a regex pattern on the affected code path? And, now with Config-General-2.61 (probably too much output, apologies) C:\Users\sinan\AppData\Local\Temp> cd Config-General-2.61 C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> perl Makefile.PL Checking if your kit is complete... Looks good Generating a nmake-style Makefile Writing Makefile for Config::General Writing MYMETA.yml and MYMETA.json C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> nmake test Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. cp General.pm blib\lib\Config\General.pm cp General/Interpolated.pm blib\lib\Config\General\Interpolated.pm cp General/Extended.pm blib\lib\Config\General\Extended.pm "C:\opt\perl\5.24.0\bin\perl.exe" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib\lib', 'blib\arch')" t\*.t t\run.t .. ok All tests successful. Files=1, Tests=75, 1 wallclock secs ( 0.06 usr + 0.03 sys = 0.09 CPU) Result: PASS Now, with the patch applied: C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> nmake clean Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. "C:\opt\perl\5.24.0\bin\perl.exe" -MExtUtils::Command -e rm_f -- General.bso General.def General.exp General.x blib\arch\auto\Config\General\extral ibs.all blib\arch\auto\Config\General\extralibs.ld Makefile.aperl *.lib *.obj *perl.core MYMETA.json MYMETA.yml blibdirs.ts core core.*perl.*.? core.[0-9 ] core.[0-9][0-9] core.[0-9][0-9][0-9] core.[0-9][0-9][0-9][0-9] core.[0-9][0-9][0-9][0-9][0-9] libGeneral.def mon.out perl perl.exe perl.exe perlmain.c p m_to_blib pm_to_blib.ts so_locations tmon.out "C:\opt\perl\5.24.0\bin\perl.exe" -MExtUtils::Command -e rm_rf -- *.pdb */*~ *~ blib t/*.out t/test.cfg "C:\opt\perl\5.24.0\bin\perl.exe" -MExtUtils::Command -e mv -- Makefile Makefile.old > NUL C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> patch -p0 < /Users/sinan/Downloads/glob-1.61.patch (Stripping trailing CRs from patch; use --binary to disable.) patching file General.pm C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> perl Makefile.PL Checking if your kit is complete... Looks good Generating a nmake-style Makefile Writing Makefile for Config::General Writing MYMETA.yml and MYMETA.json C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> nmake test Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. cp General/Extended.pm blib\lib\Config\General\Extended.pm cp General.pm blib\lib\Config\General.pm cp General/Interpolated.pm blib\lib\Config\General\Interpolated.pm "C:\opt\perl\5.24.0\bin\perl.exe" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib\lib', 'blib\a rch')" t\*.t t\run.t .. 1/75 include: <t/sub1/cfg.sub1c> include: <t/sub1/cfg.sub1d> include: <t/sub1/cfg.sub1e> done # Looks like you planned 75 tests but ran 26. # Looks like your test exited with 255 just after 26. t\run.t .. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 49/75 subtests Test Summary Report ------------------- t\run.t (Wstat: 65280 Tests: 26 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 75 tests but ran 26. Files=1, Tests=26, 1 wallclock secs ( 0.03 usr + 0.05 sys = 0.08 CPU) Result: FAIL Failed 1/1 test programs. 0/26 subtests failed. NMAKE : fatal error U1077: 'C:\opt\perl\5.24.0\bin\perl.exe' : return code '0xff' Stop. C:\Users\sinan\AppData\Local\Temp\Config-General-2.61> perl t.pl include: <t\complex\n1.cfg> include: <t\complex\n2.cfg> done *-*-*-*-* I am sorry, I am not sure if this helps at all. Please let me know anything else you'd like me to try. -- Sinan

Mon Jul 25 12:19:57 2016 sinan [...] unur.com - Correspondence added

CC:	bug-Config-General [...] rt.cpan.org
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Mon, 25 Jul 2016 12:19:43 -0400
To:	"A. Sinan Unur" <sinan [...] unur.com>
From:	"A. Sinan Unur" <sinan [...] unur.com>

Here is what happens when I run under the debugger. I never get control back, and I have to kill the script via Windows' Task Manager. When I do, I see that we are yo-yo'ing back and forth between two parts of the code. 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1> Config::General::_hook(blib\lib/Config/General.pm:1226): 1226: return (1, @arguments); DB<1> Config::General::_open(blib\lib/Config/General.pm:445): 445: return if(!$cont); DB<1> Config::General::_open(blib\lib/Config/General.pm:447): 447: my($fh, $configfile); DB<1> Config::General::_open(blib\lib/Config/General.pm:449): 449: if($basepath) { DB<1> Config::General::_open(blib\lib/Config/General.pm:454): 454: $configfile = $basefile; DB<1> Config::General::_open(blib\lib/Config/General.pm:457): 457: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:460): 460: my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); DB<1> Config::General::_open(blib\lib/Config/General.pm:463): 463: if ( !@include && defined $this->{ConfigPath} ) { DB<1> Config::General::_open(blib\lib/Config/General.pm:476): 476: for (@include) { DB<1> Config::General::_open(blib\lib/Config/General.pm:477): 477: $this->_open($_); DB<1> Config::General::_open(blib\lib/Config/General.pm:441): 441: my($this, $basefile, $basepath) = @_; DB<1> Config::General::_open(blib\lib/Config/General.pm:442): 442: my $cont; DB<1> Config::General::_open(blib\lib/Config/General.pm:444): 444: ($cont, $basefile, $basepath) = $this->_hook('pre_open', $basefile, $basepath); DB<1> Config::General::_hook(blib\lib/Config/General.pm:1220): 1220: my ($this, $hook, @arguments) = @_; DB<1> Config::General::_hook(blib\lib/Config/General.pm:1221): 1221: if(exists $this->{Plug}->{$hook}) { DB<1>

Mon Jul 25 12:37:23 2016 sinan [...] unur.com - Correspondence added

CC:	bug-Config-General [...] rt.cpan.org
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Mon, 25 Jul 2016 12:37:11 -0400
To:	"A. Sinan Unur" <sinan [...] unur.com>
From:	"A. Sinan Unur" <sinan [...] unur.com>

The docs for File::Glob say: "Win32 users should use the real slash." So, I did this: Index: General.pm =================================================================== --- General.pm (revision 111) +++ General.pm (working copy) # on debian with perl > 5.8.4 croak() doesn't work anymore without this. @@ -448,7 +448,7 @@ if($basepath) { # if this doesn't work we can still try later the global config path to use - $configfile = catfile($basepath, $basefile); + $configfile = "$basepath/$basefile"; } else { $configfile = $basefile; @@ -464,7 +464,7 @@ foreach my $dir (@{$this->{ConfigPath}}) { my ($volume, $path, undef) = splitpath($basefile); if ( -d catfile( $dir, $path ) ) { - push @include, grep { -f $_ } bsd_glob(catfile($dir, $basefile), GLOB_BRACE | GLOB_QUOTE); + push @include, grep { -f $_ } bsd_glob( "$dir/$basefile", GLOB_BRACE | GLOB_QUOTE); last; } } With that, I get: C:\Users\sinan\src\Config-General> nmake test Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. "C:\opt\perl\5.24.0\bin\perl.exe" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib\lib', 'blib\a rch')" t\*.t t\run.t .. ok All tests successful. Files=1, Tests=78, 1 wallclock secs ( 0.06 usr + 0.03 sys = 0.09 CPU) Result: PASS HTH, -- Sinan

Tue Jul 26 06:58:05 2016 tlinden [...] cpan.org - Correspondence added

Thank you very much for the tests! The backslash/slash issue is indeed weird, but I do not understand the underlying mechanics. That is, why does it work with a slash and not with backslash? Because, whichever way I try on a w32 system, the output is the same in both cases: DB<3> $dir = "t" DB<4> $basefile = "a*" DB<7> p Dumper(bsd_glob(catfile($dir, $basefile), GLOB_BRACE | GLOB_QUOTE)) $VAR1 = 't/a1'; $VAR2 = 't/a2'; $VAR3 = 't/a3'; DB<11> p Dumper(bsd_glob("$dir/$basefile", GLOB_BRACE | GLOB_QUOTE)); $VAR1 = 't/a1'; $VAR2 = 't/a2'; $VAR3 = 't/a3'; What I find much more illustrative is the debug output. _open() seems to keep working on the glob pattern all the time, it keeps calling itself with $_ as parameter which seems not to be the current file, but the glob pattern instead. So - maybe - it is a scoping problem. Or, the glob is not being resolved, but we alread printed @included after the bsd_glob() call which showed the valid file list. Therefore, here's one more test: again apply the attached patch to an unmodified General.pm and retry. If this doesn't help, I'll see if I can implement this backslash thing in a portable way, but I'd like to avoid that and to understand the issue. Thanks in advance, Tom

Subject:

glob-local.patch

*** General.pm~ 2016-07-26 12:52:03.631656544 +0200 --- General.pm 2016-07-26 12:56:05.247656524 +0200 *************** *** 457,462 **** --- 457,463 ---- if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { # Something like: *.conf (or maybe dir/*.conf) was included; expand it and # pass each expansion through this method again. + local $_; my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); # applied patch by AlexK fixing rt.cpan.org#41030 *************** *** 473,480 **** # Multiple results or no expansion results (which is fine, # include foo/* shouldn't fail if there isn't anything matching) # rt.cpan.org#79869: local $this->{IncludeGlob}; ! for (@include) { ! $this->_open($_); } return; } --- 474,482 ---- # Multiple results or no expansion results (which is fine, # include foo/* shouldn't fail if there isn't anything matching) # rt.cpan.org#79869: local $this->{IncludeGlob}; ! foreach my $file (@include) { ! print STDERR "CALLING __open($file)\n"; ! $this->_open($file); } return; }

Tue Jul 26 07:13:33 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Tue, 26 Jul 2016 07:13:20 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

On Tue, Jul 26, 2016 at 6:58 AM, T. Linden via RT <bug-Config-General@rt.cpan.org> wrote: Show quoted text

> Therefore, here's one more test: again apply the attached patch to an unmodified General.pm and retry. If this doesn't help, I'll see if I can implement this backslash thing in a portable way, but I'd like to avoid that and to understand the issue.

I'll try it as soon as I get a chance which probably will be later today. I am puzzled by the whole thing as well. By the way, "$dir/$file" is perfectly fine on Windows for internal calls. The canonical form using "\" is important when interfacing with the shell or external utilities. However, I am not proposing those two small changes as "the" solution, but, instead offering the fact that tests run to completion when catfile is replaces as a data point in our quest to find the underlying problem. Thank you for your work. -- Sinan

Tue Jul 26 07:49:18 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Tue, 26 Jul 2016 07:49:05 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

On Tue, Jul 26, 2016 at 6:58 AM, T. Linden via RT <bug-Config-General@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=116340 >

... Show quoted text

> Therefore, here's one more test: again apply the attached patch to an unmodified General.pm and retry.

Here is what I get with both `nmake test` and by running your short script. CALLING __open(t\complex\n1.cfg) ... many many times CALLING __open(t\complex\n1.cfg) CALLING __open(t\complex\n1.cfg) ... If I press CTRL-C in time, I get to observe: ... CALLING __open(t\complex\n1.cfg) # WARN: Deep recursion on subroutine "Config::General::_open" at C:\Users\sinan\AppData\Local\Temp\Config-General-2.62\blib\lib/Config/General.pm line 479, <GEN55> line 28. CALLING __open(t\complex\n1.cfg) CALLING __open(t\complex\n1.cfg) CALLING __open(t\complex\n1.cfg) ... Curious.

Tue Jul 26 08:02:47 2016 tlinden [...] cpan.org - Correspondence added

Oha, here's the problem: if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { This looks if the configfile contains any glob meta characters as of: META CHARACTERS \ Quote the next metacharacter [] Character class {} Multiple pattern * Match any string of characters ? Match any single character ~ User name home directory Backslash is part of the list and the file, _open() gets called with, contains multiple backslashes. So, I need to modify the regex so that it only matches a backslash if it occurs in front of another meta character. That's also the reason why it works if you replace catfile with "$dir/$file" :)

Tue Jul 26 08:27:55 2016 tlinden [...] cpan.org - Correspondence added

Ok, I'd propose this patch to fix the issue. It modifies the regex which looks if a given filename contains a glob. On windows systems it ignores the backslash as meta character unless it occurs in front of other meta characters.

Subject:

glob-fix1.patch

*** General.pm~ 2016-07-26 12:52:03.631656544 +0200 --- General.pm 2016-07-26 14:22:33.120658169 +0200 *************** *** 454,462 **** $configfile = $basefile; } ! if ($this->{IncludeGlob} and $configfile =~ /[*?\[\{\\]/) { # Something like: *.conf (or maybe dir/*.conf) was included; expand it and # pass each expansion through this method again. my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); # applied patch by AlexK fixing rt.cpan.org#41030 --- 454,468 ---- $configfile = $basefile; } ! my $glob = qr/[*?\[\{\\]/; ! if ($^O =~ /win/i) { ! $glob = qr/(\\[*?\[\{\\]|[*?\[\{])/; ! } ! ! if ($this->{IncludeGlob} and $configfile =~ /$glob/) { # Something like: *.conf (or maybe dir/*.conf) was included; expand it and # pass each expansion through this method again. + local $_; my @include = grep { -f $_ } bsd_glob($configfile, GLOB_BRACE | GLOB_QUOTE); # applied patch by AlexK fixing rt.cpan.org#41030 *************** *** 473,480 **** # Multiple results or no expansion results (which is fine, # include foo/* shouldn't fail if there isn't anything matching) # rt.cpan.org#79869: local $this->{IncludeGlob}; ! for (@include) { ! $this->_open($_); } return; } --- 479,487 ---- # Multiple results or no expansion results (which is fine, # include foo/* shouldn't fail if there isn't anything matching) # rt.cpan.org#79869: local $this->{IncludeGlob}; ! foreach my $file (@include) { ! print STDERR "CALLING __open($file)\n"; ! $this->_open($file); } return; }

Tue Jul 26 08:48:08 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Tue, 26 Jul 2016 08:47:55 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

Thank you. I missed that pattern because ... Let's blame syntax-highlighting ;-) In any case, I won't be in front of a Windows system for a while, but I'll check out your patch the moment I get a chance. -- Sinan

Tue Jul 26 12:52:22 2016 sinan [...] unur.com - Correspondence added

CC:	bug-Config-General [...] rt.cpan.org
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Tue, 26 Jul 2016 12:52:09 -0400
To:	"A. Sinan Unur" <sinan [...] unur.com>
From:	"A. Sinan Unur" <sinan [...] unur.com>

It works! C:\Users\sinan\AppData\Local\Temp> cd Config-General-2.62 C:\Users\sinan\AppData\Local\Temp\Config-General-2.62> copy "..\..\..\..\Downloads\glob-fix1.patch" 1 file(s) copied. C:\Users\sinan\AppData\Local\Temp\Config-General-2.62> patch -p0 < glob-fix1.patch (Stripping trailing CRs from patch; use --binary to disable.) patching file General.pm C:\Users\sinan\AppData\Local\Temp\Config-General-2.62> perl Makefile.PL Checking if your kit is complete... Looks good Generating a nmake-style Makefile Writing Makefile for Config::General Writing MYMETA.yml and MYMETA.json C:\Users\sinan\AppData\Local\Temp\Config-General-2.62> nmake test Microsoft (R) Program Maintenance Utility Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. cp General.pm blib\lib\Config\General.pm cp General/Interpolated.pm blib\lib\Config\General\Interpolated.pm cp General/Extended.pm blib\lib\Config\General\Extended.pm "C:\opt\perl\5.24.0\bin\perl.exe" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib\lib', 'blib\a rch')" t\*.t t\run.t .. 11/78 CALLING __open(t/sub1/cfg.sub1c) CALLING __open(t/sub1/cfg.sub1d) CALLING __open(t/sub1/cfg.sub1e) CALLING __open(t/included.conf) CALLING __open(t\complex\n1.cfg) CALLING __open(t\complex\n2.cfg) t\run.t .. ok All tests successful. Files=1, Tests=78, 1 wallclock secs ( 0.08 usr + 0.01 sys = 0.09 CPU) Result: PASS

Fri Jul 29 02:13:13 2016 tlinden [...] cpan.org - Correspondence added

Fixed in 2.63. Thanks again for reporting and your support to track it down! best regards, Tom

Fri Jul 29 02:13:13 2016 tlinden [...] cpan.org - Status changed from 'open' to 'resolved'

Fri Jul 29 10:35:22 2016 sinan [...] unur.com - Correspondence added

CC:	"A. Sinan Unur" <sinan [...] unur.com>
Subject:	Re: [rt.cpan.org #116340] Testing Config::General consumes extreme amounts of memory
Date:	Fri, 29 Jul 2016 10:35:11 -0400
To:	bug-Config-General [...] rt.cpan.org
From:	"A. Sinan Unur" <sinan [...] unur.com>

On Fri, Jul 29, 2016 at 2:13 AM, T. Linden via RT <bug-Config-General@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=116340 > > > Fixed in 2.63. Thanks again for reporting and your support to track it down!

Thank you very much for your work. Have a good weekend. -- Sinan