Bug #87886 for Devel-Symdump: Use of each() on hash after insertion without resetting hash iterator results in undefined behavior

Thu Aug 15 17:47:57 2013 ether [...] cpan.org - Ticket created

Subject:

Use of each() on hash after insertion without resetting hash iterator results in undefined behavior

Use of each() on hash after insertion without resetting hash iterator results in undefined behavior at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 53. Devel::Symdump::_symdump('Devel::Symdump=HASH(0x7fe3830e2430)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 43 Devel::Symdump::_doit('Devel::Symdump=HASH(0x7fe3830e2430)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 37 Devel::Symdump::new('Devel::Symdump', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Pod/Coverage.pm line 316 Pod::Coverage::_get_syms('Pod::Coverage::TrustPod=HASH(0x7fe3830daf08)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Pod/Coverage.pm line 152 Pod::Coverage::coverage('Pod::Coverage::TrustPod=HASH(0x7fe3830daf08)') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Test/Pod/Coverage.pm line 162 Test::Pod::Coverage::pod_coverage_ok('Catalyst::Exception::Basic', 'HASH(0x7fe380c0d7b8)') called at t/author/podcoverage.t line 17 (from Catalyst-Runtime's t/author/podcoverage.t)

Thu Feb 06 12:49:11 2014 POPEL [...] cpan.org - Correspondence added

A patch was proposed in https://rt.cpan.org/Ticket/Display.html?id=86910#txn-1298792

Thu Feb 06 12:49:11 2014 The RT System itself - Status changed from 'new' to 'open'

Fri Feb 07 02:12:17 2014 ANDK [...] cpan.org - Correspondence added

Can anybody explain, why this happens? I can apply the suggested patch, but I can't believe this is a bug in Devel::Symdump. So I would be grateful for an explanation. Thanks,

Fri Feb 07 12:48:20 2014 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #87886] Use of each() on hash after insertion without resetting hash iterator results in undefined behavior
Date:	Fri, 7 Feb 2014 09:48:11 -0800
To:	Andreas Koenig via RT <bug-Devel-Symdump [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Fri, Feb 07, 2014 at 02:12:17AM -0500, Andreas Koenig via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=87886 > > > Can anybody explain, why this happens? I can apply the suggested patch, but I can't believe this is a bug in Devel::Symdump. So I would be grateful for an explanation.

perldoc -f each: "If you add or delete a hash's elements while iterating over it, the effect on the iterator is unspecified; for example, entries may be skipped or duplicated--so don't do that."

Fri Feb 07 15:27:14 2014 ANDK [...] cpan.org - Correspondence added

OK. I'm just a bear of small brains. So I'm awaiting the why. That shows me who adds or deletes what where.

Fri Feb 07 19:27:07 2014 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #87886] Use of each() on hash after insertion without resetting hash iterator results in undefined behavior
Date:	Fri, 7 Feb 2014 16:26:56 -0800
To:	Andreas Koenig via RT <bug-Devel-Symdump [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Fri, Feb 07, 2014 at 03:27:14PM -0500, Andreas Koenig via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=87886 > > > OK. I'm just a bear of small brains. So I'm awaiting the why. That shows me who adds or deletes what where.

Are you asking who is modifying this hash while we're iterating on it? It would have to be something inside that while loop, or a sub that is called from it. It does look like this sub calls itself recursively... -- "You will not become a saint through other people's sins." - Anton Chekhov . . . . . Karen Etheridge, karen@etheridge.ca GCS C+++$ USL+++$ P+++$ w--- M++

Fri Feb 07 23:23:26 2014 ANDK [...] cpan.org - Correspondence added

A couple of things that would help, in increasing usefulness: - versions of perl, Catalyst-Runtime, Devel::Symdump, Pod::Coverage, whatever else was involved - a small code snippet that reproduces the problem Show quoted text

> It would have to be something inside that > while loop,

Yes, that's something I also believe. Notwithstanding... if that perl had a bug, anything outside that while loop might also be it. Show quoted text

> or a sub that is called from it.

You mean a sub called from within the loop? I cannot imagine that there is a call to a sub hidden in that loop. Maybe some new fangled magic hidden in a GLOB dereference? Or somebody redefined $Devel::Symdump::MAX_RECURSION. Scary idea but not impossible. Show quoted text

> It does look like this sub calls itself > recursively.

I do not see why this might be relevant. Thanks for your help.

Sat Feb 08 00:40:47 2014 ether [...] cpan.org - Reference to ticket #86910 added

Sat Feb 08 01:13:58 2014 ether [...] cpan.org - Correspondence added

I've managed to reproduce this by going back to my 5.19.2 perlbrew (where I had the full list of Catalyst prereqs installed), and the latest Catalyst-Runtime release (5.90053) downloaded from the cpan. Devel::Symdump is version 2.10. Pod::Coverage is 0.23. After simply untarring the distribution: $; perl -Ilib -MCarp::Always t/author/podcoverage.t ... ok 33 - Pod coverage on Catalyst::Engine::HTTP Use of each() on hash after insertion without resetting hash iterator results in undefined behavior at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 53. Devel::Symdump::_symdump('Devel::Symdump=HASH(0x1031f5550)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 43 Devel::Symdump::_doit('Devel::Symdump=HASH(0x1031f5550)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Devel/Symdump.pm line 37 Devel::Symdump::new('Devel::Symdump', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Pod/Coverage.pm line 316 Pod::Coverage::_get_syms('Pod::Coverage::TrustPod=HASH(0x102ce0d20)', 'Catalyst::Exception::Basic') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Pod/Coverage.pm line 152 Pod::Coverage::coverage('Pod::Coverage::TrustPod=HASH(0x102ce0d20)') called at /Users/ether/.perlbrew/libs/19.2@std/lib/perl5/Test/Pod/Coverage.pm line 162 Test::Pod::Coverage::pod_coverage_ok('Catalyst::Exception::Basic', 'HASH(0x100c19390)') called at t/author/podcoverage.t line 17 ok 34 - Pod coverage on Catalyst::Exception::Basic ... Then, I switched to a 5.18.2 installation, downloaded Catalyst-Runtime-5.90053 and installed its prereqs, then ran t/author/podcoverage.t. I saw no failure -- so perhaps this is a new change in 5.19.x?

Sat Feb 08 13:02:24 2014 ANDK [...] cpan.org - Correspondence added

The ordering of the tests differs for me. All 43 tests pass and here is the output for tests 34 and 35, all others are boring. ok 34 - Pod coverage on Catalyst::Exception::Basic You are loading Catalyst::Engine::HTTP explicitly. This is almost certainly a bad idea, as Catalyst::Engine::HTTP has been removed in this version of Catalyst. Please update your application's scripts with: catalyst.pl -force -scripts MyApp to update your scripts to not do this. at lib/Catalyst/Engine/HTTP.pm line 8. require Catalyst/Engine/HTTP.pm called at (eval 783) line 2 eval ' require Catalyst::Engine::HTTP ' called at /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.2/a2da/lib/site_perl/5.19.2/Pod/Coverage/CountParents.pm line 12 Pod::Coverage::CountParents::_get_pods('Pod::Coverage::TrustPod=HASH(0x5281ee8)') called at /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.2/a2da/lib/site_perl/5.19.2/Pod/Coverage.pm line 149 Pod::Coverage::coverage('Pod::Coverage::TrustPod=HASH(0x5281ee8)') called at /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.2/a2da/lib/site_perl/5.19.2/Test/Pod/Coverage.pm line 162 Test::Pod::Coverage::pod_coverage_ok('Catalyst::Engine::HTTP', 'HASH(0x49e98f8)') called at t/author/podcoverage.t line 17 ok 35 - Pod coverage on Catalyst::Engine::HTTP No word about each or hash. The perl is 5.19.2, all other versions are the ones you say in your posting. As long as I cannot reproduce this, I'm not willing to accept the uggested patch. It uses more resources and most probably just papers over a much more severe problem in some perls. Let me know when you have any more evidence how this can be reproduced.

Sat Feb 08 13:14:29 2014 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #87886] Use of each() on hash after insertion without resetting hash iterator results in undefined behavior
Date:	Sat, 8 Feb 2014 10:14:18 -0800
To:	Andreas Koenig via RT <bug-Devel-Symdump [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Sat, Feb 08, 2014 at 01:02:24PM -0500, Andreas Koenig via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=87886 > > > The ordering of the tests differs for me.

Random hash ordering, I'd expect. Show quoted text

> All 43 tests pass and here is the output for tests 34 and 35, all others are boring. > The perl is 5.19.2, all other versions are the ones you say in your posting.

I just tried in 5.19.7 and also do not see a warning, so it's possible this was a transient issue in blead that has now been resolved. Show quoted text

> As long as I cannot reproduce this, I'm not willing to accept the uggested patch. It uses more resources and most probably just papers over a much more severe problem in some perls. > Let me know when you have any more evidence how this can be reproduced.

That's fine with me, since I'm not seeing this in 5.18.2 nor 5.19.7 (I haven't installed 5.19.8 yet). I'll reopen this ticket if I see it again; until then I'm content to classify this as "not reproducible". thanks!

Sat Feb 08 14:43:06 2014 ANDK [...] cpan.org - Status changed from 'open' to 'stalled'

Mon Apr 07 10:09:54 2014 ANDK [...] cpan.org - Correspondence added

I was just pointed at http://blogs.perl.org/users/rurban/2014/04/do-not-use-each.html Maybe related and worth recording in any case.

Mon Apr 07 10:09:54 2014 The RT System itself - Status changed from 'stalled' to 'open'