Skip Menu |

This queue is for tickets about the Regexp-Grammars CPAN distribution.

Report information
The Basics
Id: 48170
Status: open
Priority: 0/
Queue: Regexp-Grammars

People
Owner: Nobody in particular
Requestors: mschwern [...] cpan.org
Cc: jpeacock [...] cpan.org
AdminCc:

Bug Information
Severity: Important
Broken in: 1.001004
Fixed in: (no value)



Subject: hash tests fail
Both the hash tests failed on my machine (OS X 10.5.7 with a source compiled 5.10.0). Attached is the verbose output of those tests.
Subject: test.out
Download test.out
application/octet-stream 2.7k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Sat, 25 Jul 2009 18:43:44 -0700
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Show quoted text
> Both the hash tests failed on my machine (OS X 10.5.7 with a source > compiled 5.10.0). Attached is the verbose output of those tests.
I cannot replicate this on the same system and version of Perl. Can you give me a perl -V so I can compare the Perl versions more closely. More importantly, the error messages don't make sense. There's nothing in the test's code that would undefine the value of $first_only. That's the root cause here, because the corresponding match then fails and the eq comparison in the test is to the result of the previous successful match. Can you rerun the following variation on hash.t so we can track down the mysterious undefiner? Thanks, Damian -----cut----------cut----------cut----------cut----------cut----- use 5.010; use warnings; use Test::More 'no_plan'; my %hash = ( do => 'a deer', re => 'a drop of golden sun', dore => 'a portal', me => 'a name I call myself', fa => 'a long long way to run', ); my $other = 1; # Needed to counteract bug in scoping (?) my $listified = do { use Regexp::Grammars; qr{ <[WORD=%hash]>+ }xms; }; my $first_only = do { use Regexp::Grammars; qr{ <WORD=%hash> }xms; }; warn defined $first_only; my $no_cap = do { use Regexp::Grammars; qr{ <%hash>+ }xms; }; warn defined $first_only; while (my $line = <DATA>) { warn defined $first_only; my ($input, $expected) = split /\s+/, $line; warn defined $first_only; if ($input =~ $listified) { is_deeply $/{WORD}, eval($expected), "list: $input"; } else { is 'FAIL', $expected, "list: $input"; } warn defined $first_only; if ($input =~ $first_only) { is $/{WORD}, eval($expected)->[0], "scalar: $input"; } else { is 'FAIL', $expected, "scalar: $input"; } if ($input =~ $no_cap) { isnt 'FAIL', $expected, "no-cap: $input"; } else { is 'FAIL', $expected, "no-cap: $input"; } } __DATA__ dorefameredo ['dore','fa','me','re','do'] dorefamell ['dore','fa','me'] zzzzz FAIL zzzdoremezzz ['dore','me']
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Sat, 25 Jul 2009 23:08:27 -0700
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
damian@conway.org via RT wrote: Show quoted text
> <URL: http://rt.cpan.org/Ticket/Display.html?id=48170 > >
>> Both the hash tests failed on my machine (OS X 10.5.7 with a source >> compiled 5.10.0). Attached is the verbose output of those tests.
> > I cannot replicate this on the same system and version of Perl. Can you > give me a perl -V so I can compare the Perl versions more closely.
Attached as perlV Show quoted text
> More importantly, the error messages don't make sense. There's nothing > in the test's code that would undefine the value of $first_only. > That's the root cause here, because the corresponding match then fails > and the eq comparison in the test is to the result of the previous > successful match. > > Can you rerun the following variation on hash.t so we can track down the > mysterious undefiner?
Attached as test.out -- 184. When operating a military vehicle I may *not* attempt something "I saw in a cartoon". -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Download perlV.out
application/octet-stream 2.9k

Message body not shown because it is not plain text.

Download test.out
application/octet-stream 2.2k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Mon, 27 Jul 2009 14:18:23 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Show quoted text
>> Can you >> give me a perl -V so I can compare the Perl versions more closely.
> > Attached as perlV >
>> Can you rerun the following variation on hash.t so we can track down the >> mysterious undefiner?
> > Attached as test.out
Many thanks. According to the test the matching of a completely unrelated regex is causing a second regex to become undef. Sounds like a problem in Perl itself to me. The only obvious differences between our set-ups is that you have PERL_IMPLICIT_CONTEXT USE_ITHREADS USE_REENTRANT_API set whereas I don't. So I'm theorizing maybe a thread-related issue? Damian
On Mon Jul 27 14:19:01 2009, damian@conway.org wrote: Show quoted text
> The only obvious differences between our set-ups is that you have > > PERL_IMPLICIT_CONTEXT > USE_ITHREADS > USE_REENTRANT_API
FWIW, I have precisely the same test failures as schwern, but on RHEL5.2 with a custom-built Perl at $WORK. Indeed, I also have the same compile-time flags as he does, with the addition of USE_64_BIT_ALL and USE_64_BIT_INT (not that those should make any difference)... John
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Tue, 28 Jul 2009 10:25:32 -0700
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
damian@conway.org via RT wrote: Show quoted text
> <URL: http://rt.cpan.org/Ticket/Display.html?id=48170 > >
>>> Can you >>> give me a perl -V so I can compare the Perl versions more closely.
>> Attached as perlV >>
>>> Can you rerun the following variation on hash.t so we can track down the >>> mysterious undefiner?
>> Attached as test.out
> > Many thanks. According to the test the matching of a completely unrelated regex > is causing a second regex to become undef. Sounds like a problem in Perl itself > to me. > > The only obvious differences between our set-ups is that you have > > PERL_IMPLICIT_CONTEXT > USE_ITHREADS > USE_REENTRANT_API > > set whereas I don't. > > So I'm theorizing maybe a thread-related issue?
Sounds like a bug introduced by compiling in threads, yeah. 5.10.1RC0 still has it. That's going to eliminate a lot of perls. :( -- 60. "The Giant Space Ants" are not at the top of my chain of command. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
CC: mschwern [...] cpan.org
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Tue, 28 Jul 2009 14:09:03 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: John Peacock <john.peacock [...] havurah-software.org>
Michael G Schwern via RT wrote: Show quoted text
> Sounds like a bug introduced by compiling in threads, yeah. 5.10.1RC0 still > has it. That's going to eliminate a lot of perls. :(
Is it possible that this is at all related to the recent p5p discussion regarding "Regex inside regex"? http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-07/msg01095.html John
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Tue, 28 Jul 2009 17:15:54 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Michael G Schwern via RT <bug-Regexp-Grammars@rt.cpan.org> wrote: Show quoted text
> Sounds like a bug introduced by compiling in threads, yeah.  5.10.1RC0 still > has it.  That's going to eliminate a lot of perls. :(
Hmmmm. That's most frustrating. Particularly because it's utterly outside my control. The module doesn't use threading, so it shouldn't be affected by threading. Not sure how to proceed. :-( Damian
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Tue, 28 Jul 2009 17:17:37 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
John Peacock via RT <bug-Regexp-Grammars@rt.cpan.org> wrote: Show quoted text
> Is it possible that this is at all related to the recent p5p discussion > regarding "Regex inside regex"?
I'll have to check again, but I was trying to be exceptionally careful *not* to use regexes inside the generated regexes. For precisely that reason. Maybe I missed one. I'll investigate that possibility when I get a change. Damian
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 15:57:11 +0100
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
I finally managed to reproduce a similar error on my unthreaded version of 5.10. So could I get you both to try this version of the hash test? And then repeat the trials, removing one of the $other variables each time? If the behaviour changes as I expect then we're in deep trouble (as you'll see). Thanks, Damian

Message body is not shown because sender requested not to inline it.

CC: mschwern [...] cpan.org, jpeacock [...] cpan.org
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 11:35:29 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: John Peacock <john.peacock [...] havurah-software.org>
damian@conway.org via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=48170 > > > I finally managed to reproduce a similar error on my unthreaded version of 5.10. > So could I get you both to try this version of the hash test? > > And then repeat the trials, removing one of the $other variables each time? > > If the behaviour changes as I expect then we're in deep trouble (as you'll see).
Seems _very_ sensitive. The first time I tried it I could get passes with all (obviously) and with both #3 and #4 commented out. Now when I try it I can only get passes with any single $other variable commented out. This is very bad. Since you managed to reproduce locally, you seem to have a better idea of what is getting stomped on. Do you think you can conceive of a test that doesn't rely on Regexp::Grammars that we can send to p5p so we can get the groupmind working on it? I'm not sure this needs to block 5.10.1, but it would be good to document this failure mode (since it seems to be fairly deep magic that is going wildly off target)... John
CC: mschwern [...] cpan.org
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 11:45:04 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: John Peacock <john.peacock [...] havurah-software.org>
John Peacock via RT wrote: Show quoted text
> Seems _very_ sensitive. The first time I tried it I could get passes > with all (obviously) and with both #3 and #4 commented out. Now when I > try it I can only get passes with any single $other variable commented > out. This is very bad.
Interesting to note that if I move the $other assignments to between the block for $listified and $first_only, then I only need two $other's and not three. And if I move them to between $first_only and $no_cap, then I only need a single $other assignment. I don't know whether this is significant... John
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 18:06:50 +0100
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
John Peacock via RT <bug-Regexp-Grammars@rt.cpan.org>: Show quoted text
> Do you think you can conceive of a test that doesn't rely on > Regexp::Grammars that we can send to p5p so we can get the groupmind > working on it?
It turns out not to rely on Regexp::Grammars at all. Here's a minimal example. -----cut----------cut----------cut----------cut----------cut----- use 5.010; my $rx = qr{ (?{ @arr = [1]; $arr[-1][0] = 0; my $var; }) }x; my $line = ""; $line =~ $rx; say 'fail' if !defined $line; -----cut----------cut----------cut----------cut----------cut----- It seems to be a "carnage-à-trois" between regexes, lexicals, and autovivification. Because if you remove either of the final two statements in the (?{...}) block, then it stops killing $line. %-) That may suggest a workaround for Regexp::Grammars: namely, eschewing lexicals in its magically inserted code blocks. I'm exploring that now. Interestingly, injecting one *extra* lexical declaration anywhere in the code (outside the (?{...}) that is) has no effect. But injecting *two* lexicals anywhere in the code also stops the match from undefinng $line. Whereas injecting a second lexical inside the (?{...}) is giving me a bus error. :-( John, if you'd be willing to submit this conundrum to p5p, I'd be extremely grateful. (I'm in full conference travel mode at the moment, so I wouldn't be able to shepherd this through the groupminding for at least a month). Damian
CC: mschwern [...] cpan.org
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 13:14:38 -0400
To: bug-Regexp-Grammars [...] rt.cpan.org
From: John Peacock <john.peacock [...] havurah-software.org>
damian@conway.org via RT wrote: Show quoted text
> It turns out not to rely on Regexp::Grammars at all. Here's a minimal example. > > -----cut----------cut----------cut----------cut----------cut----- > > use 5.010; > > my $rx = qr{ (?{ @arr = [1]; $arr[-1][0] = 0; my $var; }) }x; > > my $line = ""; > $line =~ $rx; > > say 'fail' if !defined $line; > > -----cut----------cut----------cut----------cut----------cut-----
Nice. Show quoted text
> John, if you'd be willing to submit this conundrum to p5p, I'd be > extremely grateful. (I'm in full conference travel mode at the moment, > so I wouldn't be able to shepherd this through the groupminding for at > least a month).
Not a problem at all. I'll write this up and send it out tonight (I have 9 million other things going on right this minute at $WORK, so it isn't a good time for that)... John
Subject: Re: [rt.cpan.org #48170] hash tests fail
Date: Wed, 29 Jul 2009 18:36:26 +0100
To: bug-Regexp-Grammars [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Show quoted text
> Not a problem at all.  I'll write this up and send it out tonight (I > have 9 million other things going on right this minute at $WORK, so it > isn't a good time for that)...
Deeply appreciated, John. Meanwhile, I'll see if I can't kludge around the lexicals in Regexp::Grammars. Damian