Bug #79149 for Regexp-Grammars: 5.17.3: Eval-group not allowed at runtime

Wed Aug 22 17:16:18 2012 d.thomas [...] its.uq.edu.au - Ticket created

Subject:

5.17.3: Eval-group not allowed at runtime

RHEL5, my first build with 5.17.x Regexp-Grammars-1.021.tar.gz [worked in 5.16.1] t/alias_literal.t ................... Eval-group not allowed at runtime, use re 'eval' in regex m/(?{; *Regexp::Grammars::LOGFILE = Regexp::Grammars::_open_log('>>','-'); })((?{; @! = () if !.../ at t/alias_literal.t line 11. # Looks like your test exited with 255 before it could output anything. t/alias_literal.t ................... Dubious, test returned 255 (wstat 65280, 0xff00) No subtests run ... Result: FAIL Failed 59/61 test programs. 0/2 subtests failed.

Mon Jun 03 17:03:32 2013 CARNIL [...] cpan.org - Cc CARNIL added

Sun Jun 16 11:00:31 2013 gregoa [...] cpan.org - Cc GREGOA added

Sat Jun 22 09:33:54 2013 SHLOMIF [...] cpan.org - Correspondence added

Hi all, On Wed Aug 22 17:16:18 2012, d.thomas@its.uq.edu.au wrote: Show quoted text

> > RHEL5, my first build with 5.17.x > > Regexp-Grammars-1.021.tar.gz [worked in 5.16.1] > > t/alias_literal.t ................... Eval-group not allowed at > runtime, use re 'eval' in regex m/(?{; > *Regexp::Grammars::LOGFILE > = Regexp::Grammars::_open_log('>>','-'); > })((?{; @! = () if !.../ at > t/alias_literal.t line 11. > # Looks like your test exited with 255 before it could output > anything. > t/alias_literal.t ................... Dubious, test returned 255 > (wstat 65280, 0xff00) > No subtests run >

This now happens with Regexp-Grammars-1.028 consistently on all perl-5.18.0: http://www.cpantesters.org/distro/R/Regexp-Grammars.html#Regexp-Grammars-1.028 Please fix it. Regards, -- Shlomi Fish Show quoted text

> ... > > > Result: FAIL > Failed 59/61 test programs. 0/2 subtests failed.

Sat Jun 22 09:33:55 2013 The RT System itself - Status changed from 'new' to 'open'

Tue Jun 25 01:56:27 2013 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Tue, 25 Jun 2013 15:55:33 +1000
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Show quoted text

> This now happens with Regexp-Grammars-1.028 consistently on all perl-5.18.0 > Please fix it

Unfortunately, it appears to be unfixable. Perl 5.18 changed how 'qr' constant overloadings are parsed and the scope in which they are subsequently compiled. This change appears to make it impossible to reliably create 'qr' overloadings that inject code blocks into a regex, as it prevents the overloaded regexes from compiling properly in many cases, even with an explicit 'use re "eval"' in scope. In a few cases, the use of a 'qr' overloading causes Perl 5.18 to segfault. Because Regexp::Grammars relies on 'qr' overloads to inject code blocks into regexes, the module is simply not compatible with Perl 5.18. After well over a month of pounding on the problem, I have been unable to find a workaround for this fundamental incompatibility, and there seems little prospect of a fix unless the behaviour of regex parsing/compilation is changed again in some future release of Perl. At present, if you rely on Regexp::Grammars for your parsing needs, your alternatives are either not to upgrade to Perl 5.18, or else to consider switching to another parsing system, such as Marpa. I deeply regret that Regexp::Grammars can no longer be maintained due to these backwards-incompatible changes in Perl 5.18. When I have more time (i.e. not soon), I will look at whether I can construct a simple-enough failure case to report the issue to P5P. But I'm not particularly hopeful this issue can be fixed, even by them. Meanwhile, I have just uploaded release 1.029, which carps most of the above warning/explanation when run under 5.18, and which also sidesteps the testing problem by the simple expedient of skipping the tests that fail or segfault under 5.18. This means you can install and try the module under 5.18 (in case your usage is one of the few that Perl 5.18 doesn't break), but with no guarantees, warranty, or support. It's far from ideal, but it's all I can offer at present. Damian

Tue Jun 25 08:51:32 2013 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Tue Jun 25 01:56:27 2013, damian@conway.org wrote: Show quoted text

> > This now happens with Regexp-Grammars-1.028 consistently on all

> perl-5.18.0

> > Please fix it

> > Unfortunately, it appears to be unfixable. > > Perl 5.18 changed how 'qr' constant overloadings are parsed and the > scope in which they are subsequently compiled. This change appears to > make it impossible to reliably create 'qr' overloadings that inject > code > blocks into a regex, as it prevents the overloaded regexes from > compiling properly in many cases, even with an explicit 'use re > "eval"' > in scope. In a few cases, the use of a 'qr' overloading causes Perl > 5.18 to segfault. > > Because Regexp::Grammars relies on 'qr' overloads to inject code > blocks > into regexes, the module is simply not compatible with Perl 5.18. > > After well over a month of pounding on the problem, I have been unable > to find a workaround for this fundamental incompatibility, and there > seems little prospect of a fix unless the behaviour of regex > parsing/compilation is changed again in some future release of Perl. > > At present, if you rely on Regexp::Grammars for your parsing needs, > your > alternatives are either not to upgrade to Perl 5.18, or else to > consider > switching to another parsing system, such as Marpa. > > I deeply regret that Regexp::Grammars can no longer be maintained due > to > these backwards-incompatible changes in Perl 5.18. > > When I have more time (i.e. not soon), I will look at whether I can > construct a simple-enough failure case to report the issue to P5P. But > I'm not particularly hopeful this issue can be fixed, even by them. > > Meanwhile, I have just uploaded release 1.029, which carps most of the > above warning/explanation when run under 5.18, and which also > sidesteps > the testing problem by the simple expedient of skipping the tests that > fail or segfault under 5.18. This means you can install and try the > module under 5.18 (in case your usage is one of the few that Perl 5.18 > doesn't break), but with no guarantees, warranty, or support. It's far > from ideal, but it's all I can offer at present.

That is all most unfortunate. If there are crashes, there is definitely a bug in perl. I suspect the main problem may have to do with concatenation overloading. It has bothered me for some time that perl regexp compilation will trigger concatenation overloading, which makes little sense to me, as it is for *strings*, not regexps, which, as you see, are a completely different creature. I wonder whether disabling string concatenation overloading will fix it. In any case, I think this should really be fixed at perl’s end. Please do report a bug, so this can be fixed for 5.20 (or 5.18.1 or .2, if the changes are not too sweeping).

Tue Jun 25 17:17:09 2013 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Wed, 26 Jun 2013 07:16:13 +1000
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Show quoted text

> If there are crashes, there is definitely a bug in perl. I suspect the > main problem may have to do with concatenation overloading. It has > bothered me for some time that perl regexp compilation will trigger > concatenation overloading, which makes little sense to me, as it is > for *strings*, not regexps, which, as you see, are a completely > different creature. > > I wonder whether disabling string concatenation overloading will fix > it.

The problem with doing that is that the 'qr' constant overloading is only passed the compile-time constant parts of a regex. So if you have: /abc $var def/ then the overloading is called twice (with 'abc ' and ' def' respectively). The only way to grab and process the *entire* regex is to have the first call return an object with concatenation overloading, so that the eventual run-time concatenation of the contents of $var can be trapped and produce yet another object, which can trap the second concatenation to produce a final object...whose stringify or 'qr' coercion can then finally process the entire regex pattern to produce the desired effect (in this case, inserting the necessary parsing machinery). Without that elaborate work-around, Regexp::Grammars regexes could not contain any interpolations, which would make them vastly less useful. It may still come to that, if I can find no other solutions, but it would be a significant loss of utility, if it does. Show quoted text

> In any case, I think this should really be fixed at perl’s end.

Agreed. Show quoted text

> Please do report a bug, so this can be fixed for 5.20 (or 5.18.1 or > .2, if the changes are not too sweeping).

I certainly intend to, as soon as I can spare enough time to boil an example down to less than the 2500 lines of the module itself. ;-) But, as I'm in final preparation for a five-week round-the-world speaking tour, "as soon as I can" will probably not be "as soon as I'd prefer". Damian Damian

Sun Aug 18 02:03:48 2013 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

Lexical::Var 0.008 has just been released, and works with 5.18. In <CAATtAp7gmiU3rsPGY51wE7HfZyV-SU08vqn0DE=6E6CabQVKVQ@mail.gmail.com> you said: Show quoted text

> I will definitely revisit it as a possible solution on its next release.

I hope that is still true. :-)

Sun Aug 18 19:44:41 2013 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Sun, 18 Aug 2013 16:43:48 -0700
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Father Chrysostomos reported: Show quoted text

> Lexical::Var 0.008 has just been released, and works with 5.18.

And with Regexp::Grammars. :-) Between the recent release of 5.18.1 and this release, the number of outstanding issues has been significantly reduced. Though not, alas, to zero yet, as it appears that Dave's patch for lexical hashes used inside regex code blocks didn't make it into 5.18.1 (???) Anyway, I'll release an update the R::G this morning, with now only a few skips in the test suite under 5.18. Many thanks to everyone who has worked to resurrect my module (and, more importantly, to fix these fundamental issues in Perl itself). Damian

Thu Feb 13 10:24:17 2014 paul [...] city-fan.org - Correspondence added

From:

paul [...] city-fan.org

Now that perl 5.18.2 has been out for a while, what's the compatibility situation for Regexp-Grammars?

Fri Feb 14 20:54:56 2014 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Sat, 15 Feb 2014 12:54:08 +1100
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Show quoted text

> Now that perl 5.18.2 has been out for a while, what's the > compatibility situation for Regexp-Grammars?

I haven't had a chance to revisit the issue since 5.18.2 was released. I will look into it as soon as I am a little less swamped with work. Probably in early March. I apologize for the delay. Damian

Sat Feb 15 01:23:28 2014 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Sat, 15 Feb 2014 17:22:38 +1100
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Show quoted text

> Now that perl 5.18.2 has been out for a while, what's the > compatibility situation for Regexp-Grammars?

Well, despite the prediction of a delay until March, I've now had a chance to investigate. The bug in Perl that is breaking Regexp::Grammars is still present in 5.18.2. I have raised the issue again with the folks who may be able to expedite a fix. Sorry this is not better news. Damian

Sun Jun 08 14:31:09 2014 CARNIL [...] cpan.org - Correspondence added

Hi Damian! On Sat Feb 15 01:23:28 2014, damian@conway.org wrote: Show quoted text

> > Now that perl 5.18.2 has been out for a while, what's the > > compatibility situation for Regexp-Grammars?

> > Well, despite the prediction of a delay until March, I've now had a > chance to investigate. > > The bug in Perl that is breaking Regexp::Grammars is still present in > 5.18.2. I have raised the issue again with the folks who may be able to > expedite a fix. > > Sorry this is not better news. > > Damian

Perl 5.20 was released at end of may. Do you know if all problems are now resolved affecting Regexp::Grammars? Thanks a lot for any information, Regards, Salvatore

Tue Jun 10 17:43:30 2014 damian [...] conway.org - Correspondence added

Subject:	Re: [rt.cpan.org #79149] 5.17.3: Eval-group not allowed at runtime
Date:	Wed, 11 Jun 2014 07:42:41 +1000
To:	bug-Regexp-Grammars [...] rt.cpan.org
From:	Damian Conway <damian [...] conway.org>

Hi Salvatore, I have good news regarding Regexp::Grammars. I have now made all the modifications necessary to allow the module to run correctly under Perl 5.10 to Perl 5.20 (except under Perl 5.18, whose regex overloading is still broken under 5.18.2). I just uploaded the new release to CPAN, so you should be able to grab it within the next few hours. My thanks to everyone for their patience and help in overcoming this problem. Damian

Tue Jun 10 18:06:53 2014 DCONWAY [...] cpan.org - Status changed from 'open' to 'resolved'