Bug #125059 for Devel-Tokenizer-C: Fails to generate properly

Mon Apr 09 17:40:47 2018 khw [...] cpan.org - Ticket created

Subject:

Fails to generate properly

Attached is a program that reproduces a bug wherein the generated code fails to account for some of the input. In particular, the input includes a line 'ccc=107', but there is no rule generated for that. This example does not compile, because for clarity I omitted the function that converts the returned strings into suitable C identifiers. Using the default strategy also fails, but the narrow strategy does cause a rule for 107 to be emitted. But for that strategy, I notice that no rule for age=v100 gets omitted.

Subject:

tokenizer_bug.pl

Message body is not shown because it is too large.

Mon Apr 09 20:56:26 2018 khw [...] cpan.org - Correspondence added

Subject:

Fails to generate all required rules

On Mon Apr 09 15:40:47 2018, khw wrote: Show quoted text

> Attached is a program that reproduces a bug wherein the generated code > fails to account for some of the input. > > In particular, the input includes a line 'ccc=107', but there is no > rule generated for that. > > This example does not compile, because for clarity I omitted the > function that converts the returned strings into suitable C > identifiers. > > Using the default strategy also fails, but the narrow strategy does > cause a rule for 107 to be emitted. But for that strategy, I notice > that no rule for age=v100 gets omitted.

I believe that below is a complete list of the missing rules. All these have digits in their strings, and I didn't see any missing one that didn't have digits (though I could have missed some). age=v100 age=v20 age=v30 age=v40 age=v50 age=v60 age=v70 age=v80 age=v90 canonicalcombiningclass=ccc10 canonicalcombiningclass=ccc103 canonicalcombiningclass=ccc107 canonicalcombiningclass=ccc130 canonicalcombiningclass=ccc20 canonicalcombiningclass=ccc30 ccc=10 ccc=103 ccc=130 ccc=20 ccc=30 in=v100 in=v20 in=v30 in=v40 in=v50 in=v60 in=v70 in=v80 in=v90 numericvalue=0 numericvalue=10 numericvalue=100 numericvalue=1000 numericvalue=10000 numericvalue=100000 numericvalue=1/10 numericvalue=1/20 numericvalue=1/40 numericvalue=20 numericvalue=200 numericvalue=2000 numericvalue=20000 numericvalue=200000 numericvalue=216000 numericvalue=30 numericvalue=300 numericvalue=3000 numericvalue=30000 numericvalue=300000 numericvalue=3/20 numericvalue=3/80 numericvalue=40 numericvalue=400 numericvalue=4000 numericvalue=40000 numericvalue=400000 numericvalue=432000 numericvalue=500 numericvalue=5000 numericvalue=50000 numericvalue=500000 numericvalue=600 numericvalue=6000 numericvalue=60000 numericvalue=600000 numericvalue=700 numericvalue=7000 numericvalue=70000 numericvalue=700000 numericvalue=800 numericvalue=8000 numericvalue=80000 numericvalue=800000 numericvalue=900 numericvalue=9000 numericvalue=90000 numericvalue=900000 nv=0 nv=10 nv=100 nv=1000 nv=10000 nv=100000 nv=1/10 nv=1/160 nv=1/20 nv=1/40 nv=20 nv=200 nv=2000 nv=20000 nv=200000 nv=216000 nv=30 nv=300 nv=3000 nv=30000 nv=300000 nv=3/20 nv=3/80 nv=40 nv=400 nv=4000 nv=40000 nv=400000 nv=432000 nv=500 nv=5000 nv=50000 nv=500000 nv=600 nv=6000 nv=60000 nv=600000 nv=700 nv=7000 nv=70000 nv=700000 nv=800 nv=8000 nv=80000 nv=800000 nv=900 nv=9000 nv=90000 nv=900000

Tue Apr 10 20:21:36 2018 MHX [...] cpan.org - Correspondence added

Nice catch, thanks! It turns out it's missing all items for which the final stage has multiple candidates and one of the candidates is a zero. Depending on the strategy chosen, those are likely different candidates. The simplest repro is using two keywords 'k0' and 'k1' with the ordered (default) strategy. Only the code for 'k1' will be generated. This is easy to fix, I'll try to get a new release out on CPAN as soon as possible.

Tue Apr 10 20:21:36 2018 The RT System itself - Status changed from 'new' to 'open'

Tue Apr 10 20:37:28 2018 MHX [...] cpan.org - Correspondence added

I've uploaded v0.09 to pause. Please let me know if that works for you.

Wed Apr 11 00:18:02 2018 khw [...] cpan.org - Correspondence added

That seems to have done the trick. Thank you very much

Fri Apr 13 14:58:56 2018 MHX [...] cpan.org - Correspondence added

On Wed Apr 11 00:18:02 2018, khw wrote: Show quoted text

> That seems to have done the trick.

Great, thanks! :) I released 0.09 too quickly, forgot to update the test plan after adding a test, so there's fails everywhere for that version. I've added some more tests, and released 0.10. The actual code is unchanged, though, so you should be fine.

Fri Apr 13 14:58:58 2018 MHX [...] cpan.org - Status changed from 'open' to 'resolved'

Fri Apr 13 14:58:58 2018 MHX [...] cpan.org - Fixed in 0.09 added