Bug #30385 for HOP-Lexer:

Wed Oct 31 05:47:45 2007 CASIANO [...] cpan.org - Ticket created

Dear Ovid, Thank you for such helpful module. Let me ask you for two features: - Parse::Yapp and Parse::Eyapp generated parsers use a pair protocol (TOKEN, VALUE) instead of a reference [TOKEN, VALUE]. Can we have an option 'PROTOCOL' that will change the protocol to suit Parse::E?Yapp needs? That means that EOF is marked by the pair ('', undef) instead of undef. - On any serious parser - in order to produce accurate error messages - we need to keep record of line numbers. Can we have an option "LINENUMBERS" that when activated will change the behavior of the generated lexer, returning as an additional information the line number where the token begins? something like: (TOKEN, [ VALUE, LINENUMBER]) or may be this instead (TOKEN, { VAL => VALUE, LINENUMBER => LINENUMBER}) we can certainly do it with the current API adding some code but I feel it clutters the description of the lexer. Many thanks for your work Casiano

Wed Oct 31 06:06:18 2007 ovid [...] cpan.org - Correspondence added

Subject:	Line numbers and YAPP
From:	OVID [...] cpan.org

Hi Casiano, You raise excellent concerns. Due to the nature of the lexer, line numbers could potentially be difficult to detect. I'd have to figure out a clean way of tracking newlines along with matching the existing tokens. I'll give this some thought. As for the pair protocol, could you show me a programmatic example of the interface you're thinking about? It seems to me that what you're asking for is this, but I'm unsure: my $lexer = make_lexer( $iter, @input_tokens ); while ( my ( $token, $value ) = $lexer->() ) { ... } Or with line numbers: my $lexer = make_lexer( $iter, @input_tokens ); while ( my ( $token, $value, $line ) = $lexer->() ) { ... } If that's really what you're looking for, you could easily wrap the lexer sub to transform what you're looking for (and keep with the functional nature of the module: my $lexer = make_lexer( $iter, @input_tokens ); my $yapp_lexer = sub { my $token = $lexer->(); return ( '', undef ) unless $token; return @$token; } while ( my ( $token, $value ) = $yapp_lexer->() ) { ... } Does that satisfy your YAPP needs? If so, it's a more appropriate solution and I can add this example to the documentation and then try to figure out the line number issue. Cheers, Ovid

Wed Oct 31 06:06:42 2007 The RT System itself - Status changed from 'new' to 'open'

Wed Oct 31 09:20:31 2007 casiano.rodriguez.leon [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #30385] Line numbers and YAPP
Date:	Wed, 31 Oct 2007 13:19:22 +0000
To:	bug-HOP-Lexer [...] rt.cpan.org
From:	"Casiano Rodriguez-Leon" <casiano.rodriguez.leon [...] gmail.com>

Thanks a lot, Ovid! Show quoted text

> > You raise excellent concerns. Due to the nature of the lexer, line > numbers could potentially be difficult to detect. I'd have to figure > out a clean way of tracking newlines along with matching the existing > tokens. I'll give this some thought.

It seems difficult to solve. May be using tr on the matching string, but I don't know. As for the pair protocol, could you show me a programmatic example of Show quoted text

> the interface you're thinking about? It seems to me that what you're > asking for is this, but I'm unsure:

Yes, you are right. It can be solved using a simple wrapper. In fact this is what I am doing now. Follows an example taken from my notes (see http://nereida.deioc.ull.es/~pl/perlexamples/node148.html): 54 sub Run { 55 my($self)=shift; 56 my $input = shift; 57 58 my @lexemes = ( 59 ['SPACE', qr{[ \t]+}, sub { () } ], 60 ['NUM', qr{[0-9]+(?:\.[0-9]+)?}], 61 ['VAR', qr{[A-Za-z][A-Za-z0-9_]*}], 62 ["\n", qr{\n}], 63 ['ANY', qr{.}, 64 sub { 65 my ($label, $val) = @_; 66 return [ $val, $val ]; 67 } 68 ] 69 70 ); 71 my $lexer = string_lexer($input, @lexemes); 72 73 return $self->YYParse( 74 yylex => sub { 75 my $parser = shift; 76 my $token = $lexer->(); 77 return @$token if defined($token); 78 return ('', undef); 79 }, 80 yyerror => \&_Error, 81 #yydebug => 0xF 82 ); 83 } And certainly is not much trouble. Many thanks for your fast reply and your great modules Casiano