| Subject: | Nested patterns of same precedence are not lexed correctly |
| Date: | Fri, 20 Mar 2009 09:45:00 -0400 |
| To: | <bug-hop-lexer [...] rt.cpan.org> |
| From: | <dmitriy.sokolov [...] barclayscapital.com> |
#!perl -w
require HOP::Lexer;
use strict;
my @tokens = (
['DOUBLEQUOTE', qr/"[^"]*"/],
['SINGLEQUOTE', qr/'[^']*'/],
['SPACES', qr/\s+/, sub { return (); }]
);
my $string = q/'"a"' "'b'"/;
print "<$string>\n";
my $lexer = HOP::Lexer::string_lexer($string, @tokens);
while ((my $token = $lexer->())) {
next unless (ref($token) eq 'ARRAY');
print "[", join(", ", @$token), "]\n";
}
One would expect the output to look like this:
<'"a"' "'b'">
[SINGLEQUOTE, '"a"']
[DOUBLEQUOTE, "'b'"]
However, it looks like this:
<'"a"' "'b'">
[DOUBLEQUOTE, "a"]
[DOUBLEQUOTE, "'b'"]
While this may not be a problem for the quote-mixing under certain
circumstances, it surely creates the complications when quotes need to
be distinguished from, say, comments (e.g. /* ... */). Both quoted
strings and comments can include each other and in this case preference
to a token should be given based on the leftmost match rather than the
token sequence.
Dmitriy Sokolov
Show quoted text
_______________________________________________
This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________