| Subject: | Whitespace-only line is concatenated with the whitespaces in the beginning of the next line |
Please take a look on these three cases:
one: "\n foo". it will translate to two tokens: whitespace "\n " and
word "foo".
two: "\n \nfoo". now we will have three tokens: whitespace "\n",
whitespace " \n" and word "foo".
three: "foo\n foo", will translate to: word "foo", whitespace "\n",
whitespace " ", word "foo".
So I think that the first case is a bug, and the whitespace should be
splitted to newline and the rest.
For doing this, you should add in Whitespace.pm the following line: (in
pseudo diff)
sub __TOKENIZER__on_line_start {
my $t = $_[1];
my $line = $t->{line};
# Can we classify the entire line in one go
if ( $line =~ /^\s*$/ ) {
# A whitespace line
$t->_new_token( 'Whitespace', $line );
+ $t->_finalize_token;
return 0;
Thanks,
Shmuel.