Bug #6746 for RTF-Tokenizer: Whitespace in RTF::Tokenizer

Thu Jun 24 09:31:12 2004 Guest - Ticket created

Subject:

Whitespace in RTF::Tokenizer

As mentioned in a personal e-mail this morning, the tokenizer misses whitespace in control codes, resulting in the input \info{author Me being turned into: \info{authorme which breaks the document. The attached file - not yet thoroughly tested - corrects this by removing the (?:) cluster in a substitution in _grab_token(). Further more, the fixed version does away with the module's (undocumented?) attempt at converting [\n\r] to \par, since it did not always work. All [\n\r] are now left as they are found. Hope it helps, Lee Goddard (lee -at- pro check up -dot- com)

Message body is not shown because it is too large.

Thu Jun 24 09:43:54 2004 Guest - Correspondence added

Show quoted text

> As mentioned in a personal e-mail this morning, the tokenizer misses > whitespace in control codes, resulting in the input > > \info{author Me > > being turned into: > > \info{authorme

No, this isn't what happens. Firstly, presumably you mean: \info{\author Me As that's what was in your email. The tokenizer interprets this, correctly, as: ['control', 'info', ''], ['group', '1', ''], ['control', 'author', ''], ['text', 'Me', ''] This is expected and correct behaviour. Your problem is that you are trying to write RTF in an incorrect manner; if a control word is followed by a text field, you *must* place a piece of whitespace between them, which is to be ignored. Show quoted text

> Further more, the fixed version does away with the module's > (undocumented?) attempt at converting [\n\r] to \par, since it did > not always work. All [\n\r] are now left as they are found.

Conversion to \par is correct behaviour. Can you give me an example file where this doesn't work as expected. Thanks +Pete

Thu Jun 24 09:53:10 2004 sargie [...] cpan.org - Status changed from 'new' to 'resolved'