Bug #78490 for App-OrgUtils: list-org-todos may fail if org file contains non-ascii (or non-latin1) characters

Fri Jul 20 10:50:54 2012 SREZIC [...] cpan.org - Ticket created

Subject:

list-org-todos may fail if org file contains non-ascii (or non-latin1) characters

The attached file test4 cannot be processed with list-org-todos: $ /opt/perl-5.14.2/bin/list-org-todos /tmp/test4 Cannot decode string with wide characters at /opt/perl-5.14.2/lib/site_perl/5.14.2/x86_64-linux/Encode.pm line 176. The other attached file test5 does not fail, but it seems that the a-umlaut character gets lost: $ /opt/perl-5.14.2/bin/list-org-todos /tmp/test5 .-----------. | rodjend� | '-----------' Note that both files are in utf8 encoding. The first contains a character from the latin2 set (central european), the second a character from the latin1 set (western european). Regards, Slaven

Subject:

test5

Download test5
application/octet-stream 61b

Message body not shown because it is not plain text.

Subject:

test4

Download test4
application/octet-stream 60b

Message body not shown because it is not plain text.

Fri Jul 20 10:53:13 2012 SREZIC [...] cpan.org - Cc srezic@iconmobile.com added

Fri Jul 20 12:04:02 2012 sharyanto [...] cpan.org - Correspondence added

Are you sure you are using the latest version of App::OrgUtils and Org::Parser? Works fine here (Perl 5.10.1 though, but still).

Fri Jul 20 12:04:03 2012 The RT System itself - Status changed from 'new' to 'open'

Fri Jul 20 12:07:16 2012 SREZIC [...] cpan.org - Correspondence added

On 2012-07-20 12:04:02, SHARYANTO wrote: Show quoted text

> Are you sure you are using the latest version of App::OrgUtils and > Org::Parser? Works fine here (Perl 5.10.1 though, but still).

Yes: cpan[3]> m App::OrgUtils Module id = App::OrgUtils CPAN_USERID SHARYANTO (Steven Haryanto <stevenharyanto@gmail.com>) CPAN_VERSION 0.13 CPAN_FILE S/SH/SHARYANTO/App-OrgUtils-0.13.tar.gz UPLOAD_DATE 2012-06-13 MANPAGE App::OrgUtils - Some utilities for Org documents INST_FILE /opt/perl-5.14.2/lib/site_perl/5.14.2/App/OrgUtils.pm INST_VERSION 0.13 cpan[4]> m Org::Parser Module id = Org::Parser CPAN_USERID SHARYANTO (Steven Haryanto <stevenharyanto@gmail.com>) CPAN_VERSION 0.24 CPAN_FILE S/SH/SHARYANTO/Org-Parser-0.24.tar.gz UPLOAD_DATE 2012-07-17 MANPAGE Org::Parser - Parse Org documents INST_FILE /opt/perl-5.14.2/lib/site_perl/5.14.2/Org/Parser.pm INST_VERSION 0.24

Fri Jul 20 13:59:13 2012 sharyanto [...] cpan.org - Correspondence added

Does dump-org-structure work there? Anyway, I'm putting Unicode test files in the distribution (Org::Parser 0.25). Will wait for results from CPAN Testers. Regards, Steven

Mon Jul 23 04:15:18 2012 SREZIC [...] cpan.org - Correspondence added

On 2012-07-20 13:59:13, SHARYANTO wrote: Show quoted text

> Does dump-org-structure work there?

Yes, as expected: $ /opt/perl-5.14.2/bin/dump-org-structure t/data/unicode/latin2.org Document: Text: "TODO -*- mode: org; coding: utf-8; -*-\n\n" Headline: l=1 todo=TODO "* TODO rođendan\n" (title) Text: "rođendan" Show quoted text

> Anyway, I'm putting Unicode test files in the distribution (Org::Parser > 0.25). Will wait for results from CPAN Testers.

The new test even passes on my system. It seems it's the output which is failing here, not the parsing. Regards, Slaven

Wed Jul 25 02:11:24 2012 sharyanto [...] cpan.org - Correspondence added

Found the culprit, it only happens under --format=text (which is the default) because Text::ASCIITable tries to decode utf8 when it shouldn't have to (because string already contains unicode and I/O stream is also already :utf8). Try the new 0.15. Regards, Steven

Wed Jul 25 02:11:25 2012 sharyanto [...] cpan.org - Status changed from 'open' to 'resolved'