Bug #71377 for FBP-Perl: Unicode support

Sat Oct 01 03:55:21 2011 dolmen [...] cpan.org - Ticket created

Subject:

Unicode support

The FBP dialogs may contain non-ascii labels. The FBP file is encoded in UTF8 and so is correctly loaded. But FBP-Perl should be able to output Perl code that reproduce them. This is a major issue for the Padre About dialog. Currently, "Olivier Mengué" in Padre.fbp is output as "Olivier Mengu\x{e9}". The perl file has no specified encoding, so it will be interpreted differently depending on the OS and the user environment (for example, works on Windows/French, not on Ubuntu/fr_FR.utf8). FBP-Perl should output Perl code using "use utf8" and generate valid UTF-8 sequences. The documentation should also indicate that the flatten output must be written in a file using utf8 encoding. I have a commit bit on svn.ali.as. Is it ok to make changes directly in the FBP-Perl repo? -- Olivier Mengué - http://perlresume.org/DOLMEN

Sat Oct 01 07:23:41 2011 dolmen [...] cpan.org - Correspondence added

Data::Dumper::Dumper::qquote (used in FBP::Perl::quote) does not work correctly for Unicode strings: it outputs byte strings. -- Olivier Mengué - http://perlresume.org/DOLMEN

Sat Oct 01 07:23:42 2011 dolmen [...] cpan.org - Status changed from 'new' to 'open'

Thu Oct 06 00:37:09 2011 adam [...] ali.as - Correspondence added

Subject:	Re: [rt.cpan.org #71377] Unicode support
Date:	Thu, 6 Oct 2011 15:36:59 +1100
To:	bug-FBP-Perl [...] rt.cpan.org
From:	Adam Kennedy <adamkennedybackup [...] gmail.com>

If you have commit, by all means go ahead and make the necessary changes. The one thing that I would request as you implement this change is that you implement the addition of the "use utf8;" and UTF-8 writing in such a way as that it is ONLY done if there is non-ASCII content in the file. I have a strong bias against unnecessary boilerplate, and it has served me well. In addition, it would mean that seeing "use utf8" in a file can serve as an indicator that there is actual utf8 content present in the file. Best Regards Adam K 2011/10/1 Olivier Mengué via RT <bug-FBP-Perl@rt.cpan.org>: Show quoted text

> Sat Oct 01 03:55:21 2011: Request 71377 was acted upon. > Transaction: Ticket created by DOLMEN > Queue: FBP-Perl > Subject: Unicode support > Broken in: 0.64 > Severity: Important > Owner: Nobody > Requestors: dolmen@cpan.org > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=71377 > > > > The FBP dialogs may contain non-ascii labels. The FBP file is encoded in > UTF8 and so is correctly loaded. > But FBP-Perl should be able to output Perl code that reproduce them. > > > This is a major issue for the Padre About dialog. > > Currently, "Olivier Mengué" in Padre.fbp is output as "Olivier > Mengu\x{e9}". The perl file has no specified encoding, so it will be > interpreted differently depending on the OS and the user environment > (for example, works on Windows/French, not on Ubuntu/fr_FR.utf8). > > > FBP-Perl should output Perl code using "use utf8" and generate valid > UTF-8 sequences. > The documentation should also indicate that the flatten output must be > written in a file using utf8 encoding. > > > > I have a commit bit on svn.ali.as. Is it ok to make changes directly in > the FBP-Perl repo? > > -- > Olivier Mengué - http://perlresume.org/DOLMEN >

Thu Oct 06 18:06:40 2011 dolmen [...] cpan.org - Correspondence added

Le 2011-10-06 06:37:09, adam@ali.as a écrit : Show quoted text

> If you have commit, by all means go ahead and make the necessary > changes. > > The one thing that I would request as you implement this change is > that you implement the addition of the "use utf8;" and UTF-8 writing > in such a way as that it is ONLY done if there is non-ASCII content in > the file.

Show quoted text

> I have a strong bias against unnecessary boilerplate, and it has > served me well. > > In addition, it would mean that seeing "use utf8" in a file can serve > as an indicator that there is actual utf8 content present in the file.

The current FBP::Perl API does not write by itself: this is a task delegated to the user. So the user of the API will have to do the job of detecting non-ascii bytes and choose to output with the proper encoding and the proper Perl header (byte order mark or "use utf8/encoding..."). Note also that we have to output raw bytes because AFAIK the gettext tool do not recognize Perl escapes (and anyway Perl escapes do not work well for unicode between 128 and 255). -- Olivier Mengué - http://perlresume.org/DOLMEN