Bug #57340 for PDF-Create: Inserting images fails with LANG=UTF8

Mon May 10 05:58:36 2010 frankie [...] etsetb.upc.edu - Ticket created

Subject:	Inserting images fails with LANG=UTF8
From:	frankie [...] etsetb.upc.edu

Hi. I spotted a serious problem. It only happens when some circumstances are met. Images are added wrong in the PDF when LANG=UTF8 and a DBI networking connection is made. I've been digging the root of the problem. I will try to explain the best that I can. The PDFs created are different in the line where the image size is defined in the PDF file. When LANG=ca_ES@euro PDF: 510.236220472441 0 0 841.889763779528 0 0 cm When LANG=ca_ES.UTF-8 PDF: 510,236220472441 0 0 841,889763779528 0 0 cm Notice when the PDF is wrong the decimal separator is "," instead ".". Something happens somewhere in the DBI networking process that triggers a change in that. Then it affects the image size definition inside the PDF. I tried many different DBI releases and it always happen the same. I also tried connection to mysql and postgress servers. The connection doesn't need to complete right. A bogus hostname does the trick : eval {DBI->connect("DBI:Pg:host=WHATEVER") }; PDF::Create v1.05 DBI v1.609 perl v5.10.1 I can also send a minimal script that creates the wrong PDF if you need it.

Mon May 10 06:34:39 2010 frankie [...] etsetb.upc.edu - Correspondence added

I just added a test to help spot the bug.

Subject:

60-dbi.t

#!/usr/bin/perl -w # # PDF::Create - Test Script # BEGIN { unshift @INC, "lib", "../lib" } use strict; use PDF::Create; use Test::More tests => 3; # we want the resulting pdf file to have the same name as the test my $pdfname = $0; $pdfname =~ s/\.t/\.pdf/; my $MM = 25.4 / 72; my $PAGE_WIDTH = 210 / $MM; my $PAGE_HEIGHT = 297 / $MM; my ($X,$Y)= (300,369); $ENV{LANG}='es_ES.UTF-8'; ################################################################### # # start testing # SKIP: { eval { require DBI; require DBD::mysql; }; $@ =~ s/(.*?\.pm) in /$1/; skip $@,3 if $@; eval { DBI->connect("DBI:mysql:host=WHATEVER",undef,undef ,{PrintError=>0}) }; my $pdf = new PDF::Create( 'filename' => "$pdfname", 'Version' => 1.2, 'PageMode' => 'UseOutlines', 'Author' => 'Markus Baertschi', 'Title' => 'Testing Basic Stuff', ); my $root = $pdf->new_page( 'MediaBox' => $pdf->get_page_size('A4') ); ok( defined $root, "Create page root" ); my $image_file = "t/pdf-logo.jpg"; my $image = $pdf->image($image_file) or die $!; my $page = $root->new_page(); $page->image( image => $image , xpos => 0,ypos => 0 , xscale => $PAGE_WIDTH/$Y, yscale => $PAGE_HEIGHT/$Y); ok( !$pdf->close(), "Close PDF" ); my $SIZE = "483.963894757058 0 0 841.889763779528 0 0 cm"; open PDF,"<$pdfname" or die "$! $pdfname"; my $line; for ( 1..8){ $line=<PDF>; } chomp $line; ok ( $line eq $SIZE); close PDF; };

Mon May 10 06:34:40 2010 frankie [...] etsetb.upc.edu - Status changed from 'new' to 'open'

Mon May 10 07:03:19 2010 MARKUSB [...] cpan.org - Taken

Mon May 10 07:04:07 2010 MARKUSB [...] cpan.org - Broken in 1.05 added

Mon May 10 07:28:16 2010 MARKUSB [...] cpan.org - Correspondence added

It looks to me like this comes from the implicit float->string conversion in the image method of the page object (page.pm, lines 526- 538). Could you try to insert $ENV{LANG}='C' on line 525 ?

Mon May 10 07:51:54 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Mon, 10 May 2010 13:51:10 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Mon, May 10, 2010 at 07:28:17AM -0400, Markus Baertschi via RT wrote: Show quoted text

> > It looks to me like this comes from the implicit float->string > conversion in the image method of the page object (page.pm, lines 526- > 538).

In my source for Page.pm from PDF::Create v1.05 I have init_widths sub: sub init_widths { { 'Courier' => [ 599, 599, 599, 599, 599, 599, Show quoted text

> Could you try to insert $ENV{LANG}='C' on line 525 ?

I inserted it in the beginning of sub image like this: ####################################################################### # Place an image on the current page # sub image { $ENV{LANG}='C'; my $self = shift; my %params = @_; But I see no effect. Actually I tried doing several $ENV{LANG}='C' all through the test code: right before calling image, before calling new_page and even before doing PDF::Create->new. It never worked. It only works if I change the LANG before DBI->connect. Weird !

Mon May 10 08:02:28 2010 frankie [...] etsetb.upc.edu - Correspondence added

Well, it looks like I call image with "," separators already. I found in perldata: "If use locale is in effect, and POSIX::setlocale() has been called, the character used for the decimal separator is affected by the LC_NUMERIC locale." So ,the DBI code must be calling those functions, it should be cleared before the output to the pdf file. I will look a little deeper tomorrow.

Mon May 10 08:16:42 2010 MARKUSB [...] cpan.org - Correspondence added

On Mon May 10 08:02:28 2010, FRANKIE wrote: Show quoted text

> So ,the DBI code must be calling those functions, it should be cleared > before the output to the pdf file. I will look a little deeper

tomorrow. You might need $loc = setlocale( LC_NUMERIC, "C" ); before calculating the image size/scaling in your code. Markus

Mon May 10 10:07:38 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Mon, 10 May 2010 16:06:52 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Mon, May 10, 2010 at 08:16:43AM -0400, Markus Baertschi via RT wrote: Show quoted text

> > You might need $loc = setlocale( LC_NUMERIC, "C" ); before calculating > the image size/scaling in your code. >

Absolutely. I added this in the beginning of sub image in Page.pm, then I did a setlocale( LC_NUMERIC, $loc); in the end. In addition, I had to do use POSIX in Page.pm

Mon May 10 11:11:26 2010 markus [...] markus.org - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Mon, 10 May 2010 17:10:45 +0200
To:	bug-PDF-Create [...] rt.cpan.org
From:	Markus Baertschi <markus [...] markus.org>

I don't quite understand: You said before that your program is passing the comma instead of the dot to $page->image(). Now it sounds like adding setlocale to page.pmfixes things. If it is the first (you pass the comma) then changing page.pm does not help. If it is the second then I'll have to see what is the best/most elegant way to make sure I always generate floats with dots instead of commas. Show quoted text

> > You might need $loc = setlocale( LC_NUMERIC, "C" ); before calculating > > the image size/scaling in your code.

> Absolutely. I added this in the beginning of sub image in Page.pm, > then I did a setlocale( LC_NUMERIC, $loc); in the end. > In addition, I had to do use POSIX in Page.pm > > >

-- Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Rossé 16 Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: markus@markus.org Switzerland Homepage: www.markus.org

Tue May 11 03:28:27 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 09:27:50 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Mon, May 10, 2010 at 11:11:27AM -0400, Markus Baertschi via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=57340 > > > I don't quite understand: You said before that your program is passing the > comma instead of

Yes, my wrong. I added some warns to check the contents of the xscale values. And when I printed in the screen it was with a comma. In my opinion, that is a float internally. When I ask to show it on screen, the locale makes it print in my language format. So it has a comma. But if I changed the locale to 'C' right before printing it it is shown with a dot. Show quoted text

> the dot to $page->image(). Now it sounds like adding setlocale to > page.pmfixes things. > If it is the first (you pass the comma) then changing page.pm does not help. > > If it is the second then I'll have to see what is the best/most elegant way > to make > sure I always generate floats with dots instead of commas.

In my opinion the PDF spec requires a dot as a decimal separator. To make sure it is like this a change is required to Page.pm so float values are created in 'C' locale. After inserting the image the locale should be restored to its initial value. So if I use a a locale in my code it should be like this when it returns from page->image. This way you are sure the PDFs are created right, no matter what locale is set in the main code.

Tue May 11 04:40:10 2010 markus [...] markus.org - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 10:39:33 +0200
To:	bug-PDF-Create [...] rt.cpan.org
From:	Markus Baertschi <markus [...] markus.org>

On Tue, May 11, 2010 at 9:28 AM, Francesc Guasch via RT < bug-PDF-Create@rt.cpan.org> wrote: Show quoted text

> > In my opinion the PDF spec requires a dot as a decimal separator. > To make sure it is like this a change is required to Page.pm so > float values are created in 'C' locale. After inserting the image > the locale should be restored to its initial value. So if I use a > a locale in my code it should be like this when it returns from > page->image. > > This way you are sure the PDFs are created right, no matter what > locale is set in the main code. >

There is something I don't know about the locale thing. Isn't the locale setting a global thing ? That means if I change the locale to 'C' in page.pm, then suddenly this changes the locale for the calling program too. This would introduce a unwanted side-effect and break things. -- Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Rossé 16 Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: markus@markus.org Switzerland Homepage: www.markus.org

Tue May 11 04:54:59 2010 markus [...] markus.org - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 10:54:27 +0200
To:	bug-PDF-Create [...] rt.cpan.org
From:	Markus Baertschi <markus [...] markus.org>

I just did some rtfm, can you insert the 'no locale' command at the beginning of the image() subroutine in page.pm ? This pragma is tells perl to ignore the locale settings until the end of the current block. This would be a good fix with no unintended collateral damage. no locale Sorry for the work, but I'm at my money-earning day-job and don't have a test-environment here. Markus -- Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Rossé 16 Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: markus@markus.org Switzerland Homepage: www.markus.org

Tue May 11 05:12:08 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 11:11:22 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Tue, May 11, 2010 at 04:40:13AM -0400, Markus Baertschi via RT wrote: Show quoted text

> > > > In my opinion the PDF spec requires a dot as a decimal separator. > > To make sure it is like this a change is required to Page.pm so > > float values are created in 'C' locale. After inserting the image

> > There is something I don't know about the locale thing. Isn't the > locale setting a global thing ? > That means if I change the locale to 'C' in page.pm, then suddenly > this changes the locale for the calling program too.

I'm testing what happens when there is code that messes with locales. When a call is made to DBI->connect the LC_NUMERIC is changed because internally there must be some code that uses the locale. When it returns from connect it is changed globally. print setlocale(LC_NUMERIC)."\n"; *C* $dbh = DBI->connect("dbi:mysql:host... print setlocale(LC_NUMERIC)."\n"; *es_ES.UTF-8* Show quoted text

> This would introduce a unwanted side-effect and break things.

Yes it does, but you can also consider it a desired behaviour. If I retrieve some number from a database and I want to show it on screen it should be shown in a format I will understand. In that case I have configured my terminal as es_ES.UTF-8 and that is what I get. But then if I create a PDF and its internal numbering spec requires a dot instead a comma the locale should be changed locally and returned to the previous format later.

Tue May 11 05:16:34 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 11:15:59 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Tue, May 11, 2010 at 04:55:00AM -0400, Markus Baertschi via RT wrote: Show quoted text

> > I just did some rtfm, can you insert the 'no locale' command at the > beginning of the image() subroutine in page.pm ? > This pragma is tells perl to ignore the locale settings until the end of the > current block. This would be a good fix with no unintended collateral > damage.

It failed, doing the setlocale and restoring it at the end worked fine. Show quoted text

> Sorry for the work, but I'm at my money-earning day-job and don't have a > test-environment here.

Don't worry, I'm glad to help.

Tue May 11 05:22:08 2010 markus [...] markus.org - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Tue, 11 May 2010 11:21:33 +0200
To:	bug-PDF-Create [...] rt.cpan.org
From:	Markus Baertschi <markus [...] markus.org>

On Tue, May 11, 2010 at 11:12 AM, Francesc Guasch via RT < bug-PDF-Create@rt.cpan.org> wrote: Show quoted text

> I'm testing what happens when there is code that messes with locales. > When a call is made to DBI->connect the LC_NUMERIC is changed because > internally there must be some code that uses the locale. When it > returns from connect it is changed globally. > > print setlocale(LC_NUMERIC)."\n"; > > *C* > > > $dbh = DBI->connect("dbi:mysql:host... > > print setlocale(LC_NUMERIC)."\n"; > > *es_ES.UTF-8* > >

> > This would introduce a unwanted side-effect and break things.

>

In my opinion this is a bug in DBI. If the application/database requires a change in locale this should have to be done explicitly by the app. Changing locales behind the scene breaks things, as we have seen. Yes it does, but you can also consider it a desired behaviour. If I Show quoted text

> retrieve some number from a database and I want to show it on screen > it should be shown in a format I will understand. In that case I > have configured my terminal as es_ES.UTF-8 and that is what I get. >

Yes, but you, as developer know about the locales you want to support and can set it according to your needs. But then if I create a PDF and its internal numbering spec requires Show quoted text

> a dot instead a comma the locale should be changed locally and > returned to the previous format later. >

I accept this as a bug in PDF::Create. It should not depend on a specific local/set of locales, but it should not change them globally for others either. Imagine your app depends on the correct locale, but calls to PDF::Create change it, so you have to reset if after each invocation. Markus -- Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Rossé 16 Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: markus@markus.org Switzerland Homepage: www.markus.org

Wed May 12 06:32:45 2010 frankie [...] etsetb.upc.edu - Correspondence added

Subject:	Re: [rt.cpan.org #57340] Inserting images fails with LANG=UTF8
Date:	Wed, 12 May 2010 12:32:04 +0200
To:	Markus Baertschi via RT <bug-PDF-Create [...] rt.cpan.org>
From:	Francesc Guasch <frankie [...] etsetb.upc.edu>

On Tue, May 11, 2010 at 05:22:09AM -0400, Markus Baertschi via RT wrote: Show quoted text

> > In my opinion this is a bug in DBI. If the application/database requires a > change in locale this should have to be done explicitly by the app. Changing > locales behind the scene breaks things, as we have seen.

you must be right, I'll file a bug to DBI so. Show quoted text

> > I accept this as a bug in PDF::Create. It should not depend on a specific > local/set of locales, but it should not change them globally for others

Right, so before returning from Page->image it should restore the locale. That is what I said yesterday. This is a diff from my Page.pm: 23a24 Show quoted text

> use POSIX;

464a466,468 Show quoted text

> my $loc = setlocale(LC_NUMERIC); > setlocale(LC_NUMERIC,'C'); >

515a520,521 Show quoted text

> > setlocale(LC_NUMERIC,$loc);

I think that makes Page->image locale transparent and won't create undesirable consequences to the caller. And I also attach a test file. It checks whether PDF with images are correctly created when utf spanish locales are used. It skips the test if the es_ES.utf-8 locale is not available.

Message body is not shown because sender requested not to inline it.

Tue Jul 06 09:29:47 2010 MARKUSB [...] cpan.org - Status changed from 'open' to 'patched'

Tue Jul 06 09:30:08 2010 MARKUSB [...] cpan.org - Severity Normal added

Wed Jul 07 16:30:58 2010 MARKUSB [...] cpan.org - Status changed from 'patched' to 'resolved'

Wed Jul 07 16:30:59 2010 MARKUSB [...] cpan.org - Fixed in 1.06 added