Bug #101372 for Text-CSV_XS: Should allow_loose_quotes fix error 2027 - EIQ

Wed Jan 07 06:23:55 2015 bugs [...] supernover.de - Ticket created

Subject:	Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?
Date:	Wed, 07 Jan 2015 12:23:35 +0100
To:	bug-Text-CSV_XS [...] rt.cpan.org
From:	" - bugs [...] supernover.de" <bugs [...] supernover.de>

I'd think that allow_loose_quotes should allow for constructs which otherwise trigger error 2027... perl -e 'use Text::CSV_XS;my $csv = Text::CSV_XS->new ({sep_char => ";", allow_loose_quotes => 1, escape_char => undef, binary => 1 }); $csv->parse (qq{;"Bad" ;});print "". $csv->error_diag(), "\n";' gives EIQ - Quoted field not terminated Environment: Text-CSV_XS 1.13 perl, v5.10.0 built for x86_64-linux-thread-multi SUSE Linux Enterprise Server 11 (x86_64) Kernel 3.0.13-0.27 regards, j.

Wed Jan 07 06:44:21 2015 h.m.brand [...] xs4all.nl - Correspondence added

Subject:	Re: [rt.cpan.org #101372] Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?
Date:	Wed, 7 Jan 2015 12:44:04 +0100
To:	bug-Text-CSV_XS [...] rt.cpan.org
From:	"H.Merijn Brand" <h.m.brand [...] xs4all.nl>

On Wed, 7 Jan 2015 06:23:56 -0500, " - bugs@supernover.de via RT" <bug-Text-CSV_XS@rt.cpan.org> wrote: Show quoted text

> I'd think that allow_loose_quotes should allow for constructs which > otherwise trigger error 2027... > > perl -e 'use Text::CSV_XS;my $csv = Text::CSV_XS->new ({sep_char => ";", > allow_loose_quotes => 1, escape_char => undef, binary => 1 }); > $csv->parse (qq{;"Bad" ;});print "". $csv->error_diag(), "\n";' > > gives > EIQ - Quoted field not terminated

As it should. For what you show, you'd need allow_white_space => 1 $ perl -MCSV -we'dcsv (in => \q{,"Bad" ,}, escape_char => undef, allow_loose_quotes => 1)' # CSV_XS ERROR: 2027 - EIQ - Quoted field not terminated @ rec 0 pos 8 field 2 [] $ perl -MCSV -we'dcsv (in => \q{,"Bad" ,}, escape_char => undef, allow_whitespace => 1)' [ [ '', 'Bad', '' ] ] If what you suggest would be true, this line: ;"Bad" ;"Good";1 Would result in [ "", "Bad\" ;\"Good", 1 ] most likely NOT what you would expect. I will await your comment, but I will resolve this ticket as invalid -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.21 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

Download (untitled)
application/pgp-signature 490b

Message body not shown because it is not plain text.

Wed Jan 07 06:44:22 2015 The RT System itself - Status changed from 'new' to 'open'

Wed Jan 07 13:00:54 2015 bugs [...] supernover.de - Correspondence added

Subject:	Re: [rt.cpan.org #101372] Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?
Date:	Wed, 07 Jan 2015 18:59:11 +0100
To:	bug-Text-CSV_XS [...] rt.cpan.org
From:	" - bugs [...] supernover.de" <bugs [...] supernover.de>

Am 07.01.2015 um 12:44 schrieb h.m.brand@xs4all.nl via RT: Show quoted text

>> EIQ - Quoted field not terminated

> > As it should. For what you show, you'd need allow_white_space => 1

Ah, I see. I read about allow_white_space in the docs, but included only this one example, and allow_white_space => 1 does not allow for the following (which I, sadly, also encounter): perl -e 'use Text::CSV_XS;my $csv = Text::CSV_XS->new ({sep_char => ";", allow_loose_quotes => 1, escape_char => undef, binary => 1, allow_whitespace => 1 }); $csv->parse (qq{;"Bad" Bank;});print "". $csv->error_diag(), "\n";' EIQ - Quoted field not terminated Best regards, j.

Wed Jan 07 14:07:51 2015 h.m.brand [...] xs4all.nl - Correspondence added

Subject:	Re: [rt.cpan.org #101372] Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?
Date:	Wed, 7 Jan 2015 20:07:29 +0100
To:	bug-Text-CSV_XS [...] rt.cpan.org
From:	"H.Merijn Brand" <h.m.brand [...] xs4all.nl>

On Wed, 7 Jan 2015 13:00:55 -0500, " - bugs@supernover.de via RT" <bug-Text-CSV_XS@rt.cpan.org> wrote: Show quoted text

> Queue: Text-CSV_XS > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=101372 > > > Am 07.01.2015 um 12:44 schrieb h.m.brand@xs4all.nl via RT:

> >> EIQ - Quoted field not terminated

> > > > As it should. For what you show, you'd need allow_white_space => 1

> > Ah, I see. I read about allow_white_space in the docs, but included only > this one example, and allow_white_space => 1 does not allow for the > following (which I, sadly, also encounter):

*) Show quoted text

> perl -e 'use Text::CSV_XS;my $csv = Text::CSV_XS->new ({sep_char => ";", > allow_loose_quotes => 1, escape_char => undef, binary => 1, > allow_whitespace => 1 }); $csv->parse (qq{;"Bad" Bank;});print "". > $csv->error_diag(), "\n";' > > EIQ - Quoted field not terminated

;"Bad" Bank; is unparsable. Your *eyes* see exactly what is meant, but a quotation following a separation (by definition) is the start of a quoted field having " also be - by default - the escape character will make the parser see the second " as escape (unless disabled as you did) because the " is not (directly) followed by a separation character. That second " can be acceptable with allow_loose_quotes (being unescaped) but as the field *started* with a quotation, the field *has to* be terminated with quotation. If you want this to be parsed as [ "", "\"Bad\" Bank", "" ] there is no other option than to just use split, as this is not CSV anymore, but you'll be lost when data turns nasty as like ;"Bad" foo "the ; f" Bank; *) That leaves you with two options 1. Make sure the generator does not give you shit like that 2. Make a very customized parser using hooks and error-checking functions. I won't be helping you in there, but the docs have a very simplistic example of how to use callbacks $csv->callbacks (error => \&my_error_handler); -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.21 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

Download (untitled)
application/pgp-signature 490b

Message body not shown because it is not plain text.

Thu Jan 08 07:34:41 2015 bugs [...] supernover.de - Correspondence added

Subject:	Re: [rt.cpan.org #101372] Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?
Date:	Thu, 08 Jan 2015 13:34:09 +0100
To:	bug-Text-CSV_XS [...] rt.cpan.org
From:	" - bugs [...] supernover.de" <bugs [...] supernover.de>

Am 07.01.2015 um 20:07 schrieb h.m.brand@xs4all.nl via RT: Show quoted text

> If you want this to be parsed as [ "", "\"Bad\" Bank", "" ] there is no > other option than to just use split, as this is not CSV anymore, but > you'll be lost when data turns nasty as like > > ;"Bad" foo "the ; f" Bank;

Yes, I'm well aware of that... *sigh*. In fact I do use split right now in the error case and check the resulting number of fields to avoid fups like above (although not as elegantly as hooking a handler into the callback). I still thought Text::CSV_XS could be dumbed down to sth similar (I was hoping for somewhat less stupid) via switch, but I now see why that would be a bad idea, even if it was available. Show quoted text

> *) That leaves you with two options > > 1. Make sure the generator does not give you shit like that

Yes, that'd certainly be the way to go. My influence on that one is rather limited, alas. Show quoted text

> 2. Make a very customized parser using hooks and error-checking > functions. I won't be helping you in there, but the docs have a very > simplistic example of how to use callbacks > > $csv->callbacks (error => \&my_error_handler);

I'll try that. Thanks for your help and insight, best regards, jo

Thu Jan 08 07:44:03 2015 HMBRAND [...] cpan.org - Status changed from 'open' to 'rejected'

Bug #101372 for Text-CSV_XS: Should allow_loose_quotes fix error 2027 - EIQ - Quoted field not terminated?