Skip Menu |

This queue is for tickets about the Text-CSV_XS CPAN distribution.

Report information
The Basics
Id: 86155
Status: resolved
Priority: 0/
Queue: Text-CSV_XS

People
Owner: Nobody in particular
Requestors: fitz.elliott [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 0.80
Fixed in: 1.01



Subject: getline_all() only returns one line on Mac formatted files
Hello, I'm not sure if the following is a bug or expected behavior, but when calling getline_all() on a file with \r-only eols, only one row is returned. Subsequent calls to getline() or getline_all() return an empty arrayref. Here is a one-liner that can be run in the dist directory to demonstrate: perl -e 'open my $fh, "<", "files/macosx.csv" or die "Oh no."; use Text::CSV_XS; my $csv=Text::CSV_XS->new({binary => 1}); use Data::Dumper; print Dumper($csv->getline_all($fh));' This happens in perls 5.10, 5.12, 5.14, 5.16, and 5.18. I've tested this back to tag 0.78, and it doesn't seem to have ever worked. I've attached a failing test case for 45_eol.t. I wasn't sure if I should add it there or in 70_rt.t. I'd be happy to reformat it if you would prefer it in 70_rt.t. Alternatively, if this is not a bug I can submit a doc patch for getline_all() explaining this. Thank you for your effort in maintaining this module. Have a great weekend! Cheers, Fitz
Subject: 0001-failing-test-case-for-getline_all-on-os-x-linebreaks.patch
From b1b08a835faf9620b36a0f824183d05181e581d1 Mon Sep 17 00:00:00 2001 From: Fitz Elliott <fitz.elliott@gmail.com> Date: Fri, 14 Jun 2013 15:23:05 -0400 Subject: [PATCH] failing test case for getline_all() on os x linebreaks diff --git a/t/45_eol.t b/t/45_eol.t index bc843c1..525355b 100644 --- a/t/45_eol.t +++ b/t/45_eol.t @@ -3,7 +3,7 @@ use strict; use warnings; -use Test::More tests => 1065; +use Test::More tests => 1068; BEGIN { require_ok "Text::CSV_XS"; @@ -254,4 +254,11 @@ $/ = $def_rs; close $fh; } +{ open my $fh, "<", "files/macosx.csv" or die "Ouch $!"; + ok (1, "MacOSX exported file - getline_all"); + ok (my $csv = Text::CSV_XS->new ({ auto_diag => 1, binary => 1 }), "new csv"); + diag (); + is (@{ $csv->getline_all($fh) }, 16, "getline_all gets all lines for osx"); +} + 1; -- 1.8.3.1
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Fri, 14 Jun 2013 22:36:13 +0200
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
On Fri, 14 Jun 2013 15:24:00 -0400, "Fitz Elliott via RT" <bug-Text-CSV_XS@rt.cpan.org> wrote: Show quoted text
> Hello, > > I'm not sure if the following is a bug or expected behavior, but when > calling getline_all() on a file with \r-only eols, only one row is > returned. Subsequent calls to getline () or getline_all () return an > empty arrayref. > > Here is a one-liner that can be run in the dist directory to demonstrate:
I can reprocude $ perl -MDP -MText::CSV_XS -E'open my$fh,"<","files/macosx.csv";my$csv=Text::CSV_XS->new({binary=>1});DDumper($csv->getline_all($fh));' Show quoted text
> This happens in perls 5.10, 5.12, 5.14, 5.16, and 5.18. I've tested this back to tag 0.78, and it doesn't seem to have ever worked.
But it works with _PP :/ $ perl -MDP -MText::CSV_PP -E'open my$fh,"<","files/macosx.csv";my$csv=Text::CSV_PP->new({binary=>1});DDumper($csv->getline_all($fh));' Show quoted text
> I've attached a failing test case for 45_eol.t. I wasn't sure if I should > add it there or in 70_rt.t. I'd be happy to reformat it if you would > prefer it in 70_rt.t. Alternatively, if this is not a bug I can > submit a doc patch for getline_all () explaining this.
I will choose 75_hashref.t Show quoted text
> Thank you for your effort in maintaining this module. Have a great weekend!
I now have something now to dig into :) Show quoted text
> Cheers, > Fitz
-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Sat, 15 Jun 2013 15:42:05 -0400
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "fitz.elliott" <fitz.elliott [...] gmail.com>
Show quoted text
>> I've attached a failing test case for 45_eol.t. I wasn't sure if I should >> add it there or in 70_rt.t. I'd be happy to reformat it if you would >> prefer it in 70_rt.t. Alternatively, if this is not a bug I can >> submit a doc patch for getline_all () explaining this.
> > I will choose 75_hashref.t
Just checking, did you mean 77_getall.t? And if it's 75_hashref.t, would you like me to adapt the test to be more in line with the others, or is it okay just to copy and paste it? Cheers, Fitz
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Sat, 15 Jun 2013 16:34:06 -0400
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "fitz.elliott" <fitz.elliott [...] gmail.com>
Show quoted text
>> I've attached a failing test case for 45_eol.t. I wasn't sure if I should >> add it there or in 70_rt.t. I'd be happy to reformat it if you would >> prefer it in 70_rt.t. Alternatively, if this is not a bug I can >> submit a doc patch for getline_all () explaining this.
> > I will choose 75_hashref.t
Hello again, I went ahead and adapted the test for 77_getall.t and discovered something interesting in the process: only the first getline_all on a particular file fails. With the attached patch, the first $sub->() in do_tests() always fails, and the rest always succeed. It's not a matter of the arguments; if you switch the order of the $sub->(...) calls, it's always the first that fails. That's... weird. ...and it just got weirder. I just tried adding tests for \r\n line endings, and those all failed. When I moved them above the \r tests, it started working again. \r seems to mess up everything that comes after. The attached patch only adds the mac line endings tests. I'm sending it so you can see the behavior I described, and so you don't think I'm a lunatic. :) I'll send a new one with the windows line endings after I run some errands. Cheers, Fitz

Message body is not shown because sender requested not to inline it.

Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Sun, 16 Jun 2013 10:02:48 +0200
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
On Sat, 15 Jun 2013 15:42:17 -0400, "fitz.elliott@gmail.com via RT" <bug-Text-CSV_XS@rt.cpan.org> wrote: Show quoted text
> > I will choose 75_hashref.t
> > Just checking, did you mean 77_getall.t? And if it's 75_hashref.t, > would you like me to adapt the test to be more in line with the > others, or is it okay just to copy and paste it?
Well, thinking about it after your mails, what I end up doing in other cases is to pick the file where I think it would fit best when reading the initial ticket. If it is small and simple, that usually is 70_rt If it is something weird or complex, I'd choose the t-file that meets the intuition best, and when I solved/fixed the problem, I move the tests to the file where the problem was solved. It is rather new to me that someone reporting a bug also digs deeper him/herself, I think I like it. If you are serious about digging, feel free to choose whatever t-file at first and I can move the test somewhere else later. I have not much time to dig myself, so anything you can come up with might help. If you want *full* credits if you actually fix this bug in a way I like it, be very very sure to follow my style. Show quoted text
> Cheers, > Fitz
-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Sun, 16 Jun 2013 18:39:12 +0200
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
On Sat, 15 Jun 2013 16:34:19 -0400, "fitz.elliott@gmail.com via RT" <bug-Text-CSV_XS@rt.cpan.org> wrote: Show quoted text
> Hello again, > > I went ahead
The test I will be using for this ticket will be in this form: -->8--- use Test::More; use Data::Peek; use Text::CSV_XS; my $data = join "\r" => qq{1,"aap",3}, qq{2,"noot",5}, qq{3,"mies",7}; ok (my $csv = Text::CSV_XS->new ({ binary => 1 }), "new csv"); ok (open (my $fh, "<", \$data), "open < data"); ok (my $r = $csv->getline_all ($fh), "getline_all"); is (ref $r, "ARRAY", "ref is ARRAY"); is (scalar @$r, 3, "Three lines"); is ($r->[0][0], 1, "(0, 0)"); is ($r->[1][1], "noot", "(1, 1)"); is ($r->[2][2], 7, "(2, 2)"); DDumper $r; done_testing (); -->8000 -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Sun, 16 Jun 2013 20:28:06 +0200
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
This is the fix: --8<--- diff --git a/CSV_XS.pm b/CSV_XS.pm index c19a86c..eb94a23 100644 --- a/CSV_XS.pm +++ b/CSV_XS.pm @@ -27,7 +27,7 @@ use DynaLoader (); use Carp; use vars qw( $VERSION @ISA ); -$VERSION = "1.00"; +$VERSION = "1.01"; @ISA = qw( DynaLoader ); bootstrap Text::CSV_XS $VERSION; diff --git a/CSV_XS.xs b/CSV_XS.xs index 1698f1a..a7f3d30 100644 --- a/CSV_XS.xs +++ b/CSV_XS.xs @@ -1549,6 +1549,9 @@ static SV *cx_xsParse_all (pTHX_ SV *self, HV *hv, SV *io, SV *off, SV *len) length = SvIV (len); while (c_xsParse (csv, hv, row, NULL, io, 1)) { + + SetupCsv (&csv, hv, self); + if (skip > 0) { skip--; av_empty (row); /* re-use */ -->8--- Ill do some more tests before I release -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Subject: Re: [rt.cpan.org #86155] getline_all() only returns one line on Mac formatted files
Date: Mon, 17 Jun 2013 10:31:59 -0400
To: bug-Text-CSV_XS [...] rt.cpan.org
From: "fitz.elliott" <fitz.elliott [...] gmail.com>
On Jun 16, 2013, at 2:28 PM, h.m.brand@xs4all.nl via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=86155 > > > This is the fix: > > Ill do some more tests before I release
Wow, you work fast! I'm sorry for not responding sooner. I've installed 1.01, and everything is working perfectly, and all the tests are passing. Thanks for your effort and have a great day! Cheers, Fitz