Bug #17500 for XML-Twig: *** glibc detected *** double free or corruption (!prev): 0x08d2b4d0 ***

Mon Feb 06 02:06:31 2006 Guest - Ticket created

Subject:

*** glibc detected *** double free or corruption (!prev): 0x08d2b4d0 ***

Hello, It seems that XML::Twig causes some sort of segfault for the attached file: main::update(./verify.pl:86): eval { DB<2> n main::update(./verify.pl:87): my($fh,$parser); DB<2> n main::update(./verify.pl:88): open($fh, "-|", "bzcat", "$fullfile.xml.bz2") main::update(./verify.pl:89): or die "Can't fork bzcat: $!\n"; DB<2> n main::update(./verify.pl:90): my $data = {}; main::update(./verify.pl:91): $parser = new XML::Twig main::update(./verify.pl:92): ( twig_roots => { DB<2> n main::update(./verify.pl:108): }); DB<2> n main::update(./verify.pl:109): $parser->parse($fh); DB<2> n *** glibc detected *** double free or corruption (!prev): 0x08d2b4d0 ***

Subject:

ct-usb-audio-test.xml.bz2

Download ct-usb-audio-test.xml.bz2
application/x-bzip 11k

Message body not shown because it is not plain text.

Mon Feb 06 02:08:36 2006 Guest - Correspondence added

Here's the full invocation code: $parser = new XML::Twig ( twig_roots => { "title" => sub { $oldval->{title} = $_[1]->text(); shift()->purge(); }, "expires" => sub { $oldval->{expiry} = $_[1]->text(); shift()->purge(); }, "timestamp" => sub { $oldval->{timestamp} = $_[1]->text(); shift()->purge(); }, "keywords" => sub { $oldval->{keywords} = $_[1]->text(); shift()->purge(); }, "hardcopy" => sub { $oldval->{hardcopy} = $_[1]->text(); shift()->purge(); }, "tag" => sub { push @{$oldval->{tags}}, $_[1]->text(); shift()->purge(); }, "file" => sub { $oldval->{indexfile}->{$_[1]->att("name")} = $_[1]->att("index") ? 'y' : 'n'; shift()->purge(); }, }); After the "***glib ..." message, the complete perl process (including the debugger) is aborted.

Mon Feb 06 02:08:37 2006 The RT System itself - Status changed from 'new' to 'open'

Mon Feb 06 06:12:20 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Mon, 06 Feb 2006 12:10:45 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

You are not giving me enough information here. The XML you are working with is missing. Could you send me a complete test case that includes sample data? Something along the lines of the attached file, that reproduces the bug would be fine. The output of perl -v and of the t/zz_config.t file would also be useful. Thanks -- Michel Rodriguez Perl & XML xmltwig.com

#!/usr/bin/perl use strict; use warnings; use XML::Twig; use YAML; my $xml="rt_17500.xml"; open( my $xml_fh, '>', $xml) or die "can't create $xml: $!"; while(<DATA>) { print {$xml_fh} $_; } close $xml_fh; system "/usr/bin/bzip2 -f $xml"; my $oldval; eval { my($fh,$parser); open($fh, "-|", "bzcat", "$xml.bz2"), or die "Can't fork bzcat: $!\n"; $oldval = {}; $parser = new XML::Twig ( twig_roots => { "title" => sub { $oldval->{title} = $_[1]->text(); shift()->purge(); }, "expires" => sub { $oldval->{expiry} = $_[1]->text(); shift()->purge(); }, "timestamp" => sub { $oldval->{timestamp} = $_[1]->text(); shift()->purge(); }, "keywords" => sub { $oldval->{keywords} = $_[1]->text(); shift()->purge(); }, "hardcopy" => sub { $oldval->{hardcopy} = $_[1]->text(); shift()->purge(); }, "tag" => sub { push @{$oldval->{tags}}, $_[1]->text(); shift()->purge(); }, "file" => sub { $oldval->{indexfile}->{$_[1]->att("name")} = $_[1]->att("index") ? 'y' : 'n'; shift()->purge(); }, } ); $parser->parse($fh); }; if( $@) { die "error parsing: $@"; } print Dump $oldval; __DATA__ <doc> <item> <title>title</title> <expires>2006-12-31</expires> <timestamp>2005-04-01</timestamp> <hardcopy>yes</hardcopy> <tag>big</tag> <tag>bigger</tag> <file name="file1" index="1"/> <keywords>k1 k2</keywords> </item> <item> <title>title 2</title> <expires>2007-12-31</expires> <timestamp>2004-04-01</timestamp> <hardcopy>no</hardcopy> <tag>small</tag> <tag>smaller</tag> <file name="file2" index="0"/> <keywords>k1 k2</keywords> </item> </doc>

Mon Feb 06 10:58:36 2006 Nikolaus [...] rath.org - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Mon, 06 Feb 2006 16:57:12 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Nikolaus Rath <Nikolaus [...] rath.org>

"mirod@xmltwig.com via RT" <bug-XML-Twig@rt.cpan.org> writes: Show quoted text

> You are not giving me enough information here. The XML you are working > with is missing. > > Could you send me a complete test case that includes sample data? > Something along the lines of the attached file, that reproduces the bug > would be fine.

I thought I attached the data to the bug report... Anyway, I've attached a testcase.

Download ct-usb-audio-test.xml.bz2
application/octet-stream 11k

Message body not shown because it is not plain text.

#!/usr/bin/perl use strict; use warnings; use XML::Twig; use YAML; my $xml="ct-usb-audio-test.xml.bz2"; my $oldval; eval { my($fh,$parser); open($fh, "-|", "bzcat", "$xml"), or die "Can't fork bzcat: $!\n"; $oldval = {}; $parser = new XML::Twig ( twig_roots => { "title" => sub { $oldval->{title} = $_[1]->text(); shift()->purge(); }, "expires" => sub { $oldval->{expiry} = $_[1]->text(); shift()->purge(); }, "timestamp" => sub { $oldval->{timestamp} = $_[1]->text(); shift()->purge(); }, "keywords" => sub { $oldval->{keywords} = $_[1]->text(); shift()->purge(); }, "hardcopy" => sub { $oldval->{hardcopy} = $_[1]->text(); shift()->purge(); }, "tag" => sub { push @{$oldval->{tags}}, $_[1]->text(); shift()->purge(); }, "file" => sub { $oldval->{indexfile}->{$_[1]->att("name")} = $_[1]->att("index") ? 'y' : 'n'; shift()->purge(); }, } ); $parser->parse($fh); }; if( $@) { die "error parsing: $@"; } print Dump $oldval;

Show quoted text

> The output of perl -v and of the t/zz_config.t file would also be useful.

[134] nokile:~/Work/perlbug$ perl -v This is perl, v5.8.7 built for i486-linux-gnu-thread-multi (with 1 registered patch, see perl -V for more detail) Copyright 1987-2005, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using `man perl' or `perldoc perl'. If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page. [0] nokile:~/Work/perlbug$ locate zz_config.t [1] nokile:~/Work/perlbug$ I've got no zz_config.t. Where does it belong to? Perl or your module? I'm using Ubuntu packages. --Nikolaus -- Der Vorteil der Klugheit liegt darin, dass man sich dumm stellen kann. Das Gegenteil ist schon schwieriger. (Kurt Tucholsky)

Mon Feb 06 11:23:03 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Mon, 06 Feb 2006 17:21:43 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

Nikolaus Rath via RT wrote: Show quoted text

> [134] nokile:~/Work/perlbug$ perl -v > > This is perl, v5.8.7 built for i486-linux-gnu-thread-multi > (with 1 registered patch, see perl -V for more detail) > [...] > I've got no zz_config.t. Where does it belong to? Perl or your module? > I'm using Ubuntu packages.

zz_config.t is in the t directory for the module. I ran your test here and did not get an arror (perl 5.8.7 on linux too, unthreaded though). Could you: - try with the latest version of XML::Twig, which can be found at http://xmltwig.com/xmltwig/ ('make test' should give you configuration information) - try without the fork to bzcat Thanks In the meantime I'll compile a threaded version of Perl and see if I can reproduce the problem. -- Michel Rodriguez Perl & XML xmltwig.com

Mon Feb 06 13:07:52 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Mon, 06 Feb 2006 19:07:03 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

mirod@xmltwig.com via RT wrote: Show quoted text

> In the meantime I'll compile a threaded version of Perl and see if I can > reproduce the problem.

The threaded version (v5.8.7 built for i686-linux-thread-multi) still doesn't show the problem. At this point, if the latest version does not solve your problem, I would try to rewrite the code by having the twig_roots on a single, englobing, element, and avoiding the many calls to purge. -- Michel Rodriguez Perl & XML xmltwig.com

Mon Feb 06 14:58:57 2006 Nikolaus [...] rath.org - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Mon, 06 Feb 2006 20:57:50 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Nikolaus Rath <Nikolaus [...] rath.org>

"mirod@xmltwig.com via RT" <bug-XML-Twig@rt.cpan.org> writes: Show quoted text

>> [134] nokile:~/Work/perlbug$ perl -v >> >> This is perl, v5.8.7 built for i486-linux-gnu-thread-multi >> (with 1 registered patch, see perl -V for more detail) >> [...] >> I've got no zz_config.t. Where does it belong to? Perl or your module? >> I'm using Ubuntu packages.

> > zz_config.t is in the t directory for the module. > > I ran your test here and did not get an arror (perl 5.8.7 on linux too, > unthreaded though). > > Could you: > > - try with the latest version of XML::Twig, which can be found at > http://xmltwig.com/xmltwig/ ('make test' should give you > configuration information)

I tried to reinstall, but: Failed Test Stat Wstat Total Fail Failed List of Failed -------------------------------------------------------------------------------------------------------- t/test_additional.t 647 2 0.31% 346 348 t/test_bugs_3.18.t 157 13 8.28% 11-23 t/test_bugs_3.19.t 26 6 23.08% 5-6 20-23 Failed 3/82 test scripts, 96.34% okay. 21/3795 subtests failed, 99.45% okay. make: *** [test_dynamic] Error 255 Before that, I got a lot of UTF-8 encoding warnings, which allowed me to narrow the problem: Both 'make test' and my testcase work if I unset PERL_UNICODE (which is set to SDAL here, because it's a pure UTF-8 system). I hope this helps to pin down the problem. Regards, --Nikolaus -- Übermorgen bekommen sie mein endgültiges Vielleicht!

Tue Feb 07 02:04:48 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 08:03:38 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

Nikolaus Rath via RT wrote: Show quoted text

> I tried to reinstall, but: > > Failed Test Stat Wstat Total Fail Failed List of Failed > -------------------------------------------------------------------------------------------------------- > t/test_additional.t 647 2 0.31% 346 348 > t/test_bugs_3.18.t 157 13 8.28% 11-23 > t/test_bugs_3.19.t 26 6 23.08% 5-6 20-23 > Failed 3/82 test scripts, 96.34% okay. 21/3795 subtests failed, 99.45% okay. > make: *** [test_dynamic] Error 255 > > > Before that, I got a lot of UTF-8 encoding warnings, which allowed me > to narrow the problem: Both 'make test' and my testcase work if I > unset PERL_UNICODE (which is set to SDAL here, because it's a pure > UTF-8 system). > > I hope this helps to pin down the problem.

It does, nice job, With PERL_UNICODE set to SDAL, I was able to reproduce the bug, and indeed to get the same error message you got. It goes away if, instead of parsing a pipe from bzcat, I parse a regular XML file. It also works fine if you do the bzcat and pipe it to the code: PERL_UNICODE=SDAL bzcat -c rt_17500.xml.bz2 | perl bug.pl and parse \*STDIN in the code. Stepping through the code with debugger it looks like actually the problem occurs before even reaching any XML::Twig specific code. It looks like it's a problem with IO::File ignoring PERL_UNICODE (or -C). XML::Parser, on which XML::Twig is based, uses IO::File. This is confirmed by this thread: http://www.mhonarc.org/archive/html/perl-unicode/2004-04/msg00004.html As the problem is due to the 'D' option, telling perl to use the perlIO layer, the easiest workaround is to use the -CSAL option on the shebang line (or on the command line). As for the test failures: you end telling the system to read as unicode, files that are in iso-8859-1. I will have to take this case into account in the test (baring in mind that XML::Twig should still work with very old versions of perl!). So I will fix the tests and add a note about the PERL_UNICODE/-CD option in the docs (and in the FAQ). Does this solve the problem? Thanks -- Michel Rodriguez Perl & XML xmltwig.com

Tue Feb 07 04:10:01 2006 Nikolaus [...] rath.org - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 10:01:04 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Nikolaus Rath <Nikolaus [...] rath.org>

"mirod@xmltwig.com via RT" <bug-XML-Twig@rt.cpan.org> writes: Show quoted text

> Stepping through the code with debugger it looks like actually the > problem occurs before even reaching any XML::Twig specific code. It > looks like it's a problem with IO::File ignoring PERL_UNICODE (or -C). > XML::Parser, on which XML::Twig is based, uses IO::File. > > This is confirmed by this thread: > http://www.mhonarc.org/archive/html/perl-unicode/2004-04/msg00004.html

You're absolutely right. Actually, I already stumbled over this bug some time ago: http://rt.perl.org/rt3/Ticket/Display.html?id=38181 But in this case I somehow missed to see the connection. Thanks for your help. Regards, --Nikolaus -- *Catholism*: If shit happend I deserve it. *Voodoo*: Shit will hapen to you. *Ateism*: Shit happens, but I don't care anyway. *Judaism*: Why does shit always happen to us? *Mormonism*: If shit happens we still got enough women to clean it up. *Moslemism*: If shit happens take a hostage.

Tue Feb 07 05:25:14 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 11:24:08 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

Nikolaus Rath via RT wrote: Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=17500 > > > "mirod@xmltwig.com via RT" <bug-XML-Twig@rt.cpan.org> writes: >

>>Stepping through the code with debugger it looks like actually the >>problem occurs before even reaching any XML::Twig specific code. It >>looks like it's a problem with IO::File ignoring PERL_UNICODE (or -C). >>XML::Parser, on which XML::Twig is based, uses IO::File. >> >>This is confirmed by this thread: >>http://www.mhonarc.org/archive/html/perl-unicode/2004-04/msg00004.html

> > > You're absolutely right. Actually, I already stumbled over this bug > some time ago: > > http://rt.perl.org/rt3/Ticket/Display.html?id=38181 > > But in this case I somehow missed to see the connection. Thanks for > your help.

I have a version of XML::Twig that traps this error and issues a nicer error message. Now the problem remains that tests fail. More important, whether tests pass or fail, PERL_UNICODE and -C are user-level options, and once set, there doesn't seem to be anything I can do at XML::Twig level to reset the unicode flags to a value that would work. At runtime, I think all I can do is raise an error if {^UNICODE} tells me that the UTF8 perIO layer is used and the XML encoding is not utf8. -- Michel Rodriguez Perl & XML xmltwig.com

Tue Feb 07 06:28:54 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 12:27:39 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

Hi, OK, the development version should now trap this error, and pass the tests with PERL_UNICODE set to SDAL (or to any other value hopefully!). Let me know if it actually does ;--) Thanks -- Michel Rodriguez Perl & XML xmltwig.com

Tue Feb 07 07:10:16 2006 Nikolaus [...] rath.org - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 13:09:11 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Nikolaus Rath <Nikolaus [...] rath.org>

"mirod@xmltwig.com via RT" <bug-XML-Twig@rt.cpan.org> writes: Show quoted text

>>>Stepping through the code with debugger it looks like actually the >>>problem occurs before even reaching any XML::Twig specific code. It >>>looks like it's a problem with IO::File ignoring PERL_UNICODE (or -C). >>>XML::Parser, on which XML::Twig is based, uses IO::File. >>> >>>This is confirmed by this thread: >>>http://www.mhonarc.org/archive/html/perl-unicode/2004-04/msg00004.html

>> >> >> You're absolutely right. Actually, I already stumbled over this bug >> some time ago: >> >> http://rt.perl.org/rt3/Ticket/Display.html?id=38181 >> >> But in this case I somehow missed to see the connection. Thanks for >> your help.

> > I have a version of XML::Twig that traps this error and issues a > nicer error message. > > At runtime, I think all I can do is raise an error if {^UNICODE} > tells me that the UTF8 perIO layer is used and the XML encoding is > not utf8.

You need some quite sophisticated tests to do that properly, because: [0] nokile:~$ locale LANG=en_US.UTF-8 LC_CTYPE=de_DE.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TIME=de_DE.UTF-8 LC_COLLATE=de_DE.UTF-8 LC_MONETARY=de_DE.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=de_DE.UTF-8 LC_NAME=de_DE.UTF-8 LC_ADDRESS=de_DE.UTF-8 LC_TELEPHONE=de_DE.UTF-8 LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_ALL= [0] nokile:~$ perl -e 'print "${^UNICODE}\n";' 127 and [0] nokile:~$ unset LC_NUMERIC LC_COLLATE LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION [0] nokile:~$ locale LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= [0] nokile:~$ perl -e 'print "${^UNICODE}\n";' 127 so you have the check the locale yourself. Honestly, if I were you I wouldn't try to implement any workarounds for this perl bug. It's hard to do properly and the problem is already fixed in newer versions. Show quoted text

> Now the problem remains that tests fail.

IMO this isn't a problem at all. I'd just make the output nicer by checking $^UNICODE and the perl version and aborting the test with a warning if there's a problematic combination. The user will stumble over other problems sooner or later anyway, so you're doing him a favor by telling him that (and what) is wrong on his system. Regards, --Nikolaus -- Man gewöhnt sich an allem - sogar am Dativ.

Tue Feb 07 07:34:11 2006 mirod [...] xmltwig.com - Correspondence added

Subject:	Re: [rt.cpan.org #17500] * glibc detected * double free or corruption (!prev): 0x08d2b4d0 ***
Date:	Tue, 07 Feb 2006 13:32:37 +0100
To:	bug-XML-Twig [...] rt.cpan.org
From:	Michel Rodriguez <mirod [...] xmltwig.com>

Nikolaus Rath via RT wrote: Show quoted text

> You need some quite sophisticated tests to do that properly, because: > [...] > so you have the check the locale yourself. Honestly, if I were you I > wouldn't try to implement any workarounds for this perl bug. It's hard > to do properly and the problem is already fixed in newer versions.

Ok, it makes sense, and I don't know enough about the subject to get it right anyway. Show quoted text

>

>>Now the problem remains that tests fail.

> > IMO this isn't a problem at all. I'd just make the output nicer by > checking $^UNICODE and the perl version and aborting the test with a > warning if there's a problematic combination. The user will stumble > over other problems sooner or later anyway, so you're doing him a > favor by telling him that (and what) is wrong on his system.

That's what I did, I just skip the tests. In any case the problems are triggered by a runtime configuration for the user. Which cannot be tested at install time. I'll stop here then. Thanks for the help -- Michel Rodriguez Perl & XML xmltwig.com

Mon May 15 07:15:20 2006 MIROD [...] cpan.org - Status changed from 'open' to 'resolved'

Mon May 15 07:15:21 2006 MIROD [...] cpan.org - Fixed in 3.24 added