Skip Menu |

This queue is for tickets about the XML-Twig CPAN distribution.

Report information
The Basics
Id: 67271
Status: rejected
Priority: 0/
Queue: XML-Twig

People
Owner: Nobody in particular
Requestors: Thorsten.Dombach [...] mabunta.de
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 3.38
Fixed in: (no value)



Subject: Having seg fault with script on multi thread perl systems
Hello, when I use XML-Twig-3.38 in the stripped down script to extract some information from the XML File, I get a segmentation fault with the return statement. I get the seg fault on the following systems: /usr/bin/perl -v This is perl, v5.8.8 built for i586-linux-thread-multi uname -a Linux sl90113 2.6.16.60-0.58.1-xenpae #1 SMP Wed Dec 2 12:27:56 UTC 2009 i686 i686 i386 GNU/Linux ==== perl -v This is perl, v5.10.0 built for x86_64-linux-thread-multi uname -a Linux sl01166 2.6.27.39-0.2-xen #1 SMP 2009-11-23 12:57:38 +0100 x86_64 x86_64 x86_64 GNU/Linux ===== Then I recompiled a new perl on system sl90113 without the following options: If this doesn't make any sense to you, just accept the default 'n'. Build a threading Perl? [n] If this doesn't make any sense to you, just accept the default 'n'. Build Perl for multiplicity? [n] I now have: /usr/local/bin/perl -v This is perl, v5.10.0 built for i686-linux-64int Now the programm works fine. The problem for reproducing the bug is, that the xml file need a size bigger than 25 MB. I can offer such kind of file but not for public availability. I have a smaller one added with dummy data for seeing the xml structure. Regards, Thorsten
Subject: ex_seg_fault.pl
#!/usr/local/bin/perl use Data::Dumper; use XML::Twig; # define the functions sub f_create_twig { # create the twig of group and group members my $group = $_[0]; my $cma = $_[1]; my $conf = ${$_[2]}; my $DEBUG = $_[3]; my %all_mem_twig; my $group_found = 0; my $twig = XML::Twig->new(pretty_print => 'indented'); $twig->parsefile("./eg_seg_fault.xml"); my $root = $twig->root; # first find the searched group and all it's members foreach my $no ($root->children('network_object')){ if (($no->first_child_text('type') eq "group") and ($no->first_child_text('Name') eq $group)) { print "group: " . $no->first_child_text('Name') . "\n" if $DEBUG; $group_twig = $no->sprint(0); foreach $mem ($no->first_child('members')->children('reference')) { print "mem: " . $mem->first_child_text('Name') . "\n" if $DEBUG; push(@all_members,$mem->first_child_text('Name')); $group_found = 1; } # end foreach } # end if last if $group_found; } # end foreach if (!$group_found) { # group was not found on this cma print "The group $group was not found on CMA $cma\n"; return 0; } print "all mem: @all_members\n" if $DEBUG; foreach my $no ($root->children('network_object')) { foreach my $mem_name (@all_members) { if ($no->first_child_text('Name') eq $mem_name) { $all_mem_twig{$mem_name} = $no->sprint(0); } # end if } # end foreach } # enf foreach return $group_twig, %all_mem_twig; } # end sub f_create_twig $DEBUG = 1; my @cmas; @cmas =("site", "third_customer"); # normally is here the config parameters from config file $conf = "config"; # get master xml my ($master_group_twig,%master_mem_twig) = &f_create_twig("g_xTD-test-lok","application",\$conf,$DEBUG); # create cma xmls # print "before cma loop: @cmas\n"; # foreach my $cma (@cmas) { # print "cma-loop:$groupname - cma: $cma\n" if $DEBUG; # # create from every cma one twig of the group and an array of twigs for all members of the group # my ($cma_group_twig,%cma_mem_twig) = &f_create_twig($groupname,$cma,\$conf,$DEBUG); #print "after return of f_create_twig\n"; # # group is not found in this cma # next if (!$cma_group_twig); # # compare the member names of the groups # # this print out when something is different in the amount or names of the members between master # &f_cmp_mem(\%master_mem_twig,\%cma_mem_twig,$cma,$groupname,$config{groups}->{$groupname}->{master},$DEBUG); # # now compare each member of the cma to the member of the master # #print "cmp_xml: $master_xml-$tmp_xml-$cma-$group-$DEBUG\n" if $DEBUG; # #&f_cmp_xml($master_group_twig,$cma_group_twig,$cma,$groupname,$DEBUG); # } print "\nEND\n" if $DEBUG; exit 0;
Subject: dev_network_objects.xml

Message body is not shown because it is too large.

Hi, I am very sorry, I never saw this ticket for some reason. So more for the sake of future readers: in your script you load the entire XML in memory. There is no magic there, if the file is too big you're going to use all of the system memory. If you want to process bigger files than can fit in memory, you need to use the tools that XML::Twig give you to limit the size that's held in memory at one time. Usually it means setting handlers, purging the twig after processing certain elements... __ mirod
Subject: AW: [rt.cpan.org #67271] Having seg fault with script on multi thread perl systems
Date: Fri, 23 Sep 2011 13:22:30 +0200
To: "bug-XML-Twig [...] rt.cpan.org" <bug-XML-Twig [...] rt.cpan.org>
From: Thorsten Dombach <Thorsten.Dombach [...] mabunta.de>
Hi, no problem. But sorry I cannot understand your answer. Because, when I use the 32 Bit not multithreaded Version with the SAME XML File it works. For my understanding, I would expect that the 64 Bit Version did not crash, because it can address more memory than the 32 Bit Version. So how to understand this? Regards, Thorsten Show quoted text
-----Ursprüngliche Nachricht----- Von: MIROD via RT [mailto:bug-XML-Twig@rt.cpan.org] Gesendet: Freitag, 23. September 2011 12:05 An: Thorsten Dombach Betreff: [rt.cpan.org #67271] Having seg fault with script on multi thread perl systems <URL: https://rt.cpan.org/Ticket/Display.html?id=67271 > Hi, I am very sorry, I never saw this ticket for some reason. So more for the sake of future readers: in your script you load the entire XML in memory. There is no magic there, if the file is too big you're going to use all of the system memory. If you want to process bigger files than can fit in memory, you need to use the tools that XML::Twig give you to limit the size that's held in memory at one time. Usually it means setting handlers, purging the twig after processing certain elements... __ mirod
On Fri Sep 23 07:22:43 2011, Thorsten.Dombach@mabunta.de wrote: Show quoted text
> Hi, > > no problem. But sorry I cannot understand your answer. Because, > when I use the 32 Bit not multithreaded Version > with the SAME XML > File it works. For my understanding, I would expect that the 64 Bit > Version did not crash, because > it can address more memory than the 32 > Bit Version. > So how to understand this?
My guess would be that the 64 bit version requires more space, so when you use the 32 bit Perl the entire XML still fits in memory, but when you switch to the 64 bit Perl then you go over the machine size. You could check by looking at top while running your code, to see how much memory it uses. __ mirod
On Fri Sep 23 10:01:17 2011, MIROD wrote: Show quoted text
> On Fri Sep 23 07:22:43 2011, Thorsten.Dombach@mabunta.de wrote:
> > Hi, > > > > no problem. But sorry I cannot understand your answer. Because, > > when I use the 32 Bit not multithreaded Version > > with the SAME XML > > File it works. For my understanding, I would expect that the 64 Bit > > Version did not crash, because > > it can address more memory than the 32 > > Bit Version. > > So how to understand this?
> > My guess would be that the 64 bit version requires more space, so when > you use the 32 bit Perl the entire XML still fits in memory, but when > you switch to the 64 bit Perl then you go over the machine size. > > You could check by looking at top while running your code, to see how > much memory it uses. > > __ > mirod
-- __ mirod