Skip Menu |

This queue is for tickets about the Net-SFTP CPAN distribution.

Report information
The Basics
Id: 7065
Status: resolved
Priority: 0/
Queue: Net-SFTP

People
Owner: Nobody in particular
Requestors: doug.hendricks [...] tnzi.com
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 0.08
Fixed in: (no value)



Subject: PARI stack overflow in long running perl scripts using Net::SFTP
bug in Net-SFTP-0.08 uname -a: OSF1 tnzieisp.***.com V5.1 2650 alpha which is Tru64 V5.1B on HP alpha perl -V is below. I have a (complicated) daemon script that creates and undefs Net::SFTP objects, so that each object lasts only a minute or two, but the script itself is intended to run for weeks. After running for less than an hour, I get the message: PARI: *** the PARI stack overflows ! current stack size: 4.0 Mbytes [hint] you can increase GP stack with allocatemem() and all useful work stops. I know that I am not the only person having this problem. I have found (using Google) other victims on other hardware platforms. Nobody has shared an answer. I and one other victim (Jussi Mononen) have tried fiddling with allocatemem() with no avail. Jussi Mononen has posted this bug in comp.lang.perl.modules. Jussi is using Solaris. I have tried patching GP/Pari with a patch from Math::Pari , no effect. Somebody named Wiggins D'Anconia has posted about the same problem in Net::SFTP in perl.beginners under the subject line "Re: Net::Scp" here is a test script that fails in a few minutes, provided that directory 'data' on the remote host has hundreds of files in it: #!/usr/bin/perl -w use strict; use Net::SFTP; while (1) { my $ftp = Net::SFTP->new('*your*remote*host*here*',user=>'ftp'); my (@files) = (); my $subref = sub { push @files, $_[0]->{filename} }; ### $ftp->ls('.', $subref); # a small directory doesn't fail (quick) $ftp->ls('data', $subref); # big directory, lots of files. ### $ftp->get('afile','afile'); # not even necessary to get! print @files; sleep 5; } ###### end of script perl -V: Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration: Platform: osname=dec_osf, osvers=5.1, archname=alpha-dec_osf uname='osf1 zen.xko.dec.com t5.1 2617 alpha ' config_args='-des -Dlibpth=/usr/shlib /usr/lib /usr/ccs/lib -Dusemymalloc=no -Dprefix=/usr/lib/perl-5.8.0 -Dman1dir=/usr/share/doclib/annex/man/man1 -Dman3d ir=/usr/share/doclib/annex/man/man3 -Dccflags=-fprm d -ieee -Doptimize=-O4 -Dhos tcat=cat /etc/hosts -Dmyhostname=localhost -Umydomain= -Dcf_by=perlhelp -Dcf_ema il=root@localhost' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=unde f useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fprm d -ieee -std -fprm d -ieee -DLANGUAGE_C', optimize='-O4', cppflags='-fprm d -ieee -std -fprm d -ieee -DLANGUAGE_C' ccversion='V6.5-011', gccversion='', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='ld', ldflags ='' libpth=/usr/shlib /usr/lib /usr/ccs/lib libs=-ldbm -ldb -lm -lutil perllibs=-lm -lutil libc=/usr/shlib/libc.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-rpath,/usr/ lib/perl-5.8.0/lib/5.8.0/alpha-dec_osf/CORE' cccdlflags=' ', lddlflags='-shared -expect_unresolved "*" -O4 -msym -std -s' Characteristics of this binary (from libperl): Compile-time options: USE_64_BIT_INT USE_64_BIT_ALL USE_LARGE_FILES Built under dec_osf Compiled at Aug 26 2002 18:28:54 @INC: /usr/lib/perl-5.8.0/lib/5.8.0/alpha-dec_osf /usr/lib/perl-5.8.0/lib/5.8.0 /usr/lib/perl-5.8.0/lib/site_perl/5.8.0/alpha-dec_osf /usr/lib/perl-5.8.0/lib/site_perl/5.8.0 /usr/lib/perl-5.8.0/lib/site_perl . ======
[guest - Fri Jul 23 00:27:58 2004]: the ls() method appears to be the big problem below. Getting or putting files doesn't seem to trigger the bug, but ls() will. The larger the directory, the faster the bug appears. Doug Show quoted text
> > bug in Net-SFTP-0.08 > uname -a: OSF1 tnzieisp.***.com V5.1 2650 alpha > which is Tru64 V5.1B on HP alpha > perl -V is below. > > I have a (complicated) daemon script that creates and undefs Net::SFTP > objects, so that each object lasts only a minute or two, but the > script itself is intended to run for weeks. > After running for less than an hour, I get the message: > PARI: *** the PARI stack overflows ! > current stack size: 4.0 Mbytes > [hint] you can increase GP stack with allocatemem() > > and all useful work stops. > I know that I am not the only person having this problem. I have > found (using Google) other victims on other hardware platforms. > Nobody has shared an answer. > I and one other victim (Jussi Mononen) have tried fiddling with > allocatemem() with no avail. > Jussi Mononen has posted this bug in comp.lang.perl.modules. Jussi is > using Solaris. > I have tried patching GP/Pari with a patch from Math::Pari , no > effect. > Somebody named Wiggins D'Anconia has posted about the same problem in > Net::SFTP in perl.beginners under the subject line "Re: Net::Scp" > > here is a test script that fails in a few minutes, provided that > directory 'data' on the remote host has hundreds of files in it: > #!/usr/bin/perl -w > use strict; > use Net::SFTP; > while (1) { > my $ftp = Net::SFTP->new('*your*remote*host*here*',user=>'ftp'); > my (@files) = (); > my $subref = sub { push @files, $_[0]->{filename} }; > ### $ftp->ls('.', $subref); # a small directory doesn't fail > (quick) > $ftp->ls('data', $subref); # big directory, lots of files. > ### $ftp->get('afile','afile'); # not even necessary to get! > print @files; > sleep 5; > } > > ###### end of script > > perl -V: > Summary of my perl5 (revision 5.0 version 8 subversion 0) > configuration: > Platform: > osname=dec_osf, osvers=5.1, archname=alpha-dec_osf > uname='osf1 zen.xko.dec.com t5.1 2617 alpha ' > config_args='-des -Dlibpth=/usr/shlib /usr/lib /usr/ccs/lib > -Dusemymalloc=no > -Dprefix=/usr/lib/perl-5.8.0 > -Dman1dir=/usr/share/doclib/annex/man/man1 -Dman3d > ir=/usr/share/doclib/annex/man/man3 -Dccflags=-fprm d -ieee > -Doptimize=-O4 -Dhos > tcat=cat /etc/hosts -Dmyhostname=localhost -Umydomain= > -Dcf_by=perlhelp -Dcf_ema > il=root@localhost' > hint=recommended, useposix=true, d_sigaction=define > usethreads=undef use5005threads=undef useithreads=undef > usemultiplicity=unde > f > useperlio=define d_sfio=undef uselargefiles=define usesocks=undef > use64bitint=define use64bitall=define uselongdouble=undef > usemymalloc=n, bincompat5005=undef > Compiler: > cc='cc', ccflags ='-fprm d -ieee -std -fprm d -ieee -DLANGUAGE_C', > optimize='-O4', > cppflags='-fprm d -ieee -std -fprm d -ieee -DLANGUAGE_C' > ccversion='V6.5-011', gccversion='', gccosandvers='' > intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 > d_longlong=define, longlongsize=8, d_longdbl=define, > longdblsize=16 > ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', > lseeksize=8 > alignbytes=8, prototype=define > Linker and Libraries: > ld='ld', ldflags ='' > libpth=/usr/shlib /usr/lib /usr/ccs/lib > libs=-ldbm -ldb -lm -lutil > perllibs=-lm -lutil > libc=/usr/shlib/libc.so, so=so, useshrplib=true, > libperl=libperl.so > gnulibc_version='' > Dynamic Linking: > dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' > -Wl,-rpath,/usr/ > lib/perl-5.8.0/lib/5.8.0/alpha-dec_osf/CORE' > cccdlflags=' ', lddlflags='-shared -expect_unresolved "*" -O4 > -msym -std -s' > > > Characteristics of this binary (from libperl): > Compile-time options: USE_64_BIT_INT USE_64_BIT_ALL USE_LARGE_FILES > Built under dec_osf > Compiled at Aug 26 2002 18:28:54 > @INC: > /usr/lib/perl-5.8.0/lib/5.8.0/alpha-dec_osf > /usr/lib/perl-5.8.0/lib/5.8.0 > /usr/lib/perl-5.8.0/lib/site_perl/5.8.0/alpha-dec_osf > /usr/lib/perl-5.8.0/lib/site_perl/5.8.0 > /usr/lib/perl-5.8.0/lib/site_perl > . > > ======
[guest - Mon Jul 26 22:12:30 2004]: I'm also having problems after calling ls. Because there is not mget, I'm using an ls to grab a list of files, then using get to download them. After a couple of minutes, the script dies. I can also get the script to die immediately by pushing the RETURN key.
Show quoted text
> [guest - Mon Jul 26 22:12:30 2004]: > > I'm also having problems after calling ls. > > Because there is not mget, I'm using an ls to grab a list of files, > then > using get to download them. > After a couple of minutes, the script dies. > > I can also get the script to die immediately by pushing the RETURN > key.
My workaround was to create both a Net::SFTP and a Net::SCP object. The Net::SFTP was wrapped in an eval block that populated an array with filenames. Net::SCP then did the work.
This was fixed in version 1.26. The problem was that the Math::Pari big integer conversion for 'ls' file sizes coming over the wire was leaking. There's a workaround which stops the leak but it's not a complete fix, but the Math::Pari maintainer has been apprised of the issue.