Bug #68579 for File-ReadBackwards: horrible non-killable test process when not enough disk space

i have heard about the BIG test problem. the large_file.t test was copied from the perl core tests and modified to use ReadFileBackwards. it doesn't create a 5GB file but just seeks out there and should create a file which looks that large. what OS are you one? there is a list of them which skip this test and it currently is set to this: qw( MSWin32 NetWare VMS unicos ) ; i can add more entries to that list to skip if the os doesn't support this test. the newest perl has redone the test and it is called op/lfs.t. it does a real check on whether sparse files are supported and not just a check on OS names. i will steal that part and hopefully it will fix this issue. uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

On 05/31/2011 02:22 PM, Uri Guttman via RT wrote: Show quoted text
This is Ubuntu 10.04, not exactly anything exotic. The only possibly-exotic thing is that my home dir (and thus ~/.cpanm) is on an ecryptfs volume, which might be getting pissed off about the seek that calls for more space than it has?

On 05/31/2011 02:47 PM, Uri Guttman via RT wrote: Show quoted text
Yep, running it on the regular ext4 FS works OK, even with only 200MB free. Show quoted text
Yep, the t/op/lfs.t in the perl 5.14.0 release skips on the ecryptfs volume: rob@nightshade perl-5.14.0$ ./perl t/op/lfs.t # checking whether we have sparse files... # s1 = 20 3052238 33188 1 1000 1000 0 1000003 1306881462 1306881462 1306881462 4096 1976 # s2 = 20 3052324 33188 1 1000 1000 0 2000003 1306881462 1306881462 1306881462 4096 3928 1..0 # Skip no sparse files? also, it runs when I move the compiled dist to /tmp. So yep, sounds reasonable to swipe that sparse-checking code.

On 05/31/2011 04:58 PM, Uri Guttman via RT wrote: Show quoted text
Here's a rewrite, with the sparse-files-detection code lifted from t/op/lfs.t. Look OK? ----------------------------------- #!/usr/local/bin/perl -ws use strict ; use Carp ; use Config ; use Fcntl qw( :seek ) ; use File::Temp qw( tempfile ); use Test::More ; use File::ReadBackwards ; # NOTE: much of this code was taken from the core perl test script # ops/lfs.t. it was modified to test File::ReadBackwards and large files my $test_file = 'bw.data' ; my @test_lines = ( "3rd from last line\n", "2nd from last\n", "last line\n", ) ; my $test_text = join '', @test_lines ; sub skip_all_tests { my( $skip_text ) = @_ ; # unlink $test_file ; plan skip_all => $skip_text ; } if( $Config{lseeksize} < 8 ) { skip_all_tests( "no 64-bit file offsets\n" ) ; } unless( $Config{uselargefiles} ) { skip_all_tests( "no large file support\n" ) ; } unless ( have_sparse_files() ) { skip_all_tests( "no sparse file support\n" ) ; } # run the long seek code below in a subprocess in case it exits with a # signal my $rc = system $^X, '-e', <<"EOF"; open(BIG, ">$test_file"); seek(BIG, 5_000_000_000, 0); print BIG "$test_text" ; exit 0; EOF if( $rc ) { my $error = 'signal ' . ($rc & 0x7f) ; skip_all_tests( "seeking past 2GB failed: $error" ) ; } open(BIG, ">$test_file"); unless( seek(BIG, 5_000_000_000, 0) ) { skip_all_tests( "seeking past 2GB failed: $!" ) ; } # Either the print or (more likely, thanks to buffering) the close will # fail if there are are filesize limitations (process or fs). my $print = print BIG $test_text ; my $close = close BIG; unless ($print && $close) { print "# print failed: $!\n" unless $print; print "# close failed: $!\n" unless $close; if( $! =~/too large/i ) { skip_all_tests( 'writing past 2GB failed: process limits?' ) ; } if( $! =~ /quota/i ) { skip_all_tests( 'filesystem quota limits?' ) ; } skip_all_tests( "large file error: $!" ) ; } plan tests => 2 ; my $bw = File::ReadBackwards->new( $test_file ) or die "can't open $test_file: $!" ; my $line = $bw->readline() ; is( $line, $test_lines[-1], 'last line' ) ; $line = $bw->readline() ; is( $line, $test_lines[-2], 'next to last line' ) ; unlink $test_file ; exit ; ######## subroutines # this is lifted wholesale from t/op/lfs.t in perl. Also, Uri is the # wind beneath my wings. sub have_sparse_files { # don't even try for spare files on some OSs return 0 if { map { $_ => 1 } qw( MSWin32 NetWare VMS unicos ) }->{ $^O }; # take that, readability. my (undef,$big0) = tempfile(); my (undef,$big1) = tempfile(); my (undef,$big2) = tempfile(); # We'll start off by creating a one megabyte file which has # only three "true" bytes. If we have sparseness, we should # consume less blocks than one megabyte (assuming nobody has # one megabyte blocks...) open(BIG, ">$big1") or die "open $big1 failed: $!"; binmode(BIG) or die "binmode $big1 failed: $!"; seek(BIG, 1_000_000, SEEK_SET) or die "seek $big1 failed: $!"; print BIG "big" or die "print $big1 failed: $!"; close(BIG) or die "close $big1 failed: $!"; my @s1 = stat($big1); diag "s1 = @s1"; open(BIG, ">$big2") or die "open $big2 failed: $!"; binmode(BIG) or die "binmode $big2 failed: $!"; seek(BIG, 2_000_000, SEEK_SET) or die "seek $big2 failed: $!"; print BIG "big" or die "print $big2 failed: $!"; close(BIG) or die "close $big2 failed: $!"; my @s2 = stat($big2); diag "s2 = @s2"; unless ( $s1[7] == 1_000_003 && $s2[7] == 2_000_003 && $s1[11] == $s2[11] && $s1[12] == $s2[12] && $s1[12] > 0 ) { diag 'no sparse files. sad face.'; return 0; } diag 'we seem to have sparse files...'; }

On 06/01/2011 06:52 PM, Uri Guttman via RT wrote: Show quoted text
Yep, test skips on my encrypted fs, and runs on my regular fs, as desired. R

hi robert, i have uploaded the new backwards to http://sysarch.com/File-ReadBackwards-1.05.tar.gz i made one minor change in your test, i have the have_sparse_files sub return 1 at the end. it was saying it had sparse files but it didn't run the planned tests since the boolean test of that call failed. also i commented out the diag calls. if this works for you let me know. thanx for your help here. you are acked in the Changes file. also i plan to post on blogs.perl.org about this. note that i haven't touched this module in 6(!!) years until the sparse file issue came up. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------