Skip Menu |

This queue is for tickets about the File-Path CPAN distribution.

Report information
The Basics
Id: 71825
Status: rejected
Priority: 0/
Queue: File-Path

People
Owner: RICHE [...] cpan.org
Requestors: robert [...] achronix.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 2.10_001



Subject: rmtree returns successfully before work is complete in Win7-x64
Date: Thu, 20 Oct 2011 14:09:19 -0700
To: "bug-File-Path [...] rt.cpan.org" <bug-File-Path [...] rt.cpan.org>
From: Robert Sigel <robert [...] achronix.com>
Bug observed in: Perl Version(s): (full details from -V included at the end of the message) 64-bit Strawberry Perl 5.12.3.0 64-bit ActivePerl 5.14.2.1402 OS vendor/version: Microsoft Windows 7 Professional version 6.1 (Build 7601: Service Pack 1) code snippet exhibiting the bug: rmtree "$outdir" || die "Can't delete $outdir directory tree !\n\t$!"; print "DEBUG: removed $outdir tree\n"; if (!(-r $outdir)) { mkdir($outdir,0755) || die "Can't make release output directory !\n\t$!"; print "DEBUG: $outdir recreated\n" } else { print "WARNING: $outdir still exists!\n"; } if (!(-r $outdir."/system")) { mkdir("$outdir/system",0755) || die "Can't make release output directory (2)!\n\t$!"; print "DEBUG: ".$outdir."/system recreated\n" } else { print "WARNING: ".$outdir."/system still exists!\n"; } When $outdir is a large directory tree (in this case, 2700+ files, 305+ subdirs, 550MB+ of data), the first -r readability check (for $outdir itself) passes, finding the $outdir directory, despite the prior tree deletion. Then the second readability check (for the $outdir/system subdir) fails, and we try to create the subdir, but cannot, since now the parent dir does not exist. (Despite the error message about permissions, observing the directory structure after the code dies proves that the $outdir does not exist.) Running the above code (where $outdir is an existing populated dir tree named 'Release-Output') produces the output below. DEBUG: removed Release-Output tree WARNING: Release-Output still exists! Can't make release output directory (2)! Permission denied at d:/src/win3/main/software/build-utils/build_release.pl line 475. Details which may be relevant: This is in a single-threaded build system - no other threads are manipulating the directory structure at this time. An antivirus suite is running on this workstation - the impact upon low-level filesystem operations is unknown. The D: drive containing both the script and the $outdir is a 2-disk RAID0 array. (The OS should abstract this away, but...) This perl code behaves correctly on WinXP-32 SP3 and various flavors of Linux (multiple different Perl vendors/versions). I'm a beginning perl developer who's inherited this 'mature' script. This bug was found during our migration to Win7-64. I've added the debug and warning print statements. ================= Strawberry perl -V reports the following: ------- Summary of my perl5 (revision 5 version 12 subversion 3) configuration: Platform: osname=MSWin32, osvers=6.1, archname=MSWin32-x64-multi-thread uname='Win32 strawberryperl 5.12.3.0 #1 Sun May 15 09:43:50 2011 x64' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags =' -s -O2 -DWIN32 -DHAVE_DES_FCRYPT -DWIN64 -DCONSERVATIVE -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -mms-bitfields -DPERL_MSVCRT_READFIX', optimize='-s -O2', cppflags='-DWIN32' ccversion='', gccversion='4.4.3', gccosandvers='' intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags ='-s -L"C:\strawberry_perl_5.12.3.0_64\perl\lib\CORE" -L"C:\strawberry_perl_5.12.3.0_64\c\lib"' libpth=C:\strawberry_perl_5.12.3.0_64\c\lib C:\strawberry_perl_5.12.3.0_64\c\x86_64-w64-mingw32\lib libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 libc=, so=dll, useshrplib=true, libperl=libperl512.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-mdll -s -L"C:\strawberry_perl_5.12.3.0_64\perl\lib\CORE" -L"C:\strawberry_perl_5.12.3.0_64\c\lib"' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_SITECUSTOMIZE Built under MSWin32 Compiled at May 15 2011 09:52:35 %ENV: PERL_JSON_BACKEND="JSON::XS" PERL_YAML_BACKEND="YAML" @INC: C:/strawberry_perl_5.12.3.0_64/perl/site/lib C:/strawberry_perl_5.12.3.0_64/perl/vendor/lib C:/strawberry_perl_5.12.3.0_64/perl/lib . ============ ActivePerl perl -V reports the following: ------- Summary of my perl5 (revision 5 version 14 subversion 2) configuration: Platform: osname=MSWin32, osvers=5.2, archname=MSWin32-x64-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -Ox -GL -fp:precise -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO', optimize='-MD -Zi -DNDEBUG -Ox -GL -fp:precise', cppflags='-DWIN32' ccversion='16.0.30319', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='__int64', ivsize=8, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"C:\ActivePerl64\lib\CORE" -machine:AMD64' libpth=\lib libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32. lib comctl32.lib bufferoverflowU.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl514.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"C:\ActivePerl64\lib\CORE" -machine:AMD64' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PL_OP_SLAB_ALLOC USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_SITECUSTOMIZE Locally applied patches: ActivePerl Build 1402 [295342] Built under MSWin32 Compiled at Oct 7 2011 15:19:36 %ENV: PERL_JSON_BACKEND="JSON::XS" PERL_YAML_BACKEND="YAML" @INC: C:/ActivePerl64/site/lib C:/ActivePerl64/lib .
Subject: RE: [rt.cpan.org #71825]
Date: Thu, 20 Oct 2011 14:35:20 -0700
To: "bug-File-Path [...] rt.cpan.org" <bug-File-Path [...] rt.cpan.org>
From: Robert Sigel <robert [...] achronix.com>
Error correction: I previously stated this was observed on a RAID0 drive. I was wrong - it is instead on a 2 disk RAID1 array. Again, it shouldn't matter, but...
Subject: Re: [rt.cpan.org #71825]
Date: Fri, 21 Oct 2011 00:35:42 +0200
To: bug-File-Path [...] rt.cpan.org
From: David Landgren <david [...] landgren.net>
Le 20/10/2011 23:35, Robert Sigel via RT a écrit : Show quoted text
> Queue: File-Path > Ticket<URL: https://rt.cpan.org/Ticket/Display.html?id=71825> > > Error correction: > I previously stated this was observed on a RAID0 drive. I was wrong - it is instead on a 2 disk RAID1 array. Again, it shouldn't matter, but...
No, it shouldn't matter. Thanks for the report, I'll try and have a look this weekend. Thanks, DAvid
I have witnessed this bug too. While using Path::Class (which in turn uses this). I first noticed it on my Win7x64 box and I have used this module lots in the past. In my case I'm working in a directory with a lot of files so when I need to clean it out, I just remove the directory and recreate it (sometimes in a tight loop). When I start to put more files in it I'm finding that some of the new files are removed. Sometimes I can run the script again and have it bomb at the same place, other times it will keep going. I have tried to sleep a second after it and that didn't help.
RT-Send-CC: david [...] landgren.net
On 2011-10-20 15:35:55, david@landgren.net wrote: Show quoted text
> Le 20/10/2011 23:35, Robert Sigel via RT a écrit :
> > Queue: File-Path > > Ticket<URL: https://rt.cpan.org/Ticket/Display.html?id=71825> > > > > Error correction: > > I previously stated this was observed on a RAID0 drive. I was wrong > > - it is instead on a 2 disk RAID1 array. Again, it shouldn't matter, > > but...
> > No, it shouldn't matter. Thanks for the report, I'll try and have a > look > this weekend.
Many weekends have passed; any luck?
Win32 API only has deferred deletion of files and directories (NtSetInformationFile/FileDispositionInformation). Only when the last kernel handle (in any process) is closed to the resource, is it deleted from disk. Native API has instant deletion (NtDeleteFile) but Native API requires XS and there is a big outcry from peanut gallery when you try to use Native API/ At NYPM Hackathon the community discussion basically figured out 4 ways to resolve this ticket. 1. the user states he is running an anti-virus program, if his AV prog has a handle to the dir or a file in the dir open, it is the user's problem on why they are deleting an in use file, at most, the cavet should be documented in POD alone, it is not File::Path's mistake, it is the users 2. File::Path should on Win32 should use inotify/FindFirstChangeNotification (that requires XS) to wait until the DIR disappears, with a user selectable timeout on how many seconds to wait, one person said the default could be 0, which means do not block, and fatally error if the file is still -e after unlink even though unlink succeeded. The other idea is to retry rmtree for a fixed number of iterations, possibly with sleep, to delete the file or dir. Another idea was to unlink/mark for delete, then rename the dir to something else, and that dir will disappear naturally, which might be minutes, hours or a reboot away. EUMM does this to inuse DLLs in blib. 3. tell the user it is the users problem, and the user should call rmtree in a loop since it is their particular Win32 system with AV app is the freak, not normal Win32 systems. 4. use Win32 C debugging apis to find all processes with open handles (requires UAC/root) to the files/dirs in question, then force close the handle in that process (see "Unlocker" explorer context menu tool on google). This is very unsafe without explicit human review (imagine breaking/fatally erroring/SEGVing out a production DB daemon or webserver proces). IMO this ticket is heading towards rejected.
per buik88's comments.