Skip Menu |

This queue is for tickets about the File-ChangeNotify CPAN distribution.

Report information
The Basics
Id: 52325
Status: open
Priority: 0/
Queue: File-ChangeNotify
People
Owner: Nobody in particular
Requestors: curtis.fletcher [...] n3k.co.uk
pjs [...] cpan.org
steve [...] yewtc.demon.co.uk
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)

History Show all quoted text
  Tue Dec 01 11:42:39 2009 curtis.fletcher [...] n3k.co.uk - Ticket created
Subject: Bug in File::ChangeNotify::Watcher::Inotify __build__mask
Date: Tue, 1 Dec 2009 16:42:01 -0000
To: <bug-File-ChangeNotify [...] rt.cpan.org>
From: "Curtis Fletcher" <curtis.fletcher [...] n3k.co.uk>
Package: File-ChangeNotify-0.09 Perl: 5.10.0 os: Linux ubuntu 2.6.27-7-generic In File::ChangeNotify::Watcher::Inotify sub __build__mask I believe that IN_MOVED_TO should be included in $mask. Otherwise renaming/moving files into a watched directory are not picked up. Curtis Fletcher Software Developer tuscany networks Limited Tuscany House, White Hart Lane, Basingstoke, RG21 4AF Tel: 01256 303 726 Fax: 01256 303 701 www.tuscanynetworks.com Important Notice The information contained in or attached to this email is intended only for the use of the correct addressee(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are not authorised to and must not disclose, copy, distribute, or retain this message or any part of it. It may contain information which is confidential and/or covered by legal professional or other privilege (or other rules or laws with similar effect in jurisdictions outside England and Wales). The views expressed in this email are those of the author and not necessarily the views of the company, its directors, officers or employees and we make no representation or accept any liability for its accuracy or completeness unless expressly stated to the contrary. The company will not be liable for direct, special, indirect or consequential damages arising from alteration of the contents of this message by a third party or as a result of any virus being passed on. The company has implemented virus detection systems to minimise the risk of passing on viruses. The company records all e-mail messages sent to and from this address and reserves the right to monitor and examine content for the purposes of training, quality control and for investigating or detecting any unauthorised use of its system and for ensuring its effective operation. tuscany networks is registered in England, company number 320 3561.
  Tue Dec 01 13:12:53 2009 autarch [...] urth.org - Correspondence added
Subject: Re: [rt.cpan.org #52325] Bug in File::ChangeNotify::Watcher::Inotify __build__mask
Date: Tue, 1 Dec 2009 12:12:42 -0600 (CST)
To: Curtis Fletcher via RT <bug-File-ChangeNotify [...] rt.cpan.org>
From: Dave Rolsky <autarch [...] urth.org>
On Tue, 1 Dec 2009, Curtis Fletcher via RT wrote: Show quoted text
> In File::ChangeNotify::Watcher::Inotify sub __build__mask I believe that > IN_MOVED_TO should be included in $mask. Otherwise renaming/moving files > into a watched directory are not picked up.
Can you come up with a test case? That'd be really helpful. -dave /*============================================================ http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) ============================================================*/
  Tue Dec 01 13:12:54 2009 The RT System itself - Status changed from 'new' to 'open'
  Mon Jan 23 04:18:37 2012 sjq-perl [...] jadevine.org.uk - Correspondence added
On Tue Dec 01 11:42:39 2009, curtis.fletcher@n3k.co.uk wrote: Show quoted text
> Package: File-ChangeNotify-0.09 > Perl: 5.10.0 > os: Linux ubuntu 2.6.27-7-generic > > In File::ChangeNotify::Watcher::Inotify sub __build__mask I believe
that Show quoted text
> IN_MOVED_TO should be included in $mask. Otherwise renaming/moving
files Show quoted text
> into a watched directory are not picked up. >
I'm seeing something very similar to this when trying to watch a directory of files which are maintained by an rsync process. The rsync process makes a temporary file elsewhere in the filesystem and then renames the file after the download has been completed. File::ChangeNotify does not spot any changes to the file contents. When I add IN_MOVED_TO into the mask for the Inotify watcher it spots the changed files. The event type was reported as "unknown" so I also altered the Event class. Attached is a patch which works for me. I started trying to add some tests but I wasn't clear on the best way to write them so that they worked with different event watchers. Regards, Stephen Quinney
Subject: File-ChangeNotify-movedto.diff
diff -ruNp File-ChangeNotify-0.16/lib/File/ChangeNotify/Event.pm File-ChangeNotify-0.16.new/lib/File/ChangeNotify/Event.pm --- File-ChangeNotify-0.16/lib/File/ChangeNotify/Event.pm 2010-07-12 17:07:03.000000000 +0100 +++ File-ChangeNotify-0.16.new/lib/File/ChangeNotify/Event.pm 2012-01-23 08:37:32.000000000 +0000 @@ -18,7 +18,7 @@ has path => ( has type => ( is => 'ro', - isa => enum( [qw( create modify delete unknown )] ), + isa => enum( [qw( create modify delete rename unknown )] ), required => 1, ); @@ -66,8 +66,8 @@ The full path to the file or directory t =item * type => $type -The type of event. This must be one of "create", "modify", "delete", or -"unknown". +The type of event. This must be one of "create", "modify", "delete", +"rename" or "unknown". =back diff -ruNp File-ChangeNotify-0.16/lib/File/ChangeNotify/Watcher/Inotify.pm File-ChangeNotify-0.16.new/lib/File/ChangeNotify/Watcher/Inotify.pm --- File-ChangeNotify-0.16/lib/File/ChangeNotify/Watcher/Inotify.pm 2010-07-12 17:07:03.000000000 +0100 +++ File-ChangeNotify-0.16.new/lib/File/ChangeNotify/Watcher/Inotify.pm 2012-01-23 08:30:40.000000000 +0000 @@ -102,7 +102,7 @@ sub _build__mask { my $self = shift; my $mask - = IN_MODIFY | IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_MOVE_SELF; + = IN_MODIFY | IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_MOVE_SELF | IN_MOVED_TO; $mask |= IN_DONT_FOLLOW unless $self->follow_symlinks(); return $mask; @@ -190,6 +190,7 @@ sub _convert_event { $event->IN_CREATE() ? 'create' : $event->IN_MODIFY() ? 'modify' : $event->IN_DELETE() ? 'delete' + : $event->IN_MOVED_TO() ? 'rename' : 'unknown' ), );
  Wed Jan 25 10:13:11 2012 autarch [...] urth.org - Correspondence added
Subject: Re: [rt.cpan.org #52325] Bug in File::ChangeNotify::Watcher::Inotify __build__mask
Date: Wed, 25 Jan 2012 09:13:00 -0600 (CST)
To: Stephen Quinney via RT <bug-File-ChangeNotify [...] rt.cpan.org>
From: Dave Rolsky <autarch [...] urth.org>
On Mon, 23 Jan 2012, Stephen Quinney via RT wrote: Show quoted text
> Attached is a patch which works for me. I started trying to add some > tests but I wasn't clear on the best way to write them so that they > worked with different event watchers.
I really want the behavior of every watcher to be the same, so I think you should just write tests to test the behavior. If some of the watchers fail, that's ok, now we at least have a failing test. -dave /*============================================================ http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) ============================================================*/
  Mon Apr 09 23:03:02 2012 pjs [...] cpan.org - Ticket #76460: Ticket created
Subject: Directory renames not tracked
As best I can tell, this is inherent to File::ChangeNotify and not Linux::Inotify2. If I rename a directory, subsequent operations on it still have the original name. Program: use Data::Dumper; use File::ChangeNotify; my $watcher = File::ChangeNotify->instantiate_watcher( directories => [ '/tmp' ]); while ( my @events = $watcher->wait_for_events ) { print "Got events:\n"; print Dumper $_ for @events; } Actions taken (wd: /tmp): mkdir foo mv foo bar touch bar/baz Output: Got events: $VAR1 = bless( { 'type' => 'create', 'path' => '/tmp/foo' }, 'File::ChangeNotify::Event' ); Got events: $VAR1 = bless( { 'type' => 'unknown', 'path' => '/tmp/foo' }, 'File::ChangeNotify::Event' ); Got events: $VAR1 = bless( { 'type' => 'create', 'path' => '/tmp/foo/baz' }, 'File::ChangeNotify::Event' ); Last event should of course have a path of /tmp/bar/baz.
  Mon Apr 09 23:12:19 2012 pjs [...] cpan.org - Ticket #76460: Correspondence added
Just realized that 52325 is addressing the same issue - the subject line threw me off. Close this one by all means; it may be useful for the test case.
  Mon Apr 09 23:12:20 2012 pjs [...] cpan.org - Ticket #76460: Status changed from 'new' to 'open'
  Mon Apr 09 23:48:00 2012 DROLSKY [...] cpan.org - Ticket #76460: Merged into ticket #52325
  Mon Apr 09 23:48:00 2012 DROLSKY [...] cpan.org - Merged into ticket #52325
  Tue Apr 10 00:01:33 2012 pjs [...] cpan.org - Correspondence added
I found that patch incomplete for tracking moves; it recognized a move but didn't track the new location. I had to apply the following patch on top of it. I will see if I can come up with a test. +++ /usr/local/lib/perl5/site_perl/5.14.2/File/ChangeNotify/Watcher/Inotify.pm~2012-04-09 20:15:48.000000000 -0700 @@ -85,15 +85,16 @@ # excluded or when the exclusion excludes a file, not a dir. next if $self->_path_is_excluded( $event->fullname() ); - if ( ($event->IN_CREATE() || $event->IN_MOVED_TO()) && $event->IN_ISDIR() ) { + if ( $event->IN_CREATE() && $event->IN_ISDIR() ) { $self->_watch_directory( $event->fullname() ); push @interesting, $event; push @interesting, $self->_fake_events_for_new_dir( $event->fullname() ); } - elsif ( $event->IN_DELETE_SELF() || $event->IN_MOVE_SELF() ) { + elsif ( $event->IN_DELETE_SELF() ) { $self->_remove_directory( $event->fullname() ); } + # We just want to check the _file_ name elsif ( $event->name() =~ /$filter/ ) { push @interesting, $event;
  Tue Apr 10 00:01:33 2012 The RT System itself - System error
Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.
  Tue Apr 10 00:01:34 2012 The RT System itself - System error
Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.
  Tue Apr 10 00:22:30 2012 pjs [...] cpan.org - Correspondence added
Okay, this will test for correct behavior as provided by the combined patches above. I do not have KQueue, I have no idea how to patch that. use Test::More tests => 2; use File::Temp qw(tempdir); use File::ChangeNotify; my $dir = tempdir( CLEANUP => 1 ); my $watcher = File::ChangeNotify->instantiate_watcher( directories => [ $dir ] ); $watcher->new_events; mkdir "$dir/foo"; # Event 1: create rename "$dir/foo", "$dir/bar"; # Event 2: rename symlink "..", "$dir/bar/baz"; # Event 3: create my @events = $watcher->new_events; ok @events == 3, 'Saw all events'; is $events[2]->{path}, "$dir/bar/baz", 'Saw correct destination path';
  Tue Apr 10 12:49:55 2012 pjs [...] cpan.org - Correspondence added
Oops, got the patch backwards, sorry...
  Tue Apr 10 12:49:56 2012 The RT System itself - System error
Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.
  Tue Apr 10 12:49:56 2012 The RT System itself - System error
Sending the previous mail has failed. Please contact your admin, they can find more details in the logs.
  Fri Apr 19 12:26:11 2013 steve [...] yewtc.demon.co.uk - Ticket #84747: Ticket created
Subject: File Closed
Date: Fri, 19 Apr 2013 17:25:52 +0100
To: bug-File-ChangeNotify [...] rt.cpan.org
From: Steve Rogerson <steve [...] yewtc.demon.co.uk>
hi, I've just read 52325 and this affects me also. I have a specific need to check directories that have files added, typically by an ftp process. When files have finished being downloaded that are then processed in some way. I thought File::ChangeNotify would do what I want. In fact I have a working process that uses Linux::Inotify2 but I like the fact the File::ChangeNotify is os neutral. I can't use it for two reasons. 1. Essentially #52325 - it doesn't detect files moved to the directory, as is done sometimes and 2. It doesn't detect that the file has been closed, in my context the file transfer has been completed. We get the creates and modifies, but that doesn't tell me when the file has been closed and is ready to process. It seems to me that my usage is not that strange, and that what's needed is to expose the mask to the user [ ok and process the extra events]. In my case I don't care about modified, I just want to know when it's finished. I do have a "test" of sorts - but it fails sort of by definition as I know that the events I want are not generated. Actually it sits and waits forever, which isn't too good for a test, but I'm working on it. Steve
  Sun Apr 21 11:59:40 2013 DROLSKY [...] cpan.org - Ticket #84747: Merged into ticket #52325
  Sun Apr 21 11:59:40 2013 DROLSKY [...] cpan.org - Merged into ticket #52325
  Sun Apr 21 12:02:15 2013 DROLSKY [...] cpan.org - Correspondence added
On Fri Apr 19 12:26:11 2013, steve@yewtc.demon.co.uk wrote: Show quoted text
> I've just read 52325 and this affects me also. I have a specific need to > check directories that have files added, typically by an ftp process. When > files have finished being downloaded that are then processed in some way. I > thought File::ChangeNotify would do what I want. In fact I have a working > process that uses Linux::Inotify2 but I like the fact the File::ChangeNotify > is os neutral.
I'm all for making this more featureful. There are two things that need to happen: 1. There needs to be tests for new features. 2. There needs to be a way to make it work with all implementations. However, #2 can be a huge PITA since without something like inotify I have no idea how you can watch for close events or renames. So ... for people who need that, it might be better to just use inotify or kqueue directly rather than trying to use this module. Alternately, if someone came up with a Win32 implementation that used something Win32-specific to do this, I could consider getting rid of the pure Perl implementation, since that platform is the main reason it exists.
  Mon Apr 22 06:47:09 2013 steve.bitcard [...] yewtc.demon.co.uk - Correspondence added
From: steve.bitcard [...] yewtc.demon.co.uk
On Sun Apr 21 12:02:15 2013, DROLSKY wrote: Show quoted text
> On Fri Apr 19 12:26:11 2013, steve@yewtc.demon.co.uk wrote: >
> > I've just read 52325 and this affects me also. I have a specific
> need to
> > check directories that have files added, typically by an ftp
> process. When
> > files have finished being downloaded that are then processed in
> some way. I
> > thought File::ChangeNotify would do what I want. In fact I have a
> working
> > process that uses Linux::Inotify2 but I like the fact the
> File::ChangeNotify
> > is os neutral.
> > I'm all for making this more featureful. There are two things that > need to happen: > > 1. There needs to be tests for new features. > 2. There needs to be a way to make it work with all implementations. > > However, #2 can be a huge PITA since without something like inotify I > have no idea how you can watch for close events or renames. > > So ... for people who need that, it might be better to just use > inotify or kqueue directly rather than trying to use this module. > > Alternately, if someone came up with a Win32 implementation that used > something Win32-specific to do this, I could consider getting rid > of the pure Perl implementation, since that platform is the main > reason it exists.
I agree that probably I'm better off continue using inotify (Linux::Inotify2) directly. It doesn't look as though there is a simple Win32 alternative. Having said that, if the mask was exposed, it would allow linux users to check for move/close, though I agree that sort of contradicts the notion of os independence. Though the other choice is "lowest common denominator".