Bug #76746 for CHI: CHI::Driver::File - speed up get

Sat Apr 21 13:10:49 2012 vitaliy.tokarev [...] gmail.com - Ticket created

Subject:

CHI::Driver::File - speed up get_keys() function

I use File driver in situations when I have a small amount of keys, but I often call get_keys(). Even when we use L1_CACHE, get_keys() calls find(). More over, the subdirs is never deletes via this driver. So, we call often get_keys() which calls find() on every subdir in the directory where keys stored. I wrote a micro patch, which can be useful for people with same situation. I don't know about overhead on scalar function, but I consider it is better then we are calling filesystem functions every time. I put here full code to easy understanding and discuss: my $is_empty = 0; sub get_keys { my ($self) = @_; return () if ($is_empty); my @filepaths; my $re = quotemeta($self->file_extension); my $wanted = sub { push( @filepaths, $_ ) if -f && /${re}$/ }; my @keys = $self->_collect_keys_via_file_find( \@filepaths, $wanted ); $is_empty = (scalar @keys) ? 0 : 1; return @keys; } Much better will be store results somewhere and just look for modifications, but then in situations where a lot of keys CHI will eat more memory. What are you say about this patch?

Sat Apr 21 13:20:43 2012 vitaliy.tokarev [...] gmail.com - Correspondence added

From:

vitaliy.tokarev [...] gmail.com

Also, if someone will use this patch, dont forget to reset $is_empty via store() function: sub store { ... write_file(...); $is_empty = 0; ... }

Sun Apr 22 00:55:54 2012 swartz [...] pobox.com - Correspondence added

Subject:	Re: [rt.cpan.org #76746] CHI::Driver::File - speed up get_keys() function
Date:	Sat, 21 Apr 2012 21:55:38 -0700
To:	bug-CHI [...] rt.cpan.org
From:	Jonathan Swartz <swartz [...] pobox.com>

This won't work in a situation with multiple processes accessing the same file cache - imagine that $is_empty is set in process 1, then process 2 stores a new key; process 1 will still think there are no keys. Best thing might be to have a periodic process that deletes empty directories or old files in your cache file hierarchy. Jon On Apr 21, 2012, at 10:10 AM, Vitaliy Tokarev via RT wrote: Show quoted text

> Sat Apr 21 13:10:49 2012: Request 76746 was acted upon. > Transaction: Ticket created by gh0stwizard > Queue: CHI > Subject: CHI::Driver::File - speed up get_keys() function > Broken in: 0.52 > Severity: Wishlist > Owner: Nobody > Requestors: vitaliy.tokarev@gmail.com > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=76746 > > > > I use File driver in situations when I have a small amount of keys, but > I often call get_keys(). Even when we use L1_CACHE, get_keys() calls > find(). More over, the subdirs is never deletes via this driver. > > So, we call often get_keys() which calls find() on every subdir in the > directory where keys stored. I wrote a micro patch, which can be useful > for people with same situation. I don't know about overhead on scalar > function, but I consider it is better then we are calling filesystem > functions every time. > > I put here full code to easy understanding and discuss: > > my $is_empty = 0; > > sub get_keys { > my ($self) = @_; > > return () if ($is_empty); > > my @filepaths; > my $re = quotemeta($self->file_extension); > my $wanted = sub { push( @filepaths, $_ ) if -f && /${re}$/ }; > my @keys = $self->_collect_keys_via_file_find( \@filepaths, > $wanted ); > $is_empty = (scalar @keys) ? 0 : 1; > return @keys; > } > > > Much better will be store results somewhere and just look for > modifications, but then in situations where a lot of keys CHI will eat > more memory. What are you say about this patch?

Sun Apr 22 00:55:55 2012 The RT System itself - Status changed from 'new' to 'open'

Sun Apr 22 06:11:38 2012 vitaliy.tokarev [...] gmail.com - Correspondence added

From:

vitaliy.tokarev [...] gmail.com

Yes, you are right. Ticket may be closed due specific of a situation. We are trying to resolve our problems and this was my mistake. Some months ago I wrote the program that looking for modifications of files, event-based (EV module). And I may say that is not best practice, because if we have a lot of files EV could not solve this task ideally. So, this is an old programming problem what to choose: of cpu time or a memory usage. Memory is cheap, cpu is not still. I will try find best solution for this question and post here if luck is still with me. Thanks! On Sun Apr 22 00:55:54 2012, swartz@pobox.com wrote: Show quoted text

> This won't work in a situation with multiple processes accessing the > same file cache - imagine that $is_empty is set in process 1, then > process 2 stores a new key; process 1 will still think there are no > keys. > > Best thing might be to have a periodic process that deletes empty > directories or old files in your cache file hierarchy. > > Jon

Wed May 30 07:57:36 2012 JSWARTZ [...] cpan.org - Status changed from 'open' to 'resolved'

Bug #76746 for CHI: CHI::Driver::File - speed up get_keys() function