Skip Menu |

This queue is for tickets about the KiokuDB-Backend-BDB CPAN distribution.

Report information
The Basics
Id: 46370
Status: open
Priority: 0/
Queue: KiokuDB-Backend-BDB

People
Owner: Nobody in particular
Requestors: j.borevich [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 0.14
Fixed in: (no value)



Subject: Secondary indices not cleared aftter deleting objects (KiokudDB::Backend::DBD::GIN)
Secondary indices not cleared aftter deleting objects (KiokudDB::Backend::DBD::GIN) perl version: 5.8.8 os version: Linux fireball 2.6.18-6-686 #1 SMP Tue Jun 17 21:31:27 UTC 2008 i686 GNU/Linux KiokuDB version: 0.27 KiokiDB::Backend::BDB version: 0.14 BerkeleyDB (perl package) version: 0.38 BerkeleyDB version: 4.7.25 Execute following code: { package App::Object; use Moose; has indexed_field => ( is => 'rw', isa => 'Int' ); __PACKAGE__->meta->make_immutable; 1; } package main; use KiokuDB; use Search::GIN::Extract::Callback; my $dir = KiokuDB->connect( "bdb-gin:dir=~/tmp/db/", create => 1, extract => Search::GIN::Extract::Callback->new(extract => sub { my ( $object, $extractor, @args) = @_; my $index = {}; if ( $object->isa('App::Object') ){ $index->{indexed_field} = $object->indexed_field; } return $index; }) ); { my $scope = $dir->new_scope; my $obj = new App::Object indexed_field => 123; $dir->store($obj); $dir->delete($obj); } Then in shell: # ~/bin/db_dump -p gin_index VERSION=3 format=print type=btree duplicates=1 dupsort=1 db_pagesize=4096 HEADER=END indexed_field:123 E41BB608-4A08-11DE-8F2D-9288FFEE0A87 DATA=END # ~/bin/db_dump -p gin_index VERSION=3 format=print type=btree duplicates=1 dupsort=1 db_pagesize=4096 HEADER=END indexed_field:123 E41BB608-4A08-11DE-8F2D-9288FFEE0A87 DATA=END Simplest way i found to fix this bug is to add method modifiler (before delete). In KiokuDB::Backend::DBD::GIN : before delete => sub { my ( $self, @entries ) = @_; if ( $self->extract ) { foreach my $entry ( @entries ) { if ($entry->has_object){ my @keys = $self->extract_values( $entry->object, entry => $entry ); if ( @keys ) { my $d = $entry->backend_data || $entry->backend_data({}); $d->{keys} = \@keys; $keys{$entry->id} = \@keys; } } } } }; Possible there is better way to fix this problem.
Subject: Re: [rt.cpan.org #46370] Secondary indices not cleared aftter deleting objects (KiokudDB::Backend::DBD::GIN)
Date: Tue, 26 May 2009 18:02:12 +0200
To: bug-KiokuDB-Backend-BDB [...] rt.cpan.org
From: Yuval Kogman <nuffin [...] cpan.org>
Ouch, that's a little disconcerting, I was assuming that BDB's secondary index support would ensure that is deleted. It looks like your fix is somehow related by triggering the secondary database callback somehow. I will have to look into this with carefully. Thanks for the report!
From: j.borevich [...] gmail.com
Втр. Май 26 12:02:32 2009, NUFFIN писал: Show quoted text
> Ouch, that's a little disconcerting, I was assuming that BDB's > secondary index support would ensure that is deleted. > > It looks like your fix is somehow related by triggering the secondary > database callback somehow. > > I will have to look into this with carefully. > > Thanks for the report!
BDB call function given to DB->associate method on each - db_put, db_get and db_del methods, and waits for keys in secondary db to know which records should be updated/deleted. I found some better way to fix it. Look at after delete modifiler at pathed package. Patched package and test attached.
#!/usr/bin/perl package KiokuDB::Backend::BDB::GIN; # this is patched version of package # search patched according to https://rt.cpan.org/Ticket/Display.html?id=46377 # delete (with secondary indicie) according to https://rt.cpan.org/Ticket/Display.html?id=46370 # # documantation removed (this file is just a suggestion) use Moose; use Data::Stream::Bulk::Util qw(cat); use BerkeleyDB; use namespace::clean -except => 'meta'; extends qw(KiokuDB::Backend::BDB); with qw( KiokuDB::Backend::Role::Query::GIN Search::GIN::Extract::Delegate ); sub BUILD { shift->secondary_db } # early has secondary_db => ( is => "ro", lazy_build => 1, ); sub _build_secondary_db { my $self = shift; $self->_open_secondary( name => "secondary", file => "gin_index" ); } has root_only => ( isa => "Bool", is => "ro", default => 0, ); has '+extract' => ( required => 0, ); my %keys; sub _open_secondary { my ( $self, @args ) = @_; my $secondary = $self->manager->open_db( dup => 1, dupsort => 1, @args ); $self->manager->associate( secondary => $secondary, primary => $self->primary_db, callback => sub { my $id = shift; return delete $keys{$id} || []; } ); return $secondary; } before insert => sub { my ( $self, @entries ) = @_; if ( $self->extract ) { foreach my $entry ( @entries ) { if ( $entry->deleted || !$entry->has_object || ( !$entry->root && $self->root_only ) ) { $entry->clear_backend_data; } else { my @keys = $self->extract_values( $entry->object, entry => $entry ); if ( @keys ) { my $d = $entry->backend_data || $entry->backend_data({}); $d->{keys} = \@keys; $keys{$entry->id} = \@keys; # temp } } } } }; before delete => sub { my ( $self, @entries ) = @_; if ( $self->extract ) { foreach my $entry ( @entries ) { # want to find stored entry that wasn't changed while ($entry->has_prev) { $entry = $entry->prev }; my $b = $entry->backend_data; # entry should contain link to secondary db keys if ($b && exists $b->{keys}){ $keys{$entry->id} = $b->{keys}; } } } }; sub search { my ( $self, $query, @args ) = @_; my %args = ( distinct => $self->distinct, @args, ); my $live_objects = $args{live_objects}; my %spec = $query->extract_values($self); my @cursors; foreach my $key (@{ $spec{values} }){ my $cur = $self->secondary_db->db_cursor; my $data = ''; my $ret = $cur->c_get($key, $data, DB_SET); if ($ret == DB_NOTFOUND){ return new Data::Stream::Bulk::Nil; } confess "Can't join tables: $ret" unless $ret == 0; push @cursors, $cur; } my $cursor = scalar @cursors ? $self->primary_db->db_join(\@cursors) : $self->secondary_db->db_cursor; my $flags = scalar @cursors ? 0 : DB_NEXT; my @result; return new Data::Stream::Bulk::Callback callback => sub { my ($id, $data) = ('', ''); my $ret = $cursor->c_get($id, $data, $flags); if ($ret == 0){ my ($entry) = $live_objects->ids_to_entries($id); unless (defined $entry){ $entry = $self->deserialize($data); } return [$entry]; } elsif ($ret == DB_NOTFOUND){ $cursor->c_close; return; } else { throw LGame::Err "Error while fetching db: $ret"; } } } sub fetch_entry { die "TODO" } sub remove_ids { my ( $self, @ids ) = @_; die "Deletion the GIN index is handled implicitly by BDB"; } sub insert_entry { my ( $self, $id, @keys ) = @_; die "Insertion to the GIN index is handled implicitly by BDB"; } __PACKAGE__->meta->make_immutable; __PACKAGE__
# For pass all tests required following: # commit http://github.com/nothingmuch/kiokudb/commit/b60fffb61974b8133f87d260e70140050e0ce1e6 { package App::Object; use Moose; has indexed_field => ( is => 'rw', isa => 'Int' ); __PACKAGE__->meta->make_immutable; 1; } package main; use KiokuDB; use Search::GIN::Extract::Callback; use Search::GIN::Query::Manual; use Test::Simple tests => 5; my $dir = KiokuDB->connect( "bdb-gin:dir=/home/self/tmp/db/", create => 1, extract => Search::GIN::Extract::Callback->new(extract => sub { my ( $object, $extractor, @args) = @_; my $index = { class => ref $object }; if ( $object->isa('App::Object') ){ $index->{indexed_field} = $object->indexed_field; } return $index; }) ); { my $scope = $dir->new_scope; $dir->store(App::Object->new(indexed_field => 123)); $dir->store(App::Object->new(indexed_field => 777)); } { my $scope = $dir->new_scope; my $query = Search::GIN::Query::Manual->new( values => { class => 'App::Object', indexed_field => 123 }, ); @result = $dir->search($query)->all; ok @result == 1, 'Search works fine'; my $obj = shift @result; $dir->delete($obj); @result = $dir->search($query)->all; ok @result == 0, 'Deleting object without it changing'; $dir->store(App::Object->new(indexed_field => 123)); @result = $dir->search($query)->all; ok @result == 1; $obj = shift @result; $obj->indexed_field(111); $dir->delete($obj); @result = eval { $dir->search($query)->all }; my $err = $@; ok !$err, 'Checking secondary indicies cleared'; ok @result == 0, 'Deleting object after it was changed'; }