| Subject: | FTS tokenizer segfault |
| Date: | Sat, 10 Dec 2011 08:21:17 -0500 |
| To: | bug-DBD-SQLite [...] rt.cpan.org |
| From: | Ephraim Stevens <ephraim.stevens [...] gmail.com> |
Gentleman,
With DBD::SQLite 1.33 and 1.35, I am trying to use a custom perl tokenizer
with the DBD::SQLite module and it segfaults. In fact, I'm using the
example taken right out of the documentation. Can you please provide
guidance? Maybe I'm missing something here.
I tried this code USING FTS4 and FTS3 with the same results. Forgive me if
I overlooked something basic. Thanks in advance for your help. Here is the
code:
--------------------------------------------------------------------------------------------------------------------------------------------
#!/usr/bin/perl
use DBD::SQLite;
my $dbh = DBI->connect("dbi:SQLite:dbname=fts.db","","", {
ShowErrorStatement => 1}
) or die $DBI::errstr;
sub locale_tokenizer {
return sub {
my $string = shift;
use locale;
my $regex = qr/\w+/;
my $term_index = 0;
return sub { # closure
$string =~ /$regex/g or return; # either match, or no more token
my ($start, $end) = ($-[0], $+[0]);
my $len = $end-$start;
my $term = substr($string, $start, $len);
return ($term, $len, $start, $end, $term_index++);
}
};
}
$dbh->do('CREATE VIRTUAL TABLE FIXMESSAGE USING FTS3(MESSAGE,
tokenize=perl, "main::locale_tokenizer");');
------------------------------------------------------------------------------------------------------------------------------------------------
Test run:
root@ln:/usr/local/dbperl# ./test.pl
Segmentation fault