Bug #6068 for DBD-SQLite: UTF-8 flag not enabled ?

Mon Apr 19 09:20:03 2004 Guest - Ticket created

Subject:

UTF-8 flag not enabled ?

I am using debian testing (Perl 5.8.3, DBD-SQLite 0.29) I recompiled libssqlite0 and sqlite to work in UTF8, everything seems to goes well but When I am fetching data from SQLite, UTF-8 flag seems to be unset, and I net to do map ({Encode::_utf8_on($_);} @row); then everything is ok ($dbh->{sqlite_encoding}\n"; return 'UTF8') at the end everything is ok but I shouldn't call _utf8_on() witch is supposed to be private. despite this little thing, good job, thank you !

Mon Apr 26 01:07:31 2004 Guest - Correspondence added

[guest - Mon Apr 19 09:20:03 2004]: Show quoted text

> everything seems > to goes well but When I am fetching data from SQLite, UTF-8 flag seems > to be unset, and I net to do > map ({Encode::_utf8_on($_);} @row); > then everything is ok

can you supply a piece of code to show what is happening?

Tue Jan 04 15:28:40 2005 BULB [...] cpan.org - Correspondence added

From:

bulb [...] ucw.cz

Show quoted text

> can you supply a piece of code to show what is happening?

It pretty trivial. The strings returned from queries should have SvUTF8 flag set, but they don't. Especialy now, that sqlite3 is always in utf8 mode... (you see, it's 9 months since the original report). Anyway, if you want, I can post a small testcase. Sorry for the czech text, but I needed some non-ascii text and I happen to use cs_CZ.ISO-8859-2 locale. You don't need to understand it. Just compare whether the three strings are the same. First the string (which is UTF8, because the source does use utf8) is written to a database, then it's printed from memory and then it's fetched from database. First time without any mangling and second time the SvUTF8 flag is forced on it (utf8::decode and Encode:_utf_on are equivalent). One would expect, that the string comes out three times the same. But the second time, it causes a LOT of warnings and prints garbage. That is because though the string is in UTF-8 internaly, perl thinks it's iso-8859-1 (DBD::SQLite didn't tell it better) and tries to convert it -- which fails.

Download 2
application/x-perl 544b

Message body not shown because it is not plain text.

Tue Jan 04 15:42:43 2005 BULB [...] cpan.org - Correspondence added

Show quoted text

> > to goes well but When I am fetching data from SQLite, UTF-8 flag

seems Show quoted text

> > to be unset, and I net to do > > map ({Encode::_utf8_on($_);} @row); > > then everything is ok

dbdimp.c:493 says: SvUTF8_off(AvARRAY(av)[i]); It should not be there!

Tue Jan 04 15:49:17 2005 BULB [...] cpan.org - Correspondence added

Show quoted text

> dbdimp.c:493 says: > SvUTF8_off(AvARRAY(av)[i]); > > It should not be there!

I have to correct myself once more. SvUTF8_off is the default. Thus the line 493 (in TEXT case) must say SvUTF8_on (strings are assumed utf-8 in sqlite3), the next occurence in BLOB case is probably correct (blobs are octet-streams) and the last one (in default case) is completely bogus, because the flag does not make sense for PL_undef.

Mon Feb 21 16:50:16 2005 msergeant [...] cpan.org - Status changed from 'new' to 'resolved'