Skip Menu |

This queue is for tickets about the DBD-mysql CPAN distribution.

Report information
The Basics
Id: 17829
Status: resolved
Worked: 3 hours (180 min)
Priority: 0/
Queue: DBD-mysql

People
Owner: Nobody in particular
Requestors: geoff [...] laxan.com
Cc: ABH [...] cpan.org
JETEVE [...] cpan.org
AdminCc:

Bug Information
Severity: Important
Broken in: 3.0002
Fixed in: (no value)



Subject: No UTF-8 support
DBD::mysql provides no way to get values from the database as strings with the UTF8 flag set. There should be an option which can be passed to connect() to make this happen when appropriate. Of course, the strings may or may not be UTF8. One possible solution might be to have a single option which sets the conncetion character set to UTF8 like this: $dbh->do("set names 'utf8'"); and then turns on UTF8 flags on strings returned from SELECT statements. MySQL will convert everything it sends to UTF-8, so it would then be appropriate to pass things off to Perl with the UTF8 flag on. Note that DBD::Pg has a feature for turning on UTF8 flags, so code could be borrowed from there. According to the Changes file it was added in version 1.22.
There's also a suggested patch at http://lists.mysql.com/perl/3786
This issue is also tracked at http://bugs.mysql.com/18904
Hi all. I found a patch for the version 3.0002_6 here: http://lists.mysql.com/perl/3786 Unfortunately, it was for a old version. I used this patch as a basis to make a patch for the last stable version: 3.0006 . Usage: [jerome@saab todif]$ ls DBD-mysql-3.0006 DBD-mysql-3.0006.tar.gz DBD-mysql-3.0006_utf8.patch [jerome@saab todif]$ patch -p0 < DBD-mysql-3.0006_utf8.patch patching file DBD-mysql-3.0006/dbdimp.c patching file DBD-mysql-3.0006/dbdimp.h patching file DBD-mysql-3.0006/INSTALL.html patching file DBD-mysql-3.0006/lib/DBD/mysql.pm patching file DBD-mysql-3.0006/META.yml patching file DBD-mysql-3.0006/t/._prepare_noerror.t Then compile/install the stuff and use the new option: $dbh->{'mysql_enable_utf8'} = 1 ; To have the utf8 string tagged as utf8/unicode ! I guess soon that feature will be included in the next official release, but while waiting you can use this ! J.
diff -NBaur DBD-mysql-3.0006/dbdimp.c DBD-mysql-3.0006_utf8/dbdimp.c --- DBD-mysql-3.0006/dbdimp.c 2006-06-10 00:47:26.000000000 +0100 +++ DBD-mysql-3.0006_utf8/dbdimp.c 2006-08-10 17:38:43.000000000 +0100 @@ -1379,6 +1380,22 @@ } #endif + + /* UTF8 */ +#if defined(sv_utf8_decode) && MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION + + if ((svp = hv_fetch(hv, "mysql_enable_utf8", 17, FALSE)) && *svp) { + /* Do not touch imp_dbh->enable_utf8 as we are called earlier + * than it is set and mysql_options() must be before: + * mysql_real_connect() + */ + mysql_options(sock, MYSQL_SET_CHARSET_NAME, + (SvTRUE(*svp) ? "utf8" : "latin1")); + } +#endif + /* END OF UTF8 */ + + #if defined(DBD_MYSQL_WITH_SSL) && !defined(DBD_MYSQL_EMBEDDED) && \ (defined(CLIENT_SSL) || (MYSQL_VERSION_ID >= 40000)) if ((svp = hv_fetch(hv, "mysql_ssl", 9, FALSE)) && *svp) @@ -1579,6 +1596,12 @@ /* Safer we flip this to TRUE perl side if we detect a mod_perl env. */ imp_dbh->auto_reconnect = FALSE; + /* UTF8 */ +#if defined(sv_utf8_decode) && MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION + imp_dbh->enable_utf8 = FALSE; /* initialize mysql_enable_utf8 */ +#endif + /* END OF UTF8 */ + if (!my_login(dbh, imp_dbh)) { do_error(dbh, mysql_errno(&imp_dbh->mysql), @@ -1897,6 +1920,16 @@ else if (kl == 31 && strEQ(key,"mysql_unsafe_bind_type_guessing")) imp_dbh->bind_type_guessing = SvIV(valuesv); + + /* UTF8 */ +#if defined(sv_utf8_decode) && MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION + else if (kl == 17 && strEQ(key, "mysql_enable_utf8")){ + imp_dbh->enable_utf8 = bool_value; + } +#endif + + /* END OF UTF8 */ + else return FALSE; /* Unknown key */ @@ -1987,6 +2020,15 @@ const char* msg = mysql_error(&imp_dbh->mysql); result= sv_2mortal(newSVpv(msg, strlen(msg))); } + + /* UTF8 */ +#if defined(sv_utf8_decode) && MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION + else if (kl == strlen("enable_utf8") && strEQ(key, "enable_utf8")){ + result = sv_2mortal(newSViv(imp_dbh->enable_utf8)); + } +#endif + /* END OF UTF8 */ + break; case 'd': @@ -2787,6 +2829,7 @@ MYSQL_ROW cols; imp_sth_fbh_t *fbh; + #if MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION MYSQL_BIND *bind; #endif @@ -2923,6 +2966,12 @@ if (dbis->debug >= 2) PerlIO_printf(DBILOGFP, "st_fetch string data %s\n", fbh->data); sv_setpvn(sv, fbh->data, fbh->length); + /* UTF8 */ +#ifdef sv_utf8_decode + if(imp_dbh->enable_utf8) + sv_utf8_decode(sv); +#endif + /* END OF UTF8 */ break; default: @@ -2974,6 +3023,12 @@ { --len; } } sv_setpvn(sv, col, len); + /* UTF8 */ +#ifdef sv_utf8_decode + if(imp_dbh->enable_utf8) + sv_utf8_decode(sv); +#endif + /* END OF UTF8 */ } else (void) SvOK_off(sv); /* Field is NULL, return undef */ @@ -3930,6 +3985,13 @@ ptr= SvPV(str, len); result= newSV(len*2+3); + + /* UTF8 */ +#ifdef SvUTF8 + if (SvUTF8(str)) SvUTF8_on(result); +#endif + /* END OF UTF8 */ + sptr= SvPVX(result); *sptr++ = '\''; diff -NBaur DBD-mysql-3.0006/dbdimp.h DBD-mysql-3.0006_utf8/dbdimp.h --- DBD-mysql-3.0006/dbdimp.h 2006-05-17 23:29:40.000000000 +0100 +++ DBD-mysql-3.0006_utf8/dbdimp.h 2006-08-10 14:49:24.000000000 +0100 @@ -139,6 +139,12 @@ */ int use_server_side_prepare; int has_autodetect_prepare; + + /* UTF8 */ +#if defined(sv_utf8_decode) && MYSQL_VERSION_ID >=SERVER_PREPARE_VERSION + bool enable_utf8; +#endif + /* END OF UTF8 */ }; diff -NBaur DBD-mysql-3.0006/INSTALL.html DBD-mysql-3.0006_utf8/INSTALL.html --- DBD-mysql-3.0006/INSTALL.html 2006-06-10 17:46:45.000000000 +0100 +++ DBD-mysql-3.0006_utf8/INSTALL.html 2006-08-10 17:49:54.000000000 +0100 @@ -2,7 +2,7 @@ <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>INSTALL - How to install and configure DBD::mysql</title> -<link rev="made" href="mailto:root@b28.apple.com" /> +<link rev="made" href="mailto:root@localhost" /> </head> <body style="background-color: white"> diff -NBaur DBD-mysql-3.0006/lib/DBD/mysql.pm DBD-mysql-3.0006_utf8/lib/DBD/mysql.pm --- DBD-mysql-3.0006/lib/DBD/mysql.pm 2006-06-10 17:45:40.000000000 +0100 +++ DBD-mysql-3.0006_utf8/lib/DBD/mysql.pm 2006-08-10 17:40:08.000000000 +0100 @@ -9,7 +9,7 @@ use Carp (); @ISA = qw(DynaLoader); -$VERSION = '3.0006'; +$VERSION = '3.0006_utf8'; bootstrap DBD::mysql $VERSION; @@ -952,6 +952,20 @@ It is possible to set/unset the C<mysql_use_result> attribute after creation of statement handle. See below. +=item mysql_enable_utf8 + +This attribute determines whether DBD::mysql should assume strings +stored in the database are utf8. This feature defaults to off. + +When set, a data retrieved from a textual column type (char, varchar, +etc) will have the UTF-8 flag turned on if necessary. This enables +character semantics on that string. + +Additionally, turning on this flag tells MySQL that incoming data should +be treated as UTF-8. This will only take effect if used as part of the +call to connect(). + +This option is experimental and may change in future versions. =head1 STATEMENT HANDLES diff -NBaur DBD-mysql-3.0006/META.yml DBD-mysql-3.0006_utf8/META.yml --- DBD-mysql-3.0006/META.yml 2006-06-10 17:46:45.000000000 +0100 +++ DBD-mysql-3.0006_utf8/META.yml 2006-08-10 17:49:54.000000000 +0100 @@ -1,7 +1,7 @@ # http://module-build.sourceforge.net/META-spec.html #XXXXXXX This is a prototype!!! It will change in the future!!! XXXXX# name: DBD-mysql -version: 3.0006 +version: 3.0006_utf8 version_from: lib/DBD/mysql.pm installdirs: site requires: diff -NBaur DBD-mysql-3.0006/t/._prepare_noerror.t DBD-mysql-3.0006_utf8/t/._prepare_noerror.t --- DBD-mysql-3.0006/t/._prepare_noerror.t 2006-05-03 19:42:43.000000000 +0100 +++ DBD-mysql-3.0006_utf8/t/._prepare_noerror.t 1970-01-01 01:00:00.000000000 +0100 @@ -1 +0,0 @@ -Mac OS X  2 RBINAhDmp \ No newline at end of file
Thanks for this patch, sorry for taking a long time to get to adding it! Working on adding this to the latests release. On Thu Aug 10 13:16:47 2006, JETEVE wrote: Show quoted text
> Hi all. > I found a patch for the version 3.0002_6 here: > http://lists.mysql.com/perl/3786 > > Unfortunately, it was for a old version. > I used this patch as a basis to make a patch for the last stable version: > 3.0006 . > > Usage: > [jerome@saab todif]$ ls > DBD-mysql-3.0006 DBD-mysql-3.0006.tar.gz DBD-mysql-3.0006_utf8.patch > [jerome@saab todif]$ patch -p0 < DBD-mysql-3.0006_utf8.patch > patching file DBD-mysql-3.0006/dbdimp.c > patching file DBD-mysql-3.0006/dbdimp.h > patching file DBD-mysql-3.0006/INSTALL.html > patching file DBD-mysql-3.0006/lib/DBD/mysql.pm > patching file DBD-mysql-3.0006/META.yml > patching file DBD-mysql-3.0006/t/._prepare_noerror.t > > Then compile/install the stuff and use the new option: > $dbh->{'mysql_enable_utf8'} = 1 ; > To have the utf8 string tagged as utf8/unicode ! > > I guess soon that feature will be included in the next official release, > but while waiting you can use this ! > > J. >
On Fri Apr 07 00:34:31 2006, ABH wrote: Show quoted text
> There's also a suggested patch at > http://lists.mysql.com/perl/3786
Hi! Sorry to take so long to get into this patch! I applied this patch to 3.0006_1, and have this failure on the utf test: 104:~/DBD/subversion/DBD-mysql/branches/Dev-3_0 patg$ PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/utf8.t t/utf8....Wide character in print at t/lib.pl line 258. t/utf8....FAILED test 10 Failed 1/13 tests, 92.31% okay Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- t/utf8.t 13 1 7.69% 10 Failed 1/1 test scripts, 0.00% okay. 1/13 subtests failed, 92.31% okay. It's this test: # Finally, check that we got back UTF-8 correctly. Test( $state or ($ref->[0] eq $utf8_str) ) or ErrMsgF( "got back '$ref->[0]' instead of '$utf8_str'.\n" ); Any ideas?
On Tue Aug 15 13:36:24 2006, CAPTTOFU wrote: Show quoted text
> On Fri Apr 07 00:34:31 2006, ABH wrote:
> > There's also a suggested patch at > > http://lists.mysql.com/perl/3786
> > Hi! Sorry to take so long to get into this patch! I applied this patch > to 3.0006_1, and have this failure on the utf test: > > 104:~/DBD/subversion/DBD-mysql/branches/Dev-3_0 patg$ > PERL_DL_NONLAZY=1 > /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, > 'blib/lib', 'blib/arch')" t/utf8.t > t/utf8....Wide character in print at t/lib.pl line 258. > t/utf8....FAILED test 10
Try to add: binmode STDOUT , ':utf8' ;
I have a patch from Dominic Mitchell that I'm adding to the upcoming release 3.0008 later this week that will solve this. regards, Patrick On Thu Feb 23 10:05:51 2006, GEOFFR wrote: Show quoted text
> DBD::mysql provides no way to get values from the database as strings > with the UTF8 flag set. There should be an option which can be passed > to connect() to make this happen when appropriate. > > Of course, the strings may or may not be UTF8. One possible solution > might be to have a single option which sets the conncetion character set > to UTF8 like this: > > $dbh->do("set names 'utf8'"); > > and then turns on UTF8 flags on strings returned from SELECT statements. > MySQL will convert everything it sends to UTF-8, so it would then be > appropriate to pass things off to Perl with the UTF8 flag on. > > Note that DBD::Pg has a feature for turning on UTF8 flags, so code could > be borrowed from there. According to the Changes file it was added in > version 1.22.
CC: GEOFFR [...] cpan.org, JETEVE [...] cpan.org
Subject: Re: [rt.cpan.org #17829] No UTF-8 support
Date: Mon, 2 Oct 2006 15:03:38 -0700
To: bug-DBD-mysql [...] rt.cpan.org
From: Ask Bjørn Hansen <ask [...] perl.org>
On Oct 2, 2006, at 3:02 PM, Patrick Galbraith via RT wrote: Show quoted text
> I have a patch from Dominic Mitchell that I'm adding to the upcoming > release 3.0008 later this week that will solve this.
That's great news, thanks Patrick! - ask -- http://askask.com/ - http://develooper.com/
Patrick applied the patch from Dominic, and it's in the latest release. Therefore, I'm closing this ticket.