Bug #21288 for MediaWiki: Unicode/UTF8 not fully handled

Sat Sep 02 16:22:46 2006 stuart.caie [...] gmail.com - Ticket created

Subject:

Unicode/UTF8 not fully handled

There are a lot of places where strings with UTF8 characters are included are not passed on correctly to the wiki. libwww-perl does not promise to do any conversion of data, it just takes the raw byte values of data sent. So in order to pass UTF8-encoded data, it has to be done by the caller. I would appreciate if I did not explicitly have to encode utf8 data before passing it to MediaWiki, I think MediaWiki should do this itself. For example, a call to upload currently looks like this: $BOT->upload(encode_utf8($image_name), $image_data, encode_utf8($description)); It would be good if it was simply $BOT->upload($image_name, $image_data, $description);

Thu Sep 07 12:54:13 2006 edwardspec [...] gmail.com - Correspondence added

On Суб. Сен. 02 16:22:46 2006, stuart.caie@gmail.com wrote: Show quoted text

> There are a lot of places where strings with UTF8 characters are > included are not passed on correctly to the wiki. > > libwww-perl does not promise to do any conversion of data, it just takes > the raw byte values of data sent. So in order to pass UTF8-encoded data, > it has to be done by the caller. I would appreciate if I did not > explicitly have to encode utf8 data before passing it to MediaWiki, I > think MediaWiki should do this itself. > > For example, a call to upload currently looks like this: > > $BOT->upload(encode_utf8($image_name), $image_data, > encode_utf8($description)); > > It would be good if it was simply $BOT->upload($image_name, $image_data, > $description);

The only place to fix was in MediaWiki::page::_wiki_url(). There is no need in escaping description - it's not done by LWP but this is handled in HTTP::Request::Common (in this syntax: my $res = $obj->{ua}->request( POST $url, Content => [( $key => $val, # both $key and $val are escaped $key2 => $val2 # both $key2 and $val2 are escaped )] ); Fixed version will be included into 1.09 version.

Thu Sep 07 12:54:16 2006 The RT System itself - Status changed from 'new' to 'open'

Thu Sep 07 12:54:17 2006 edwardspec [...] gmail.com - Status changed from 'open' to 'resolved'

Thu Sep 07 12:54:20 2006 edwardspec [...] gmail.com - Given to SPECTRUM

Sat Jan 27 20:33:39 2007 stuart.caie [...] gmail.com - Correspondence added

From:

stuart.caie [...] gmail.com

On Thu Sep 07 12:54:13 2006, SPECTRUM wrote: Show quoted text

> The only place to fix was in MediaWiki::page::_wiki_url(). There is no > need in escaping description - it's not done by LWP but this is handled > in HTTP::Request::Common (in this syntax: > my $res = $obj->{ua}->request( > POST $url, > Content => [( > $key => $val, # both $key and $val are escaped > $key2 => $val2 # both $key2 and $val2 are escaped > )] > );

I checked further into this - it is not handled by HTTP::Request::Common. It is bug #15294 in the URI module (http://rt.cpan.org/Public/Bug/Display.html?id=15294) Firstly, you ask that HTTP::Request::Common encode using urlencoding (application/x-www-form-urlencoded). To get the urlencoded form, HTTP::Request::Common will create a URI object and use query_form() on it. URI::query_form() passes elements to URI::Escape::uri_escape() As noted in the URI::Escape documentation, this is the absolutely wrong thing to do for unicode. Implementing the patch given in bug #15294 and also adding "use encoding 'utf8';" to the head of the script fixes the problem. I would really appreciate if you could let people know in your documentation that they need to do these two things to get UTF-8 encoding working with this module.

Sat Jan 27 20:33:41 2007 The RT System itself - Status changed from 'resolved' to 'open'