Bug #106930 for Net-OpenID-Consumer: Verification fail when in some fields contain unicode characters

Mon Sep 07 09:16:14 2015 oholecek [...] suse.com - Ticket created

Subject:	Verification fail when in some fields contain unicode characters
Date:	Mon, 7 Sep 2015 15:15:48 +0200
To:	<bug-Net-OpenID-Consumer [...] rt.cpan.org>
From:	Ondrej Holecek <oholecek [...] suse.com>

Hi, when some fields returned by openid server contain unicode characters subsequent HTTP verification fail (server returns is_valid: false). But when "charset=UTF-8" is added to request, verification succeed. Thanks, OH Module: perl-Net-OpenID-Consumer-1.16-2.1.noarch Distri: openSUSE Tumbleweed (20150903) Perl: v5.22.0 Example diff of my solution: diff --git a/lib/Net/OpenID/Consumer.pm b/lib/Net/OpenID/Consumer.pm index a2261cb..9a9953b 100644 --- a/lib/Net/OpenID/Consumer.pm +++ b/lib/Net/OpenID/Consumer.pm @@ -976,7 +976,7 @@ sub verified_identity { $post{"openid.mode"} = "check_authentication"; my $req = HTTP::Request->new(POST => $server); - $req->header("Content-Type" => "application/x-www-form-urlencoded"); + $req->header("Content-Type" => "application/x-www-form-urlencoded; charset=UTF-8"); $req->content(join("&", map { "$_=" . uri_escape_utf8($post{$_}) } keys %post)); my $ua = $self->ua;

Download signature.asc
application/pgp-signature 473b

Message body not shown because it is not plain text.

Fri Jan 15 06:19:54 2016 crew [...] cs.stanford.edu - Correspondence added

according to the spec, application/x-www-form-urlencoded does not actually take a charset parameter and according to https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 it's a bad idea to add one because various servers in the wild will get confused even if the one you're corresponding with happens to accept this. I'm also inclined to think this is a server failure since I'm not sure why the server *wouldn't* be decoding parameter values in OpenID messages as UTF-8 anyway, but without an actual example I'm not clear on what's actually happening in your case.

Fri Jan 15 06:19:55 2016 The RT System itself - Status changed from 'new' to 'open'

Fri Jan 15 08:39:26 2016 oholecek [...] suse.com - Correspondence added

Subject:	Re: [rt.cpan.org #106930] Verification fail when in some fields contain unicode characters
Date:	Fri, 15 Jan 2016 14:38:47 +0100
To:	bug-Net-OpenID-Consumer [...] rt.cpan.org
From:	Ondrej Holecek <oholecek [...] suse.com>

Show quoted text

> according to the spec, > application/x-www-form-urlencoded does not actually take a charset parameter > and according to https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 > it's a bad idea to add one because various servers in the wild will get > confused even if the one you're corresponding with happens to accept this.

Make sense. Do you have some other suggestions how to explicitly inform server it's in utf-8 encoding? Show quoted text

> > I'm also inclined to think this is a server failure since I'm not sure why > the server *wouldn't* be decoding parameter values in OpenID messages as > UTF-8 anyway, but without an actual example I'm not clear on what's > actually happening in your case.

I don't have access to server configuration. OpenID provider in question was https://www.opensuse.org/openid/user/ if you want to check. The actual problem was that some peoples names contain non ascii characters and when they tried to log in then request verification failed.

Download signature.asc
application/pgp-signature 473b

Message body not shown because it is not plain text.

Fri Jan 15 18:42:00 2016 crew [...] cs.stanford.edu - Correspondence added

Two possibilities: (1) the HTML5 solution to this is evidently to add a hidden field named '_charset_' to the form; browser then fills this in when POSTing and server is supposed to check for its existence in a POST request and decode accordingly. I'm guessing legacy OpenID servers are unlikely to know about this but it may be worth a try. (if it works it's probably the simplest solution). (2) Use multipart/form-data instead of application/x-www-form-urlencoded. This allows individual Content-Type: headers on each of the fields and seems to be the Right Thing but getting LWP::UserAgent to actually do it looks cumbersome.

Thu Nov 21 15:40:29 2019 petr [...] tesarici.cz - Correspondence added

On Mon Sep 07 09:16:14 2015, oholecek@suse.com wrote: Show quoted text

> Hi, > > when some fields returned by openid server contain unicode characters > subsequent HTTP verification fail (server returns is_valid: false). > But when > "charset=UTF-8" is added to request, verification succeed.

I don't think this analysis is correct. I have done some research, and the issue seems to be that the server returns UTF-8 data in a URL-encoded query. Since you're running a CGI script, this data is passed through an environment variable (QUERY_STRING). The CGI module does not treat that data as UTF-8. It cannot do that, because a query argument may in fact be an arbitrary byte stream. Net::OpenID does not that data either, so the UTF-8 encoded byte stream is treated as a Latin1 string, and that's why it fails. I can make it work by reverting commit c392be3a2180b742e940f8ccd42bedcb8e8e6cd6, but it seems Evgeniy had a different setup that was fixed by that commit. See also RT 80329.

Thu Nov 21 15:46:07 2019 petr [...] tesarici.cz - Correspondence added

On Thu Nov 21 15:40:29 2019, petr@tesarici.cz wrote: Show quoted text

> On Mon Sep 07 09:16:14 2015, oholecek@suse.com wrote:

> > Hi, > > > > when some fields returned by openid server contain unicode characters > > subsequent HTTP verification fail (server returns is_valid: false). > > But when > > "charset=UTF-8" is added to request, verification succeed.

> > I don't think this analysis is correct. I have done some research, and > the issue seems to be that the server returns UTF-8 data in a URL- > encoded query. Since you're running a CGI script, this data is passed > through an environment variable (QUERY_STRING). The CGI module does > not treat that data as UTF-8. It cannot do that, because a query > argument may in fact be an arbitrary byte stream. Net::OpenID does not > that data either, so the UTF-8 encoded byte stream is treated as a > Latin1 string, and that's why it fails. > > I can make it work by reverting commit > c392be3a2180b742e940f8ccd42bedcb8e8e6cd6, but it seems Evgeniy had a > different setup that was fixed by that commit. See also RT 80329.

Um, actually, no, to make things work I need both: reverting commit c392be3a _AND_ adding charset=UTF-8. At this point, I think the situation with www.opensuse.org is hopeless.