Bug #57945 for CGI: Charset is only added to text/* types

On Fri May 28 08:37:50 2010, mark@summersault.com wrote: Show quoted text
Hmm. Sorry for the delay. I don't get email on changes (having used OpenID) and I can't see a "profile" or similar link in RT where I can add an email address... Here are some bits of relevant text: RFC 2616 (HTTP), section 3.4.1: http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html "3.4.1 Missing Charset Some HTTP/1.0 software has interpreted a Content-Type header without charset parameter incorrectly to mean "recipient should guess." Senders wishing to defeat this behavior MAY include a charset parameter even when the charset is ISO-8859-1 and SHOULD do so when it is known that it will not confuse the recipient. Unfortunately, some older HTTP/1.0 clients did not deal properly with an explicit charset parameter. HTTP/1.1 recipients MUST respect the charset label provided by the sender; and those user agents that have a provision to "guess" a charset MUST use the charset from the content-type field if they support that charset, rather than the recipient's preference, when initially displaying a document. See section 3.7.1." Section 3.7.1 states, in part: "The "charset" parameter is used with some media types to define the character set (section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value." RFC 4627 (JSON) says: http://www.ietf.org/rfc/rfc4627.txt "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8." It then talks about sniffing the stream to be able to tell if other encodings have, in fact, been used. Ick. So one could argue: "Types other than text types are supposed to have a canonical representation defined when they are registered. For JSON, the canonical representation is "Unicode", and the defined method of determining the character set is by sniffing the first four bytes of the stream." Or one could argue: "My word, that's horrible. Add a charset parameter, for the love of Pete!" As for XML, the situation is more clear. RFC 3023 says: http://www.rfc-editor.org/rfc/rfc3023.txt "3.2 Application/xml Registration MIME media type name: application MIME subtype name: xml Mandatory parameters: none Optional parameters: charset Although listed as an optional parameter, the use of the charset parameter is STRONGLY RECOMMENDED, ..." So that's at least one non-text type where a charset should be added. In general, I suspect most XML MIME types (and there are _a_lot_) have this attitude to charsets. People need to be able to set this parameter. Do you need more from me here? Gerv

On Mon Jun 14 16:42:15 2010, mark@summersault.com wrote: Show quoted text
Aye, aye, sir. :-) I'll look more closely at it when I have a few minutes to myself, but I'm thinking that the fix could be to change at CGI.pm ~ line 1561 if (defined $charset) { $self->charset($charset); } else { $charset = $self->charset if $type =~ /^text\//; } for if (defined $charset) { $self->charset($charset); } else { $charset = $self->charset; } In other words, adopt the chartset if it has been configured, no matter what the type is. The logic being that if somebody doesn't want the charset to be specified, he or she would not set it to any value in the first place. Anyway, I'll try to come with a patch and some tests for it soonishly.

Changes committed to http://github.com/yanick/CGI.pm/tree/charset The only thing I'm afraid of is apps that might freak upon receiving charsets for some mimetypes. Reading the HTTP specs at http://www.w3.org/Protocols/rfc2616/rfc2616- sec3.html, we have (as was quoted by Gerv) =head1 SPECS Some HTTP/1.0 software has interpreted a Content-Type header without charset parameter incorrectly to mean "recipient should guess." Senders wishing to defeat this behavior MAY include a charset parameter even when the charset is ISO-8859-1 and SHOULD do so when it is known that it will not confuse the recipient. Unfortunately, some older HTTP/1.0 clients did not deal properly with an explicit charset parameter. HTTP/1.1 recipients MUST respect the charset label provided by the sender; and those user agents that have a provision to "guess" a charset MUST use the charset from the content-type field if they support that charset, rather than the recipient's preference, when initially displaying a document. See section 3.7.1. =cut The first paragraph seems to advocate the explicit mention of the charset (yay!), but the second one warns that some clients can indeed not like it. Now, new code can easily circumvent that problem by calling header( -charset => '' ); but old code will suddenly behave differently than it did before. Does anyone think that this could be an issue for someone out there? `/anick

On June 18, 2010 09:27:38 am mark@summersault.com via RT wrote: Show quoted text
Sounds very reasonable, very prudent. I whole-heartily agree. Joy, `/anick

Gerv, So the summary for now is that we don't plan to modify CGI.pm immediately for this. If more of consensus arises that this should be changed-- and doing so would be more spec-compliant or real-world-compliant, we'll consider revisiting it. We'll leave the ticket open so that others might find it, and you can also promote the change others to gather support for it if you'd like. Thanks again for the feedback on CGI.pm. Mark

On Tue Jul 20 00:07:33 2010, http://www.gerv.net/ wrote: Show quoted text
about Show quoted text
I see. In that case, Yanick's proposed fix is the same thing that I would recommend. I've added some more tests for charset() now and merged and pushed Yanick's work. It should be in our next release. Mark

Thanks for the bug report. A patch for it appeared in 3.51, if not sooner. Resolving. Mark