Bug #65162 for MIME-tools: Double-decoding failure in 5.500, caused by a 'fix' #63739

David, Thanks, that was quick! Show quoted text
Well, it does fix the immediate issue, although I fear it might break some expected uses (like banned filenames checks in amavisd :) The sample message (having a name: utf-8''%CE%B2CURE%2Etxt ) as returned by $head->mime_attr('content-type.name') and as $head->recommended_filename yielded a \316\262CURE.txt with 5.428, and a \x{03b2}CURE.txt with 5.500. Either of the two choices made a sensible candidate for checking rules. Now both attributes give =?UTF-8?B?zrJDVVJFLnR4dA==?=, so a caller must now do a 'MIME-Header' decoding (which is what I expected the change in #63739 tried to avoid). The docs on mime_attr() doesn't say anything about re-encoding attributes from RFC 2184 into RFC 2047 form. Perhaps we need an additional method besides mime_attr(), e.g. a mime_attr_raw, which could provide a: utf-8''%CE%B2CURE%2Etxt in case of this sample message. Then the mime_attr could continue to provide a decoded name, either as UTF-8 encoded bytes, or as perl Unicode characters. Somehow it sounds wrong that the MIME::WordDecoder::decode even tried to decode what was known to already have been decoded. Mark

On Wed, 26 Jan 2011 15:43:39 -0500 "David F. Skoll" <dfs@roaringpenguin.com> wrote: Show quoted text
Never mind; I'm an idiot... you can't change encoding within the same parameter (according to RFC 2184) Regards, David.

Show quoted text
On a second thought, your suggestion of native Perl string makes more sense to me now (not utf-8 encoded octets, and certainly not re-encoded to mime =?...? encoding). So that would be what 5.500 returns now, if it would not die. (it doesn't die if a caller supplies his own $parser->filer) Mark

Show quoted text
Well, it avoids the immediate problem, but allowing implicit nested decoding is evil. Take this one: Content-Type: text/plain; name*=''%3d%3futf-8?b?BEFORE%3f%3d The $entity->head->recommended_filename will report a filename: =?utf-8?b?BEFORE?= yet the file actually generated file will be named: %04AND.dat Mark

Show quoted text
I did. Show quoted text
There is, try it for yourself: $ perl -MMIME::Parser -e ' print MIME::Parser->new->parse(\*STDIN)->head->recommended_filename,"\n"' <1.msg =?utf-8?b?BEFORE?= $ ll -tr | tail -3 -rw-r----- 1 xxx xxx 5 Jan 27 20:13 %04AND.dat 1.msg: From: test@example.com Subject: test Date: Tue, 25 Jan 2011 14:35:04 +0100 Message-Id: <123@example.com> Content-Type: text/plain; name*=''%3d%3futf-8?b?BEFORE%3f%3d test Mark

Show quoted text
Thanks! Show quoted text
I can understand that WordDecoder.pm probably pre-dates the Perl native handling of Unicode, so its structure, documentation and examples probably reflect that. From a current point of view, its terminology can be confusing (and it was for me, took several passes of browsing through code to grasp what it is that it is supposed to do). As I (think I) understand it now, it offers a couple of simple *transcoders* with poorly defined semantics, plus one genuine decoder (the 'UTF-8'). I'll try to explain the underlying conceptual model in Perl parlance, the way I see it was supposed to be, or at least the way I would expect it (bear with me, I know the steps 2 and 3 are integrated): 1. text in RFC 2047 is *decoded* into chunks, each chunk is a pair of: a character set name (with optional language), plus a string of octets. A non- RFC 2047 string is just represented as an empty (unknown) character set name, plus a verbatim copy of octets. This first step is done by MIME::Words::decode_mimewords, called from decode(). 2. the list of chunks (chset+octets) is further *decoded* into Perl native (Unicode) characters (not bytes) (we must not care or believe-we-know-better how it is represented internally). The resulting strings from all chunks are joined, forming a correct native characters string representation of a text from a mail header field. 3. the native character string is then optionally *encoded* into a requested encoding, resulting in a string of octets, i.e. an external representation of the given text in the requested encoding. The provided 'UTF-8' "decoder" is indeed the only true *decoder*, skipping the last encoding step. It does the #1 and #2, and returns a native perl characters string. Its name 'UTF-8' is a misnomer and is misleading, one would expect from its name that it does the final step #3 and encodes into UTF-8, but it doesn't, it returns a native (Unicode) string. In my view it should be renamed to something like 'native' or 'Unicode' or 'perl string'. Remaining traditional "decoders" are actually *transcoders*. They integrate steps #2 and #3 into one operation, and do it halfheartedly: for example given an RFC 2047 text in Latin-15 and transcoding it into Latin-1, a small handful of characters does not have their equivalent and should be nuked (by '?' or whatever), the rest is fine and can be retained. For completeness one might expect there exists a 'UTF-8' *transcoder*, which would as a step #3 encode the native string into UTF-8 octets and return that. It would be even better if one could provide an arbitrary encoding (known to Perl) as an argument to such a general-purpose transcoder - it should be trivial to implement, just needs to call Encoder::encode on a native string as obtained from step #2. I think at least the documentation should be updated, some examples are misleading (like the: ### Decode a MIME string (e.g., into Latin1) via the default decoder: $str = $wd->decode('To: =?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld>'); which somehow leads a reader into thinking that the #3 encoding "into Latin1" is in any way related to the "ISO-8859-1" in the example RFC 2047 string), and perhaps the 'UTF-8' *decoder* renamed or at least documented that it is *not* a *transcoder* like all the rest, and maybe an 'UTF-8' *transcoder* implemented. Just though I should relieve it off my chest, sorry for being long and for giving you a headache :) Regards Mark

Show quoted text
Yes, that would be nice. Show quoted text
:) Show quoted text
Agreed, that's a sensible thing to do. No need to duplicate functionality offered by the Encode module. Show quoted text
Thanks for your understanding and efforts! Mark

Show quoted text
Thanks, looks good, works as expected! (tested with amavisd-new) Btw, one unrelated minor wish: could you please increase a bit the 2 kB I/O buffer size in MIME/Entity.pm and MIME/Body.pm, perhaps to 4k or 8k. For some operations I need to pass my own perl methods as a tied glob to MIME-tools, which end up being accessed (read) through MIME::Entity::print_body. Larger buffer would reduce a bit the overhead of calls for large mail. Mark

Hello, I believe the just-released 5.501 version of MIME::tools fixes this bug. Regards, David.