Bug #111853 for Encode: Encode >= 2.80 breaks HTML-DOM-0.055

Sun Feb 07 12:50:50 2016 SREZIC [...] cpan.org - Ticket created

Subject:

Encode >= 2.80 breaks HTML-DOM-0.055

See https://rt.cpan.org/Ticket/Display.html?id=111824 The test failures look like this: expected: # Content-Disposition: form-data; name="foo" got: # Content-Disposition: form-data; name="=?UTF-8?B?Zm9v?=" And while the got result is not exactly wrong, I don't think that utf-8+base64 should be used here where plain ascii would also work.

Sun Feb 07 12:51:09 2016 SREZIC [...] cpan.org - Reference to ticket #111824 added

Tue Feb 16 02:06:10 2016 SREZIC [...] cpan.org - Reference by ticket #112063 added

Sun Feb 21 15:07:10 2016 SREZIC [...] cpan.org - Reference by ticket #112231 added

Tue Feb 23 15:00:50 2016 SREZIC [...] cpan.org - Correspondence added

On 2016-02-07 12:50:50, SREZIC wrote: Show quoted text

> See https://rt.cpan.org/Ticket/Display.html?id=111824 > > The test failures look like this: > > expected: > > # Content-Disposition: form-data; name="foo" > > got: > > # Content-Disposition: form-data; name="=?UTF-8?B?Zm9v?=" > > > And while the got result is not exactly wrong, I don't think that utf- > 8+base64 should be used here where plain ascii would also work.

Also affected (possibly): https://github.com/theory/svn-notify/issues/14

Wed Feb 24 14:46:23 2016 dwheeler [...] cpan.org - Requestor DWHEELER added

Fri Mar 25 03:48:43 2016 SREZIC [...] cpan.org - Broken in 2.83 added

Fri Mar 25 03:49:58 2016 SREZIC [...] cpan.org - Reference by ticket #113333 added

Fri Mar 25 08:50:34 2016 pali [...] cpan.org - Correspondence added

On Ned Feb 07 12:50:50 2016, SREZIC wrote: Show quoted text

> The test failures look like this: > > expected: > > # Content-Disposition: form-data; name="foo" > > got: > > # Content-Disposition: form-data; name="=?UTF-8?B?Zm9v?="

This looks like a broken test. Name attribute in Content-Disposition header must be us-ascii, not MIME. So it looks like you try to MIME encode that attribute which is wrong.

Fri Mar 25 08:50:35 2016 The RT System itself - Status changed from 'new' to 'open'

Fri Mar 25 18:52:53 2016 dwheeler [...] cpan.org - Correspondence added

On 2016-03-25 08:50:34, PALI wrote: Show quoted text

> This looks like a broken test. Name attribute in Content-Disposition > header must be us-ascii, not MIME. So it looks like you try to MIME > encode that attribute which is wrong.

Well, the SVN::Notify test failure referenced by SREZIC is: # Subject: =?UTF-8?Q?=5B111=5D_Did_this,_that,_and_the_=C2=ABother=C2=BB.?=' # # doesn't match '(?^:Subject: \[111\] \=\?UTF\-8\?Q\?Did_this\,_that\,_and_the_\=C2\=ABother\=C2\=BB\.\?\=\n)' Here's where it uses Encode to MIME-Q encode the subject and other headers: https://github.com/theory/svn-notify/blob/master/lib/SVN/Notify.pm#L1464

Sun Mar 27 20:53:38 2016 pali [...] cpan.org - Cc PALI added

Sun Mar 27 21:00:06 2016 pali [...] cpan.org - Correspondence added

On Pia mar 25 18:52:53 2016, DWHEELER wrote: Show quoted text

> On 2016-03-25 08:50:34, PALI wrote: >

> > This looks like a broken test. Name attribute in Content-Disposition > > header must be us-ascii, not MIME. So it looks like you try to MIME > > encode that attribute which is wrong.

> > Well, the SVN::Notify test failure referenced by SREZIC is: > > # Subject: =?UTF- > 8?Q?=5B111=5D_Did_this,_that,_and_the_=C2=ABother=C2=BB.?=' > # > # doesn't match '(?^:Subject: \[111\] \=\?UTF\- > 8\?Q\?Did_this\,_that\,_and_the_\=C2\=ABother\=C2\=BB\.\?\=\n)'

This looks correct.. or where do you see problem? Show quoted text

> Here's where it uses Encode to MIME-Q encode the subject and other > headers: > > https://github.com/theory/svn- > notify/blob/master/lib/SVN/Notify.pm#L1464

Looks like that test try to encode whole "from" and "to" headers. That would produce incorrect email. Only phrase part(s) of "from" and "to" headers must be MIME encoded -- not email address itself.

Mon Mar 28 13:33:05 2016 dwheeler [...] cpan.org - Correspondence added

CC:	SREZIC [...] cpan.org, pali [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Mon, 28 Mar 2016 10:32:58 -0700
To:	bug-Encode [...] rt.cpan.org
From:	"David E. Wheeler" <dwheeler [...] cpan.org>

On Mar 27, 2016, at 6:00 PM, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> This looks correct.. or where do you see problem?

The test passed for like 8 years and suddenly started failing. Did something change in the MIME-Q encoder? Only change I see matching “MIME-Q" in Changes dates from 2009. I can probably change the test to use encode instead of its own regex. It’s just odd that it started breaking after all these years. Show quoted text

> Looks like that test try to encode whole "from" and "to" headers. That would produce incorrect email. Only phrase part(s) of "from" and "to" headers must be MIME encoded -- not email address itself.

Good to know, thanks. David

Mon Mar 28 13:41:07 2016 pali [...] cpan.org - Correspondence added

CC:	bug-Encode [...] rt.cpan.org, SREZIC [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Mon, 28 Mar 2016 19:40:37 +0200
To:	"David E. Wheeler" <dwheeler [...] cpan.org>
From:	pali [...] cpan.org

On Monday 28 March 2016 19:32:58 David E. Wheeler wrote: Show quoted text

> On Mar 27, 2016, at 6:00 PM, Pali via RT <bug-Encode@rt.cpan.org> wrote:

> > This looks correct.. or where do you see problem?

> > The test passed for like 8 years and suddenly started failing. Did > something change in the MIME-Q encoder? Only change I see matching > “MIME-Q" in Changes dates from 2009.

Yes, finally after years in Encode version 2.83 I fixed Encode::MIME::Header package to correctly encode and decode strings... Show quoted text

> I can probably change the test to use encode instead of its own > regex. It’s just odd that it started breaking after all these years.

Beware that parsing some of email headers is not easy by regexes... Show quoted text

> > Looks like that test try to encode whole "from" and "to" headers. > > That would produce incorrect email. Only phrase part(s) of "from" > > and "to" headers must be MIME encoded -- not email address itself.

> > Good to know, thanks. > > David

Mon Mar 28 13:42:50 2016 dwheeler [...] cpan.org - Correspondence added

CC:	SREZIC [...] cpan.org, pali [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Mon, 28 Mar 2016 10:42:45 -0700
To:	bug-Encode [...] rt.cpan.org
From:	"David E. Wheeler" <dwheeler [...] cpan.org>

On Mar 28, 2016, at 10:41 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> Yes, finally after years in Encode version 2.83 I fixed > Encode::MIME::Header package to correctly encode and decode strings...

Actually, it already does that. The test sets up the subject like this: my $subj = "Did this, that, and the «other»."; my $qsubj; if (SVN::Notify::PERL58()) { $subj = Encode::decode_utf8( $subj ); $qsubj = quotemeta Encode::encode( 'MIME-Q', $subj ); } else { $qsubj = quotemeta $subj; } So I don’t understand why the test fails. Both the test and the code Q-encode the UTF-8 subject. Best, David

Mon Mar 28 13:50:01 2016 pali [...] cpan.org - Cc PALI deleted

Mon Mar 28 13:51:24 2016 pali [...] cpan.org - Correspondence added

CC:	bug-Encode [...] rt.cpan.org, SREZIC [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Mon, 28 Mar 2016 19:51:02 +0200
To:	"David E. Wheeler" <dwheeler [...] cpan.org>
From:	pali [...] cpan.org

On Monday 28 March 2016 19:42:45 David E. Wheeler wrote: Show quoted text

> On Mar 28, 2016, at 10:41 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote:

> > Yes, finally after years in Encode version 2.83 I fixed > > Encode::MIME::Header package to correctly encode and decode > > strings...

> > Actually, it already does that. The test sets up the subject like > this: > > my $subj = "Did this, that, and the «other»."; > my $qsubj; > if (SVN::Notify::PERL58()) { > $subj = Encode::decode_utf8( $subj ); > $qsubj = quotemeta Encode::encode( 'MIME-Q', $subj ); > } else { > $qsubj = quotemeta $subj; > } > > So I don’t understand why the test fails. Both the test and the code > Q-encode the UTF-8 subject. > > Best, > > David

And what does that test compare?

Mon Mar 28 13:54:05 2016 dwheeler [...] cpan.org - Correspondence added

CC:	SREZIC [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Mon, 28 Mar 2016 10:53:57 -0700
To:	bug-Encode [...] rt.cpan.org
From:	"David E. Wheeler" <dwheeler [...] cpan.org>

On Mar 28, 2016, at 10:51 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> And what does that test compare?

The failing test: like( $email, qr/Subject: \[111\] $qsubj\n/, "Check subject" ); David

Tue Mar 29 04:42:04 2016 pali [...] cpan.org - Correspondence added

RT-Send-CC:

pali [...] cpan.org

On Mon Mar 28 13:54:05 2016, DWHEELER wrote: Show quoted text

> On Mar 28, 2016, at 10:51 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote: >

> > And what does that test compare?

> > The failing test: > > like( $email, qr/Subject: \[111\] $qsubj\n/, "Check subject" ); > > David > >

Hi! Ideally post me whole test as I still do not see what there happen...

Tue Mar 29 12:29:02 2016 dwheeler [...] cpan.org - Correspondence added

CC:	SREZIC [...] cpan.org
Subject:	Re: [rt.cpan.org #111853] Encode >= 2.80 breaks HTML-DOM-0.055
Date:	Tue, 29 Mar 2016 09:28:50 -0700
To:	bug-Encode [...] rt.cpan.org
From:	"David E. Wheeler" <dwheeler [...] cpan.org>

On Mar 29, 2016, at 1:42 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> Hi! Ideally post me whole test as I still do not see what there happen...

Probably easiest just to clone and run: git clone https://github.com/theory/svn-notify.git cd svn-notify prove -lv t/base.t Best, David

Tue Mar 29 14:16:41 2016 pali [...] cpan.org - Cc PALI added

Tue Mar 29 14:17:23 2016 pali [...] cpan.org - Correspondence added

On Uto mar 29 12:29:02 2016, DWHEELER wrote: Show quoted text

> On Mar 29, 2016, at 1:42 AM, Pali via RT <bug-Encode@rt.cpan.org> wrote: >

> > Hi! Ideally post me whole test as I still do not see what there happen...

> > Probably easiest just to clone and run: > > git clone https://github.com/theory/svn-notify.git > cd svn-notify > prove -lv t/base.t > > Best, > > David >

This this patch: https://github.com/theory/svn-notify/pull/15

Fri Apr 01 15:07:25 2016 pali [...] cpan.org - Correspondence added

So problem was in svn-notify (now fixed), not in Encode::MIME::Header. And I bet same would be also for HTML-DOM-0.055: incorrect usage of MIME-Header.

Sat Jun 25 06:04:11 2016 pali [...] cpan.org - Correspondence added

Bug for HTML-DOM is now closed, so close also this one.

Sat Jun 25 07:11:27 2016 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'