Skip Menu |

Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the CGI CPAN distribution.

Report information
The Basics
Id: 80415
Status: resolved
Priority: 0/
Queue: CGI

People
Owner: MARKSTOS [...] cpan.org
Requestors: yves [...] cpan.org
Cc: perl5-porters [...] perl.org
AdminCc:

Bug Information
Severity: Normal
Broken in: 3.60
Fixed in: 3.62



CC: perl5-porters [...] perl.org
Subject: hash order dependency bugs in CGI tests and code
I have been working on making it possible to add new hash algorithms to perl core and on making the hash seed random per process for security reasons. In my work on this I have encountered some hash order dependency bugs in the tests for CGI involving tags with more than one attribute. Making tag order optionally deterministic and then using the option in the tests solved the problem. The attached patch contains this fix. I left the new behavior undocumented as currently it exists only to make the tests happy. There was a similar bug related to the positioning of the following code: #### Method: endform # This method is DEPRECATED *endform = \&end_form; # deprecated! This was being executed before any subs were generated. Changing hash order would somehow cause subs to be loaded in an order such that this would not do the right thing and would fall through to an auto generated "end_form" tag. I moved the declaration to be part of the code for "end_form" and the bug went away. It would be really nice if the attached patch or moral equivalent could get applied to CGI sometime soon as I would like to merge my hash changes to core soon and this is a roadblock. Thanks, Yves
Subject: cgi.patch
commit 02b13840587691a7f1e1d88ccc6ddaf7980e1b87 Author: Yves Orton <demerphq@gmail.com> Date: Mon Aug 27 08:57:19 2012 +0200 fix hash key ordering dependencies in cpan/CGI/.. tests Hash seed randomization makes various tests fail as they depend on a particular hash key ordering. Note this is not a patch that has been pushed upstream. diff --git a/cpan/CGI/Changes b/cpan/CGI/Changes index 68ef980..4e88b06 100644 --- a/cpan/CGI/Changes +++ b/cpan/CGI/Changes @@ -1,3 +1,11 @@ +Version 3.61 Oct 20th, 2012 [CORE internal "bugfix" release (non-cpan)] + + [TEST FIXES] + - Made it possible to force a sorted order for things like hash + attributes so that tests are not dependent on a particular hash + ordering. This will be required in modern perls which will + change the ordering per process. + Version 3.60 Aug 15th, 2012 [BUG FIXES] diff --git a/cpan/CGI/lib/CGI.pm b/cpan/CGI/lib/CGI.pm index f510680..e200f23 100644 --- a/cpan/CGI/lib/CGI.pm +++ b/cpan/CGI/lib/CGI.pm @@ -20,7 +20,7 @@ use Carp 'croak'; # The revision is no longer being updated since moving to git. $CGI::revision = '$Id: CGI.pm,v 1.266 2009/07/30 16:32:34 lstein Exp $'; -$CGI::VERSION='3.60'; +$CGI::VERSION = '3.61'; # HARD-CODED LOCATION FOR FILE UPLOAD TEMPORARY FILES. # UNCOMMENT THIS ONLY IF YOU KNOW WHAT YOU'RE DOING. @@ -129,9 +129,6 @@ sub initialize_globals { # ------------------ START OF THE LIBRARY ------------ -#### Method: endform -# This method is DEPRECATED -*endform = \&end_form; # make mod_perlhappy initialize_globals(); @@ -1974,6 +1971,9 @@ sub end_form { } } } +#### Method: endform +# This method is DEPRECATED +*endform = \&end_form; # deprecated! END_OF_FUNC #### Method: end_multipart_form diff --git a/cpan/CGI/lib/CGI/Util.pm b/cpan/CGI/lib/CGI/Util.pm index b059281..2a98184 100644 --- a/cpan/CGI/lib/CGI/Util.pm +++ b/cpan/CGI/lib/CGI/Util.pm @@ -6,9 +6,10 @@ our @ISA = qw(Exporter); our @EXPORT_OK = qw(rearrange rearrange_header make_attributes unescape escape expires ebcdic2ascii ascii2ebcdic); -our $VERSION = '3.53'; +our $VERSION = '3.54'; use constant EBCDIC => "\t" ne "\011"; +our $SORT_ATTRIBUTES; # (ord('^') == 95) for codepage 1047 as on os390, vmesa our @A2E = ( @@ -132,8 +133,12 @@ sub make_attributes { my $quote = $do_not_quote ? '' : '"'; + my @attr_keys= keys %$attr; + if ($SORT_ATTRIBUTES) { + @attr_keys= sort @attr_keys; + } my(@att); - foreach (keys %{$attr}) { + foreach (@attr_keys) { my($key) = $_; $key=~s/^\-//; # get rid of initial - if present diff --git a/cpan/CGI/t/autoescape.t b/cpan/CGI/t/autoescape.t index 41172982..3a25c2d 100644 --- a/cpan/CGI/t/autoescape.t +++ b/cpan/CGI/t/autoescape.t @@ -6,6 +6,7 @@ use warnings; use Test::More tests => 18; use CGI qw/ autoEscape escapeHTML button textfield password_field textarea popup_menu scrolling_list checkbox_group optgroup checkbox radio_group submit image_button button /; +$CGI::Util::SORT_ATTRIBUTES = 1; is (button(-name => 'test<'), '<input type="button" name="test&lt;" value="test&lt;" />', "autoEscape defaults to On"); diff --git a/cpan/CGI/t/function.t b/cpan/CGI/t/function.t index e0c0845..7082a79 100644 --- a/cpan/CGI/t/function.t +++ b/cpan/CGI/t/function.t @@ -5,6 +5,7 @@ END {print "not ok 1\n" unless $loaded;} use Config; use CGI (':standard','keywords'); $loaded = 1; +$CGI::Util::SORT_ATTRIBUTES = 1; print "ok 1\n"; ######################### End of black magic. @@ -103,4 +104,4 @@ test(30, !charset("") && header() eq "Content-Type: text/html${CRLF}${CRLF}", "E test(31, header(-foo=>'bar') eq "Foo: bar${CRLF}Content-Type: text/html${CRLF}${CRLF}", "Custom header"); -test(32, start_form(-action=>'one',name=>'two',onsubmit=>'three') eq qq(<form method="post" action="one" enctype="multipart/form-data" onsubmit="three" name="two">), "initial dash followed by undashed arguments"); +test(32, start_form(-action=>'one',name=>'two',onsubmit=>'three') eq qq(<form method="post" action="one" enctype="multipart/form-data" name="two" onsubmit="three">), "initial dash followed by undashed arguments"); diff --git a/cpan/CGI/t/html.t b/cpan/CGI/t/html.t index 09a3e33..efa2f03 100644 --- a/cpan/CGI/t/html.t +++ b/cpan/CGI/t/html.t @@ -5,6 +5,7 @@ use Test::More tests => 33; END { ok $loaded; } use CGI ( ':standard', '-no_debug', '*h3', 'start_table' ); $loaded = 1; +$CGI::Util::SORT_ATTRIBUTES= 1; ok 1; BEGIN { @@ -98,7 +99,7 @@ is start_html( <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"> <head> <title>The world of foo</title> -<script src="foo.js" charset="utf-8" type="text/javascript"></script> +<script charset="utf-8" src="foo.js" type="text/javascript"></script> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> </head> <body>
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Fri, 26 Oct 2012 10:15:56 -0400
To: bug-CGI [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
Thanks for your work, on this Yves. I'll take a look. Mark
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Sat, 3 Nov 2012 13:55:03 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 26 October 2012 16:16, mark@summersault.com via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > > Thanks for your work, on this Yves. > > I'll take a look.
Hi, sorry to bug, any word on this? Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
RT-Send-CC: mark [...] stosberg.com, perl5-porters [...] perl.org
I've looked at this patch now, which adds a new package variable to CGI::Util, to control whether HTML attributes end up sorted or not: +$CGI::Util::SORT_ATTRIBUTES = 1; I agree on the importance of solving the compatibility problems with Perl, but I don't like this would be the first package variable like this in CGI::Util, and that it modifies the behavior of a subroutine through this action-at-distance approach. From some research on the HTML specs, it appears that no specific order for HTML attributes is defined. From reading the CGI.pm docs, I didn't see that CGI.pm advertises a particular sort order either. So, what about the approach of *always* sorting the attributes, and avoiding the option? It appears we already have test failures on Windows due to the sort order of attributes. Sure, I'm sure some existing expectations will break for someone somewhere, but it sounds like those expectations are going to break anyway if they upgrade Perl far enough. Mark
RT-Send-CC: demerphq [...] gmail.com
Regarding the "endform" part of this patch, it appeared the refactor was a subtly different behavior than the old one. While previously endform could be accessed all the time, after the patch, it was only defined after end_form() was loaded. This seemed like a potential catch-22 situation, since the point was to make endform() available instead of end_form, not only after it is defined. I chose to make an equivalent refactor which seemed safer from a compatibility perspective. I made a complete second definition for endform(), which mirrors what we did with startform()/start_form(), and avoids the catch-22 issue. Mark
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Sun, 4 Nov 2012 10:30:39 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 4 November 2012 01:59, MARKSTOS via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > Regarding the "endform" part of this patch, it appeared the refactor was > a subtly different behavior than the old one. While previously endform > could be accessed all the time, after the patch, it was only defined > after end_form() was loaded. > > This seemed like a potential catch-22 situation, since the point was to > make endform() available instead of end_form, not only after it is > defined. > > I chose to make an equivalent refactor which seemed safer from a > compatibility perspective. I made a complete second definition for > endform(), which mirrors what we did with startform()/start_form(), and > avoids the catch-22 issue.
Sounds good. I have to admit i found this particular issue to be hard to explain. There is some kind of hash order dependency involved but I could not quite work out what it was. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Sun, 4 Nov 2012 10:33:24 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 4 November 2012 01:31, MARKSTOS via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > > I've looked at this patch now, which adds a new package variable to > CGI::Util, to control whether HTML attributes end up sorted or not: > > +$CGI::Util::SORT_ATTRIBUTES = 1; > > I agree on the importance of solving the compatibility problems with > Perl, but I don't like this would be the first package variable like > this in CGI::Util, and that it modifies the behavior of a subroutine > through this action-at-distance approach.
Yeah, true. Although I am not convinced that that is intrinsically a bad thing. Show quoted text
> From some research on the HTML specs, it appears that no specific order > for HTML attributes is defined. From reading the CGI.pm docs, I didn't > see that CGI.pm advertises a particular sort order either. > > So, what about the approach of *always* sorting the attributes, and > avoiding the option?
The main argument IMO for NOT doing that, and the primary reason I did it as I did is there could be a non-trivial cost from having it sort always. IMO it would not be ideal for production, but would be preferred for testing (either of CGI itself or an app built on top of it). Show quoted text
> It appears we already have test failures on Windows due to the sort > order of attributes. Sure, I'm sure some existing expectations will > break for someone somewhere, but it sounds like those expectations are > going to break anyway if they upgrade Perl far enough.
Yes, I think it is safe to assume that the hash function will be different in 5.18. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
CC: Perl5 Porteros <perl5-porters [...] perl.org>
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 6 Nov 2012 11:06:55 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 4 November 2012 01:59, MARKSTOS via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > Regarding the "endform" part of this patch, it appeared the refactor was > a subtly different behavior than the old one. While previously endform > could be accessed all the time, after the patch, it was only defined > after end_form() was loaded. > > This seemed like a potential catch-22 situation, since the point was to > make endform() available instead of end_form, not only after it is > defined. > > I chose to make an equivalent refactor which seemed safer from a > compatibility perspective. I made a complete second definition for > endform(), which mirrors what we did with startform()/start_form(), and > avoids the catch-22 issue.
Hi. I noticed you uploaded a new version 3.62 on cpan, but it doesn't contain any changes that would resolves the test fails I am seeing. Perhaps this is an oversight? I am wondering if what i should do is merge my changes to blead, creating a 3.63 core-only release and then let you use that to test an alternative fix? Again, really sorry to be pushy about this, but these test fails are the primary reason I have not merged any of my hash related changes to core. I really want to get this done in time for 5.18 or it will have to wait for 5.20, and since this change will be turbulent we need a lot of lead up time before 5.18 is released. Which basically all means I need to get this merged ASAP. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 06 Nov 2012 09:50:34 -0500
To: bug-CGI [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
Show quoted text
> Hi. I noticed you uploaded a new version 3.62 on cpan, but it doesn't > contain any changes that would resolves the test fails I am seeing. > Perhaps this is an oversight?
No. I was falling asleep and published the bug fixes I had ready. Show quoted text
> I am wondering if what i should do is merge my changes to blead, > creating a 3.63 core-only release and then let you use that to test an > alternative fix?
As long as the sort-toggling fix remains undocumented, I can't see how it would be a problem if things were out of sync briefly. Were you opposed to sorting-by-default-in-production because of the extra sorting overhead would be a small performance hit? Considering that HTML tags generally have 0-5 attributes, my sense is that it wouldn't be that great. Show quoted text
> Again, really sorry to be pushy about this, but these test fails are > the primary reason I have not merged any of my hash related changes to > core. I really want to get this done in time for 5.18 or it will have > to wait for 5.20, and since this change will be turbulent we need a > lot of lead up time before 5.18 is released. Which basically all means > I need to get this merged ASAP.
It's helpful to know the back story. I'm interested to help the core Perl team move things along. Mark
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 6 Nov 2012 17:58:12 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 6 November 2012 15:50, mark@summersault.com via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > >
>> Hi. I noticed you uploaded a new version 3.62 on cpan, but it doesn't >> contain any changes that would resolves the test fails I am seeing. >> Perhaps this is an oversight?
> > No. I was falling asleep and published the bug fixes I had ready.
Heh. I can understand that one. ;-) Show quoted text
>> I am wondering if what i should do is merge my changes to blead, >> creating a 3.63 core-only release and then let you use that to test an >> alternative fix?
> > As long as the sort-toggling fix remains undocumented, I can't see how > it would be a problem if things were out of sync briefly.
Ok. I am sure it would make your life easier testing whatever change you decided to go with. Show quoted text
> Were you opposed to sorting-by-default-in-production because of the > extra sorting overhead would be a small performance hit? Considering > that HTML tags generally have 0-5 attributes, my sense is that it > wouldn't be that great.
I generally agree with you, I was just reluctant to take a position that people should pay a cost in order to get the tests to pass, especially as I am not really a CGI "consumer". So if you think sorting attributes always makes sense overall I have no objection. Show quoted text
> Again, really sorry to be pushy about this, but these test fails are
>> the primary reason I have not merged any of my hash related changes to >> core. I really want to get this done in time for 5.18 or it will have >> to wait for 5.20, and since this change will be turbulent we need a >> lot of lead up time before 5.18 is released. Which basically all means >> I need to get this merged ASAP.
> > It's helpful to know the back story. I'm interested to help the core > Perl team move things along.
Thanks, your support is much appreciated. And again, sorry to be a nag. :-) Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 06 Nov 2012 13:42:30 -0500
To: bug-CGI [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
Show quoted text
> Thanks, your support is much appreciated. And again, sorry to be a nag. :-)
Supporting the Perl core is more exciting than maintaining CGI.pm-- I'm happy to help. Mark
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 06 Nov 2012 13:44:54 -0500
To: bug-CGI [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
I think I'm now leaning towards an internal-use-only boolean that is intended to be used only for testing. That solution addresses backcompat, performance, and my concerns about introducing a new package scope variable into the public interface. Plus, as a non-public API, we are free-er to change the design later. Mark
CC: yves [...] cpan.org, perl5-porters [...] perl.org
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Tue, 6 Nov 2012 20:30:35 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 6 November 2012 19:45, mark@summersault.com via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > > I think I'm now leaning towards an internal-use-only boolean that is > intended to be used only for testing. > > That solution addresses backcompat, performance, and my concerns about > introducing a new package scope variable into the public interface. > > Plus, as a non-public API, we are free-er to change the design later.
Sounds good to me. :-) Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
I too have come across non-deterministic output from CGI.pm with perl 5.18, caused by hash key ordering. I suggest that it is not just CGI.pm's own test suite which is affected - many applications will expect the same program run twice to give the same output. For example, at my site there are some pages which are regularly produced and sent out as email alerts when they change. Because hash key ordering changes, with the latest perl version the HTML output changes randomly from one run to the next. Similarly, sites such as the Internet Archive monitor a page to see whether it is different from the previous version. I think the 'cost' of sorting hash keys is really a non-issue. The hash of attributes in an HTML element contains at most a dozen elements, and the sorting is done in C code. It is not going to make any real world difference to the performance of the page. So please could I ask that sorting keys be turned on by default.
...in other words I think it would be better for $CGI::Util::SORT_ATTRIBUTES to be on by default. (I would be interested to hear of any application where unsorted attributes provides a speedup. Note that sorting the attributes makes the page slightly more compressible with gzip, which would mean it would be expected to load slightly faster in many cases.)
CC: demerphq <yves [...] cpan.org>, Perl5 Porteros <perl5-porters [...] perl.org>
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Fri, 7 Mar 2014 12:10:00 +0100
To: bug-CGI [...] rt.cpan.org
From: demerphq <demerphq [...] gmail.com>
On 7 March 2014 10:58, EDAVIS via RT <bug-CGI@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=80415 > > > ...in other words I think it would be better for > $CGI::Util::SORT_ATTRIBUTES to be on by default.
I wouldn't object to that. I just didnt want to be responsible for making the decision. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
This issue has been copied to: https://github.com/leejo/CGI.pm/issues/106 please take all future correspondence there. This ticket will remain open but please do not reply here. This ticket will be closed when the github issue is dealt with.
CC: yves [...] cpan.org, perl5-porters [...] perl.org
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Thu, 22 May 2014 15:03:04 +0100
To: Lee Johnson via RT <bug-CGI [...] rt.cpan.org>
From: Zefram <zefram [...] fysh.org>
Lee Johnson via RT wrote: Show quoted text
>This issue has been copied to: https://github.com/leejo/CGI.pm/issues/106 >please take all future correspondence there.
Can't. Github requires one to have an account with them in order to add a comment there. -zefram
On Thu May 22 10:03:21 2014, zefram@fysh.org wrote: Show quoted text
> Lee Johnson via RT wrote:
> >This issue has been copied to: https://github.com/leejo/CGI.pm/issues/106 > >please take all future correspondence there.
> > Can't. Github requires one to have an account with them in order to > add a comment there.
No problem. I will obviously be keeping an eye on the RT queue(s, there are currently 2 for CGI.pm) and will be updating tickets here when resolved. Thanks, Lee.
Subject: Re: [rt.cpan.org #80415] hash order dependency bugs in CGI tests and code
Date: Thu, 22 May 2014 14:37:36 -0400
To: bug-CGI [...] rt.cpan.org
From: Adam Dutko <adam [...] runbymany.com>
Any coordination of working on bugs beyond merge requests? I'd hate to start working on something to find out someone already started or finished. -- Enjoy life! -Adam
On Thu May 22 14:37:44 2014, adam@runbymany.com wrote: Show quoted text
> Any coordination of working on bugs beyond merge requests? I'd hate to > start working on something to find out someone already started or finished. >
I think the easiest approach for the moment is to just make a comment on the github issue if you're looking at it, or i can add you as a contributor so you can assign an issue to yourself. I'm pruning issues that are no longer relevant or that have already been dealt with so the issue queue may change quite bit in the coming weeks. Cheers! Lee.
I'm considering this issue closed. Tests are passing against 5.18+ on cpantesters so i'm happy that there are no [more] hash order dependency bugs in the tests. There maybe bugs in the code, an eyeball doesn't show anything obvious and i will handle any that come along as an issue separate from this one.