Bug #75997 for WWW-Scripter: User/pass form issue with WWW::Scripter .027

Fri Mar 23 00:12:14 2012 TCamp [...] zscaler.com - Ticket created

Subject:	User/pass form issue with WWW::Scripter .027
Date:	Thu, 22 Mar 2012 21:12:02 -0700
To:	"bug-WWW-Scripter [...] rt.cpan.org" <bug-WWW-Scripter [...] rt.cpan.org>
From:	"TCamp [...] zscaler.com" <TCamp [...] zscaler.com>

Used with WWW::Mechanize 1.72 Howdy, I'm trying to script access into https://admin.Zscaler.net I can get past the first page with Mechanize, but not with Scripter. It breaks on the password entry, says "No such field 'password' at /usr/local/share/perl/5.10.0/WWW/Mechanize.pm line 1537." If I perform a dump of the content() retrieved by the Scripter object, it appears to have corrupted the form. I've attached a cut-down example script with the mirror Mechanize functions commented – they succeed. Unfortunately I cannot provide credentials for you to successfully log in, but that should not be necessary to see the problem. Regards, -- Tomki Camp Zscaler Inc. Global Support Manager Tcamp@zscaler.com<mailto:Tcamp@zscaler.com> Support Contact Information USA: 1-800-953-3897 Intl: 1-408-533-0288 Email: support@zscaler.com<mailto:support@zscaler.com>

Message body is not shown because sender requested not to inline it.

Fri Mar 23 11:41:34 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Fri Mar 23 00:12:14 2012, TCamp@zscaler.com wrote: Show quoted text

> Used with WWW::Mechanize 1.72 > > Howdy, > I'm trying to script access into https://admin.Zscaler.net > I can get past the first page with Mechanize, but not with Scripter. > It breaks on the password entry, says "No such field 'password' at > /usr/local/share/perl/5.10.0/WWW/Mechanize.pm line 1537." > > If I perform a dump of the content() retrieved by the Scripter object, > it appears to have corrupted the form. > > I've attached a cut-down example script with the mirror Mechanize > functions commented – they succeed. > Unfortunately I cannot provide credentials for you to successfully > log in, but that should not be necessary to see the problem.

Thank you for the report. Local firewall setting won’t allow me to access that site. But if you could send me the HTML source code (the output of $browser->response->content) and the corrupted content ($browser->content), I should be able to figure out what is wrong.

Fri Mar 23 11:41:37 2012 The RT System itself - Status changed from 'new' to 'open'

Fri Mar 23 18:25:52 2012 TCamp [...] zscaler.com - Correspondence added

Subject:	Re: [rt.cpan.org #75997] AutoReply: User/pass form issue with WWW::Scripter .027
Date:	Fri, 23 Mar 2012 15:25:29 -0700
To:	"bug-WWW-Scripter [...] rt.cpan.org" <bug-WWW-Scripter [...] rt.cpan.org>
From:	"TCamp [...] zscaler.com" <TCamp [...] zscaler.com>

Here you go. Let me know if I can provide anything else. I've also attached 'original.out', which is just a page-source view from my browser. Thanks! -- Tomki Camp Zscaler Inc. Global Support Manager Tcamp@zscaler.com Support Contact Information USA: 1-800-953-3897 Intl: 1-408-533-0288 Email: support@zscaler.com On 3/22/12 21:12 PM, "Bugs in WWW-Scripter via RT" <bug-WWW-Scripter@rt.cpan.org> wrote: Show quoted text

> >Greetings, > >This message has been automatically generated in response to the >creation of a trouble ticket regarding: > "User/pass form issue with WWW::Scripter .027", >a summary of which appears below. > >There is no need to reply to this message right now. Your ticket has been >assigned an ID of [rt.cpan.org #75997]. Your ticket is accessible >on the web at: > > https://rt.cpan.org/Ticket/Display.html?id=75997 > >Please include the string: > > [rt.cpan.org #75997] > >in the subject line of all future correspondence about this issue. To do >so, >you may reply to this message. > > Thank you, > bug-WWW-Scripter@rt.cpan.org > >------------------------------------------------------------------------- >Used with WWW::Mechanize 1.72 > >Howdy, >I'm trying to script access into https://admin.Zscaler.net >I can get past the first page with Mechanize, but not with Scripter. It >breaks on the password entry, says "No such field 'password' at >/usr/local/share/perl/5.10.0/WWW/Mechanize.pm line 1537." > >If I perform a dump of the content() retrieved by the Scripter object, it >appears to have corrupted the form. > >I've attached a cut-down example script with the mirror Mechanize >functions commented they succeed. > Unfortunately I cannot provide credentials for you to successfully log >in, but that should not be necessary to see the problem. > >Regards, >-- >Tomki Camp >Zscaler Inc. Global Support Manager >Tcamp@zscaler.com<mailto:Tcamp@zscaler.com> > >Support Contact Information >USA: 1-800-953-3897 >Intl: 1-408-533-0288 >Email: support@zscaler.com<mailto:support@zscaler.com> > >

Download corrupt.out
application/octet-stream 4.8k

Message body not shown because it is not plain text.

Download notcorrupt_page2.out
application/octet-stream 18.2k

Message body not shown because it is not plain text.

Download notcorrupt.out
application/octet-stream 4.9k

Message body not shown because it is not plain text.

Download original.out
application/octet-stream 4.6k

Message body not shown because it is not plain text.

Sat Mar 24 17:42:19 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Fri Mar 23 18:25:52 2012, TCamp@zscaler.com wrote: Show quoted text

> Here you go. > Let me know if I can provide anything else. > I've also attached 'original.out', which is just a page-source view from > my browser. > > Thanks!

This seems to be an HTML::DOM bug. Can you try applying this patch to the latest HTML::DOM and see whether it fixes the problem? If it works, I’ll make a new release. BTW, this has to do with the body tags being inside the form tags on the page in question. This bug is related to HTML::Tree’s bug #76021 (<https://rt.cpan.org/Ticket/Display.html? id=76021>), since HTML::DOM derives it’s parsing code from HTML::Tree.

Subject:

open_U1t48IiP.txt

diff -rup a/lib/HTML/DOM/_TreeBuilder.pm b/lib/HTML/DOM/_TreeBuilder.pm --- HTML-Tree-4.2-SxenFd-orig/lib/HTML/TreeBuilder.pm 2011-04-06 01:37:54.000000000 -0700 +++ HTML-Tree-4.2-SxenFd/lib/HTML/TreeBuilder.pm 2012-03-24 14:29:11.000000000 -0700 @@ -706,7 +706,8 @@ sub warning { for ( keys %$attr ) { $body->attr( $_, $attr->{$_} ); } - return $self->{'_pos'} = $body; # bypass tweaking. + $self->{'_pos'} = $body unless $pos->is_inside('body'); + return $self->{'_pos'}; # bypass tweaking. #---------------------------------------------------------------------- }

Sat Mar 24 17:43:22 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sat Mar 24 17:42:19 2012, SPROUT wrote: Show quoted text

> On Fri Mar 23 18:25:52 2012, TCamp@zscaler.com wrote:

> > Here you go. > > Let me know if I can provide anything else. > > I've also attached 'original.out', which is just a page-source view

> from

> > my browser. > > > > Thanks!

> > This seems to be an HTML::DOM bug. > > Can you try applying this patch to the latest HTML::DOM and see > whether it fixes the > problem?

Oops. I made a mistake. Please try this patch.

Subject:

new patch.txt

diff -rup a/lib/HTML/DOM/_TreeBuilder.pm b/lib/HTML/DOM/_TreeBuilder.pm --- a/lib/HTML/DOM/_TreeBuilder.pm 2011-04-06 01:37:54.000000000 -0700 +++ b/lib/HTML/DOM/_TreeBuilder.pm 2012-03-24 14:29:11.000000000 -0700 @@ -706,7 +706,8 @@ sub warning { for ( keys %$attr ) { $body->attr( $_, $attr->{$_} ); } - return $self->{'_pos'} = $body; # bypass tweaking. + $self->{'_pos'} = $body unless $pos->is_inside('body'); + return $self->{'_pos'}; # bypass tweaking. #---------------------------------------------------------------------- }

Sun Mar 25 01:28:09 2012 TCamp [...] zscaler.com - Correspondence added

Subject:	Re: [rt.cpan.org #75997] User/pass form issue with WWW::Scripter .027
Date:	Sat, 24 Mar 2012 22:27:53 -0700
To:	"bug-WWW-Scripter [...] rt.cpan.org" <bug-WWW-Scripter [...] rt.cpan.org>
From:	"TCamp [...] zscaler.com" <TCamp [...] zscaler.com>

That worked perfectly, thank you. Also thank you for pointing out the form-before-body mistake, I will give the team feedback on that. On 3/24/12 14:43 PM, "Father Chrysostomos via RT" <bug-WWW-Scripter@rt.cpan.org> wrote: Show quoted text

><URL: https://rt.cpan.org/Ticket/Display.html?id=75997 > > >On Sat Mar 24 17:42:19 2012, SPROUT wrote:

>> On Fri Mar 23 18:25:52 2012, TCamp@zscaler.com wrote:

>> > Here you go. >> > Let me know if I can provide anything else. >> > I've also attached 'original.out', which is just a page-source view

>> from

>> > my browser. >> > >> > Thanks!

>> >> This seems to be an HTML::DOM bug. >> >> Can you try applying this patch to the latest HTML::DOM and see >> whether it fixes the >> problem?

> >Oops. I made a mistake. Please try this patch.

Sun Mar 25 05:42:51 2012 TCamp [...] zscaler.com - Correspondence added

Subject:	Re: [rt.cpan.org #75997] User/pass form issue with WWW::Scripter .027
Date:	Sun, 25 Mar 2012 02:42:39 -0700
To:	"bug-WWW-Scripter [...] rt.cpan.org" <bug-WWW-Scripter [...] rt.cpan.org>
From:	"TCamp [...] zscaler.com" <TCamp [...] zscaler.com>

I've gotten all the results I need to get at this point with the script, but now this warning comes up: Use of uninitialized value $code in concatenation (.) or string at /usr/local/share/perl/5.10.0/WWW/Scripter/Plugin/JavaScript/JE.pm line 274. Everything seems to be working just fine regardless. Do you need sample output to track this down? ‹Tki

Sun Mar 25 15:33:28 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sat Mar 24 17:42:19 2012, SPROUT wrote: Show quoted text

> On Fri Mar 23 18:25:52 2012, TCamp@zscaler.com wrote:

> > Here you go. > > Let me know if I can provide anything else. > > I've also attached 'original.out', which is just a page-source view

> from

> > my browser. > > > > Thanks!

> > This seems to be an HTML::DOM bug.

I’ve just released HTML::DOM 0.052 with a fix.

Sun Mar 25 15:41:57 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sun Mar 25 05:42:51 2012, TCamp@zscaler.com wrote: Show quoted text

> I've gotten all the results I need to get at this point with the script, > but now this warning comes up: > > Use of uninitialized value $code in concatenation (.) or string at > /usr/local/share/perl/5.10.0/WWW/Scripter/Plugin/JavaScript/JE.pm line 274. > > Everything seems to be working just fine regardless. > > Do you need sample output to track this down?

Yes, please. Somehow, the code that turns onclick="foo" into a JavaScript function is being handed undef instead of foo. I haven’t been able figure out why.

Sun Mar 25 21:47:30 2012 TCamp [...] zscaler.com - Correspondence added

Subject:	Re: [rt.cpan.org #75997] User/pass form issue with WWW::Scripter .027
Date:	Sun, 25 Mar 2012 18:47:21 -0700
To:	"bug-WWW-Scripter [...] rt.cpan.org" <bug-WWW-Scripter [...] rt.cpan.org>
From:	"TCamp [...] zscaler.com" <TCamp [...] zscaler.com>

Ok, so that would just be within the file 'notcorrupt.out' which I sent earlier. Looking at the JS 'validate' function, it does not appear to explicitly return anything other than false - this seems to not matter when a human interacts with the form via Browser, but could that be an issue with the parser? That's the limit of my guesswork right now.. If there is better debuggery I can help provide from my end please let me know just what I can do. On 3/25/12 12:41 PM, "Father Chrysostomos via RT" <bug-WWW-Scripter@rt.cpan.org> wrote: Show quoted text

><URL: https://rt.cpan.org/Ticket/Display.html?id=75997 > > >On Sun Mar 25 05:42:51 2012, TCamp@zscaler.com wrote:

>> I've gotten all the results I need to get at this point with the script, >> but now this warning comes up: >> >> Use of uninitialized value $code in concatenation (.) or string at >> /usr/local/share/perl/5.10.0/WWW/Scripter/Plugin/JavaScript/JE.pm line >>274. >> >> Everything seems to be working just fine regardless. >> >> Do you need sample output to track this down?

> >Yes, please. > >Somehow, the code that turns onclick="foo" into a JavaScript function is >being handed undef >instead of foo. I haven¹t been able figure out why. >

Mon Mar 26 02:07:04 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sun Mar 25 21:47:30 2012, TCamp@zscaler.com wrote: Show quoted text

> Ok, so that would just be within the file 'notcorrupt.out' which I sent > earlier.

Thank you. Show quoted text

> Looking at the JS 'validate' function, it does not appear to > explicitly return anything other than false - this seems to not matter > when a human interacts with the form via Browser, but could that be an > issue with the parser?

It’s the onLoad="..." in the body tag that was causing the problem. It turns out my bug fix had a problem with it, because event handlers defined in an extra <body> tag were mistakenly being applied to the previous element encountered, but without the text of the event handler. Try this patch to HTML::DOM. It will be a while before I get a chance to make another release.

Subject:

open_Obf2VbOe.txt

diff -rup HTML-DOM-orig/lib/HTML/DOM/_TreeBuilder.pm HTML-DOM/lib/HTML/DOM/_TreeBuilder.pm --- HTML-DOM-orig/lib/HTML/DOM/_TreeBuilder.pm 2012-03-24 23:51:05.000000000 -0700 +++ HTML-DOM/lib/HTML/DOM/_TreeBuilder.pm 2012-03-25 22:59:35.000000000 -0700 @@ -709,7 +709,7 @@ sub warning { $body->attr( $_, $attr->{$_} ); } $self->{'_pos'} = $body unless $pos->is_inside('body'); - return $self->{'_pos'}; # bypass tweaking. + return $body; # bypass tweaking. #---------------------------------------------------------------------- }

Mon Apr 02 00:09:07 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Mon Mar 26 02:07:04 2012, SPROUT wrote: Show quoted text

> On Sun Mar 25 21:47:30 2012, TCamp@zscaler.com wrote:

> > Ok, so that would just be within the file 'notcorrupt.out' which I

> sent

> > earlier.

> > Thank you. >

> > Looking at the JS 'validate' function, it does not appear to > > explicitly return anything other than false - this seems to not

> matter

> > when a human interacts with the form via Browser, but could that be

> an

> > issue with the parser?

> > It’s the onLoad="..." in the body tag that was causing the problem. > It turns out my bug fix > had a problem with it, because event handlers defined in an extra > <body> tag were > mistakenly being applied to the previous element encountered, but > without the text of the > event handler. > > Try this patch to HTML::DOM.

I’ve just released HTML::DOM 0.053 with this fix.

Mon Apr 02 00:09:08 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Status changed from 'open' to 'resolved'