Bug #27453 for WWW-Myspace: Login unsuccessful (UK)

Tue Jun 05 15:47:28 2007 tobiesch [...] users.sourceforge.net - Ticket created

minor issue: I had to manually downgrade DBD::SQLite from version 1.13 to 1.12 in order to make Class::DBI work (would otherwise return: "DBD::SQLite::st fetchrow_array warning"). For more details on this see http://lists.digitalcraftsmen.net/pipermail/classdbi/2007-January/001480.html Main issue: Most of the test of the WWW::Myspace module fail. Crucially, I cannot log on. I attach the output of the test that were running during cpan install. During the tests and also once I try to run the code I get frequent warnings concerning the Locale::SubCountry module such like that: Use of uninitialized value in hash element at /usr/lib/perl5/site_perl/5.8.5/Locale/SubCountry.pm line 439, <DATA> line 15048 Trying to create a new WWW::Myspace object with login credentials results in a long delay with an eventual error of: Error getting page: 500 Server closed connection without sending any data back Trying to use methods for which I don't think I have to login such as $myspace->get_friends(id=>76959716) returns the error: Link not found on page However, as this is just a warning I doubt this is related to the problem. Relating to an earlier bug reported here, could the main problem be that I access the site from the UK? I attach the html code of several pages as accessed from the UK through my browser. Perl version: v5.8.5 built for x86_64-linux-thread-multi OS version: 2.6.9-42.0.2.ELsmp #1 SMP Wed Aug 23 13:38:27 BST 2006 x86_64 x86_64 x86_64 GNU/Linux Please let me know what other information you need.

Subject:

pageafterlogin.htm

Message body is not shown because it is too large.

Subject:

homepage.htm

Message body is not shown because it is too large.

Subject:

www.MySpace.com.htm

Message body is not shown because it is too large.

Subject:

module_install_output

Download module_install_output
application/octet-stream 18.4k

Message body not shown because it is not plain text.

Tue Jun 05 22:48:07 2007 tobiesch [...] users.sourceforge.net - Correspondence added

From:

tobiesch [...] users.sourceforge.net

Through testing so far I could verify that the test fails for the site_login method in WWW::Myspace at the following call: $self->follow_link( url_regex => qr/fuseaction=user/i, re=> 'SignOut' ); Although a dump of the response object shows that I've successfully logged on, the module will not recognize it as _page_ok fails to recognize that the regexp "SignOut" is actually on the page: # Page must match the regexp unless ( $page =~ /$regexp/i ) { #print "REGEXP",$res->decoded_content(),"\n\n\n"; $page_ok = 0; $self->error("Page doesn't match verification pattern."); # warn "Page doesn't match verification pattern.\n"; } I am not sure but what I've seen so far is that $res->content yields garbage output while $res->decoded_content() brings up plain text. Any chance that it is down to some strange locale problem? Or am I completely on the wrong track? thx for your help!

Tue Jun 05 23:16:27 2007 olaf [...] wundersolutions.com - Correspondence added

From:

olaf [...] wundersolutions.com

On Tue Jun 05 15:47:28 2007, tobiesch wrote: Show quoted text

> minor issue: I had to manually downgrade DBD::SQLite from version 1.13 > to 1.12 in order to make Class::DBI work (would otherwise return: > "DBD::SQLite::st fetchrow_array warning"). For more details on this > see > http://lists.digitalcraftsmen.net/pipermail/classdbi/2007- > January/001480.html

Thanks very much for this tip. I've made a note of it in the BUGS section of WWW::Myspace::Data All the best, Olaf

Tue Jun 05 23:16:28 2007 The RT System itself - Status changed from 'new' to 'open'

Tue Jun 05 23:19:06 2007 olaf [...] wundersolutions.com - Correspondence added

On Tue Jun 05 22:48:07 2007, tobiesch wrote: Show quoted text

> Through testing so far I could verify that the test fails for the > site_login method in WWW::Myspace at the following call: > > $self->follow_link( > url_regex => qr/fuseaction=user/i, > re=> 'SignOut' > ); >

Myspace is so unstable that failing tests are tough to diagnose. However, in your case, maybe the regex needs to be tweaked? We're trying to get at the root of the problem with logins from outside of North America. Could you tweak the regex and let us know if this solves it? $self->follow_link( url_regex => qr/fuseaction=signout/i, re=> 'SignOut' ); All the best, Olaf

Tue Jun 05 23:19:12 2007 olaf [...] wundersolutions.com - Given to OALDERS

Tue Jun 05 23:27:11 2007 olaf [...] wundersolutions.com - Subject changed from (no value) to 'Login unsuccessful (UK)'

Tue Jun 05 23:30:16 2007 grantg [...] spamarrest.com - Correspondence added

Subject:	Re: [rt.cpan.org #27453]
Date:	Tue, 5 Jun 2007 20:29:44 -0700
To:	bug-WWW-Myspace [...] rt.cpan.org
From:	Grant Grueninger <grantg [...] spamarrest.com>

I think you may have cracked the issue that's been eluding us for months. We originally thought is was a locale issue but we couldn't reproduce it in England, Canada, or California, and users in the US and elsewhere were reporting login failures. Unfortunately, none were savvy enough to report through RT and give useful information. I suspect it's a character encoding issue. I'm running tests now having changed all references to the "content" method of HTTP::Request to "decoded_content". So far so good (except for the usual myspace unreliability issues). Myspace must be using different character encoding in different places or quite possibly on different servers or different accounts. (They actually run different versions of the "application" on different systems and with different accounts - makes programming WWW::Myspace fun...). If you check out the svn repository, you can test the latest versions and help debug this (or see if that did it). There's another issue which I think affects Linux systems that I believe is a problem in Locale::Subcountry. On Jun 5, 2007, at 7:48 PM, via RT wrote: Show quoted text

> > Queue: WWW-Myspace > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=27453 > > > Through testing so far I could verify that the test fails for the > site_login method in WWW::Myspace at the following call: > > $self->follow_link( > url_regex => qr/fuseaction=user/i, > re=> 'SignOut' > ); > > Although a dump of the response object shows that I've successfully > logged on, the module will not recognize it as _page_ok fails to > recognize that the regexp "SignOut" is actually on the page: > > # Page must match the regexp > unless ( $page =~ /$regexp/i ) { > #print "REGEXP",$res->decoded_content(),"\n\n\n"; > $page_ok = 0; > $self->error("Page doesn't match verification > pattern."); > # warn "Page doesn't match verification pattern.\n"; > } > > I am not sure but what I've seen so far is that $res->content yields > garbage output while $res->decoded_content() brings up plain text. Any > chance that it is down to some strange locale problem? Or am I > completely on the wrong track? > > thx for your help! >

Tue Jun 05 23:45:24 2007 olaf [...] wundersolutions.com - Correspondence added

On Tue Jun 05 23:30:16 2007, grantg@spamarrest.com wrote: Show quoted text

> I think you may have cracked the issue that's been eluding us for > months.

That sounds like it's the root issue (encoding). Ignore my suggested changes. I've had a closer look at the code and that was not a correct fix. :) Olaf

Wed Jun 06 09:29:05 2007 tobiesch [...] users.sourceforge.net - Correspondence added

From:

tobiesch [...] users.sourceforge.net

Brilliant, I've checked out version 348 from the SVN repository and it seems to pass most tests now. I also did use some functions in a program, seems to work. Very good! I attach the log from the tests (sadly forget to pipe the error message also but still informative I think) Some minor discoveries: - The test 8 from 01/login (check friend count) seems to assume you have at least one friend (apart from Tom). - the example test config from the README file says: user_name: Username # (as it appears in "Hello, username" when you log in.) However, the hash and everything that follows have to be removed in order for the test being successful (I thought the test would treat it as a comment but it takes it as part of the username) - in order to make tests quicker and to less annoy MySpace, do we really need 20 attempts to get a URL (might 10 be sufficient?) On Tue Jun 05 23:45:24 2007, OALDERS wrote: Show quoted text

> On Tue Jun 05 23:30:16 2007, grantg@spamarrest.com wrote:

> > I think you may have cracked the issue that's been eluding us for > > months.

> > That sounds like it's the root issue (encoding). Ignore my suggested > changes. I've had a closer look at the code and that was not a correct > fix. :) > > Olaf >

Download test.log
application/octet-stream 249k

Message body not shown because it is not plain text.

Wed Jun 06 09:31:30 2007 tobiesch [...] users.sourceforge.net - Correspondence added

From:

tobiesch [...] users.sourceforge.net

Brilliant, I've checked out version 348 from the SVN repository and it seems to pass most tests now. I also did use some functions in a program, seems to work. Very good! I attach the log from the tests (sadly forget to pipe the error message also but still informative I think) Some minor discoveries: - The test 8 from 01/login (check friend count) seems to assume you have at least one friend (apart from Tom). - the example test config from the README file says: user_name: Username # (as it appears in "Hello, username" when you log in.) However, the hash and everything that follows have to be removed in order for the test being successful (I thought the test would treat it as a comment but it takes it as part of the username) - in order to make tests quicker and to less annoy MySpace, do we really need 20 attempts to get a URL (might 10 be sufficient?) On Tue Jun 05 23:45:24 2007, OALDERS wrote: Show quoted text

> On Tue Jun 05 23:30:16 2007, grantg@spamarrest.com wrote:

> > I think you may have cracked the issue that's been eluding us for > > months.

> > That sounds like it's the root issue (encoding). Ignore my suggested > changes. I've had a closer look at the code and that was not a correct > fix. :) > > Olaf >

Download test.log
application/octet-stream 249k

Message body not shown because it is not plain text.

Wed Jun 06 12:30:54 2007 grantg [...] spamarrest.com - Correspondence added

Subject:	Re: [rt.cpan.org #27453] Login unsuccessful (UK)
Date:	Wed, 6 Jun 2007 09:26:52 -0700
To:	bug-WWW-Myspace [...] rt.cpan.org
From:	Grant Grueninger <grantg [...] spamarrest.com>

On Jun 6, 2007, at 6:32 AM, via RT wrote: Show quoted text

> > Queue: WWW-Myspace > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=27453 > > > Brilliant, I've checked out version 348 from the SVN repository and it > seems to pass most tests now. I also did use some functions in a > program, seems to work. Very good!

Great! Show quoted text

> > I attach the log from the tests (sadly forget to pipe the error > message > also but still informative I think)

Actually, if you could copy/paste the normal test output (without $DEBUG on) into an attachment, that'd be a great help. Show quoted text

> > Some minor discoveries: > > - The test 8 from 01/login (check friend count) seems to assume you > have > at least one friend (apart from Tom). > > - the example test config from the README file says: > user_name: Username # (as it appears in "Hello, username" when > you log > in.) > However, the hash and everything that follows have to be removed in > order for the test being successful (I thought the test would treat it > as a comment but it takes it as part of the username)

Good catch, I've changed the README file. I also thought YAML would treat that as a comment. Apparently not. Show quoted text

> > - in order to make tests quicker and to less annoy MySpace, do we > really > need 20 attempts to get a URL (might 10 be sufficient?)

Originally the number of attempts was infinite, the idea being that an automated script would keep trying until Myspace was back up (if down), the user's connection came back up, myspace fixed an error, etc. WWW::Myspace retries only on a failed request or myspace error. Any exceptions to this are bugs (in fact, I think we re- introduced a bug recently in which an invalid username/password would be retried on login). I cut the attempts down to 20 as a "just in case". Of course it could be argued that if we're going to limit the attempts at all, anything that isn't going to work in 10 attempts probably isn't going to work in 20 either. The only problem with cutting this down is that the retries are what makes WWW::Myspace robust enough to deal with Myspace's ridiculous level of instability. For the record, the tests (and the whole module) run slowly because of the "human" feature. :) Grant

Wed Jun 06 20:51:32 2007 tobiesch [...] users.sourceforge.net - Correspondence added

From:

tobiesch [...] users.sourceforge.net

Hi Grant, thanks for all your help! See some comments below: On Wed Jun 06 12:30:54 2007, grantg@spamarrest.com wrote: Show quoted text

> > On Jun 6, 2007, at 6:32 AM, via RT wrote: >

> > > > Queue: WWW-Myspace > > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=27453 > > > > > Brilliant, I've checked out version 348 from the SVN repository and it > > seems to pass most tests now. I also did use some functions in a > > program, seems to work. Very good!

> > Great! >

> > > > I attach the log from the tests (sadly forget to pipe the error > > message > > also but still informative I think)

> > Actually, if you could copy/paste the normal test output (without > $DEBUG on) into > an attachment, that'd be a great help. >

Ok, I attach the output of "make test" Show quoted text

> > > > Some minor discoveries:

Show quoted text

> > > > - in order to make tests quicker and to less annoy MySpace, do we > > really > > need 20 attempts to get a URL (might 10 be sufficient?)

> > Originally the number of attempts was infinite, the idea being that > an automated script would keep trying until Myspace was back up (if > down), the user's connection came back up, myspace fixed an error, > etc. WWW::Myspace retries only on a failed request or myspace > error. Any exceptions to this are bugs (in fact, I think we re- > introduced a bug recently in which an invalid username/password would > be retried on login). I cut the attempts down to 20 as a "just in > case". Of course it could be argued that if we're going to limit the > attempts at all, anything that isn't going to work in 10 attempts > probably isn't going to work in 20 either. The only problem with > cutting this down is that the retries are what makes WWW::Myspace > robust enough to deal with Myspace's ridiculous level of instability. > > For the record, the tests (and the whole module) run slowly because > of the "human" feature. :)

Well alright, I understand your reasoning. However, might be useful to give people the chance to set the MaxRetry variable. I also thought for error handling, it might actually make sense to let is_private and is_invalid behave in the same way like is_band, that is to return -1 if an error occurred. For now, after each call of "is_private" I will have to check whether the result is actually because the profile is private or whether it is because some error occurred. Anyway, just to say again: really good work!! Thanks a lot for your efforts! best, tobias

Download testoutput.log
application/octet-stream 5.7k

Message body not shown because it is not plain text.

Wed Jun 06 23:40:21 2007 grantg [...] spamarrest.com - Correspondence added

Subject:	Re: [rt.cpan.org #27453] Login unsuccessful (UK)
Date:	Wed, 6 Jun 2007 20:39:37 -0700
To:	bug-WWW-Myspace [...] rt.cpan.org
From:	Grant Grueninger <grantg [...] spamarrest.com>

On Jun 6, 2007, at 5:51 PM, via RT wrote: Show quoted text

> > Well alright, I understand your reasoning. However, might be useful to > give people the chance to set the MaxRetry variable.

Hmm, probably dangerous, but easy, so I added it. Show quoted text

> > I also thought for error handling, it might actually make sense to let > is_private and is_invalid behave in the same way like is_band, that is > to return -1 if an error occurred. For now, after each call of > "is_private" I will have to check whether the result is actually > because > the profile is private or whether it is because some error occurred.

I have the feeling I didn't write those methods. I've modified them to return undef in an error, and $myspace->error should be set by the routine that found the error in the first place. See the new docs for those methods for sample implementations. Show quoted text

> > Anyway, just to say again: really good work!! Thanks a lot for your > efforts!

Thank you too - it's a great relief having that problem finally fixed! Grant Show quoted text

> > best, tobias > > > <testoutput.log>

Fri Jun 08 12:34:11 2007 olaf [...] wundersolutions.com - Correspondence added

Looks like this is all taken care of. Thanks again for all of your feedback! Olaf

Fri Jun 08 12:34:13 2007 olaf [...] wundersolutions.com - Status changed from 'open' to 'resolved'