Bug #46006 for Finance-Quote: Yahoo has changed their currency converter

Tue May 12 15:36:16 2009 develop [...] traveljury.com - Ticket created

Subject:

Yahoo has changed their currency converter

Yahoo has changed their currency converter to make it more Web 2.000, and currency conversion now just returns undef. thanks Clint

Wed May 13 05:46:02 2009 develop [...] traveljury.com - Correspondence added

On Tue May 12 15:36:16 2009, DRTECH wrote: Show quoted text

> Yahoo has changed their currency converter to make it more Web 2.000, > and currency conversion now just returns undef. > > thanks > > Clint

Interestingly, Yahoo UK have switched their currency conversion page back to the old version, so currency() works again. I have a feeling that this is a change they're going to make again soon, so I expect this to break again in the near future. thanks Clint

Wed May 13 05:46:04 2009 develop [...] traveljury.com - Status changed from 'new' to 'rejected'

Wed May 13 05:48:21 2009 develop [...] traveljury.com - Correspondence added

Show quoted text

> Interestingly, Yahoo UK have switched their currency conversion page > back to the old version, so currency() works again. > > I have a feeling that this is a change they're going to make again soon, > so I expect this to break again in the near future. > > thanks > > Clint

By the way, the US site still has the new version online, which may be useful for fixing the bug-to-be ahead of time: http://finance.yahoo.com/currency-converter#from=USD;to=EUR;amt=1 thanks Clint

Wed May 13 05:48:21 2009 The RT System itself - Status changed from 'rejected' to 'open'

Wed May 13 06:13:13 2009 develop [...] traveljury.com - Correspondence added

Show quoted text

> By the way, the US site still has the new version online, which may be > useful for fixing the bug-to-be ahead of time: > > http://finance.yahoo.com/currency-converter#from=USD;to=EUR;amt=1 >

Actually, better than that is Google's currency converter. Simple, made-to-scrape and likely to stay available: http://www.google.com/finance/converter?a=1&from=USD&to=GBP clint

Wed May 13 07:01:23 2009 develop [...] traveljury.com - Correspondence added

Here's what I'm using now, from Google. You're free to modify and reuse this code: $convert_from = uc($convert_from); $convert_to = uc($convert_to); return 1 if $convert_from eq $convert_to; require LWP::UserAgent; my $ua = LWP::UserAgent->new(); $ua->timeout(60); my $rate; eval { my $response = $ua->get( 'http://www.google.com/finance/converter?a=1&from=' . $convert_from . '&to=' . $convert_to ); die $response->status_line unless $response->is_success; my $content = $response->decoded_content; die "Couldn't find currency conversion result" unless $content =~ m/ currency_converter_result .+? <span[^>]+> ([0-9.]+)   ([A-Z]+) <\/span> /sx; die "Result currency doesn't match convert_to" unless $convert_to eq $2; { no warnings 'numeric'; $rate = 0 + $1; } die "Bad conversion - rate is zero" unless $rate; } or $class->error( "Converting from $convert_from to $convert_to : " . ( $@ || 'Unknown' ) ); return $rate;

Wed May 13 07:12:56 2009 develop [...] traveljury.com - Correspondence added

On Wed May 13 07:01:23 2009, DRTECH wrote: Show quoted text

> Here's what I'm using now, from Google. You're free to modify and reuse > this code:

Ah, only downside is that Google's list of currencies is much more limited than Yahoo's

Wed May 13 08:56:12 2009 eco [...] ecocode.net - Correspondence added

Subject:	Re: [rt.cpan.org #46006] Yahoo has changed their currency converter
Date:	Wed, 13 May 2009 14:55:38 +0200
To:	Clinton Gormley via RT <bug-Finance-Quote [...] rt.cpan.org>
From:	Erik Colson <eco [...] ecocode.net>

On Wed, May 13, 2009 at 07:12:57AM -0400, Clinton Gormley via RT wrote: Show quoted text

> Queue: Finance-Quote > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=46006 > > > On Wed May 13 07:01:23 2009, DRTECH wrote:

> > Here's what I'm using now, from Google. You're free to modify and reuse > > this code:

> > > Ah, only downside is that Google's list of currencies is much more > limited than Yahoo's

Hi Clinton, Google has also a site publishing stock quotes. They don't allow the use of scripts to retrieve data however... regards -- erik

Wed May 13 09:09:26 2009 clint [...] traveljury.com - Correspondence added

Subject:	Re: [rt.cpan.org #46006] Yahoo has changed their currency converter
Date:	Wed, 13 May 2009 15:09:01 +0200
To:	bug-Finance-Quote [...] rt.cpan.org
From:	Clinton Gormley <clint [...] traveljury.com>

Show quoted text

> > > > Ah, only downside is that Google's list of currencies is much more > > limited than Yahoo's

> > Hi Clinton, > > Google has also a site publishing stock quotes. They don't allow the > use of scripts to retrieve data however...

Ahh right.. shhhh :) Oanda.com looks like the best site for forex, but of course their Ts&Cs prohibit automated use as well, unless paid for. good luck :) clint Show quoted text

> > regards > > -- > erik >

Sun May 17 13:26:07 2009 mta [...] umich.edu - Correspondence added 60 min

One way to solve this problem is to change the code that gets currency quotes to use the existing Yahoo module which already knows how to get currency quotes. I've attached a patch to do this. It's based on 1.15, but I think it should work in 1.16 also. I don't know if this approach is deemed acceptable or not. Perhaps it's not ok for top level code to call a module this way.

--- lib/Finance/Quote.pm 2008-10-26 04:15:50.000000000 -0400 +++ /opt/local/lib/perl5/vendor_perl/5.8.9/Finance/Quote.pm 2009-05-16 01:22:08.000000000 -0400 @@ -36,14 +36,16 @@ use Finance::Quote::UserAgent; use HTTP::Request::Common; use HTML::TableExtract; +use Finance::Quote::Yahoo::Base qw/yahoo_request/; use vars qw/@ISA @EXPORT @EXPORT_OK @EXPORT_TAGS $VERSION $TIMEOUT %MODULES %METHODS $AUTOLOAD $YAHOO_CURRENCY_URL $USE_EXPERIMENTAL_UA/; -$YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/currency/convert?amt=1&submit=Convert&"; +# $YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/currency/convert?amt=1&submit=Convert&"; # If the above URL ever fails, try rewriting this module to use the URL below. # $YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/q?s=USDCAD%3DX"; +$YAHOO_CURRENCY_URL = "http://finance.yahoo.com/d"; @ISA = qw/Exporter/; @EXPORT = (); @@ -236,17 +238,13 @@ return $amount if ($from eq $to); # Trivial case. - my $ua = $this->user_agent; - - my $data = $ua->request(GET "${YAHOO_CURRENCY_URL}from=$from&to=$to")->content; - my $te = HTML::TableExtract->new( headers => ['Symbol', 'Bid', 'Ask'] ); - $te->parse($data); - - # Make sure there's a table to parse. - return undef unless ($te->tables); - - my $row = ($te->rows())[0]; - my ($exchange_rate) = $$row[1]; + my $symbol = "$from$to=X"; + my @symbols = ($symbol); + my %info = yahoo_request($this,$YAHOO_CURRENCY_URL,\@symbols); + return undef unless $info{$symbol,"success"}; + + my $exchange_rate = $info{$symbol,"last"}; + $exchange_rate =~ s/,// ; # solve a bug when conversion rate # involves thousands. yahoo inserts # a comma when thousands occur

Thu May 28 09:52:58 2009 perl [...] sattvik.com - Correspondence added

I have created a patch that uses Yahoo's interface to return a CSV file with the latest price quote. It's loosely based on the code in Finance::Quote::Yahoo, but it is fairly lightweight.

diff -Nuar Finance-Quote-1.16/lib/Finance/Quote.pm Finance-Quote-1.16.fix/lib/Finance/Quote.pm --- Finance-Quote-1.16/lib/Finance/Quote.pm 2009-04-13 09:15:29.000000000 -0500 +++ Finance-Quote-1.16.fix/lib/Finance/Quote.pm 2009-05-28 08:34:56.000000000 -0500 @@ -35,14 +35,13 @@ use Carp; use Finance::Quote::UserAgent; use HTTP::Request::Common; -use HTML::TableExtract; use Encode; use vars qw/@ISA @EXPORT @EXPORT_OK @EXPORT_TAGS $VERSION $TIMEOUT %MODULES %METHODS $AUTOLOAD $YAHOO_CURRENCY_URL $USE_EXPERIMENTAL_UA/; -$YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/currency/convert?amt=1&submit=Convert&"; +$YAHOO_CURRENCY_URL = "http://uk.old.finance.yahoo.com/d/quotes.csv?f=sl1&e=.csv&s="; # If the above URL ever fails, try rewriting this module to use the URL below. # $YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/q?s=USDCAD%3DX"; @@ -241,19 +240,14 @@ my $ua = $this->user_agent; - my $data = $ua->request(GET "${YAHOO_CURRENCY_URL}from=$from&to=$to")->content; - my $te = HTML::TableExtract->new( headers => ['Symbol', 'Bid', 'Ask'] ); - $te->parse(decode_utf8($data)); # The web page returns utf8 content which gives - # a warning when parsing $data in HTML::Parser - - # Make sure there's a table to parse. - return undef unless ($te->tables); - - my $row = ($te->rows())[0]; - my ($exchange_rate) = $$row[1]; - $exchange_rate =~ s/,// ; # solve a bug when conversion rate - # involves thousands. yahoo inserts - # a comma when thousands occur + my $response = $ua->request(GET "${YAHOO_CURRENCY_URL}$from$to%3DX"); + # Make sure there's a CSV to parse. + return undef unless $response->is_success; + + my ($query,$exchange_rate)=parse_csv($response->content); + + # Make sure correct content was returned + return undef unless $query eq "$from$to=X"; { local $^W = 0; # Avoid undef warnings.

Thu May 28 10:41:24 2009 perl [...] sattvik.com - Correspondence added

On Thu May 28 09:52:58 2009, sattvik wrote: Show quoted text

> I have created a patch that uses Yahoo's interface to return a CSV file > with the latest price quote. It's loosely based on the code in > Finance::Quote::Yahoo, but it is fairly lightweight.

It turns out the CSV seems to round the value off to a hundredth of a unit. This patch uses URL recommended in the module.

diff -Naur Finance-Quote-1.16/lib/Finance/Quote.pm Finance-Quote-1.16.fix/lib/Finance/Quote.pm --- Finance-Quote-1.16/lib/Finance/Quote.pm 2009-04-13 09:15:29.000000000 -0500 +++ Finance-Quote-1.16.fix/lib/Finance/Quote.pm 2009-05-28 09:39:18.000000000 -0500 @@ -35,16 +35,14 @@ use Carp; use Finance::Quote::UserAgent; use HTTP::Request::Common; -use HTML::TableExtract; +use HTML::TreeBuilder; use Encode; use vars qw/@ISA @EXPORT @EXPORT_OK @EXPORT_TAGS $VERSION $TIMEOUT %MODULES %METHODS $AUTOLOAD $YAHOO_CURRENCY_URL $USE_EXPERIMENTAL_UA/; -$YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/currency/convert?amt=1&submit=Convert&"; -# If the above URL ever fails, try rewriting this module to use the URL below. -# $YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/q?s=USDCAD%3DX"; +$YAHOO_CURRENCY_URL = "http://uk.finance.yahoo.com/q?s="; @ISA = qw/Exporter/; @EXPORT = (); @@ -241,16 +239,23 @@ my $ua = $this->user_agent; - my $data = $ua->request(GET "${YAHOO_CURRENCY_URL}from=$from&to=$to")->content; - my $te = HTML::TableExtract->new( headers => ['Symbol', 'Bid', 'Ask'] ); - $te->parse(decode_utf8($data)); # The web page returns utf8 content which gives - # a warning when parsing $data in HTML::Parser + my $data = $ua->request(GET "${YAHOO_CURRENCY_URL}$from$to%3DX")->content; + # The web page returns utf8 content which gives a warning when parsing $data + # in HTML::Parser + my $tb = HTML::TreeBuilder->new_from_content(decode_utf8($data)); + + # Find the <div> with the data + my $div = $tb->look_down('id','yfi_quote_summary_data'); + # Make sure there's a <div> to parse. + return undef unless $div; + + # The first <b> should contain the quote + my $rate_element=$div->look_down('_tag','b'); + # Make sure there's a <b> to parse. + return undef unless $rate_element; - # Make sure there's a table to parse. - return undef unless ($te->tables); + my $exchange_rate=$rate_element->as_text; - my $row = ($te->rows())[0]; - my ($exchange_rate) = $$row[1]; $exchange_rate =~ s/,// ; # solve a bug when conversion rate # involves thousands. yahoo inserts # a comma when thousands occur

Thu May 28 10:43:29 2009 perl [...] sattvik.com - Cc sattvik added

Sun Jun 14 14:52:52 2009 eco [...] ecocode.net - Taken

Sun Jun 14 14:53:30 2009 eco [...] ecocode.net - Correspondence added 1 min

patch applied -- Erik

Sun Jun 14 14:53:32 2009 eco [...] ecocode.net - Status changed from 'open' to 'resolved'