Skip Menu |

This queue is for tickets about the Finance-Quote CPAN distribution.

Report information
The Basics
Id: 59623
Status: resolved
Worked: 25 min
Priority: 0/
Queue: Finance-Quote

People
Owner: eco [...] ecocode.net
Requestors: laurent.wozniak [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 1.17
Fixed in: (no value)



Subject: bourso.pm broken, data source changed (patch included, module and test)
boursorama.com has changed again, bourso.pm must be updated. No "last" field is returned, making this module unusable, especially in GnuCash. 3 test cases are failing. NB: Patch included in next message.
From: laurent.wozniak [...] gmail.com
Fix: in bourso.pm: Updated numbers parsing. Fix: in bourso.t: Replaced obsolete stocks by new ones. Updated: in bourso.t: Added more tests (26 => 50). Now including all instrument types (stock, fund, warrant, bond, indice), not just the "stock" type as previously. Those instruments were specified in the comments, but not activated in the tests. Now, they are. This also explain why bug #49337 was not triggering a test case failure.
Subject: bourso.59623.patch
diff --git a/lib/Finance/Quote/Bourso.pm b/lib/Finance/Quote/Bourso.pm index 18ff338..203598e 100644 --- a/lib/Finance/Quote/Bourso.pm +++ b/lib/Finance/Quote/Bourso.pm @@ -87,8 +87,15 @@ sub methods { return ( france => \&bourso, europe => \@labels); } } -sub bourso { +sub bourso_to_number { + my $x = shift(@_); + $x =~ s/\s//g ; # remove spaces etc in number + return $x; +} + + +sub bourso { my $quoter = shift; my @stocks = @_; my (%info,$reply,$url,$te,$ts,$row,$style); @@ -192,7 +199,8 @@ sub bourso { ASSIGN: for ( $key ) { # OPCVM /Valeur liquidative/ && do { - my ($last, $currency) = ($tempinfo{$key} =~ m/(\d*.?\d*)\s*(\w*)/); + my ($last, $currency) = ($tempinfo{$key} =~ m/(\d+(?:\s\d+)*(?:\.\d+)?)(?:\(c\))?(?:\s+(\w+))?/); + $last = bourso_to_number($last); $info{$stocks,"last"} = $last; $info{$stocks,"currency"} = $currency; }; @@ -205,7 +213,8 @@ sub bourso { }; # REGULAR STOCK /Cours/ && do { - my ($last, $currency) = ($tempinfo{$key} =~ m/(\d*.?\d*)\(c\)\s*(\w*)/); + my ($last, $currency) = ($tempinfo{$key} =~ m/(\d+(?:\s\d+)*(?:\.\d+)?)(?:\(c\))?(?:\s+(\w+))?/); + $last = bourso_to_number($last); $info{$stocks,"last"} = $last; $info{$stocks,"currency"} = $currency||"EUR"; # defaults to EUR my $exchange = $key; @@ -222,20 +231,19 @@ sub bourso { $quoter->store_date(\%info, $stocks, {eurodate => $info{$stocks,"date"}}); }; /Volume/ && do { - $info{$stocks,"volume"}= $tempinfo{$key} ; - $info{$stocks,"volume"} =~ s/\s//g ; # remove spaces etc in number + $info{$stocks,"volume"}=bourso_to_number($tempinfo{$key}); }; /Ouverture/ && do { - $info{$stocks,"open"}=$tempinfo{$key} + $info{$stocks,"open"}=bourso_to_number($tempinfo{$key}); }; /Haut/ && do { - $info{$stocks,"high"}=$tempinfo{$key} + $info{$stocks,"high"}=bourso_to_number($tempinfo{$key}); }; /Bas/ && do { - $info{$stocks,"low"}=$tempinfo{$key} + $info{$stocks,"low"}=bourso_to_number($tempinfo{$key}); }; /Cloture veille/ && do { - $info{$stocks,"previous"}=$tempinfo{$key} + $info{$stocks,"previous"}=bourso_to_number($tempinfo{$key}); }; /Valorisation/ && do { $info{$stocks,"cap"}=$tempinfo{$key} ; diff --git a/t/bourso.t b/t/bourso.t index c058ccf..e7868f1 100755 --- a/t/bourso.t +++ b/t/bourso.t @@ -7,14 +7,23 @@ if (not $ENV{ONLINE_TEST}) { plan skip_all => 'Set $ENV{ONLINE_TEST} to run this test'; } -plan tests => 26; +plan tests => 50; # Test Bourso functions. my $q = Finance::Quote->new(); # my stocks = stock, fund, warrant, bond, indice -my @stocks = ("AF","MSFT","SOLB","CNP"); +my @stocks = ( + "FR0000441677", # Fund + "AF", # Stock, EUR, Nyse Euronext + "MSFT", # Stock, USD, NASDAQ + "SOLB", # Stock, EUR, BRUXELLES + "CNP", # Stock, EUR, Nyse Euronext + "FR0010371401", # Bond + "FR0010707414", # Warrant + "FR0003500008", # Index +); # Bourso tests need to cover all the possible cases: @@ -22,9 +31,9 @@ my @stocks = ("AF","MSFT","SOLB","CNP"); # Name What Test Case # # cours-action Stock AF -# cours-obligation Bond FR0010112052 +# cours-obligation Bond FR0010371401 # opcvm/opcvm Fund FR0000441677 -# cours-warrant Warrant FR0010639880 +# cours-warrant Warrant FR0010707414 # cours-indice Index FR0003500008 my $year = (localtime())[5] + 1900; @@ -38,7 +47,7 @@ foreach my $stock (@stocks) { ok($quotes{$stock,"last"} > 0); ok(length($quotes{$stock,"name"})); ok($quotes{$stock,"success"}); - ok( ($stock eq "FR0010112052") || # bonds and indexes are quoted in percents + ok( # indexes are quoted in percents ($stock eq "FR0003500008") || (($stock eq "MSFT") && ($quotes{$stock, "currency"} eq "USD")) || ($quotes{$stock, "currency"} eq "EUR") );
Hello Laurent, I'm probably too late to apply this patch. Sorry for the delay. I tried your patch but all tests failed. Do these also fail at your place ? Best -- Erik
From: laurent.wozniak [...] gmail.com
Hello, Show quoted text
> I tried your patch but all tests failed. Do these also fail at your
place ? Yes, now they all fail since the top "fetch" retrieving all stocks throws an exception. I split it into 1 fetch per stock, within a try/catch block: 1 success, 7 errors The success is for the type "Fund", which is the only one that I use on a regular basis. I looked into the html, and yes, the site has changed some of its structure again ... I'll be doing another patch soon. Cheers, Laurent
From: laurent.wozniak [...] gmail.com
Hello, Here is the new patch. All tests went fine. The site has made some big changes in the structure of its pages, passing from lists (li) to tables. Would you have time to apply this patch soon ? Thanks, Laurent
Subject: bourso.59623.v2.patch
diff --git a/lib/Finance/Quote/Bourso.pm b/lib/Finance/Quote/Bourso.pm index 18ff338..d7b4fc3 100644 --- a/lib/Finance/Quote/Bourso.pm +++ b/lib/Finance/Quote/Bourso.pm @@ -87,8 +87,15 @@ sub methods { return ( france => \&bourso, europe => \@labels); } } -sub bourso { +sub bourso_to_number { + my $x = shift(@_); + $x =~ s/\s//g ; # remove spaces etc in number + return $x; +} + + +sub bourso { my $quoter = shift; my @stocks = @_; my (%info,$reply,$url,$te,$ts,$row,$style); @@ -120,8 +127,8 @@ sub bourso { next ; }; - my $symbol = $symbolline[0]->as_text; - ($symbol) = ($symbol=~m/(\w+) ?-/); + my $symbol = ($symbolline[0]->content_list)[0]; + ($symbol) = ($symbol=~m/(\w+)/); $info{$stocks,"symbol"}=$symbol; # retrieve NAME @@ -172,18 +179,27 @@ sub bourso { } } else { + # regular stock - my $keys = ($infoclass->look_down('class','t01'))[0]; # this div contains all keys - my $data = ($infoclass->look_down('class','t03'))[0]; # this div contains all values - my @keys = $keys->look_down('_tag','li'); - my @values = $data->look_down('_tag','li'); + my $infoelem = ($infoclass->look_down('class','info-valeur'))[0]; - foreach my $i (0..$#keys) { - my $keytext = ($keys[$i])->as_text; - my $valuetext = ($values[$i])->as_text; - $tempinfo{$keytext} = $valuetext; - } + my @rows = $infoelem->look_down('_tag','tr'); + foreach my $i (0..$#rows) { + my $row = $rows[$i]; + my @cells = $row->look_down('_tag','td'); + my $j = 0; + if ($cells[0]->attr('rowspan')) { + $j = 1; + } + if ($cells[0]->attr('colspan')) { + next; + } + + my $keytext = ($cells[$j])->as_text; + my $valuetext = ($cells[$j + 1])->as_text; + $tempinfo{$keytext} = $valuetext; + } } foreach my $key (keys %tempinfo) { @@ -192,7 +208,8 @@ sub bourso { ASSIGN: for ( $key ) { # OPCVM /Valeur liquidative/ && do { - my ($last, $currency) = ($tempinfo{$key} =~ m/(\d*.?\d*)\s*(\w*)/); + my ($last, $currency) = ($tempinfo{$key} =~ m/(\d+(?:\s\d+)*(?:\.\d+)?)(?:\(c\))?(?:\s+(\w+))?/); + $last = bourso_to_number($last); $info{$stocks,"last"} = $last; $info{$stocks,"currency"} = $currency; }; @@ -205,7 +222,8 @@ sub bourso { }; # REGULAR STOCK /Cours/ && do { - my ($last, $currency) = ($tempinfo{$key} =~ m/(\d*.?\d*)\(c\)\s*(\w*)/); + my ($last, $currency) = ($tempinfo{$key} =~ m/(\d+(?:\s\d+)*(?:\.\d+)?)(?:\(c\))?(?:\s+(\w+))?/); + $last = bourso_to_number($last); $info{$stocks,"last"} = $last; $info{$stocks,"currency"} = $currency||"EUR"; # defaults to EUR my $exchange = $key; @@ -215,27 +233,26 @@ sub bourso { /Variation/ && do { $info{$stocks,"p_change"}=$tempinfo{$key} }; - /Dernier echange/ && do { + /Dernier .change/ && do { my ($day,$month,$year) = ( $tempinfo{$key} =~ m|(\d\d)/(\d\d)/(\d\d)| ); $year+=2000; $info{$stocks,"date"}= sprintf "%02d/%02d/%04d",$day,$month,$year; $quoter->store_date(\%info, $stocks, {eurodate => $info{$stocks,"date"}}); }; /Volume/ && do { - $info{$stocks,"volume"}= $tempinfo{$key} ; - $info{$stocks,"volume"} =~ s/\s//g ; # remove spaces etc in number + $info{$stocks,"volume"}=bourso_to_number($tempinfo{$key}); }; /Ouverture/ && do { - $info{$stocks,"open"}=$tempinfo{$key} + $info{$stocks,"open"}=bourso_to_number($tempinfo{$key}); }; /Haut/ && do { - $info{$stocks,"high"}=$tempinfo{$key} + $info{$stocks,"high"}=bourso_to_number($tempinfo{$key}); }; /Bas/ && do { - $info{$stocks,"low"}=$tempinfo{$key} + $info{$stocks,"low"}=bourso_to_number($tempinfo{$key}); }; - /Cloture veille/ && do { - $info{$stocks,"previous"}=$tempinfo{$key} + /Cl.ture veille/ && do { + $info{$stocks,"previous"}=bourso_to_number($tempinfo{$key}); }; /Valorisation/ && do { $info{$stocks,"cap"}=$tempinfo{$key} ; diff --git a/t/bourso.t b/t/bourso.t index c058ccf..cdca42e 100755 --- a/t/bourso.t +++ b/t/bourso.t @@ -7,14 +7,23 @@ if (not $ENV{ONLINE_TEST}) { plan skip_all => 'Set $ENV{ONLINE_TEST} to run this test'; } -plan tests => 26; +plan tests => 65; # Test Bourso functions. my $q = Finance::Quote->new(); # my stocks = stock, fund, warrant, bond, indice -my @stocks = ("AF","MSFT","SOLB","CNP"); +my @stocks = ( + "FR0000441677", # Fund + "AF", # Stock, EUR, Nyse Euronext + "MSFT", # Stock, USD, NASDAQ + "SOLB", # Stock, EUR, BRUXELLES + "CNP", # Stock, EUR, Nyse Euronext + "FR0010371401", # Bond + "FR0010707414", # Warrant + "FR0003500008", # Index +); # Bourso tests need to cover all the possible cases: @@ -22,23 +31,29 @@ my @stocks = ("AF","MSFT","SOLB","CNP"); # Name What Test Case # # cours-action Stock AF -# cours-obligation Bond FR0010112052 +# cours-obligation Bond FR0010371401 # opcvm/opcvm Fund FR0000441677 -# cours-warrant Warrant FR0010639880 +# cours-warrant Warrant FR0010707414 # cours-indice Index FR0003500008 my $year = (localtime())[5] + 1900; my $lastyear = $year - 1; -my %quotes = $q->fetch("bourso", @stocks); -ok(%quotes); +my %quotes; +#my %quotes = $q->fetch("bourso", @stocks); +#ok(%quotes); # Check that the name, last, currency and date are defined for all of the stocks. foreach my $stock (@stocks) { + eval{ + %quotes = $q->fetch("bourso", $stock); + ok(%quotes); + ok($quotes{$stock,"last"} > 0); ok(length($quotes{$stock,"name"})); + ok(length($quotes{$stock,"symbol"})); ok($quotes{$stock,"success"}); - ok( ($stock eq "FR0010112052") || # bonds and indexes are quoted in percents + ok( # indexes are quoted in percents ($stock eq "FR0003500008") || (($stock eq "MSFT") && ($quotes{$stock, "currency"} eq "USD")) || ($quotes{$stock, "currency"} eq "EUR") ); @@ -46,6 +61,11 @@ foreach my $stock (@stocks) { substr($quotes{$stock,"isodate"},0,4) == $lastyear); ok(substr($quotes{$stock,"date"},6,4) == $year || substr($quotes{$stock,"date"},6,4) == $lastyear); + }; + if ($@){ + print STDERR "Error fetching stock ", $stock, "\n", $@; + ok(!1); + }; } # Check that a bogus stock returns no-success.
On Sun Dec 12 15:53:00 2010, laurent.wozniak wrote: Show quoted text
> Hello, > > Here is the new patch. > All tests went fine. > > The site has made some big changes in the structure of its pages, > passing from lists (li) to tables. > > Would you have time to apply this patch soon ?
Thanks Laurent ! Works like a charme. Patch applied! Best -- erik