Skip Menu |

This queue is for tickets about the Finance-Quote CPAN distribution.

Report information
The Basics
Id: 132937
Status: open
Priority: 0/
Queue: Finance-Quote

People
Owner: Nobody in particular
Requestors: rclar.in.dc [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: FQ 1.49 TSP bug
Date: Fri, 3 Jul 2020 14:27:50 -0400
To: bug-Finance-Quote [...] rt.cpan.org
From: rclar <rclar.in.dc [...] gmail.com>
Since (I think) upgrading to FQ 1.49, TSP quotes fail; and I don't know enough perl to help much beyond that. rclar@corona ~/bin % gnc-fq-dump -v tsp CFUND TSP data table not recognised at /usr/share/perl5/Finance/Quote/TSP.pm line 103. rclar@corona ~/bin % gnc-fq-dump -v alphavantage IBM Finance::Quote fields Gnucash uses: symbol: IBM <=== required date: 07/02/2020 <=== recommended currency: USD <=== required last: 119.7000 <=\ nav: <=== one of these price: <=/ timezone: <=== optional All fields returned by Finance::Quote for stock IBM stock field value ----- ----- ----- IBM close: 119.7000 IBM currency: USD IBM currency_set_by_fq: 1 IBM date: 07/02/2020 IBM high: 121.4200 IBM isodate: 2020-07-02 IBM last: 119.7000 IBM low: 119.2600 IBM method: alphavantage IBM open: 119.6900 IBM success: 1 IBM symbol: IBM IBM volume: 3732768
Same problem here. I suspect this has to do with TSP having added several new target-date funds last week. The table format on the price history web page how has several extra columns now, so the indexes and labels in the perl script need to be updated. The code has a comment that the index system should be re-written to handle these kinds of changes. The short-term fix is probably to add the new funds into the index list?
Subject: Re: [rt.cpan.org #132937] FQ 1.49 TSP bug
Date: Wed, 8 Jul 2020 10:06:25 -0400
To: bug-Finance-Quote [...] rt.cpan.org
From: rclar <rclar.in.dc [...] gmail.com>
Yeah, I hadn't seen that. It would appear that they did away with the 2020 fund, and introduced a bunch of half decade lifecycle ones.... Grrrr... Thanks - I'll go with the short term fix for now, since that is within my perl abilities! On Wed, Jul 8, 2020 at 9:40 AM Quinn Baker via RT < bug-Finance-Quote@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=132937 > > > Same problem here. > I suspect this has to do with TSP having added several new target-date > funds last week. The table format on the price history web page how has > several extra columns now, so the indexes and labels in the perl script > need to be updated. The code has a comment that the index system should be > re-written to handle these kinds of changes. The short-term fix is probably > to add the new funds into the index list? >
Added the new funds in the attached version. I also fixed an error in the script that was giving 1-day-old quotes. It was taking the date from the first data row, but giving quote values from the next row down. I think the author didn't account for the header row being discarded automatically when they call HTML::TableExtract->new().
Subject: TSP.pm
#!/usr/bin/perl -w # # Copyright (C) 1998, Dj Padzensky <djpadz@padz.net> # Copyright (C) 1998, 1999 Linas Vepstas <linas@linas.org> # Copyright (C) 2000, Yannick LE NY <y-le-ny@ifrance.com> # Copyright (C) 2000, Paul Fenwick <pjf@cpan.org> # Copyright (C) 2000, Brent Neal <brentn@users.sourceforge.net> # Copyright (C) 2001, Rob Sessink <rob_ses@users.sourceforge.net> # Copyright (C) 2004, Frank Mori Hess <fmhess@users.sourceforge.net> # Trent Piepho <xyzzy@spekeasy.org> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA # 02111-1307, USA # # # This code is derived from version 0.9 of the AEX.pm module. require 5.005; use strict; package Finance::Quote::TSP; use vars qw( $TSP_URL $TSP_MAIN_URL %TSP_FUND_COLUMNS %TSP_FUND_NAMES); use LWP::UserAgent; use HTTP::Request::Common; use HTML::TableExtract; our $VERSION = '1.49'; # VERSION # URLs of where to obtain information $TSP_URL = 'https://www.tsp.gov/InvestmentFunds/FundPerformance/index.html'; $TSP_MAIN_URL=("http://www.tsp.gov"); # ENHANCE-ME: The decade target funds like 2020 appear and disappear. # Better not to hard code them. # %TSP_FUND_COLUMNS = ( LINCOME => "L INCOME", L2025 => "L 2025", L2030 => "L 2030", L2035 => "L 2035", L2040 => "L 2040", L2045 => "L 2045", L2050 => "L 2050", L2055 => "L 2055", L2060 => "L 2060", L2065 => "L 2065", G => "G FUND", F => "F FUND", C => "C FUND", S => "S FUND", I => "I FUND" ); %TSP_FUND_NAMES = ( LINCOME => 'Lifecycle Income Fund', L2025 => 'Lifecycle 2025 Fund', L2030 => 'Lifecycle 2030 Fund', L2035 => 'Lifecycle 2035 Fund', L2040 => 'Lifecycle 2040 Fund', L2045 => 'Lifecycle 2045 Fund', L2050 => 'Lifecycle 2050 Fund', L2055 => 'Lifecycle 2055 Fund', L2060 => 'Lifecycle 2060 Fund', L2065 => 'Lifecycle 2065 Fund', G => 'Government Securities Investment Fund', F => 'Fixed Income Index Investment Fund', C => 'Common Stock Index Investment Fund', S => 'Small Capitalization Stock Index Investment Fund', I => 'International Stock Index Investment Fund' ); sub methods { return (tsp => \&tsp) } { my @labels = qw/name date isodate currency close/; sub labels { return (tsp => \@labels); } } # ============================================================================== sub tsp { my $quoter = shift; my @symbols = @_; return unless @symbols; my(%info, %fundrows); my($ua, $reply, $row, $te, $ts, $first_row); $ua = $quoter->user_agent; $reply = $ua->request(GET $TSP_URL); return unless ($reply->is_success); $te = HTML::TableExtract->new( headers => ["Date", values %TSP_FUND_COLUMNS] ); $te->parse($reply->content); # First row is newest data, older data follows, maybe there # should be some way to get it (in addition to the second_row "close") $ts = $te->first_table_found || die 'TSP data table not recognised'; $first_row = $ts->row(0); # Make a hash that maps the order the columns are in for(my $i=1; my $key = each %TSP_FUND_COLUMNS ; $i++) { $fundrows{$key} = $i; } foreach (@symbols) { my $symbol = uc $_; if(exists $fundrows{$symbol}) { $info{$_, 'success'} = 1; $info{$_, 'name'} = $TSP_FUND_NAMES{$symbol}; $quoter->store_date(\%info, $_, {usdate => $$first_row[0]}); ($info{$_, 'last'} = $first_row->[$fundrows{$symbol}]) =~ s/[^0-9]*([0-9.,]+).*/$1/s; $info{$_, 'currency'} = 'USD'; $info{$_, 'method'} = 'tsp'; $info{$_, 'source'} = $TSP_MAIN_URL; $info{$_, 'symbol'} = $_; } else { $info{$_, 'success'} = 0; $info{$_, 'errormsg'} = "Fund name unknown"; } } return %info if wantarray; return \%info; } 1; =head1 NAME Finance::Quote::TSP Obtain fund prices for US Federal Government Thrift Savings Plan =head1 SYNOPSIS use Finance::Quote; $q = Finance::Quote->new; %info = Finance::Quote->fetch("tsp","c"); #get value of C "Common Stock Index Investment" Fund =head1 DESCRIPTION This module fetches fund information from the "Thrift Savings Plan" http://www.tsp.gov using its fund prices page https://www.tsp.gov/investmentfunds/shareprice/sharePriceHistory.shtml The quote symbols are C common stock fund F fixed income fund G government securities fund I international stock fund S small cap stock fund L2025 lifecycle fund year 2025 L2030 lifecycle fund year 2030 L2035 lifecycle fund year 2035 L2040 lifecycle fund year 2040 L2045 lifecycle fund year 2045 L2050 lifecycle fund year 2050 L2055 lifecycle fund year 2055 L2060 lifecycle fund year 2060 L2065 lifecycle fund year 2065 LINCOME lifecycle income fund =head1 LABELS RETURNED The following labels may be returned by Finance::Quote::TSP : name eg. "Lifecycle 2050 Fund" date latest date, eg. "21/02/10" isodate latest date, eg. "2010-02-21" last latest available price, eg. "16.1053" currency "USD" method "tsp" source TSP URL =head1 SEE ALSO Thrift Savings Plan, http://www.tsp.gov =cut
Of course TSP went and did a re-design of their website. This version seems to work for me. For now.
Subject: TSP.pm
#!/usr/bin/perl -w # # Copyright (C) 1998, Dj Padzensky <djpadz@padz.net> # Copyright (C) 1998, 1999 Linas Vepstas <linas@linas.org> # Copyright (C) 2000, Yannick LE NY <y-le-ny@ifrance.com> # Copyright (C) 2000, Paul Fenwick <pjf@cpan.org> # Copyright (C) 2000, Brent Neal <brentn@users.sourceforge.net> # Copyright (C) 2001, Rob Sessink <rob_ses@users.sourceforge.net> # Copyright (C) 2004, Frank Mori Hess <fmhess@users.sourceforge.net> # Trent Piepho <xyzzy@spekeasy.org> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA # 02111-1307, USA # # # This code is derived from version 0.9 of the AEX.pm module. require 5.005; use strict; package Finance::Quote::TSP; use vars qw( $TSP_URL $TSP_MAIN_URL %TSP_FUND_COLUMNS %TSP_FUND_NAMES); use LWP::UserAgent; use HTTP::Request::Common; use HTML::TableExtract; our $VERSION = '1.49'; # VERSION # URLs of where to obtain information $TSP_URL = 'https://secure.tsp.gov/components/CORS/getSharePrices.html?Lfunds=1&InvFunds=1'; $TSP_MAIN_URL=("http://secure.tsp.gov"); # ENHANCE-ME: The decade target funds like 2020 appear and disappear. # Better not to hard code them. # %TSP_FUND_COLUMNS = ( LINCOME => "L INCOME", L2025 => "L 2025", L2030 => "L 2030", L2035 => "L 2035", L2040 => "L 2040", L2045 => "L 2045", L2050 => "L 2050", L2055 => "L 2055", L2060 => "L 2060", L2065 => "L 2065", G => "G FUND", F => "F FUND", C => "C FUND", S => "S FUND", I => "I FUND" ); %TSP_FUND_NAMES = ( LINCOME => 'Lifecycle Income Fund', L2025 => 'Lifecycle 2025 Fund', L2030 => 'Lifecycle 2030 Fund', L2035 => 'Lifecycle 2035 Fund', L2040 => 'Lifecycle 2040 Fund', L2045 => 'Lifecycle 2045 Fund', L2050 => 'Lifecycle 2050 Fund', L2055 => 'Lifecycle 2055 Fund', L2060 => 'Lifecycle 2060 Fund', L2065 => 'Lifecycle 2065 Fund', G => 'Government Securities Investment Fund', F => 'Fixed Income Index Investment Fund', C => 'Common Stock Index Investment Fund', S => 'Small Capitalization Stock Index Investment Fund', I => 'International Stock Index Investment Fund' ); sub methods { return (tsp => \&tsp) } { my @labels = qw/name date isodate currency close/; sub labels { return (tsp => \@labels); } } # ============================================================================== sub tsp { my $quoter = shift; my @symbols = @_; return unless @symbols; my(%info, %fundrows); my($ua, $reply, $row, $te, $ts, $first_row); $ua = $quoter->user_agent; $reply = $ua->request(GET $TSP_URL); return unless ($reply->is_success); $te = HTML::TableExtract->new( headers => ["Date", values %TSP_FUND_COLUMNS] ); $te->parse($reply->content); # First row is newest data, older data follows, maybe there # should be some way to get it (in addition to the second_row "close") $ts = $te->first_table_found || die 'TSP data table not recognised'; $first_row = $ts->row(0); # Make a hash that maps the order the columns are in for(my $i=1; my $key = each %TSP_FUND_COLUMNS ; $i++) { $fundrows{$key} = $i; } foreach (@symbols) { my $symbol = uc $_; if(exists $fundrows{$symbol}) { $info{$_, 'success'} = 1; $info{$_, 'name'} = $TSP_FUND_NAMES{$symbol}; $quoter->store_date(\%info, $_, {usdate => $$first_row[0]}); ($info{$_, 'last'} = $first_row->[$fundrows{$symbol}]) =~ s/[^0-9]*([0-9.,]+).*/$1/s; $info{$_, 'currency'} = 'USD'; $info{$_, 'method'} = 'tsp'; $info{$_, 'source'} = $TSP_MAIN_URL; $info{$_, 'symbol'} = $_; } else { $info{$_, 'success'} = 0; $info{$_, 'errormsg'} = "Fund name unknown"; } } return %info if wantarray; return \%info; } 1; =head1 NAME Finance::Quote::TSP Obtain fund prices for US Federal Government Thrift Savings Plan =head1 SYNOPSIS use Finance::Quote; $q = Finance::Quote->new; %info = Finance::Quote->fetch("tsp","c"); #get value of C "Common Stock Index Investment" Fund =head1 DESCRIPTION This module fetches fund information from the "Thrift Savings Plan" http://www.tsp.gov using its fund prices page https://www.tsp.gov/investmentfunds/shareprice/sharePriceHistory.shtml The quote symbols are C common stock fund F fixed income fund G government securities fund I international stock fund S small cap stock fund L2025 lifecycle fund year 2025 L2030 lifecycle fund year 2030 L2035 lifecycle fund year 2035 L2040 lifecycle fund year 2040 L2045 lifecycle fund year 2045 L2050 lifecycle fund year 2050 L2055 lifecycle fund year 2055 L2060 lifecycle fund year 2060 L2065 lifecycle fund year 2065 LINCOME lifecycle income fund =head1 LABELS RETURNED The following labels may be returned by Finance::Quote::TSP : name eg. "Lifecycle 2050 Fund" date latest date, eg. "21/02/10" isodate latest date, eg. "2010-02-21" last latest available price, eg. "16.1053" currency "USD" method "tsp" source TSP URL =head1 SEE ALSO Thrift Savings Plan, http://www.tsp.gov =cut
On Sun Jul 12 16:17:07 2020, quinn.baker@gmail.com wrote: Show quoted text
> Of course TSP went and did a re-design of their website. > This version seems to work for me. For now.
Perhaps it is time to move this to using CSV, and not try to scrape? https://secure.tsp.gov/components/CORS/getSharePrices.html?startdate=20200826&enddate=20200925&Lfunds=1&InvFunds=1&format=CSV&download=1 That's a valid URL. If I have some time, I will attempt to rewrite the module to utilize this.
On Fri Sep 25 10:20:45 2020, brad@crochet.net wrote: Show quoted text
> On Sun Jul 12 16:17:07 2020, quinn.baker@gmail.com wrote:
> > Of course TSP went and did a re-design of their website. > > This version seems to work for me. For now.
> > > Perhaps it is time to move this to using CSV, and not try to scrape? > > https://secure.tsp.gov/components/CORS/getSharePrices.html?startdate=20200826&enddate=20200925&Lfunds=1&InvFunds=1&format=CSV&download=1 > > That's a valid URL. If I have some time, I will attempt to rewrite the > module to utilize this.
Never mind. I see that the latest code has been updated. Ignore me. :)