Skip Menu |

This queue is for tickets about the WWW-Fuel-US-Prices CPAN distribution.

Report information
The Basics
Id: 35225
Status: open
Priority: 0/
Queue: WWW-Fuel-US-Prices

People
Owner: Nobody in particular
Requestors: eric_harris_76 [...] yahoo.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Problem in WWW::Fuel::US::Prices (aka Gas::Prices?)
Date: Sat, 19 Apr 2008 11:23:32 -0500
To: bug-www-fuel-us-prices [...] rt.cpan.org, "Ashish Kasturia" <ashoooo [...] cpan.org>
From: "Eric S. Harris" <eric_harris_76 [...] mindspring.com>
It looks like WWW::Fuel::US::Prices recently quit working, perhaps because the site it screen-scrapes has changed (perhaps to thwart screen-scraping). Instead of a price, I'm now getting images (examples listed below). I'm actually using Gas::Prices, which I got via PPM, but it also has your name on it, and a version number of 0.0.4, so I assume it's the same. -Eric <img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> unleaded COSTCO at 200 COSTCO WAY, SAINT PETERS, MO 63376 <img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> unleaded SAMS at 2855 VETERANS MEMORIAL PKWY, SAINT CHARLES, MO 63303 <img src="/gimg/price.aspx?t=FXLBCbjiyFsKRnZ45WzJVA%3d%3d&b=%23FFFFFF"> unleaded SHELL at 12200 CLAYTON RD, SAINT LOUIS, MO 63131 <img src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> unleaded MOBIL at 8566 SAINT CHARLES ROCK RD, SAINT LOUIS, MO 63114 <img src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> unleaded PHILLIPS 66 at 3605 S BROADWAY, SAINT LOUIS, MO 63118 <img src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> unleaded VALERO at 1800 LUCAS AND HUNT RD, SAINT LOUIS, MO 63121 <img src="/gimg/price.aspx?t=RNBQeKU%2fSECxwAinsddXTQ%3d%3d&b=%23FFFFFF"> unleaded 7-ELEVEN at 12235 MCKELVEY RD, MARYLAND HEIGHTS, MO 63043 <img src="/gimg/price.aspx?t=yxvmgKC13I%2f%2fTvfQzUBLNw%3d%3d&b=%23FFFFFF"> unleaded UNBRANDED at 4481 CENTRAL SCHOOL RD, SAINT CHARLES, MO 63304
On Sat Apr 19 12:24:05 2008, eric_harris_76@yahoo.com wrote: Show quoted text
> It looks like WWW::Fuel::US::Prices recently quit working, perhaps > because the site it screen-scrapes has changed (perhaps to thwart > screen-scraping). Instead of a price, I'm now getting images (examples > listed below). > > I'm actually using Gas::Prices, which I got via PPM, but it also has > your name on it, and a version number of 0.0.4, so I assume it's the > same. -Eric > > > <img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> > unleaded COSTCO at 200 COSTCO WAY, SAINT PETERS, MO 63376 > <img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> > unleaded SAMS at 2855 VETERANS MEMORIAL PKWY, SAINT CHARLES, MO 63303 > <img src="/gimg/price.aspx?t=FXLBCbjiyFsKRnZ45WzJVA%3d%3d&b=%23FFFFFF"> > unleaded SHELL at 12200 CLAYTON RD, SAINT LOUIS, MO 63131 > <img > src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> > unleaded MOBIL at 8566 SAINT CHARLES ROCK RD, SAINT LOUIS, MO 63114 > <img > src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> > unleaded PHILLIPS 66 at 3605 S BROADWAY, SAINT LOUIS, MO 63118 > <img > src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> > unleaded VALERO at 1800 LUCAS AND HUNT RD, SAINT LOUIS, MO 63121 > <img > src="/gimg/price.aspx?t=RNBQeKU%2fSECxwAinsddXTQ%3d%3d&b=%23FFFFFF"> > unleaded 7-ELEVEN at 12235 MCKELVEY RD, MARYLAND HEIGHTS, MO 63043 > <img > src="/gimg/price.aspx?t=yxvmgKC13I%2f%2fTvfQzUBLNw%3d%3d&b=%23FFFFFF"> > unleaded UNBRANDED at 4481 CENTRAL SCHOOL RD, SAINT CHARLES, MO 63304 >
Yes, the formatting has changed. And this page scrap will no longer work. I do not intend to fix this module, unless someone helps me with it. -ASHOOOO
CC: Ashish Kasturia <ashoooo [...] cpan.org>, Ashish Kasturia <ashishkasturia [...] gmail.com>
Subject: Re: [rt.cpan.org #35225] Problem in WWW::Fuel::US::Prices (aka Gas::Prices?)
Date: Sat, 20 Sep 2008 19:22:38 -0500
To: bug-WWW-Fuel-US-Prices [...] rt.cpan.org
From: "Eric S. Harris" <eric_harris_76 [...] mindspring.com>

Message body is not shown because it is too large.

# Get the best known unleaded gas price for each zip code in a list, and one seller. # Version 0.0.3 #Enhancements done: #Enhancements underway: #Enhancements proposed: # o Save all places with the best zip code, not just the best; report them. # o Log each gas price, etc. encountered in a .CSV file, for later analysis use Gas::Prices2; my @zip_Chi = ("60609", "60601", "60622");# Testing. Zip in the Chicago area. my @zip_I44 = ("63025", "63039", "63069", "63079", "65441", "65453", "65559", "65560", "63128", "63129", "65401", "65479", "65559", "65571", ); # I-44 corridor. Not a zip code? "65501", my @zip_SLm = ( "63021", # Ballwin "63032", "63033", "63034", # Florissant "63101", "63108", # (downtown) "63111", # (South City) "63130", # (U City) "63141", "63146", # (Creve Coeur) "63301", "63302", "63303", "63304", # (St. Charles) "63376", # (St. Peters) ); # St. Louis metro area, not near home/work/commute my @zip = ("63031", "63043", "63074", # (work-work commute: Maryland Heights, St. Ann) "63121", # (work: Normandy) "63042", # (work commute route: Hazelwood) "63134", # (work commute route? Berkeley?) ); # St. Louis metro (parts close to home/work/commute) # my @zip = @zip_Chi; ## Test w/ stations in some Chicago zip codes # my @zip = @zip_SLm; ## Test w/ zip codes elsewhere in StL metro area # my @zip = @zip_I44; ## Run w/ stations along the I-44 corridor betw StL and Rolla # my @zip = ("63302");## one-shot value if (0) { my %tmp = (); foreach (@zip, @zip_Chi, @zip_I44, @zip_SLm) { $tmp{$_} = 1; }; @zip = keys(%tmp); }; my %PicToPrice = ( ); # Entries not found in the module but should be -- normally none my %NoPrice = (); my %results = (); # Enhancements/fixes # Done: # Doing: # To do: # Use a hash to accumulate references to (unique) gas stations; key is address # When printing, put the zip code before the station name name, remove it from the end of the address # Group results by zipcode (forget about by price) # Group results by price (forget about by zipcode) foreach (@zip) { my $zip =$_; # Move $zip into "for" stmt in line above my $gp = new Gas::Prices2($zip); if (defined($gp)) { my @gas_stations = @{$gp->get_stations}; foreach(@gas_stations) # This loop prints debug info { print "Station name:" . $_ -> {station_name} . "\n" . "Station address:" . $_ -> {station_address} . "\n" . "Unleaded price:" . $_ -> {unleaded_price} . "\n" . "Unleaded date:" . $_ -> {unleaded_date} . "\n" . "Plus price:" . $_ -> {plus_price} . "\n" . "Plus date:" . $_ -> {plus_date} . "\n" . "Premium price:" . $_ -> {premium_price} . "\n" . "Premium_date:" . $_ -> {premium_date} . "\n" . "Diesel price:" . $_ -> {diesel_price} . "\n" . "Diesel date:" . $_ -> {diesel_date} . "\n" if (0); #Don't really want to print it } my $cheapest_unleaded = $gp->get_cheapest_station("unleaded"); # print "Start of HTML_table_row:", substr($cheapest_unleaded->{HTML_table_row}, 0, 40), ":\n";#### # print "<!-- --> "; #### beg New 2008/05/01 # my $HTML_table_row = $cheapest_unleaded->{HTML_table_row}; # $cheapest_unleaded->{HTML_table_row} = ""; ## delete is better than shorten, but ... # for (my $i = 0; $i<length($HTML_table_row); $i++) { # print substr($HTML_table_row, $i, 50); # }; # my @HTML_table_row = split "<span", $HTML_table_row; # print "\n"; #### end New 2008/05/01 if ($cheapest_unleaded->{unleaded_price} == "unkwn") { my $price = $PicToPrice{$cheapest_unleaded->{unleaded_price_pic}}; if ($price) { $cheapest_unleaded->{unleaded_price} = $price; $NoPrice{$cheapest_unleaded->{unleaded_price}} = "Put this price in module: ".$price, " for this key ". $cheapest_unleaded->{unleaded_price_pic}; # print "Hey! I found a look-up price (", $cheapest_unleaded->{unleaded_price}, ") for zip ", $zip, "\n";#### } else { $NoPrice{$cheapest_unleaded->{unleaded_price_pic}} = $cheapest_unleaded->{station_address}; # print "Ho hum! I failed to look up a price (", $cheapest_unleaded->{unleaded_price_pic}, ") for zip ", $zip, "\n";#### }; } else { # print "Hey! It returned a price (", $cheapest_unleaded->{unleaded_price}, ") for zip ", $zip, "\n";#### }; #### print "<!-- --> <tr><form>Enter/correct/verify the price ", # Does not build something that reliably displays the cached image #### $cheapest_unleaded->{unleaded_price}, " </form>", #### "<span>", #### substr($cheapest_unleaded->{unleaded_price_pic},0,10), #### "http://autos.msn.com", #### substr($cheapest_unleaded->{unleaded_price_pic},10), #### "</span></tr><br>\n"; $results{$cheapest_unleaded->{station_address}} = $cheapest_unleaded; }; if (0) { print $zip . " \$" . $cheapest_unleaded->{unleaded_price} . " unleaded " . $cheapest_unleaded->{station_name} . " at " . $cheapest_unleaded->{addr_nozip} . "\n"; }; }; foreach (sort keys %results) { my $cheapest_unleaded = $results{$_}; my @addrparts = split / /,$cheapest_unleaded->{station_address}; my $zip = pop @addrparts; print $cheapest_unleaded->{unleaded_price} . " unleaded " . $zip . " " . $cheapest_unleaded->{station_name} . " at " . (join " ", @addrparts) . "\n"; }; foreach (sort keys %NoPrice) { #### print "\t\t\t\t\t\t #'", $_,"' => \"\", # ", $NoPrice{$_},"(GasPrices_Demo.pl)\n"; };
Ashish Kasturia via RT wrote: Show quoted text
><URL: http://rt.cpan.org/Ticket/Display.html?id=35225 > > >On Sat Apr 19 12:24:05 2008, eric_harris_76@yahoo.com wrote: > >
>>It looks like WWW::Fuel::US::Prices recently quit working, perhaps >>because the site it screen-scrapes has changed (perhaps to thwart >>screen-scraping). Instead of a price, I'm now getting images (examples >>listed below). >> >>I'm actually using Gas::Prices, which I got via PPM, but it also has >>your name on it, and a version number of 0.0.4, so I assume it's the >>same. -Eric >> >> >><img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> >>unleaded COSTCO at 200 COSTCO WAY, SAINT PETERS, MO 63376 >><img src="/gimg/price.aspx?t=2Jt6nc3lIrDLwLVtDeLlMg%3d%3d&b=%23FFFFFF"> >>unleaded SAMS at 2855 VETERANS MEMORIAL PKWY, SAINT CHARLES, MO 63303 >><img src="/gimg/price.aspx?t=FXLBCbjiyFsKRnZ45WzJVA%3d%3d&b=%23FFFFFF"> >>unleaded SHELL at 12200 CLAYTON RD, SAINT LOUIS, MO 63131 >><img >>src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> >>unleaded MOBIL at 8566 SAINT CHARLES ROCK RD, SAINT LOUIS, MO 63114 >><img >>src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> >>unleaded PHILLIPS 66 at 3605 S BROADWAY, SAINT LOUIS, MO 63118 >><img >>src="/gimg/price.aspx?t=I6tFJq9qYzreO9Vv%2fS9AVg%3d%3d&b=%23FFFFFF"> >>unleaded VALERO at 1800 LUCAS AND HUNT RD, SAINT LOUIS, MO 63121 >><img >>src="/gimg/price.aspx?t=RNBQeKU%2fSECxwAinsddXTQ%3d%3d&b=%23FFFFFF"> >>unleaded 7-ELEVEN at 12235 MCKELVEY RD, MARYLAND HEIGHTS, MO 63043 >><img >>src="/gimg/price.aspx?t=yxvmgKC13I%2f%2fTvfQzUBLNw%3d%3d&b=%23FFFFFF"> >>unleaded UNBRANDED at 4481 CENTRAL SCHOOL RD, SAINT CHARLES, MO 63304 >> >> >>
> > >Yes, the formatting has changed. And this page scrap will no longer >work. I do not intend to fix this module, unless someone helps me with it. > >-ASHOOOO > > >
It needs a better fix than what I've done; I'm not much of a Perl programmer. But it's some help, possibly enough. The version I've been using is attached. Its virtues and deficiencies: o It looks up the price in a hash, with the image as the key. o The hash is hard-coded in the .pm file itself, rather than in an external data file of some sort. o New images are manually entered after being manually discovered, which could introduce errors. One piece of good news: The set of images used by the web site has changed only once, soon after I started, and haven't changed since. (The old images are still in the file.) I'm thinking the way to address the deficiencies would be to: o Move the lookup info to an external file located in a suitable spot, and easily editable manually between releases. o Develop some better way to find new images and obtain their corresponding prices. I've been running a little utility that finds good prices in the zip codes along my commute. When it finds an image without a price, it generates (most of) the code to add to the hard-coded table. I fill in the blank -- the price -- by looking at part of the page (a gas station with the missing price) twice: as displayed by the browser and the HTML source. Not a good process. I'm not always able to find a price in a zip code easily. (That's why some of the lines are still commented-out and have no price.) I've included the utility, too, even though it has some extraneous crap that I haven't gotten around to deleting yet. I hope it isn't too distracting. Some extraneous non-crap: lists of zip codes in other areas. Running the utility with other lists exposes prices not covered by the hash for my usual run. (Gas prices in Chicago tend much higher than in St. Louis, so using Chicago zip codes gave me a preview of prices to come.) My efforts at improving this manual process have been thwarted by the site declining to display the image, except when part of one of its web pages. A better Perl programmer might be able to get around that. If I were a better Perl programmer -- or became a better Perl programmer -- I might be able to do those things. For now, I'll just give you the benefit of my experience, such as it is. Thank you for making the module available in CPAN, and yourself available by e-mail. -Eric -- If eric_harris_76@yahoo.com or eric_harris_76@mindspring.com ever fail, you can reach me at my new addr using http://ecoa.returnpath.net/finder or http://www.freshaddress.com/stayintouch.cfm. May 2008: The yahoo.com address is having technical difficulties.