| From: | "James" <James_Armstrong [...] cox.net> |
| To: | <bug-POE-Component-Client-HTTP [...] rt.cpan.org> |
| CC: | "'Jeff Bisbee'" <jbisbee [...] cpan.org> |
| Subject: | Many ( > 5) concurrent HTTP exchanges cause all to fail |
| Date: | Thu, 5 May 2005 00:31:16 -0700 |
--> You are now talking on #PoE
--- Topic for #PoE is 'POE Operates Efficiently' | 'Power Over Everything' |
billn: I think I just outsmarted myself. | PoCo::Lightspeed is on CPAN, play
with it!
--- Topic for #PoE set by
Apocalypse!~apoc@c-24-6-201-170.hsd1.ca.comcast.net at Sun Apr 24 11:11:11
2005
<jda2000> Hello, If I want to have concurrent HTTP requests, must I call
POE::Component::Client::HTTP->spawn() for each desired concurrent request?
<LotR> jda2000: no
<jda2000> LotR, Then it would be a bug if an error in one request causes all
concurrent requests to fail?
<LotR> jda2000: sounds like it, yes
<dngor> I would suggest running a test case against LotR's version of
Client::HTTP before reporting it as a bug, though. It may already be fixed.
And the version in my repository's currently bloxed.
* LotR would say report a bug if you have a nice small script that exhibits
the problem
<jda2000> LotR, I was using Version 0.65 from CPAN.
<LotR> jda2000: *nod*
<LotR> jda2000: so try to reproduce the problem with as few lines of code as
possible, and send that to bug-POE-Component-Client-HTTP@rt.cpan.org
<jda2000> LotR, Ok
Jeff,
*** I am resending this message to the address indicated by LotR.
*** It is plausible that the error is in Client::HTTP.
Thanks for the quick turn-around on the XML::RSS::Feed issue, I'll pick up
the fix as soon as I figure out how to get CPAN to do an update :-).
I thought I would play around a bit more to get familiar with the code. The
included code fails in an interesting way. It seems to connect to all web
servers, have a problem somewhere and then report that all of the downloads
failed. The 26 "[!!] Failed to fetch" messages all come out together. This
seems very un-POE like. I would expect the downloads to be more concurrent.
Also, I would expect that a download that works by itself would still work
even if another download should fail.
Is there a limit I need to adjust somewhere to accommodate more sockets?
I think I saw some sort of factory object being used in one of your
examples. Should I be using that?
I'd be glad to do another bug report but I'm not sure at this point which
component to write it against.
Thanks,
JDA
------------------------------------------------%<--------------------------
--------------------
#!/usr/bin/perl -w
use strict;
use POE;
use POE::Component::RSSAggregator;
my @feeds;
sub LoadFeeds {
my $Input;
my %names;
# open ($Input, "</home/jda2000/bashpodder/bp.conf") or die "Can't open
bp.conf";
while (<DATA>) {
if (m/^#/) { next; } # permit commenting out URLs
my $Item = {};
chomp;
my $url = $_;
my $name = 'whatever';
print "URL = " . $url . "\n";
$Item->{'url'} = $url;
if ($url=~m/\/$/) {
$url=~m/([^\/]+)\/$/;
$name = $1;
}
else {
$url=~m/([^\/]+)$/;
$name = $1;
}
$name =~ s/\..*//;
while (exists($names{$name})) {
$name .= 'I';
}
$names{$name} = 1;
$Item->{'name'} = $name;
print "Name = " . $name . "\n";
$Item->{'delay'} = 60;
push (@feeds, $Item);
}
# close($Input);
}
LoadFeeds;
POE::Session->create(
inline_states => {
_start => \&init_session,
handle_feed => \&handle_feed,
}
);
$poe_kernel->run();
sub init_session
{
my ($kernel, $heap, $session) = @_[KERNEL, HEAP, SESSION];
$heap->{rssagg} = POE::Component::RSSAggregator->new(
alias => 'rssagg',
debug => 1,
callback => $session->postback("handle_feed"),
tmpdir => '/tmp', # optional caching
);
$kernel->post('rssagg','add_feed',$_) for @feeds;
}
sub handle_feed
{
my ($kernel,$feed) = ($_[KERNEL], $_[ARG1]->[0]);
for my $headline ($feed->late_breaking_news) {
# do stuff with the XML::RSS::Headline object
print $headline->headline . "\n";
}
}
__END__
http://rasterweb.net/raster/feeds/rwaudio.rss
http://radio.weblogs.com/0001014/categories/dailySourceCode/rss.xml
http://radio.weblogs.com/0001014/categories/qotd/rss.xml
http://secrets.scripting.com/xml/rss.xml
http://ruk.ca/rss/formosa.xml
http://www.scripting.com/rss.xml http://feeds.feedburner.com/career
http://www.evilgeniuschronicles.org/static/index.rss
http://feeds.feedburner.com/esc
http://feeds.feedburner.com/ITConversations-EverythingMP3
http://pentdego.com/feeds/freeflow.xml
http://www.blogosphereradio.com/rss.aspx
http://feeds.feedburner.com/Techaura
http://thelinuxlink.net/tllts/tllts.rss
http://radio.linuxquestions.org/syndicate/lq.php
http://www.lugradio.org/episodes.rss
http://linux.quicksurf.com/wp-rss2.php
http://www.binrev.com/radio/podcast/
http://www.infonomicon.org/info.xml
http://feeds.feedburner.com/sliceofscifi
http://feeds.feedburner.com/YeastRadio.xml
http://leoville.tv/tlr/tlr.php http://thelinuxbox.org/show/tlbs.rss
http://www.dawnanddrew.com/rss2.xml
http://www.sciencecast.net/podcast.php
http://feeds.feedburner.com/ScienceCast