Bug #22358 for Net-Analysis: Doesn't do live capture

Fri Oct 20 09:52:01 2006 david [...] davidpashley.com - Ticket created

Subject:	Doesn't do live capture
Date:	Fri, 20 Oct 2006 14:50:59 +0100
To:	bug-Net-Analysis [...] rt.cpan.org
From:	David Pashley <david [...] davidpashley.com>

I've added a patch to provide live capturing of network traffic: It basically moves all the loop into a new function which gets passed the pcap object. I then duplicated the loop_file function. It takes a filter parameter. --- /home/david/Net-Analysis-0.05/lib/Net/Analysis/EventLoop.pm 2006-09-02 15:53:02.000000000 +0100 +++ /usr/share/perl5/Net/Analysis/EventLoop.pm 2006-10-19 21:20:38.000000000 +0100 @@ -37,14 +37,51 @@ # {{{ loop_file sub loop_file { - my ($self) = shift; - my %h = validate (@_, { filename => { type => SCALAR } }); + my ($self) = shift; + my %h = validate (@_, { filename => { type => SCALAR } }); - my ($np_err); - my ($pfd) = Net::Pcap::open_offline ($h{filename}, \$np_err); + my ($np_err); + my ($pfd) = Net::Pcap::open_offline ($h{filename}, \$np_err); - carp "event_loop('$h{filename}') failed: '$np_err'\n" if (defined $np_err); + carp "event_loop('$h{filename}') failed: '$np_err'\n" if (defined $np_err); + $self->event_loop($pfd); +} +# }}} + +# {{{ loop_net +sub loop_net { + my ($self) = shift; + my %h = validate (@_, { filter => { type => SCALAR } }); + + my $promisc = 0; + my $snaplen = 10240; + my $to_ms = 0; # timeout + my $opt=1; + my($err,$net,$mask,$filter_t); + + my ($np_err); + + my $dev = Net::Pcap::lookupdev(\$np_err); + + Net::Pcap::lookupnet($dev, \$net, \$mask, \$np_err); + + my $pfd = Net::Pcap::open_live($dev, $snaplen, $promisc, $to_ms, \$np_err); + + if ( Net::Pcap::compile($pfd, \$filter_t, $h{filter}, $opt, $net) == -1 ) { + die "Unable to compile filter string '$h{filter}'\n"; + } + + Net::Pcap::setfilter($pfd, $filter_t); + + carp "event_loop('$h{filter}') failed: '$np_err'\n" if (defined $np_err); + $self->event_loop($pfd); +} +# }}} +# {{{ event_loop +sub event_loop { + my ($self) = shift; + my $pfd = shift; $self->{dispatcher}->emit_event (name => 'setup'); while (1) { -- David Pashley david@davidpashley.com Nihil curo de ista tua stulta superstitione.

Sun Nov 12 13:30:57 2006 adam-pause [...] worrall.cc - Correspondence added

This patch has been folded into v0.06, along with some documentation. Let me know how it works out for you ! Thanks, - Adam

Sun Nov 12 13:30:57 2006 The RT System itself - Status changed from 'new' to 'open'

Sun Nov 12 13:30:58 2006 adam-pause [...] worrall.cc - Status changed from 'open' to 'resolved'

Sun Nov 12 15:08:55 2006 david [...] davidpashley.com - Correspondence added

Subject:	Re: [rt.cpan.org #22358] Doesn't do live capture
Date:	Sun, 12 Nov 2006 20:07:38 +0000
To:	via RT <bug-Net-Analysis [...] rt.cpan.org>
From:	David Pashley <david [...] davidpashley.com>

On Nov 12, 2006 at 18:30, via RT praised the llamas by saying: Show quoted text

> > <URL: http://rt.cpan.org/Ticket/Display.html?id=22358 > > > This patch has been folded into v0.06, along with some documentation. > > Let me know how it works out for you ! > > Thanks, > > - Adam

How was the vacation? I'll try to find the capture file for you. I have noticed one problem with running the live capture version. It seems to miss off certain connections. It appears that it just doesn't finish the connections. With the offline version, these get outputted at teardown time. I've been able to mostly deal with this by using a sig handler for SIGINT to emit a teardown event. I don't know if this is something that should be provided by the base distribution. It can be done in user code. You may just want to add some documentation: sub setup { my $self = shift; $SIG{'INT'} = sub { print STDERR "Caught Ctrl-C\n"; $self->emit (name => 'teardown'); $SIG{'INT'} = 'DEFAULT'; exit; }; } I don't know if there's a way to find connections that haven't had any packets recently and clear those out. -- David Pashley david@davidpashley.com Nihil curo de ista tua stulta superstitione.

Sun Nov 12 15:08:56 2006 The RT System itself - Status changed from 'resolved' to 'open'

Sun Nov 12 20:59:34 2006 adam [...] worrall.cc - Correspondence added

Subject:	Re: [rt.cpan.org #22358] Doesn't do live capture
Date:	Mon, 13 Nov 2006 01:52:42 -0000 (GMT)
To:	bug-Net-Analysis [...] rt.cpan.org
From:	"Adam Worrall" <adam [...] worrall.cc>

david@davidpashley.com via RT wrote: Show quoted text

> > How was the vacation?

Not too bad, although work piled up while I was away ? Show quoted text

> I'll try to find the capture file for you. I have noticed one problem > with running the live capture version. It seems to miss off certain > connections. It appears that it just doesn't finish the connections.

Hmm. Maybe we have some more bugs in the TCP session code. Or, perhaps, we have some naughty TCP behaviour; perhaps some sessions are being killed or interrupted, and thus not being cleanly closed. If we had a capture from tcpdump to compare against, it should be pretty easy to find out why a particular TCP session lingers in Net::Analysis. Show quoted text

> With the offline version, these get outputted at teardown time. I've > been able to mostly deal with this by using a sig handler for SIGINT to > emit a teardown event.

That's a nice idea. It will terminate *all* TCP sessions, though, which you may or may not care about. Some kind of reaper is probably useful, although I don't want to end up reaping legitimate tcp sessions just because they've not seen a packet for a few minutes. If the problem sessions turn out to be ones that have partially closed, but not fully closed, then I could implement an analogue of TIME_WAIT. If the problems are simply interrupted sessions, then maybe a 30-60m timeout would be appropriate. Would that work for you, or do you need something more immediate ? Thanks, - Adam

Sun Nov 12 22:31:04 2006 david [...] davidpashley.com - Correspondence added

Subject:	Re: [rt.cpan.org #22358] Doesn't do live capture
Date:	Mon, 13 Nov 2006 03:30:32 +0000
To:	"adam [...] worrall.cc via RT" <bug-Net-Analysis [...] rt.cpan.org>
From:	David Pashley <david [...] davidpashley.com>

On Nov 13, 2006 at 01:59, adam@worrall.cc via RT praised the llamas by saying: Show quoted text

> > <URL: http://rt.cpan.org/Ticket/Display.html?id=22358 > > > david@davidpashley.com via RT wrote:

> > > > How was the vacation?

> > Not too bad, although work piled up while I was away ? >

Always the way :) Show quoted text

> > I'll try to find the capture file for you. I have noticed one problem > > with running the live capture version. It seems to miss off certain > > connections. It appears that it just doesn't finish the connections.

> > Hmm. Maybe we have some more bugs in the TCP session code. Or, perhaps, > we have some naughty TCP behaviour; perhaps some sessions are being killed > or interrupted, and thus not being cleanly closed. >

I'm thinking that it's actually down to tcpdump losing packets. I've run tcpdump at the same time and then when comparing the two the offline always gets the data but the live loses some of it. I'm not sure if these are then displayed on exit or not as I've only just implemented the reaping on sigint code. I should be able to find out tomorrow. Show quoted text

> If we had a capture from tcpdump to compare against, it should be pretty > easy to find out why a particular TCP session lingers in Net::Analysis. >

I will attempt to find a connection that isn't in the live output but is in the offline version when I compare the two logs. It might be helpful to get the live version to store packets in a tcpdump file too for debugging. Show quoted text

> > With the offline version, these get outputted at teardown time. I've > > been able to mostly deal with this by using a sig handler for SIGINT to > > emit a teardown event.

> > That's a nice idea. It will terminate *all* TCP sessions, though, which > you may or may not care about. >

Well you're being asked to die, so I think it's fair enough to kill all sessions :) I agree though that some won't be completed and therefore corrupt. Show quoted text

> Some kind of reaper is probably useful, although I don't want to end up > reaping legitimate tcp sessions just because they've not seen a packet > for a few minutes. > > If the problem sessions turn out to be ones that have partially closed, > but not fully closed, then I could implement an analogue of TIME_WAIT. > > If the problems are simply interrupted sessions, then maybe a 30-60m > timeout would be appropriate. Would that work for you, or do you need > something more immediate ? >

I think that could well be suitable. It's not like it's vitally important to have every last packet inspected. Obviously this sounds like an option somewhere. I think this would mostly be useul for reclaiing memory by pruning some of the caches perl stores. Show quoted text

> Thanks, > > - Adam > >

-- David Pashley david@davidpashley.com Nihil curo de ista tua stulta superstitione.