So I found the bug, but it isn't in Net::Server, it is in munin-node (I
had to install munin and employee some bruteforce). And it is only on
munin-node on BSD under the defaults or under linux with bindv6only=1.
The bug was exploited by the recent behavior change of 2.005 which began
to look to see if IPv6 is supported. During a HUP information is passed
along to the next process via a parameter called $ENV{BOUND_SOCKETS}
which would contain something like "7|0.0.0.0|4949|TCP|ipv4" by default
in 2.004 and earlier where ipv defaulted to 4. In 2.005 ipv defaulted
to * meaning any of 4 or 6. On your box, your system is defaulting to
not have IPv6 also bind IPv4 (because sysctl net.inet6.ip6.v6only is
true by default on BSD (not on linux btw)). This means that you are now
binding two ports so BOUND_SOCKETS looks like "7|0.0.0.0|4949|TCP|ipv4
5|::|4949|TCP|ipv6" (notice the newline).
Well, for most servers this is just fine, but then there is this lovely
yet dangerous chunk of code in munin-node:
foreach my $key (keys %ENV) {
$ENV{$key} =~ /^(.*)$/;
$ENV{$key} = $1;
}
It applies a regular expression - and a bad one at that, and it doesn't
check for the outcome. Because there is now a newline in BOUND_SOCKETS,
the ^.*$ fails to match. But the next line assigns to $1 - which will
be the value of whatever the previous match was - which on my system
always ends up assigning $ENV{BOUND_SOCKET} = "/var/munin/cache/www" or
something.
So, what can be done to fix this issue? There is one thing on the
Net::Server side and several on the munin side.
On the Net::Server side I'll look at changing bound_sockets to not use
newlines as the separator.
On the munin side you can do any of the following:
Change the default host to be 127.0.0.1.
OR
Add a ipv => '4' configuration to prevent also binding ipv6
OR
Set sysctl net.inet6.ip6.v6only=0 (this isn't really a viable option but
it would fix the problem)
OR
Fix the regexes above in munin-node with something like this (assuming
you want to keep the dangerous untaint)
foreach my $key (keys %ENV) {
$ENV{$key} = $1 if $ENV{$key} =~ /^(.*)$/s;
}
# notice the conditional assignment as well as the /s
So version 2.006 will probably have a fix for this, but munin really
ought to fix the untainting code.
Paul
On Tue Jun 19 05:49:22 2012, paul@ifdnrg.com wrote:
Show quoted text> BUG
> p5-Net-Server: Munin dies Bad file descriptor after log rotation
>
> DESCRIPTION
> After a recent upgrade to p5-Net-Server-2.005 on various boxes, I'm
> finding that munin-node is going down regularly.
> Various replies on the munin lists suggest (i think correctly) that
> the problem is a bad file descriptor
> post log rotation.
>
> SAMPLE LOG FILE
> Jun 19 00:00:01 XXXXXXX newsyslog[24331]: logfile turned over
> Pid_file created by this same process. Doing nothing.
> 2012/06/19-00:00:02 Munin::Node::Server (type Net::Server::Fork)
> starting! pid(50897)
> sysctl: unknown oid 'net.ipv6.bindv6only'
> Resolved [*]:4949 to [::]:4949, IPv6
> Resolved [*]:4949 to [0.0.0.0]:4949, IPv4
> Binding open file descriptors
> 2012/06/19-00:00:02 Bad file descriptor
> at line 298 in file
> /usr/local/lib/perl5/site_perl/5.12.4/Net/Server.pm
> 2012/06/19-00:00:02 Server closing!
> shutdown() on unopened socket GEN0 at
> /usr/local/lib/perl5/5.12.4/mach/IO/Socket.pm line 295.
> shutdown() on unopened socket GEN1 at
> /usr/local/lib/perl5/5.12.4/mach/IO/Socket.pm line 295.
>
> Many thanks
> Paul.
>
