| Subject: | Nagios::StatusLog very slow parsing large files |
When parsing the status log for our relatively large Nagios install
(290 hosts, 1347 services), it was taking quite a while (6s+, possibly
up to ten (including a little XML processing too)), which wasn't
acceptable for our usage. We've therefore applied the following fix,
which as far as I can tell won't actually affect the output, it
certainly doesn't seem to on our system, I can think of a few cases
where it might, but I believe they would break the existing parser
anyway.
Either for future inclusion perhaps, or just as an option for people
who want a speed increase. Our total script processing, including some
XML processing is now about 1.2 seconds on average.
The subroutine becomes:
sub update_v2 ($) {
my $self = shift;
my %handlers = (
host => sub {
my $item = shift;
my $host = $item->{host_name};
if ( !exists $self->{HOST}{$host} ) {
$self->{HOST}{$host} = {};
}
_copy( $item, $self->{HOST}{$host} );
},
service => sub {
my $item = shift;
my $host = $item->{host_name};
my $svc = $item->{service_description};
if ( !exists $self->{SERVICE}{$host}{$svc} ) {
$self->{SERVICE}{$host}{$svc} = {};
}
_copy( $item, $self->{SERVICE}{$host}{$svc} );
},
info => sub {
_copy( shift, $self->{INFO} );
},
program => sub {
_copy( shift, $self->{PROGRAM} );
}
);
my $log_fh = gensym;
open( $log_fh, "<$self->{LOGFILE}" )
|| croak "could not open file $self->{LOGFILE} for reading: $!";
# change the first line of the RE to this:
# (info|program|host|service) \s* {(
# to make it a bit more careful, but it has a measurable cost on
runtime
my $entry_re = qr/
# capture the type into $1
(\w+) \s*
# capture all of the text between the brackets into $2
{( .*? )}
/xs;
my @lines = <$log_fh>;
my $file = "@lines";
#Drop comments because we don't need them.
$file =~ s/#.*\n//mg;
$file =~ s/[\r\n]+\s*/\n/g; # clean up whitespace and
newlines
while($file =~ /$entry_re/g){
( my $type, my $text) = ( $1, $2 );
my %item = map { split /\s*=\s*/, $_, 2 }
split /\n/, $text;
$handlers{$type}->( \%item );
}
close( $log_fh );
1;
}