Bug #41749 for SNMP-Extension-PassPersist: performance patch: save sorted entries

Tue Dec 16 06:31:38 2008 kbeevers [...] voxel.net - Ticket created

Subject:	performance patch: save sorted entries
Date:	Tue, 16 Dec 2008 06:31:16 -0500 (EST)
To:	bug-snmp-extension-passpersist [...] rt.cpan.org
From:	Kris Beevers <kbeevers [...] voxel.net>

Hi Sébastien, I am working with a large deployment and using SNMP to expose a lot of data on some very busy machines. I decided to try your module --- thanks, it's great! I ran into a bit of an issue exporting large-ish tables (~500 OIDs): the module consumes a _lot_ of CPU. Mostly, I found this was because it was sorting that many OIDs for every getnext request. Using Sort::Key::OID improved things dramatically, but while I was at it I also modified PassPersist to save the sorted array of OIDs and only invalidate it when a new OID or tree is added, rather than re-sorting for every getnext request. It appears this doesn't really make that big of a performance difference, but every little bit counts. (Probably the biggest remaining killer is the large number of system calls, which is partly an architectural problem with NET-SNMP's pass_persist protocol.) In any case, here's a patch with the changes I made; feel free to incorporate them if you find them useful. Thanks again for the good work! Regards, Kris

Message body is not shown because sender requested not to inline it.

Wed Dec 17 18:35:21 2008 SAPER [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #41749] performance patch: save sorted entries
Date:	Thu, 18 Dec 2008 00:35:03 +0100
To:	bug-SNMP-Extension-PassPersist [...] rt.cpan.org
From:	Sébastien Aperghis-Tramoni <saper [...] cpan.org>

Kris Beevers wrote via RT: Show quoted text

> Hi Sébastien,

Hello Show quoted text

> I am working with a large deployment and using SNMP to expose a lot > of data on some very busy machines. I decided to try your module > --- thanks, it's great!

Heh, you're welcome. Show quoted text

> I ran into a bit of an issue exporting large-ish tables (~500 > OIDs): the module consumes a _lot_ of CPU. Mostly, I found this > was because it was sorting that many OIDs for every getnext > request. Using Sort::Key::OID improved things dramatically, but > while I was at it I also modified PassPersist to save the sorted > array of OIDs and only invalidate it when a new OID or tree is > added, rather than re-sorting for every getnext request.

Indeed, that's a good idea. Show quoted text

> It appears this doesn't really make that big of a performance > difference, but every little bit counts. (Probably the biggest > remaining killer is the large number of system calls, which is > partly an architectural problem with NET-SNMP's pass_persist > protocol.)

I see some spots where it should be optimised a little further: you're making several unnecessary calls to sorted_entries(), as the reference can be stored in a local variable when the method is called more than once. For example in fetch_next_entry() you're making 6 calls to sorted_entries(), and unfortunately, function call in Perl is quite slow, and method call is slower. Show quoted text

> In any case, here's a patch with the changes I made; feel free to > incorporate them if you find them useful.

Thanks, applied and optimised. I just uploaded a new version of SNMP::Extension::PassPersist, it should soon be available at http://search.cpan.org/dist/SNMP-Extension-PassPersist/ -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.

Wed Dec 17 18:35:21 2008 The RT System itself - Status changed from 'new' to 'open'

Sat Dec 27 12:23:11 2008 SAPER [...] cpan.org - Status changed from 'open' to 'resolved'

Sat Dec 27 12:23:11 2008 SAPER [...] cpan.org - Fixed in 0.03 added

Sat Dec 27 12:23:11 2008 SAPER [...] cpan.org - Severity Normal added

Sat Dec 27 12:23:11 2008 SAPER [...] cpan.org - Broken in 0.02 added