| Subject: | More control over non-blocking callbacks and the Dispatcher |
Hi. I've been bugging you about various things with the Net::SNMP for a
while, but I'd like to talk about some serious analysis of the
non-blocking behaviors and what I (and probably others) would want to see.
I'm open to developing this and patching it myself just fine. My goal
is to take some of the code wrappers I've done and figure out exactly
how to integrate it into the fold. From what I've found, non-blocking
usage really only boils down to a few needs, and any missing
functionality could be added in without the need for an entire module or
different processing engine. Other modules usually end up re-creating
core subroutines, anyway, and why do that when it can just be added in.
So, in general, here are cases where non-blocking breakpoints/callbacks
are desired (the "problems"):
1. After a single request has been processed (get_request, set_request,
get_table, get_bulk, etc.)
2. After a single row is acquired (via get_entries)
3. After all of the requests for a single session have been processed
4. Find ways of processing multiple sessions/hosts.
Solutions:
1. This is already built-in as -callback.
2. There is an "undocumented and unsupported" feature called
-rowcallback. Why this is "undocumented and unsupported" is beyond me
because this is awesome for taking large SNMP tables and putting them
directly as rows into CSV/XML/databases/etc. without having to worry
about storing the entire table of columns before parsing them later.
You throw a row into the file and forget about it. What's required to
take this lovely feature into a "supported" status?
3. There are a number of potential solutions just for 3:
3a. Use the -callback function of the single request and collect the
data until you run your own callback - This has been my reasoning for
using stuff like snmp_dispatch_once. However, it's turned into my own
set of pre-loading and looping routines that really belong in the module
in some form. I end up caching all of the data until the host is done
with the data dump and then call my own callback. This should be
Net::SNMP::Dispatcher's job, since it already has caching and callback
functions of its own. Ideally, every single non-blocking operation
should just be able to throw in the entire list of events, and fire
snmp_dispatcher() once to finish everything up.
3b. Use something like a single get_bulk request for everything plus
-callback - I'm not sure if that even works. Can you request, say, 20
different tables and expect it not to choke? Would that be chopped up
into multiple get_table requests, or is this just going to be a
low-level get_bulk SNMP request that takes your requests verbatim as a
single packet and immediately pukes at the size of the request? This
also doesn't solve for (admittedly rare) scenarios like getting and
setting in the same session.
3c. Use EV or AnyEvent - That doesn't actually solve anything. It's
just a different dispatching framework, and there's nothing there that
says "wait until I'm done processing requests for this host". Besides,
it's too open-ended. You end up having to roll your own subroutines and
pray that you've done them right.
3d. Create a -hostcallback option tied to the $session - This sounds
like the best solution. It's built into Net::SNMP, requires the least
amount of code, and provides a way of adding that hook easily. Requests
can be kept tracked with an internal variable bound to the $session
object, similar to how requests are registered and deregistered on the
file descriptors. When it reaches zero on the final request, the
dispatcher sends all of the cached data to the callback sub.
4. There's more than one way for this one, too:
4a. When the host is finished, queue another one - If 3d was solved
and implemented, the hostcallback sub itself could do that job. Though,
it's a common operation to say "okay, I've processed 50 hosts and I want
to add one to continue the batch". Why should that common code be
outside the module? However, that would also result in multiple event
queues: one for real-time processing, and one for processing after a
host has finished up its results. That's doable with another internal
"queuing" variable (or even some extra variables in the event objects).
But what would be the queue size? It could just be a variable for
the dispatcher, but that just gives the random figure to user to
"control". AnyEvent::SNMP does the same thing with a MIN/MAX queue
size, but it's just an arbitrary figure. I could process 500 hosts and
do just fine for one set of data, or process 20 hosts and have problems
on another set of data. Another way would be to...
4b. ...Be able to queue all of the hosts/requests prior to calling
the dispatcher - Sounds better than messing with queue numbers, but now
there could be some potential overload issues here. Let's say I want to
query 4400 hosts to grab 75 different SNMP (real) tables. (Seriously,
this is my real-world scenario right now.) All of the send PDUs are
sent to all of the hosts first, so it would end up being a massive
cluster UDP traffic getting dumped on the buffer (and eventually thrown
out). Some of the hosts would respond quickly enough to bring back the
results, but it's questionable if the zero-event queuing of replies
(including multi-reply/requests, like get_bulk and get_table) would
balance out the massive list of requests.
A saving grace here is that the event scheduler somewhat prioritizes
existing reply/requests. If the file descriptor is constantly available
for data (as it should be, but not so much that it tosses out data,
either), it would only send one new request, and then process the file
descriptor right away. That file descriptor would process the results,
produce a new PDU, send it out, and then control would return back to
the dispatcher. You still run into the problem of new/old IP ratios
there, though. Every single new request would likely have many
different reply/requests, but you're only looking at one file descriptor
for UDP, and thus, only process a single reply/request from an existing
IP. The result is that all 4400 hosts would eventually be asked to grab
data while the dispatcher is trying to process the many more replies
that generates. Again, that results in buffer overload. (One fix would
be to add a file descriptor for each IP, regardless if it's UDP or not.
This would force the event handler to process all of the busy
descriptors before exiting the handler.)
All of the individual hosts would be not be limited to one request at
a time, either, as it could be sending requests to a busy host before it
has received the result back from it. Depending on the quantity of
requests, this could overload the other side, or it might end up
handling it well, and sending even more junk to your UDP buffer,
overloading THAT.
4c. Find a smart way to detect overload - Okay, so spamming thousands
of requests to many hosts is not a good idea, but being able dump the
entire list of events in the event queue is ideal and would more or less
be proper use of the queue. Limits on numbers of events is arbitrary.
The UDP buffer (unfortunately) can't be queried for its existing length.
(And even if it could, would you want to read a large buffer over and
over again like that?)
I believe the file descriptor idle time is the answer, though. If
it's always busy, we're either exactly keeping up or are overloaded. If
it's idle at any point, then the buffer is empty most of the time. So,
an array of times could keep track of the last 30-50 FD idle times vs.
the code processing time between FD select calls. If 95% of the total
processing time is from the code and only 5% is from idle time, that
would be ideal. The number of sessions with active events (has
processed through its first PDU) can be tracked and then the event
handler can be forced to stop sending new requests to new IPs if it
meets that threshold. This threshold could be adjustable to how
"friendly" the user wants the process to be, in regards to CPU/resource
time. Setting it to 99.9% would also be an option, which would require
30 selects of zero response time to prevent new requests.
A hard limit on the number of multiple requests to a single IP can
also be managed, possibly tied to individual IP response time. A second
process % threshold could also start adding delay seconds to the
existing "Send PDU" events, so that the FD queries can keep up.
Sorry for the long bug report, but let me know what you think. I want
to help drive that direction and release some patches.