Bug #61695 for jmx4perl: Plugin output

Mon Sep 27 11:01:40 2010 giuseppe.furlan [...] hotmail.com - Ticket created

Subject:

Plugin output

Hi Roland, I've been using v0.72_1 and multicheck for a couple of days. this is my output : check_jmx4perl --config ../etc/jmx4perl.cfg --server myserver --check overview OK - All 3 checks OK [1] OK used_heap: HeapMemory= 38.03% used [2] OK jvm_cpu: JVM-CPU-Load: 0.08% used [3] OK machine_cpu: Machine-CPU-Load: 7.49% used | Heap=204172632B;429496729.6;483183820.8;0;536870912 'JVM CPU Usage'=7.651109410864575E- 4;0.8;0.9;0;1 'Machine CPU Usage'=0.07492354740061162;0.8;0.9;0;1 Basically when multicheck is enabled there are some newlines (\n) for the label's output while performance data is ok. I prefer using label's output than performance data because it's more readable but it seems not to be compatible with Nagios format. Nagios is cutting the Status Info when there is a newline and this is a problem for me because I'm using Regular Expressions to parse Status Information. In my Nagios Log I've got this detail for check_jmx4perl (note \n in long_plugin_output) : plugin_output=OK - All 3 checks OK long_plugin_output=[1] OK used_heap: HeapMemory= 35.48% used\n[2] OK jvm_cpu: JVM-CPU-Load: 0.01% used\n[3] OK machine_cpu: Machine-CPU-Load: 1.41% used performance_data=Heap=190506376B;429496729.6;483183820.8;0;536870912 'JVM CPU Usage'=8.555417718270094E-5;0.8;0.9;0;1 'Machi ne CPU Usage'=0.014115231618118826;0.8;0.9;0;1 I've noticed a difference with all my other Nagios scripts because I use "plugin_output" not "long_plugin_output" for all other Nagios scripts and they work ok, for example : plugin_output=GPING OK - rtt min/avg/max/mdev = 0.401/0.670/1.029/0.264 ms long_plugin_output= performance_data=time=0.670ms;20;50;; ok=1 Please do you have a workaround ? or could you use the same "Performance data logic" (without newlines) for Label's output too ? Thanks in advance

Tue Sep 28 13:49:24 2010 roland [...] cpan.org - Correspondence added

Hi Guiseppe,

On Mon Sep 27 11:01:40 2010, furlang75 wrote:

Show quoted text

> I've been using v0.72_1 and multicheck for a couple of days. this is my

(BTW, 0.72 is out since last friday. However, there is no big change to the latest
0.72_1 developer release)

Show quoted text

> Basically when multicheck is enabled there are some newlines (\n) for
> the label's output while performance data is ok.
> I prefer using label's output than performance data because it's more
> readable but it seems not to be compatible with Nagios format.

I'm no big Nagios expert, but I asked one (Gerhard Lausser), and it seems, that
the current output of jmx4perl is perfectly valid according to the Nagios format.
Starting with Nagios 3 multi line plugin output is allowed, hence
the new 'long_plugin_output'. 'plugin_output' itself in fact only show the first
line of the output. The reason for multiple lines for multi checks are that they are
more readable in the Nagios Webinterface. BTW, check_multi (another very
interesting plugin) does it the same way. In fact, i really tried to be as close to
check_multi as possible.

Show quoted text

> I've noticed a difference with all my other Nagios scripts because I use
> "plugin_output" not "long_plugin_output" for all other Nagios scripts
> and they work ok, for example :

Wouldn't it be an option for you, to change to 'long_plugin_output' in your scripts ?
Since multi line output is a *feature* of Nagios 3, I expect that not only
check_jmx4perl uses these (check_multi is another example for a plugin which uses
multi line output and there are probably others as well).

Show quoted text

> Please do you have a workaround ? or could you use the same "Performance
> data logic" (without newlines) for Label's output too ?

Since I believe, that the current way is a perfect valid output
(see also http://nagios.sourceforge.net/docs/3_0/pluginapi.html 
Plugin Output Examples, Case 3) I won't change it for the regular release.

If you are really desparate, you can patch the output on your own.
Got to lib/JMX/Jmx4Perl/Nagios/CheckJmx4Perl.pm, and insert after
line 139 the following substitution:

$message =~ s/\n/ - /sg;

Run 'Build install' afterwards.

If you dont mind, I would like to close this ticket.

Tue Sep 28 13:49:26 2010 The RT System itself - Status changed from 'new' to 'open'

Wed Sep 29 14:50:21 2010 giuseppe.furlan [...] hotmail.com - Correspondence added

From:

giuseppe.furlan [...] hotmail.com

Thanks mate for giving me the right direction. I enabled long plugin output, changed the regular expression and now it's working ok. Feel free to close the ticket asap. Cheers Giuseppe Il Mar 28 Set 2010 13:49:24, ROLAND ha scritto: Show quoted text

> Hi Guiseppe, > > On Mon Sep 27 11:01:40 2010, furlang75 wrote: >

> > I've been using v0.72_1 and multicheck for a couple of days. this is

> my > > (BTW, 0.72 is out since last friday. However, there is no big change > to the > latest > 0.72_1 developer release) > >

> > Basically when multicheck is enabled there are some newlines (\n)

> for

> > the label's output while performance data is ok. > > I prefer using label's output than performance data because it's

> more

> > readable but it seems not to be compatible with Nagios format.

> > I'm no big Nagios expert, but I asked one (Gerhard Lausser), and it > seems, that > the current output of jmx4perl is perfectly valid according to the > Nagios > format. > Starting with Nagios 3 multi line plugin output is allowed, hence > the new 'long_plugin_output'. 'plugin_output' itself in fact only show > the > first > line of the output. The reason for multiple lines for multi checks are > that > they are > more readable in the Nagios Webinterface. BTW, check_multi (another > very > interesting plugin) does it the same way. In fact, i really tried to > be as > close to > check_multi as possible. >

> > I've noticed a difference with all my other Nagios scripts because I

> use

> > "plugin_output" not "long_plugin_output" for all other Nagios

> scripts

> > and they work ok, for example :

> > Wouldn't it be an option for you, to change to 'long_plugin_output' in > your > scripts ? > Since multi line output is a *feature* of Nagios 3, I expect that not > only > check_jmx4perl uses these (check_multi is another example for a plugin > which > uses > multi line output and there are probably others as well). >

> > Please do you have a workaround ? or could you use the same

> "Performance

> > data logic" (without newlines) for Label's output too ?

> > Since I believe, that the current way is a perfect valid output > (see also http://nagios.sourceforge.net/docs/3_0/pluginapi.html > Plugin Output Examples, Case 3) I won't change it for the regular > release. > > If you are really desparate, you can patch the output on your own. > Got to lib/JMX/Jmx4Perl/Nagios/CheckJmx4Perl.pm, and insert after > line 139 the following substitution: > > $message =~ s/\n/ - /sg; > Run 'Build install' afterwards. > > If you dont mind, I would like to close this ticket.

Thu Sep 30 05:42:53 2010 giuseppe.furlan [...] hotmail.com - Correspondence added

From:

giuseppe.furlan [...] hotmail.com

Roland, I have found another thing could be interesting to investigate further. basically I'm using jmx4perl with weblogic 10.0.1 and jrockit. Apparently some application-server specific MBeans can be read only when predetermined MBeans are instantiated (eg bea.jrockit.management attributes can only be read after the JRockitConsole MBean has been instantiated). This is my output with jmx4perl : check_jmx4perl --config /usr/local/groundwork/nagios/etc/jmx4perl.cfg --server PreProd-SALESesesalesiptvweb01 --check overview UNKNOWN - Error: 404 javax.management.InstanceNotFoundException: bea.jrockit.management:type=Runtimejavax.management.InstanceNotFoundException: bea.jrockit.management:type=Runtime at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1010) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:627) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659) at com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:299) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1387) at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1350) at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:599) at sun.reflect.GeneratedMethodAccessor408.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) at sun.rmi.transport.Transport$1.run(Transport.java:153) at sun.rmi.transport.Transport.serviceCall(Transport.java:149) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) at java.lang.Thread.run(Thread.java:595) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:857) at org.jmx4perl.handler.ReadHandler.fetchAttributes(ReadHandler.java:122) at org.jmx4perl.handler.ReadHandler.doHandleRequest(ReadHandler.java:58) at org.jmx4perl.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:86) at org.jmx4perl.jsr160.Jsr160RequestDispatcher.dispatchRequest(Jsr160RequestDispatcher.java:59) at org.jmx4perl.backend.BackendManager.callRequestDispatcher(BackendManager.java:181) at org.jmx4perl.backend.BackendManager.handleRequest(BackendManager.java:160) at org.jmx4perl.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:127) at org.jmx4perl.http.HttpRequestHandler.handleRequestInputStream(HttpRequestHandler.java:97) at org.jmx4perl.http.AgentServlet$1.handleRequest(AgentServlet.java:165) at org.jmx4perl.http.AgentServlet.handle(AgentServlet.java:131) at org.jmx4perl.http.AgentServlet.doPost(AgentServlet.java:119) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(Unknown Source) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200) at weblogic.work.ExecuteThread.run(ExecuteThread.java:172) If I start jrockit misson control (it's a tool which include a MBean browser and jrockit runtime debug interface) then magically the previous jmx4perl command will work ok. Additionally I have found this url which this guy claims the same : http://www.torsten-horn.de/techdocs/jmx.htm Basically he created a JMX client and is claiming that you cannot read some MBeans if you don't instantiate JRockitConsole MBean before.

Thu Sep 30 06:08:36 2010 giuseppe.furlan [...] hotmail.com - Correspondence added

From:

giuseppe.furlan [...] hotmail.com

Additional url with info about JRockitConsole MBean : http://blogs.oracle.com/hirt/2008/08/jrcmd_goes_remote_r271_edition.html Giuseppe Il Gio 30 Set 2010 05:42:53, furlang75 ha scritto: Show quoted text

> Roland, > > I have found another thing could be interesting to investigate > further. > > basically I'm using jmx4perl with weblogic 10.0.1 and jrockit. > > Apparently some application-server specific MBeans can be read only > when > predetermined MBeans are instantiated (eg bea.jrockit.management > attributes can only be read after the JRockitConsole MBean has been > instantiated). > > This is my output with jmx4perl : > > check_jmx4perl --config /usr/local/groundwork/nagios/etc/jmx4perl.cfg > --server PreProd-SALESesesalesiptvweb01 --check overview > UNKNOWN - Error: 404 javax.management.InstanceNotFoundException: >

bea.jrockit.management:type=Runtimejavax.management.InstanceNotFoundException: Show quoted text

> bea.jrockit.management:type=Runtime > at >

com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1010) Show quoted text

> at >

com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:627) Show quoted text

> at >

com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659) Show quoted text

> at >

com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:299) Show quoted text

> at >

javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1387) Show quoted text

> at >

javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81) Show quoted text

> at >

javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247) Show quoted text

> at >

javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1350) Show quoted text

> at >

javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:599) Show quoted text

> at sun.reflect.GeneratedMethodAccessor408.invoke(Unknown > Source) > at >

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) Show quoted text

> at java.lang.reflect.Method.invoke(Method.java:592) > at > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) > at sun.rmi.transport.Transport$1.run(Transport.java:153) > at sun.rmi.transport.Transport.serviceCall(Transport.java:149) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) > at >

sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) Show quoted text

> at java.lang.Thread.run(Thread.java:595) > at >

sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247) Show quoted text

> at > sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126) > at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) > at > javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown > Source) > at >

javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:857) Show quoted text

> at > org.jmx4perl.handler.ReadHandler.fetchAttributes(ReadHandler.java:122) > at > org.jmx4perl.handler.ReadHandler.doHandleRequest(ReadHandler.java:58) > at >

org.jmx4perl.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:86) Show quoted text

> at >

org.jmx4perl.jsr160.Jsr160RequestDispatcher.dispatchRequest(Jsr160RequestDispatcher.java:59) Show quoted text

> at >

org.jmx4perl.backend.BackendManager.callRequestDispatcher(BackendManager.java:181) Show quoted text

> at > org.jmx4perl.backend.BackendManager.handleRequest(BackendManager.java:160) > at >

org.jmx4perl.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:127) Show quoted text

> at >

org.jmx4perl.http.HttpRequestHandler.handleRequestInputStream(HttpRequestHandler.java:97) Show quoted text

> at > org.jmx4perl.http.AgentServlet$1.handleRequest(AgentServlet.java:165) > at > org.jmx4perl.http.AgentServlet.handle(AgentServlet.java:131) > at > org.jmx4perl.http.AgentServlet.doPost(AgentServlet.java:119) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at >

weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226) Show quoted text

> at >

weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124) Show quoted text

> at >

weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283) Show quoted text

> at >

weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175) Show quoted text

> at >

weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395) Show quoted text

> at >

weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) Show quoted text

> at weblogic.security.service.SecurityManager.runAs(Unknown > Source) > at >

weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140) Show quoted text

> at >

weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046) Show quoted text

> at >

weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366) Show quoted text

> at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200) > at weblogic.work.ExecuteThread.run(ExecuteThread.java:172) > > > If I start jrockit misson control (it's a tool which include a MBean > browser and jrockit runtime debug interface) then magically the > previous > jmx4perl command will work ok. > > Additionally I have found this url which this guy claims the same : > http://www.torsten-horn.de/techdocs/jmx.htm > Basically he created a JMX client and is claiming that you cannot read > some MBeans if you don't instantiate JRockitConsole MBean before.

Thu Sep 30 06:27:52 2010 roland [...] cpan.org - Correspondence added

Hi Guiseppe,

that's interesting, because there's a similar issue with Glassfish. There
you need to execute a certain operation on an MBean which in turn
registers all statistics AMX MBeans:

On MBean amx-support:type=boot-amx you need to call the operation 'bootAMX', then
tons of extra MBeans get registered.

However, if I understand the blog post right, thinks a bit more difficult for JRockit. Here
you trigger remotely a registration of a certain MBean. Unfortunately, this is not possible
with jmx4perl yet.

My takeaway for the jmx4perl roadmap is:

* Extend the jmx4perl agent to allow for creation and destruction of MBeans
* Allow for extra JMX calls from within the check_jmx4perl configuration
(like calling an initial MBean operation or registration of an MBean).

Since I'm quite busy right now, I can't promise any schedule, though ;-)

You might be also interested in some blog post I've written some time
ago, when I experimented with remote JMX and Weblogic:
http://labs.consol.de/blog/jmx4perl/configuring-remote-jmx-access-for-weblogic/

Thanks for sharing this info ...
...roland

Fri Oct 01 06:54:57 2010 giuseppe.furlan [...] hotmail.com - Correspondence added

From:

giuseppe.furlan [...] hotmail.com

Hi Roland, If you can implement the remote MBean registration would be great. I'm looking forward to hear news from you. Security should be already guaranteed by this file $JAVA_HOME/jre/lib/management/jmxremote.access because you can insert an entry to grant access for a specific MBean. Thanks Giuseppe Il Gio 30 Set 2010 06:27:52, ROLAND ha scritto: Show quoted text

> Hi Guiseppe, > > that's interesting, because there's a similar issue with Glassfish. > There > you need to execute a certain operation on an MBean which in turn > registers all statistics AMX MBeans: > > On MBean amx-support:type=boot-amx you need to call the operation > 'bootAMX', > then > tons of extra MBeans get registered. > > However, if I understand the blog post right, thinks a bit more > difficult for > JRockit. Here > you trigger remotely a registration of a certain MBean. Unfortunately, > this is > not possible > with jmx4perl yet. > > My takeaway for the jmx4perl roadmap is: > > * Extend the jmx4perl agent to allow for creation and destruction of > MBeans > * Allow for extra JMX calls from within the check_jmx4perl > configuration > (like calling an initial MBean operation or registration of an MBean). > > Since I'm quite busy right now, I can't promise any schedule, though > ;-) > > You might be also interested in some blog post I've written some time > ago, when I experimented with remote JMX and Weblogic: > http://labs.consol.de/blog/jmx4perl/configuring-remote-jmx-access-for- > weblogic/ > > Thanks for sharing this info ... > ...roland