All Apps and Add-ons

Why am I getting this error in splunkd.log? "Error executing modular input : Connection timed out"

jorymatson
New Member

I've installed the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box (Fedora 26, OpenJDK 1.8, Python 2.7). I'm getting a "connection timed out" error that I can't figure out. I've successfully install the Splunk_TA_Unix app, so I think my forwarder is working. Any help would be appreciated.

my application is running with PID 66850 and I can connect to it with Jconsole

Troubleshooting:
I've verified that port 8089 is listening on 127.0.0.1

[splunk@fedora1]$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN

This is the error I'm seeing in splunkd.log

08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py" Error executing modular input : Connection timed out (Connection timed out) : java.lang.RuntimeException: Connection timed out (Connection timed out)
08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.HttpService.send(HttpService.java:370)
08-15-2017 14:30:36.415 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.Service.send(Service.java:1268)
.............

splunk cmd splunkd print-modinput-config jmx

    <?xml version="1.0" encoding="UTF-8"?>
    <input>
      <server_host>fedora1</server_host>
      <server_uri>https://127.0.0.1:8089</server_uri>
      <session_key>bP7pl5adKsfxtVR9Fm67EsrA3CJMdnIJbpDflNpYX50zuvc6M4XljlG9y4rwWX^vzoHtVEqgVoRwQydOvFayndEauEM^AYOXh7vEuPC1h_J6zT_eAC</session_key>
      <checkpoint_dir>/opt/splunkforwarder/var/lib/splunk/modinputs/jmx</checkpoint_dir>
      <configuration>
        <stanza name="jmx://helloworld" app="SPLUNK4JMX">
          <param name="config_file">config.xml</param>
          <param name="disabled">0</param>
          <param name="host">fedora1</param>
          <param name="index">jmx</param>
          <param name="polling_frequency">5</param>
          <param name="sourcetype">jmx</param>
        </stanza>
      </configuration>
</input>

Here is my config.xml:

<jmxpoller>
<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

<jmxserver pid="66850" jvmDescription="nlrs">

<mbean domain="java.lang" properties="type=Memory">

<attribute name="NonHeapMemoryUsage:used" outputname="nonHeapUsed"/>
<attribute name="NonHeapMemoryUsage:max" outputname="nonHeapMax"/>

</mbean>
</jmxserver>
</jmxpoller>

~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py --validate-arguments

ERROR Error parsing XML : unexpected element (uri:"", local:"input"). Expected elements are <{}items>

I'm not sure what this error is or if it's related to my problem?

0 Karma

jorymatson
New Member

I've created a new fedora box and did a fresh install the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box. I see a different error now?

-bash-4.4$ ~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py

[Fatal Error] :-1:-1: Premature end of file.
ERROR Error executing modular input : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>
 : com.splunk.HttpException: HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>

        at com.splunk.HttpException.create(HttpException.java:84)
        at com.splunk.HttpService.send(HttpService.java:399)
        at com.splunk.Service.send(Service.java:1268)
        at com.splunk.HttpService.get(HttpService.java:151)
        at com.splunk.Entity.refresh(Entity.java:381)
        at com.splunk.Entity.refresh(Entity.java:1)
        at com.splunk.Resource.validate(Resource.java:174)
        at com.splunk.Entity.validate(Entity.java:462)
        at com.splunk.Entity.getContent(Entity.java:157)
        at com.splunk.Entity.getString(Entity.java:279)
        at com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)
        at com.splunk.modinput.ModularInput.runStateCheckerThread(Unknown Source)
        at com.splunk.modinput.ModularInput.init(Unknown Source)
        at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)
0 Karma

aurelien_cattez
New Member

Hello,
i have exactly the same error message since few days. Splunk_TA_jmx suddenly stopped to work (with jdk1.7, Splunk 6.6.1 on redhat6)

0 Karma

aurelien_cattez
New Member

One of inputs was not able to connect, so authentication failed and all jmx add on stopped. It could be improved to avoid full fail in case of 1 inputs over hundred is failing... try to review your jmx authentication or inputs config.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...