Splunk DB Connect jbridge failing

[aholzer]

We had a working installation of the Splunk DB Connect app until Friday.

While no changes were made to the configuration of the app nor the network, the Splunk DB Connect app has started failing.

Here is an extract from the jbridge.log:

2013-07-24 09:56:21,221 DEBUG win32 handler registered
2013-07-24 09:56:21,243 ERROR Java process returned error code 1! Error: Unknown
2013-07-24 09:56:21,243 ERROR Command output: None
2013-07-24 09:56:21,243 DEBUG Error waiting for process: Java process returned error code 1! Error: Unknown
2013-07-24 09:56:21,243 DEBUG JavaBridgeServer terminated

Here is the relevant error message in the splunkd.log:

07-24-2013 09:58:51.387 -0400 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Can't open DatePaser XML configuration file "current": The system cannot find the file specified. - data_source="C:\Program Files\Splunk\etc\apps\dbx\bin\jbridge_server.py", data_host="********", data_sourcetype="dbx_jbridge"
07-24-2013 09:58:51.387 -0400 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Can't open DatePaser XML configuration file "current": The system cannot find the file specified. - data_source="C:\Program Files\Splunk\etc\apps\dbx\bin\jbridge_server.py", data_host="********", data_sourcetype="dbx_jbridge"
07-24-2013 09:58:51.402 -0400 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Can't open DatePaser XML configuration file "current": The system cannot find the file specified. - data_source="C:\Program Files\Splunk\etc\apps\dbx\bin\jbridge_server.py", data_host="********", data_sourcetype="dbx_jbridge"

Assistance is greatly appreciated.

[aholzer · Accepted solution]

Turns out there were multiple Java installations on the machine and the java.conf for the app was pointing to the wrong one. The PATH for java was also set incorrectly.

I have uninstalled the unnecessary javas, and reconfigured the PATH and java.conf.

Thanks @gregbujak for the assist!

[linu1988]

So is it working now? Need to install the JAVA SE and make the ENV variable properly...

[aholzer]

Huh... I thought that was working. I'm getting an error:
c:\Program Files\Splunk\bin>java -version
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object

[gregbujak]

Can you run java from the command line?

[aholzer]

Tried that several times. It doesn't start up automatically. I believe it depends on the jbridge, and that is clearly having trouble starting. So it never gets to the dbx part.

[gregbujak]

Ah, dbx is a java process. I am assuming that you have restarted the splunkd where its running? It should auto start with the process.

[aholzer]

That is the latest log. I don't think dbx is running.

[gregbujak]

Is this the latest log? Is dbx running? Your jbridge.log is reporting
2013-07-24 09:56:21,243 DEBUG Error waiting for process: Java process returned error code 1!
But the dbx.log is reporting from last Friday
2013-07-19 11:32:27.108

[aholzer]

The last thing that appears there is:

2013-07-19 11:32:27.108 dbx7025:DEBUG:PooledConnectionFactory - Closing connection pool...
2013-07-19 11:32:27.108 dbx7025:DEBUG:PooledConnectionFactory - Destroying database connection for database=NYC_CRD_PROD
2013-07-19 11:32:27.108 dbx7025:INFO:Splunkd - Stopping Splunkd REST session keep-alive...
2013-07-19 11:32:27.108 dbx7025:DEBUG:XStreamStore - Performing shutdown...
2013-07-19 11:32:27.108 dbx7025:DEBUG:XStreamStore - Saving state...

[gregbujak]

What does your dbx.log file say?

[aholzer]

I've tried uninstalling and reinstalling the app, but that didn't work.