JMS Messaging Modular Input: Why are messages missing intermittently?

[ksoori]

We are using Message Broker to drop messages into MQ Queue and Splunk uses JMS Input to read messages out of these queue.
Recently we have started noticing that some of the messages are missing from splunk. Though those messages are dropped into the MQ queue and Splunk reads out of it, we do not see those messages in Splunk.

For eg: if we drop 5 messages related to 5 different IDs, Splunk displays 1 or 2 messages and the remaining messages are missing.

This behavior is intermittent. Sometimes Splunk shows all 5 messages, but other times it displays different messages, even though the same set of messages (content wise) is dropped again and again.
We are not sure why these messages go missing. Is there any way to increase JMS Input log level? We checked the default Splunk system log and did not find any clues there.

Can anyone help us out?

[ksoori]

yes, it looks like enabling the jms input nodes thru web ui creates some zombie process which runs in the background and we believe those missing message are consumed by those zombie process.

this issue got fixed after we started enabling the jms input process thru command lines after clearing up all zombie java processes.

[Damien_Dallimor]

Any error messages in logs ?

Search with : index=_internal ExecProcessor error jms.py

Can you describe in detail the specifics of the messages ie : data format , payload size etc....

Can you provide an example of a message that shows up vs a message that doesn't show up ?

[ksoori]

The App is installed on one of the indexer node.

we did edit the inputs.conf file thru Splunk Web UI.

Anyways, will try updating the inputs.conf file manually and see if that fixes this issue. just curious to know what 'issue' happens when we update is via Splunk Web UI.

[jimodonald]

Did you get a resolution on this issue?

[ksoori]

Hi Damien, based on our other conversations these are things we have done.

We upgraded JMS app to 1.3.9 and set logging level to INFO. We do not see any Java based Exception.

What we observed was that

The issue starts to happen when we have multiple inputs spawning java processes. When we ran the JMS App for only one queue, everything worked perfectly. But once we activated more than one input queue. We started seeing this scenario of missing messages.

Looks like the JMS App spawns more java process on the backend and when we have more than one Java Process, we start to lose messages.

Also, disabling a JMS App thru the splunk UI does not close the java process on the backend. The java process gets killed only after restarting the splunk instance.

do you have any suggestions for this?

[Damien_Dallimor]

Multiple inputs don't spawn multiple processes.
Each input runs as a seperate thread inside 1 JVM (1 process).

So perhaps you have done something non-standard in your environment or have not deployed the App correctly for a distributed Splunk architecture, I'm only guessing here. In a distributed Splunk architecture , you don't install the App on a Search Head node. It gets deployed to 1 or more Forwarder or Indexer nodes and to configure your inputs you edit inputs.conf manually (not via the Splunk Web UI).

[ksoori]

These are the 5 messages

Message 1:

AB123C26310NABCDUS60POLARIS
PROTECTIONTOTAL]]>