Getting Data In

How to troubleshoot why I am not receiving data for two sources I created on a universal forwarder?

daddyoh
Explorer

I have one forwarder that is working for 6+ sources. I created two sources today and no data is showing up.

If I do this search:

source="/usr/local/exist/latest/webapp/WEB-INF/logs/scheduler.log" host="vmweb3"` 

for all time, or this one:

source="/usr/local/exist/latest/webapp/WEB-INF/logs/exist.log" host="vmweb3"

for all time, I don't see any events showing up. I can search for other events from the same forwarder and that data does display and is current.

From the Universal Forwarder Splunk account, I can cat the files so I know that Splunk can read the files.

I have set the source type to auto detect.
For the scheduler, a few records look like:

2016-07-11 11:58:27,547 [DefaultQuartzScheduler_Worker-3] DEBUG (SystemTaskManager.java [runSystemTask]:86) - Running system maintenance task: org.exist.storage.sync.SyncTask
2016-07-11 11:58:27,549 [DefaultQuartzScheduler_Worker-3] DEBUG (SystemTaskManager.java [runSystemTask]:89) - System task completed.
2016-07-11 11:58:30,047 [DefaultQuartzScheduler_Worker-4] DEBUG (SystemTaskManager.java [runSystemTask]:86) - Running system maintenance task: org.exist.storage.sync.SyncTask
2016-07-11 11:58:30,049 [DefaultQuartzScheduler_Worker-4] DEBUG (SystemTaskManager.java [runSystemTask]:89) - System task completed.

For the exist.log file, here are a few (but this is a log file that can contain stacktraces).

2016-07-11 11:55:52,372 [eXistThread-172] INFO  (NativeBroker.java [removeXMLResource]:2705) - Removing document august-8-newyorkcity.xml (30217) ...
2016-07-11 11:57:52,564 [DefaultQuartzScheduler_Worker-4] INFO  (NativeBroker.java [sync]:3669) - Memory: 1,454,592K total; 1,454,592K max; 490,528K free

I have verified that when I created the input in Splunk for these files, I used the default index.

A scan of the splunkd.log file on the UF does not show anything different.

Any help you can provide for next steps diagnosing this would be much appreciated.

Thanks
Eric

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Whenever you add a new monitor in an inputs.conf file, you need to restart the splunk instance (UF or HF).

Also when you search for the specific sourcetype or source, you should start with a time picker range of all time. I can't tell you how many times the data has the wrong time zone or else there is significant latency between the forwarder and the indexers. With all time you will at least see the data if it is being forwarded.

0 Karma

muebel
SplunkTrust
SplunkTrust

Hi daddyoh, as part of splunk startup you should see some events in splunkd.log as splunk loads up the particular input stanzas in question. I'd take a look at that, and ensure that you see those file monitors properly loaded.

Based on your comment, that's interesting that you see the initial two files showing up after adding a third, but not the third itself. I look into splunkd.log for that restart event when the two starting worked, but the third did not.

Past that, I think you covered most of the bases, that is

  • Making sure the files exist
  • have current events
  • sensible timestamp
  • user splunk is running as can read the files

Past that my other thoughts are potential delays from the forwarder to the indexer (indexer is under heavy load, latency in events being written to disk)

There could also be something strange going on with the timezone, potentially events are being written in the future, which would lead to some delay before you find them in search using the normal time range settings. You can override this by including something like "earliest=-4h latest=+4h" in the search string in order to search potential future events.

Please let me know if this helps to answer you question!

0 Karma

daddyoh
Explorer

@muebel Thanks. I was able to get all of the files forwarding by restarting the UF several times.

0 Karma

daddyoh
Explorer

I added one more file to monitor from the same UF. I restarted splunk indexer/search/deployment server and reloaded deployment

I restarted the UF and now the first two files are showing up in search but the 3rd file is not. I have checked and there are recent event records in the 3rd file that should display.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...