After installing and configuring universal forwarders, why can't I see the data in the indexer?

[lanilim16]

I have a universal forwarder installed in a few servers and I also have added the logs to be monitored for each. I'm not able to see the data in the indexer for some reason though. I've done the same steps before using the same versions and script, I'm not sure where else to look.

Splunk Universal Forwarder 6.2.1
Splunk 6.1.0 build - indexer

[mchapman5611]

In Splunk, go to "Settings" | "Forwarding and receiving"
In the Receive Data section, click 'Configure Receiving'
Click 'New'
Add port 9997

[lanilim16 · Accepted solution]

issue fixed on its own, the log had to rotate before it got indexed.

[frmaasdam]

If it is a new index don't forget to set the rights correctly in de settings security for the admin rule. There you can set the index as one of the standard indexers to be able to search in.

[MuS]

Hi lanilim16,

first thing to do, run as admin the following search index=* earliest=0 latest=now
If you still don't see your events, run this as admin index=_internal sourcetype=splunkd metrics and check if your forwarders are sending anything
If you don't get anything from the forwarders, check any possible firewall blocking traffic or routing issues.
Last but not least login to the forwarder and check its config, like is it really configured to forward:
$SPLUNK_HOME/bin/ splunk list forward-server
or does it use the correct monitor stanza:
$SPLUNK_HOME/bin/ splunk cmd btool inputs list

Hope that helps ...

cheers, MuS

[lanilim16]

This is a new index which I've already added. I see result from

 index=_internal sourcetype=splunkd metrics host="<server>"

so now I'm not sure why it's not working for that particular index. When I check in Indexes, there are no events for that index, however I'm sure I've added it right since I can see these from splunkd.log

06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/app_prod/jboss-as/domain/servers/server-one/log/server.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server-one-*.log.
06-30-2015 15:55:16.757 -0400 INFO  TailingProcessor - Adding watch on path: /home/app_prod/jboss-as/domain/servers/server-one/log/server.log.

and I see the file when I list monitor from the forwarder.

[MuS]

check for possible typos in the index option in inputs.conf for this monitor

[nce054]

Are you indexing the data to an index that exists? Try to specify index=main in inputs.conf on the Universal Forwarder.