Splunk Search

I have installed the Splunk forwarder, but why am I unable to search the event logs from all machines?

Frederik
New Member

Sorry but this is probably a stupid question. I have set up Splunk to be able to have centralized collection of all the event logs from my servers. Now that I have installed all the agents, I cannot seem to search all the machines' event logs. I put in host=MYSERVERNAME and there are several machines that do not return anything.

Does the agent need to have an app deployed to collect event logs?

0 Karma

Raghav2384
Motivator

@Frederik,

Let's say you have 5 machines that you want to collect xyz.log (which exists on all 5 hosts)
1. You need to have forwarders on all 5 hosts
2. On each of the 5 hosts, in inputs.conf under $SPLUNK_HOME/etc/system/local directory, host should be FQDN of that particular machine (eg, on host1, it would be host=host1) next is your monitor/batch (read inputs.conf from docs) should point to the actual log location on that host.
3. outputs.conf , since all these 5 hosts are sending to the same splunk (assumption) they can be pretty much be same on all 5 hosts. This is the typical process configuring a forwarder.

With the little information provided, we can only assume what might be wrong, example.
1. As one of the answers suggested, is forwarder service up and running on all the hosts?
2. What user is your splunk forwarder running as? Does that user have Read access to the log file you are trying to consume?
3. Have you checked the connectivity from all the data sources/hosts to Central splunk instance? telnet central splunk's ip 9997?
4. Is it NAT'd or probably firewall is blocking the communication? Completely different issue

What can be done is, go to the splunkd.log on the machines that are not forwarding logs (Located under $SPLUNK_HOME/var/log/ and do a tail -f splunkd.log) see any ERRORS and abnormal stuff? If yes, that would be the first step of your triage.

For more accurate answers, please provide more information and ERRORS if any from your splunkd.log. Hope this helps!

Thanks,
Raghav

skoelpin
SplunkTrust
SplunkTrust

Well written @Raghav2384

ddrillic
Ultra Champion

The "official" documentation at I can't find my data!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Can you verify that the Splunk forwarder is running on the host machines you installed it on? You can do this by going into Splunk_Home/bin and run ./splunk status

Also, you will need to go into Splunk_Home/etc/system/local and edited the outputs.conf and make sure it's pointing to your indexer

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...