How do I edit my inputs.conf to monitor logs from two servers (webserver) with the same location?

[pranav_agile]

Hi,

I am trying to get logs from two different servers running Tomcat application, but have the same location. The following is the inputs.conf

[default]
host = $decideOnStartup

[monitor://C:\Tomcat\logs\]
disabled=false
sourcetype=access_combined_wcookie
index=globalscreening
whitelist=localhost_access_log.*txt$


[monitor://C:\Tomcat\logs\app-errors.log]
sourcetype=catalina
index=globalscreening

[monitor://C:\Tomcat\logs\app-events.log]
sourcetype=catalina
index=globalscreening

[monitor://C:\Tomcat\logs\ws28.log]
sourcetype=catalina
index=globalscreening

The issue is I am getting logs into Splunk from server A, but not from server B. I have tried to include host information into the above stanza, but the logs stopped ingesting into Splunk.

Could you please let me know where I am going wrong and how to correct it?

thank you.

[Richfez · Accepted solution]

[Edit: fixed btool command]

From your responses above it seems your inputs probably work correctly. FritzWittwer's advice is good, a quick check of /opt/splunkuniversalforwarder/var/log/splunk/splunkd.log may tell you something. Obviously, confirm the paths I gave there.

If the log file doesn't quickly pinpoint the issue, check that outputs are correctly specified. The two systems should both have pretty much the same outputs.conf, I'd guess. (In my case, pretty much ALL my UFs have the same outputs.conf file).

If outputs.conf appears right as well, you might want to run btools in debug mode to see if you can spot the problem. On each UF, you'll run something like

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug outputs list > myoutputs.txt

and possibly

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug inputs list >myinputs.txt

(More btool info here) Then review those two files. It'll take a little searching around (at least in the inputs, the outputs is probably pretty small), but compare the two and I suspect you'll find your problem.

[pranav_agile]

thank you rich7177 and everyone here. It has truly helped me to nail down the issue and logs have started ingesting into Splunk.

Yes I did compare the inputs/outputs debug logs from the two servers and found that wb030 wasn't added to one of the App that forwards the indexers config to it.

I added the host and could see the logs started ingesting into splunk.

[FritzWittwer_ol]

did you check the log file in $SPLUNK_HOME/var/log/splunk/splunkd.log ?

[meenal901]

Is $decideOnStartup being populated on both hosts properly. Can you try the following entry as default:
host-1
[default]
host = wb030

host-2
[default]
host = wb031 --> OTHER HOST NAME

These input.conf are present in $SPLUNK_HOME/etc/system/local or app level?

Check in splunkd.log in case there is any error while reading the file? May be path is not exactly same or there is read permission issue?

[pranav_agile]

I have updated inputs.conf as, but still its not getting data from wb030:
[default]
host = $decideOnStartup

[monitor://C:\Tomcat\logs]
disabled=false
sourcetype=access_combined_wcookie
index=globalscreening
whitelist=localhost_access_log.*txt$

[monitor://C:\Tomcat\logs\app-errors.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

[monitor://C:\Tomcat\logs\app-events.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

[monitor://C:\Tomcat\logs\ws28.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

[pranav_agile]

wb030 is the second web server. I would like to confirm that wb030 does have the data files in the C:\Tomcat\logs.

[somesoni2]

You mentioned you've two tomcat servers. Do you have Splunk forwarder installed on both the Tomcat servers and have deployed this inputs.conf on both the forwarders?

[pranav_agile]

yes, splunk forwarders is installed on both the servers. inputs.conf is also deployed on each of the servers.