As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

[withrythm]

From a particular Server, we are able to see logs on Splunk from certain locations, but for others, there are no logs. If there is a permission or other such issues while accessing these logs, where would such errors be logged?

[woodcock]

The main reason that a log is ignored is because of the problem described (and solved) here:

http://answers.splunk.com/answers/231959/why-are-files-in-a-monitored-directory-being-skipp.html

http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/Howlogfilerotationishandled

The easiest thing to try is to add this to your inputs.conf stanzas for each input:

crcSalt = <SOURCE>

Make sure that you copy this exactly (do not change capitalization).

[grijhwani]

There are so many different reasons why logs might not be appearing, that no, a simple search access would not be the full route to diagnosis. It might supply the answers, if they are the sort of issues logged in the audit logs.

But it could simply be that the data is not being logged at all, and that could be down to any or all of misconfigured endpoint forwarder or indexer, SSL faults, broken network, local file permissions (all of which could be absent from the indexer because of the problem itself).

[withrythm]

Thanks Grijhwani

But since we are receiving logs from the same server from another path shouldnt it rule out SSL, broken network, miconfigured endpoint forwarder/indexer?

[withrythm]

Below is what is required to be available on SPLUNK:

Server 1- logs from Path 1
Server 1- logs from Path 2

The permissions for Path 1 and Path 2 are same. However, I am able to see logs from Path 1 but not from Path 2.

My question is, supposedly if there is some issue in Path 2 while Server 1 is trying to access the logs/files on those path; where are such error logs captured. Are these available on SPLUNK GUI anywhere? or on Server 1? or on SPLUNK Servers? and how to access such error logs.

Thanks in advance!

[richgalloway]

The forwarder's logs should be on your indexers. You can also find them on the forwarder itself at SPLUNK_HOME/var/log/splunk.

[maciep]

The Splunk logs from that sever should be in Splunk:

index=_internal host=Server1 sourcetype=splunkd

You can obviously sift through the results, maybe searching for "error" or "monitor" or "tailing" (don't recall off hand the best search to find these kinds of problems).

Did you ever get logs from Path 2 into Splunk? If not, it might be worth posting your inputs on here to see if something may be off there.

[withrythm]

Thanks Maciep for your response. No these are new paths added to the inputs.conf. below are the details from the input file. All of these are directories:

[monitor:///opt/app/weblogic/oracle/middleware/user_projects/domains/endeca_server_domain/servers/Managed-2/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///opt/app/weblogic/oracle/middleware/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///var/opt/app/endecaserver/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///var/opt/app/.snapshot/daily*]
index = ftts
sourcetype = snapshot_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///home/endeca/oraInventory/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[maciep]

Yep, it would be helpful to know which stanzas you're not getting logs from. Or if you're getting logs from all of the stanzas, but not every log you'd expect with a certain stanza if that makes sense.

As mentioned by woodcock, if any of those files have large headers that are the same, then that might explain it as well.

[richgalloway]

Which of these is not being forwarded?

[woodcock]

It depends. Be as specific as you possibly can and maybe somebody here can help.

[richgalloway]

To be clear, do you want to see Splunk log (splunkd.log, etc.) or logs forwarded from monitored systems?

[diogofgm]

have you checked your user roles in both servers? you might have different permission for index access and (depending on the search you are doing) you might have different default indexes for search.