I was under the impression that if I did index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"
realtime that it would be the same as doing a tail -f /opt/splunk/var/log/splunk/splunkd.log
(in Linux). That seems to not be quite so.
I was explaining this to a co-worker and showed a tail -f while on another screen running the search in real-time. Yes, I saw everything showing in the tail -f in the search window, but in the search window I saw two other log entry types that were not showing on the other screen: INOFO HttpPubSubConnection and ERROR DiskMon. Here are a couple of samples (with IP redacted):
And, if I grep for HttpPubSubConnection or DiskMon in /opt/splunk/var/log/splunk/splunkd.log I get nothing back. So where are these log entries coming from, and why do I not see exactly the same thing on both screens?
How many Splunk systems are in your environment?
The following search will return events from any host that is monitoring the file /opt/splunk/var/log/splunk/splunkd.log
index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"
If, for example, you have two different search heads, or a search head and an indexer, than your splunk search might be returning data from multiple hosts.
How many Splunk systems are in your environment?
The following search will return events from any host that is monitoring the file /opt/splunk/var/log/splunk/splunkd.log
index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"
If, for example, you have two different search heads, or a search head and an indexer, than your splunk search might be returning data from multiple hosts.
Yes, that is what it was. And as @somesoni2 suggested, I added a host filter and the results now match. Thanks guys.
Were you running your query for the exact same host (host filter explicitly specified)?