Splunk Search

Why do I see log entries monitoring splunkd.log via search that do not appear in splunkd.log?

wrangler2x
Motivator

I was under the impression that if I did index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" realtime that it would be the same as doing a tail -f /opt/splunk/var/log/splunk/splunkd.log (in Linux). That seems to not be quite so.

I was explaining this to a co-worker and showed a tail -f while on another screen running the search in real-time. Yes, I saw everything showing in the tail -f in the search window, but in the search window I saw two other log entry types that were not showing on the other screen: INOFO HttpPubSubConnection and ERROR DiskMon. Here are a couple of samples (with IP redacted):

  1. 07-12-2016 10:38:47.075 -0700 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xxx.xxx.xxx_8089_oda1b.oit.uci.edu_oda1b_18CB2BD0-0207-47E0-B4E7-C62BAC751304
  2. 07-12-2016 10:38:42.159 -0700 ERROR DiskMon - None such on disk: /opt/splunk/var/run/splunk/dispatch

And, if I grep for HttpPubSubConnection or DiskMon in /opt/splunk/var/log/splunk/splunkd.log I get nothing back. So where are these log entries coming from, and why do I not see exactly the same thing on both screens?

0 Karma
1 Solution

ryanoconnor
Builder

How many Splunk systems are in your environment?

The following search will return events from any host that is monitoring the file /opt/splunk/var/log/splunk/splunkd.log

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"

If, for example, you have two different search heads, or a search head and an indexer, than your splunk search might be returning data from multiple hosts.

View solution in original post

ryanoconnor
Builder

How many Splunk systems are in your environment?

The following search will return events from any host that is monitoring the file /opt/splunk/var/log/splunk/splunkd.log

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"

If, for example, you have two different search heads, or a search head and an indexer, than your splunk search might be returning data from multiple hosts.

wrangler2x
Motivator

Yes, that is what it was. And as @somesoni2 suggested, I added a host filter and the results now match. Thanks guys.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Were you running your query for the exact same host (host filter explicitly specified)?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...