Getting Data In

Why isn't my data reaching the index?

ddrillic
Ultra Champion

We have a small farm with no access to the forwarders. The forwarders do phone home but the following returns nothing when running it against all time -

index=_internal host=<fwdhost1>* OR host=<fwdhost2>* OR host=<fwdhost3>* OR host=<fwdhost4>* OR host=<fwdhost5>*

Nothing reaches the index.

How can we debug it further?

0 Karma
1 Solution

mdsnmss
SplunkTrust
SplunkTrust

Is it a distributed or standalone deployment? If distributed, I would check if it is connecting to the indexers. You can get a phone home without connecting to the indexers since the phone home goes through the deployment server. You can use the search to look for the hosts to see if they are connecting:

index=_internal source=*metrics.log tcpin_connections

If not, it could be for a variety of reasons. Firewall preventing it, no forward-server listed on the forwarder, or monitor not working on the forwarders.

View solution in original post

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Is it a distributed or standalone deployment? If distributed, I would check if it is connecting to the indexers. You can get a phone home without connecting to the indexers since the phone home goes through the deployment server. You can use the search to look for the hosts to see if they are connecting:

index=_internal source=*metrics.log tcpin_connections

If not, it could be for a variety of reasons. Firewall preventing it, no forward-server listed on the forwarder, or monitor not working on the forwarders.

0 Karma

ddrillic
Ultra Champion

It's a distributed deployment and this query returns events for the two indexers. I don't see the forwarders listed. Should they?

You said -

-- If not, it could be for a variety of reasons.

Meaning, the forwarders apparently can't reach the indexers, right?

I ran this query on our core farm - index=_internal source=*metrics.log tcpin_connections and it doesn't show the forwarders...

I ran - index=_internal sourcetype=splunkd component=TcpOutputProc (host=<fwdhost1>* OR host=<fwdhost2>**)

Nothing came back. So, I guess no connectivity exists between the forwarders and the indexers...

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Your forwarders would be listed as the sourceHost field. Likely as an IP rather than a hostname. If you don't see the forwarders listed it would likely mean they are not reaching the indexers.

0 Karma

ddrillic
Ultra Champion

Perfect - much appreciated.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...