Getting Data In

How can we find out whether a set of forwarders are connected to all indexers?

ddrillic
Ultra Champion

Is there a way to find out whether a set of forwarders are connected to all intended indexers? On a regular basis we find sets of forwarders that are not connected to all indexers...

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Give this a try.

index=_internal  sourcetype=splunkd component=TcpOutputProc  (host=host1 OR host=host2....)  earliest=-30m | eval Indexer=mvindex(split(idx,":"),0) | stats dc(Indexer) as Idx_Count values(Indexer) as Indexers by host

This will get you the indexers a forwarder (specified as host=...) is connecting to with their names. Now you add appropriate where clause compare it against. Another variations is below, which add column to all available indexers (which are added as distributed search peers).

above search | appendcols[| rest /services/search/distributed/peers | table title | eval Indexer=mvindex(split(title,":"),0) | stats values(Indexer) as AllIndexers dc(Indexer) as TotalIndexers ]

View solution in original post

0 Karma

adonio
Ultra Champion

@ddrillic,
i am positive there are other ways to do it but here is a quick and dirty solution:
lets assume you have 3 indexers
you can search:

index = * | timechart span=5m dc(splunk_server) as unique_indexers by host

every host that has shows less then 3 in the chart is a suspect.
remember however that this test is not 100% accurate as maybe sometimes there is no new data for a while and therefore the forwarders will not ave much data to send.
i think that the best is to verify that you have the correct outputs.conf on your forwarders and verify the

forceTimebasedAutoLB = true

then you can enforce:

autoLBFrequency = <seconds>

or

autoLBVolume = <bytes>

read here more:
https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Outputsconf
hope it helps

0 Karma

ddrillic
Ultra Champion

Gorgeous!!!!

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try.

index=_internal  sourcetype=splunkd component=TcpOutputProc  (host=host1 OR host=host2....)  earliest=-30m | eval Indexer=mvindex(split(idx,":"),0) | stats dc(Indexer) as Idx_Count values(Indexer) as Indexers by host

This will get you the indexers a forwarder (specified as host=...) is connecting to with their names. Now you add appropriate where clause compare it against. Another variations is below, which add column to all available indexers (which are added as distributed search peers).

above search | appendcols[| rest /services/search/distributed/peers | table title | eval Indexer=mvindex(split(title,":"),0) | stats values(Indexer) as AllIndexers dc(Indexer) as TotalIndexers ]
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Just realized, the first search gives IP addresses, so if you're just comparing count, you can use above query as is. If you're comparing Indexer names as well, you need to do a dnslookukp before the stats.

0 Karma

ddrillic
Ultra Champion

Amazing !!!

We see the AllIndexers column for the first row only out of two...

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Yes. The rest is returning just single row. Just add |filldown at the end so it'll get copied to all the rows.

0 Karma

adonio
Ultra Champion

do you mean if the forwarders sends data to all relevant indexers?

0 Karma

ddrillic
Ultra Champion

Exactly that....

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...