How can we find out whether a set of forwarders are connected to all indexers?

[ddrillic]

Is there a way to find out whether a set of forwarders are connected to all intended indexers? On a regular basis we find sets of forwarders that are not connected to all indexers...

[adonio]

@ddrillic,
i am positive there are other ways to do it but here is a quick and dirty solution:
lets assume you have 3 indexers
you can search:

index = * | timechart span=5m dc(splunk_server) as unique_indexers by host

every host that has shows less then 3 in the chart is a suspect.
remember however that this test is not 100% accurate as maybe sometimes there is no new data for a while and therefore the forwarders will not ave much data to send.
i think that the best is to verify that you have the correct outputs.conf on your forwarders and verify the

forceTimebasedAutoLB = true

then you can enforce:

autoLBFrequency = <seconds>

or

autoLBVolume = <bytes>

read here more:
https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Outputsconf
hope it helps

[ddrillic]

Gorgeous!!!!

[somesoni2 · Accepted solution]

Give this a try.

index=_internal  sourcetype=splunkd component=TcpOutputProc  (host=host1 OR host=host2....)  earliest=-30m | eval Indexer=mvindex(split(idx,":"),0) | stats dc(Indexer) as Idx_Count values(Indexer) as Indexers by host

This will get you the indexers a forwarder (specified as host=...) is connecting to with their names. Now you add appropriate where clause compare it against. Another variations is below, which add column to all available indexers (which are added as distributed search peers).

above search | appendcols[| rest /services/search/distributed/peers | table title | eval Indexer=mvindex(split(title,":"),0) | stats values(Indexer) as AllIndexers dc(Indexer) as TotalIndexers ]

[somesoni2]

Just realized, the first search gives IP addresses, so if you're just comparing count, you can use above query as is. If you're comparing Indexer names as well, you need to do a dnslookukp before the stats.

[ddrillic]

Amazing !!!

We see the AllIndexers column for the first row only out of two...

[somesoni2]

Yes. The rest is returning just single row. Just add |filldown at the end so it'll get copied to all the rows.

[adonio]

do you mean if the forwarders sends data to all relevant indexers?

[ddrillic]

Exactly that....