Greetings,
I have set up 17 micro AWS boxes, One running a splunk 6.2.0 indexer, 8 with databases (8 mongo and 4 mongo and neo4j), 8 with Node.JS, and set them up with splunk 6.2.0 heavy forwarders monitoring relevant files and forwarding to the splunk receiver/indexer. The problem is but only 10 of the forwarding instances ever show up in the indexer.
The receiver seems to only see the most-recent 10 of them in the data summary. 6 of the newest boxes seem to just not show up in the data summary on the main receiver/indexer.
These 6 boxes appear to be configured properly: They have monitors on the relevant files when I 'splunk list monitor'. They also show the receiver as an "active forward" when I 'splunk list forward-server'. As far as I can tell they are set up the same as the other 10 boxes that work.
So where is the block/issue? Is there some 10-forwarder limit I am hitting? Is there a concurrent search limit manifesting as a 10-forwarder limit? Do I need to do a split across load balancers if the receiver and indexer are on the same machine?
Thanks in advance for any assistance.
There is no 10-forwarder limit.
How are you deploying your configurations (I assume manually)? For the servers where the configuration are working, where did you deploy your configuration files? Was it in $SPLUNK_HOME/etc/system/local/
(hopefully it was in $SPLUNK_HOME/etc/system/MyApp/default/
). In any case, wherever that was, just copy that subdirectory as-is from a working server to a non-working server. In particular, I suspect you have a mismatch with outputs.conf
. Also pay careful attention to server.conf
which may include configurations that are supposed to be host/server specific and should*NOT* be copied without being adjusted. It might help to do a search like this from your search head (lets' say your hostname
is "myhost" and your IP address is "172.0.0.1"):
index=_internal host=hostname OR host=172.0.0.1 OR hostname OR 172.0.0.1
Also, login to a broken forwarder with your browser and search locally for problems like this:
index=_internal WARN or ERR*
There is no 10-forwarder limit.
How are you deploying your configurations (I assume manually)? For the servers where the configuration are working, where did you deploy your configuration files? Was it in $SPLUNK_HOME/etc/system/local/
(hopefully it was in $SPLUNK_HOME/etc/system/MyApp/default/
). In any case, wherever that was, just copy that subdirectory as-is from a working server to a non-working server. In particular, I suspect you have a mismatch with outputs.conf
. Also pay careful attention to server.conf
which may include configurations that are supposed to be host/server specific and should*NOT* be copied without being adjusted. It might help to do a search like this from your search head (lets' say your hostname
is "myhost" and your IP address is "172.0.0.1"):
index=_internal host=hostname OR host=172.0.0.1 OR hostname OR 172.0.0.1
Also, login to a broken forwarder with your browser and search locally for problems like this:
index=_internal WARN or ERR*
Alas, the outputs.conf
was in 'SPLUNK_HOME/etc/system/local/' but it was identical to the outputs.conf of the other forwarders.
The thing that was also the same were the inputs.conf
and the server.conf
files: They were identical to one of the earlier servers.
So it looks like my rush to copy directories (although I might have copied an entire splunk-forwarder installation and not just the local directory) is what is causing all the trouble: They missing 6 were reporting, just under a hostname of another box.
So, I updated the hostnames and things suddenly started appearing under those new hostnames, in addition to the 10 I was seeing before.
Many thanks for pointing me to a new place to look, woodcock.