Best Practice for ingesting script as data input in indexing cluster

[EricLloyd79]

We have an index cluster with two indexers, a cluster master, and a cluster search head. We want to deploy scripts that ingest data from snmp queries to network devices at five minute intervals.

We are looking for a recommendation for the best practice on how to deploy these input scripts on our indexing cluster so that they will be fault tolerant like all of our other data ingested into the cluster. The output of the python scripts are to be directly ingested into Splunk.

[mattymo]

Hey Eric,

Just to further clarify that I understood your ask when I answered below...

Your scripts are doing the snmp polling,correct?

And are looking for splunk to run them and ingest the stdout (aka a scripted input)?

Or are you writing the output to a file which we would tail?

Does your script have failure handling built in?

indexer clustering provides fault tolerance through data replication, so we can ensure once we receive the data, we can lose an indexer and still have a copy.

But indexer clustering does not do dynamic job scheduling across the peers or ensure that a script/job runs/retries if it fails...

Thus my advice below.

Let me know if I have misunderstood any part of what you are looking to do

[mattymo · Accepted solution]

Ideally you would deploy the scripts in the forwarding layer of your environment, not directly on the indexer cluster.

example, on a host with universal forwarder or heavy forwarder.

On the topic of fault tolerance, Splunk does not provide fault tolerance on scripted input data collection. You will need to look at external solutions to ensure that your collection scheme can survive a fault. (VMotion, OS clustering, etc), or maybe an active/standby setup, where you could manually cut over and enable the inputs on the standby if you lost the active forwarder.

That being said, Splunk can most definitely alert you quickly to any fault or failure to run a poll to ensure the least amount of interruption possible.

On the topic of snmp polling, unless your scale is small, I would suggest you take a hard look at running an external poller, like cacti (my fav- see splunk compatible plugin https://docs.cacti.net/userplugin:mirage), nagios, etc and simply integrate those into Splunk.

[EricLloyd79]

Thank you. We are going to deploy on our forwarding layer and have to monitor them.

[mattymo]

Nice! Protip: checkout Meta Woot - https://splunkbase.splunk.com/app/2949/

Provides a great jump off point for alarming any hosts/sourcetypes that go missing...great short cut for data integrity monitoring!