Getting Data In

Best Practice for ingesting script as data input in indexing cluster

EricLloyd79
Builder

We have an index cluster with two indexers, a cluster master, and a cluster search head. We want to deploy scripts that ingest data from snmp queries to network devices at five minute intervals.

We are looking for a recommendation for the best practice on how to deploy these input scripts on our indexing cluster so that they will be fault tolerant like all of our other data ingested into the cluster. The output of the python scripts are to be directly ingested into Splunk.

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Ideally you would deploy the scripts in the forwarding layer of your environment, not directly on the indexer cluster.

example, on a host with universal forwarder or heavy forwarder.

On the topic of fault tolerance, Splunk does not provide fault tolerance on scripted input data collection. You will need to look at external solutions to ensure that your collection scheme can survive a fault. (VMotion, OS clustering, etc), or maybe an active/standby setup, where you could manually cut over and enable the inputs on the standby if you lost the active forwarder.

That being said, Splunk can most definitely alert you quickly to any fault or failure to run a poll to ensure the least amount of interruption possible.

On the topic of snmp polling, unless your scale is small, I would suggest you take a hard look at running an external poller, like cacti (my fav- see splunk compatible plugin https://docs.cacti.net/userplugin:mirage), nagios, etc and simply integrate those into Splunk.

- MattyMo

View solution in original post

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hey Eric,

Just to further clarify that I understood your ask when I answered below...

Your scripts are doing the snmp polling,correct?

And are looking for splunk to run them and ingest the stdout (aka a scripted input)?

Or are you writing the output to a file which we would tail?

Does your script have failure handling built in?

indexer clustering provides fault tolerance through data replication, so we can ensure once we receive the data, we can lose an indexer and still have a copy.

But indexer clustering does not do dynamic job scheduling across the peers or ensure that a script/job runs/retries if it fails...

Thus my advice below.

Let me know if I have misunderstood any part of what you are looking to do

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

Ideally you would deploy the scripts in the forwarding layer of your environment, not directly on the indexer cluster.

example, on a host with universal forwarder or heavy forwarder.

On the topic of fault tolerance, Splunk does not provide fault tolerance on scripted input data collection. You will need to look at external solutions to ensure that your collection scheme can survive a fault. (VMotion, OS clustering, etc), or maybe an active/standby setup, where you could manually cut over and enable the inputs on the standby if you lost the active forwarder.

That being said, Splunk can most definitely alert you quickly to any fault or failure to run a poll to ensure the least amount of interruption possible.

On the topic of snmp polling, unless your scale is small, I would suggest you take a hard look at running an external poller, like cacti (my fav- see splunk compatible plugin https://docs.cacti.net/userplugin:mirage), nagios, etc and simply integrate those into Splunk.

- MattyMo
0 Karma

EricLloyd79
Builder

Thank you. We are going to deploy on our forwarding layer and have to monitor them.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Nice! Protip: checkout Meta Woot - https://splunkbase.splunk.com/app/2949/

Provides a great jump off point for alarming any hosts/sourcetypes that go missing...great short cut for data integrity monitoring!

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...