Getting Data In

How to use single inputs.conf across multiple forwarders with different set of monitored directories?

tusharsaran1
Path Finder

Hi All,

Is it possible to configure inputs.conf in such a way that universal forwarders running on different hosts can read the same file but scan a different set of directories? As an example, we want to create 1 inputs.conf with 3 stanzas . Now we want 3 different forwarders to read the same inputs.conf but monitor data from 1 directory each.
In other words, is it possible to link directory stanzas in inputs.conf with particular forwarders?

Our actual use case is given below:
We have a NFS mounted log directory having ~2000 subdirectories. We want to split the load across 4 universal forwarders with each forwarder scanning ~25% of the sub directories. We want to avoid managing 4 different inputs.conf files. Is that possible?

jonmargulies
Path Finder

There is no way to do what you want by just using one inputs.conf file. The closest I could think of would be to have one primary inputs.conf app that defines all the monitoring stanzas and disables them, and then a set of secondary inputs.conf apps, each assigned to just one forwarder, that just enables the appropriate stanzas for that forwarder. But I don't recommend doing that, as it's just another layer of complexity to manage (so now when you change something you have to change it in at least two places instead of one).

In the past, when I've had the need for very complex inputs.conf configurations, with hosts collecting data on behalf of thousands of other hosts, I've had a lot of success with building a spreadsheet that tracked all my inputs and then a Python script that processed the spreadsheet into the inputs.conf file(s) I needed. This had major benefits: I was just editing things in one place (the spreadsheet), which was a lot easier to read, sort, and search than an inputs.conf file; if I needed to change the way things were broken up, I could make the change in the Python script and have it automate the changes downstream; and much lower typo risk. I highly recommend this approach.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...