How do you manage inputs.conf?

[alanzchan]

We have more than 100 applications in our deployment. Sometimes a log path/name is changed or added on the server side and I have to update inputs.conf to reflect the new change. Over the years, each change was just appended in the inputs.conf and logs that we once monitored but no longer do are now clogging up the inputs.conf.

Of course, I can log into one of the servers in each serverclass and see which logs we are actually monitoring, but as you can imagine that would be a hefty job. Is there a smarter way to clean up inputs.conf? Any suggestions on how to manage it?

[gcusello]

alanzchan,
I created a TA for each app input.
Then I created an App on my Search Head that reads all the inputs.conf on the Deployment Server (there's a TA also on Deployment Server) and then shows all the inputs in a dedicated dashboard.
In this way I can search if an input is already configured.
Bye.
Giuseppe

[diogofgm]

you can try to use REST to get that info. Try this search

| rest splunk_server=local /services/admin/monitor
| fields title index sourcetype eai:acl.app host
| join sourcetype [| metadata type=sourcetypes index=_* index=* | eval diff = tostring(now() - recentTime, "duration") | fields - type]
| convert ctime(recentTime)

This targets the local server (you can change that to target other server) to return all monitor inputs, joins it with info from metadata which lists the source types and the last time you got data from each one. This can be modified to do the join by source.

[solarboyz1]

We create a app for each serverClass, using a standard naming convention we created:

Acme_CHI_hfw_inputs/local/inputs.conf

All inputs are maintained in this server class.
We do the same for indexes, and well as baseline settings:

Acme_CHI_idx_indexes/local/indexes.conf
Acme_ALL_idx_base/local/limits.conf, distsearch.conf, etc..

By using a standardized naming convention, we know what settings are applied to which servers.
By seperating these settings from the original Add-On, we have one location to update.

We also use the ConfigQuest app to report on settings deployed, and serverClass members.

Mileage may vary.

[alanzchan]

We also have a central configuration for each application. Since we have been just appending log inputs into the configuration file, I'm more interested in how I can find out which input stanza in inputs.conf is actually being used.

[solarboyz1]

So, you can use your metrics logs to see the per-source thruput

index=_*  component=Metrics group=per_source_thruput

And match that to your inputs/serverClass?