Deployment Architecture

Specify Index when using Deployment Server

jfaldmo
Explorer

Our deployment server will use the client name to send out web/app server specific inputs.conf files. I'd like to also use the client name to specify a separate app which would say which index to use, but I'm not sure which configuration file to create to make it happen.

Tags (1)
0 Karma

jfaldmo
Explorer

Thanks for the response, Kristian. I get all that you shared. Let me give a better background to what I am looking for.

I work for a large IT department. We have 17+ business units with their own front line support engineers. These engineers are the ones who install the forwarding agent and set a client name using an install script that I created. The client name starts with the business unit's name followed by an underscore and technology name, and then followed by the server name. For example HR_websphere_server1, or OPS_tomcat_websphere_server2 (if that server has both tomcat and websphere on it.) The log locations are the same for each technology. So I only have to define 10 or so apps with the inputs.conf file in it and the proper apps get pushed out to forwarders with the proper inputs.conf file for the technology.

If I wanted to put the index location as well using this method then I would have to create 10*17 apps. I would rather define 10+17 apps if there was a way to specify the index based off of the host name instead of based off of the "monitor::" stanza.

Right now I am using props and transform files on the Indexers. But this requires a manual update of the props file each time a new forwarder is installed, and the subsequent Indexer restart. I would rather have the forwarder specify the index based on an app that was pushed to it through the deployment server.

Does all of that make sense?

0 Karma

kristian_kolb
Ultra Champion

see update above
/k

0 Karma

kristian_kolb
Ultra Champion

To specify into which index a log should go, you'll add index=yourindex to your monitor stanza in inputs.conf.

In your case, your configuration would look like;

serverclass.conf

...
[serverClass:yourClass]
whitelist.0 = yourClientName

[serverClass:yourClass:app:yourApp]
...

$SPLUNK_HOME/etc/deployment-apps/yourApp/default/inputs.conf

[monitor:///var/yourlogfile]
sourcetype = yourtype
index = yourindex

UPDATE:

Hmm, you don't mention this, but I assume that you want to put each Business Unit in a separate index.

If you were running Heavy Forwarders, you could probably push out an app that does what you currently do on the indexers. (the 10+17 scenario)

I've not (yet) had to deal with a situation like yours - sorry.

Perhaps you could set up an intermediate Heavy Forwarder per BU for relaying only, and do the parsing (and index rewrite) there? (still 10+17, but less strain on the generating hosts)

UGLY BELOW THIS LINE:

Depending on how your serverclasses are defined, and that my initial assumption is correct, you may perhaps get away with the less-than-beautiful solution of deploying one app per BU. This app would contain input stanzas for ALL technologies, regardless of whether they are present on the machines or not. The apps would be identical, except for the index=your_business_unit in each monitor stanza.

Naturally, this would mean the major part of the inputs configuration for any given host would be incorrect. This will probably generate a few errors on startup of each forwarder, but I'm not sure that it will do any more harm than that.

As I said - not beautiful - but it means that you'll only have 17 (almost identical) apps in total.

hope this helps,

Kristian

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...