Hello all,
We are trying to take a new approach on handling indexes and such. We want to switch indexes to be like OS specific, app specific,etc. We want to be able to spin up a forwarder and have it automatically get the correct OS app to start gathering stats. By doing so, there will be about 2000 forwarders to start and end goal would be to get around 5000 forwarders.
The real question is - how do you manage searching on multiple indexes with a set of hosts? (sorry if wording isn't the best) We currently have a lookup for an app to test out and it works great (only about 100 servers listed in a csv) but will this approach work for the long run? Would a key-value set be better?
Looking to see how most people approach this situation.
Thanks
In terms of searching I would use eventtypes (first) and tags (second) as others have said.
For getting the data from a sets of forwarders to specific indexes I would use a combination of setting the clientName string in the UF deploymentclient.conf file and then use those clientName strings as whitelists in the deployment server serverclass.conf stanzas. For example if you have a bunch of Windows and Linux servers in various lines of business you could set the appropriate clientName string like
LoB1_win_<hostname>
LoB1_nix_<hostname>
LoB2_win_<hostname>
By inserting the server name you give yourself granularity to deploy configs to specific systems. Perhaps in your organization though you could simply leverage the hostname based on local naming conventions. In your serverclass.conf file you would use
[serverClass:LoB_win_default]
restartSplunkd = true
whitelist.0 = LoB1_win_*
[serverClass:LoB_win_default:app:LoB1_win_default_inputs]
etc etc
Then in your input 'apps' you are pushing from the DS you specify things like monitoring paths, scripts, etc and what indexes the data goes to.
Use macros and eventtypes.
You will most likely enjoy tags
and eventtypes
.
You can tag
a host for specific function. Let's say web
. (I'll do this from tags.conf
on the command line)
[host=mywebhost01]
web = enabled
Next, use that in a search:
index=linux_servers tag::host=web
This search will pull all Linux servers that are designated as web
hosts. Note that this won't scale well if you do it by hand. You may want to use an Asset database/csv/file/whatever to pull the list, then use a script to create a tags.conf file to use.
Event types are almost the same thing, but let's you specify a non-reporting/transforming search to match. In eventtypes.conf
:
[linux_web_servers]
search = index=linux_servers tag::host=web
You would then simply search like this:
eventtype=linux_web_servers
These techniques will keep your searches clean and readable.
So, say we went the tag route. We had a script to populate the tags.conf based on some criteria stored in a csv sheet. Everything works. Lets use the example of web server.
Now one day, that hostname gets migrated to be an app server. We change the csv, and the tags.conf gets updated (assuming we were smart enough ahead of time to handle that in our script) and we restart the server.
Does the old data get changed to the new tag or does it keep the tag that was assigned when that event was indexed?
The tag is handled in search time. the tags will be applied to the events that fulfill the tag stanza, In the case the host field. If you change the tag to something else all the events from that host will have the new tag. You can work this out by using a different host name or using a proper naming convention for web servers vs app servers.