Splunk Search

Splunk Architecture: How do you manage searching on multiple indexes that are OS specific, app specific, etc based on a set of hosts?

chawagon03
Path Finder

Hello all,

We are trying to take a new approach on handling indexes and such. We want to switch indexes to be like OS specific, app specific,etc. We want to be able to spin up a forwarder and have it automatically get the correct OS app to start gathering stats. By doing so, there will be about 2000 forwarders to start and end goal would be to get around 5000 forwarders.

The real question is - how do you manage searching on multiple indexes with a set of hosts? (sorry if wording isn't the best) We currently have a lookup for an app to test out and it works great (only about 100 servers listed in a csv) but will this approach work for the long run? Would a key-value set be better?

Looking to see how most people approach this situation.

Thanks

0 Karma

Runals
Motivator

In terms of searching I would use eventtypes (first) and tags (second) as others have said.

For getting the data from a sets of forwarders to specific indexes I would use a combination of setting the clientName string in the UF deploymentclient.conf file and then use those clientName strings as whitelists in the deployment server serverclass.conf stanzas. For example if you have a bunch of Windows and Linux servers in various lines of business you could set the appropriate clientName string like

LoB1_win_<hostname>
LoB1_nix_<hostname>
LoB2_win_<hostname>

By inserting the server name you give yourself granularity to deploy configs to specific systems. Perhaps in your organization though you could simply leverage the hostname based on local naming conventions. In your serverclass.conf file you would use

[serverClass:LoB_win_default]
restartSplunkd = true
whitelist.0 = LoB1_win_*
[serverClass:LoB_win_default:app:LoB1_win_default_inputs]

etc etc

Then in your input 'apps' you are pushing from the DS you specify things like monitoring paths, scripts, etc and what indexes the data goes to.

0 Karma

bmacias84
Champion

Use macros and eventtypes.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

You will most likely enjoy tags and eventtypes.

You can tag a host for specific function. Let's say web. (I'll do this from tags.conf on the command line)

[host=mywebhost01]
web = enabled

Next, use that in a search:

index=linux_servers tag::host=web

This search will pull all Linux servers that are designated as web hosts. Note that this won't scale well if you do it by hand. You may want to use an Asset database/csv/file/whatever to pull the list, then use a script to create a tags.conf file to use.

Event types are almost the same thing, but let's you specify a non-reporting/transforming search to match. In eventtypes.conf:

[linux_web_servers]
search = index=linux_servers tag::host=web

You would then simply search like this:

eventtype=linux_web_servers

These techniques will keep your searches clean and readable.

0 Karma

chawagon03
Path Finder

So, say we went the tag route. We had a script to populate the tags.conf based on some criteria stored in a csv sheet. Everything works. Lets use the example of web server.

Now one day, that hostname gets migrated to be an app server. We change the csv, and the tags.conf gets updated (assuming we were smart enough ahead of time to handle that in our script) and we restart the server.

Does the old data get changed to the new tag or does it keep the tag that was assigned when that event was indexed?

0 Karma

diogofgm
SplunkTrust
SplunkTrust

The tag is handled in search time. the tags will be applied to the events that fulfill the tag stanza, In the case the host field. If you change the tag to something else all the events from that host will have the new tag. You can work this out by using a different host name or using a proper naming convention for web servers vs app servers.

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...