Any tips to quickly learn an existing Splunk setup?

[gabriel_vasseur]

I am new to Splunk and so far I find that the real difficulty is not learning Splunk itself but understanding my organisation's data and the way they set up Splunk. I wish Splunk would make this easier, but it doesn't. Allow me to use an example to explain.

So I see the "New Attacks - Last 30 Days" table in the "Intrusion Center" dashboard in Splunk Enterprise Security. I want to see where it gets its data from, so I click on "Open in Search" in the bottom left corner.

First hurdle: the search in question starts with a macro. So I need to do "Settings" > "Advanced Search" > "Search macros", copy-paste the macro name to see what the macro does. I have to do that every single time I meet a macro, which is painful.

Second hurdle: the macro is actually simply inputting a lookup. So I do "Settings" > "Lookups" > "Lookup definitions", search for the lookup name, find out what lookup file it uses, search for it under "Settings" > "Lookups" > "Lookup table files" and... find there's no way of knowing where this lookup comes from.

The "App" field does mention "SA-NetworkProtection" and after searching in the "Content Management" of ES, I did find a saved search called "Network - IDS Attack Tracker - Lookup Gen". I don't have permissions to see it (yet), but that's another issue.

This is just an example but my point is: Splunk is like an onion - when trying to understand where things come from, one painstakingly goes through lots of layers, whether these are macros, lookups, data models, etc... and that is frustrating. I understand that complexity is partially unavoidable because of the flexibility of Splunk and the need for abstracting and normalising the data. I just wonder if there is any tips or tricks people have found useful to cut through all these layers and gain insight faster?

One such trick I'm considering using, when I have my admin account, is to log on the linux box and grep through the configuration files...

[martin_mueller · Accepted solution]

Trying to unravel ES on your own is about the hardest thing you can do, and there's no quick win approach. I'd say it's best to take some time with those who set up ES for your organization, or at least with someone familiar with ES in general to get a good overview - there's also education available from Splunk. ES is highly modular, resulting in - you got it - a deeply-layered onion.

Regarding macros, to some degree you can see expanded search strings in the job inspector's normalizedSearch field, but that can be overwhelming in and of itself (shameless plug: https://splunkbase.splunk.com/app/2871/ :).

Regarding lookups, you can run | inputlookup lookup_name to get a clue about the lookup's contents as well.

Regarding grep, consider Splunk's own btool, for example:

$SPLUNK_HOME/bin/splunk cmd btool --debug transforms list lookup_name

[gabriel_vasseur]

Thanks Martin. That Knowledge Object Explorer app looks really useful!