Looking for thoughts on using lookup tables when data is indexed

[pfabrizi]

I know I can create lookup tables and use them during a search. We would like to apply that same process to fields as they are indexed.
so rather making field user Paul instead of Xxxad during a search we want to do this when the event is indexed.

Is this possible?
Does this impact indexing and what are the impacts on searching?

Thanks!

[gcusello · Accepted solution]

Hi pfabrizi,
if you want to denormalize logs at index time, the only way is pre-parse them before indexing by scripting, probably with the same performance impact on indexing time but a greater license consuption.
I usually use lookups to do this, also because lookups are usually quick in searches (it depends by the rows number).
Bye.
Giuseppe

[pfabrizi]

can lookup tables be indexed? I have a lookup table that has 4000+ rows and it takes much longer to search than my lookup tables with only a few hundred.

[gcusello]

If you have many rows, lookups aren't the best solution, think to summaries.
Bye.
Giuseppe

[pfabrizi]

Thank You!

I am new to SPLUNK and haven't had any training yet.

[gcusello]

Training is the best way!
anyway there are interesting free courses on https://www.splunk.com/view/SP-CAAAHSM and http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial
Bye.
Giuseppe

[pfabrizi]

is some documentation that describes this?

[gcusello]

No, to pre parse you have to create script in the language you like: we used php because we have in our team an expert on this language, choose the one you know!

Anyway this is an extreme solution that we perform only when we are obliged to pre parse a log.

You could eventually think to use summary indexes, in which you can create a denormalized table and use it for your searches.
This is the best method to accelerate dashboards, the only limit I found is that you need to have a continuos data flow because you have to schedule your summarization, if you have logs that arrive in discontinous way is more difficoult to use this solution.

Bye.
Giuseppe