Getting Data In

Can you specify the order of operations for sourcetype definitions in props.conf to run an eval after a lookup?

mcrawford44
Communicator

From: http://docs.splunk.com/Documentation/Splunk/6.4.1/admin/Propsconf

You cannot use a field added through a lookup in an eval statement for a calculated field.

Will we ever be able to choose the order of operations? I've run into a situation where I need an eval to run AFTER a lookup.

Is there an existing workaround to this, besides including the eval in every search in the environment?

0 Karma

mcrawford44
Communicator

Ok, this begs the question; Why?

Why on earth would we not be able to control the order of operations?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Because there has to be an architecture. Some (not many) parts of an architecture are in essence purely arbitrary initial design decisions, but once those decisions get made, other things follow necessarily from the design... and changing those fixed elements becomes more and more complicated and unwise. (See - any initial Microsoft major release)

I believe, if you ever go to a new splunk shop, you will breathe a sigh of relief that certain orders of execution are fixed, so that when you are researching an issue, or trying to understand what your new system is doing -- a system designed by someone else but now YOUR responsibility to keep tuned and running -- that you can read the conf files in a particular order, and eventually trace down exactly what is happening.

If someone could alter that order -- including someone who did not know all the ramifications of that change -- then if could become quite a nightmare.

It's hard enough when there are local conf files in play in clustered environments...

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Here is the order of search operations:

#Search-Time Operation ORDER
Sourcetype RENAME
EXTRACT-xxx
REPORT-xxx
KV_MODE
FIELDALIAS-xxx
EVAL-xxx
LOOKUP-xxx
MILLISECONDS
FILTER
EVENTTYPING
TAGGING

As you can see EVAL occurs before LOOKUP.

What you might consider is not coding the lookup into the props.conf but doing the lookup as part of your search then doing an eval after the lookup.

If it's something you need to do a lot perhaps a macro would simplify it.

0 Karma

mcrawford44
Communicator

Regarding "LOOKUP-xxx";

  1. What order are the "xxx" processed in? Alphanumerical?
  2. If you can order lookups, can you use the first lookup as a field for the second lookup?

This is for a sourcetype that feeds into numerous reports and data models. The raw search must produce results from both lookups. Keeping it in the sourcetype is more logical in the environment than a macro would be unfortunately.

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Props.conf based lookups are processed on the precedence order (alpha sort sequence) as other operations. I've not tried using lookups based on lookups myself but it SEEMS logical that they would work - YMMV.

woodcock
Esteemed Legend

You cannot change the order of operations but you can change the method of your modification. Many of these operations can be twisted to do the same thing as one of the others and this conversion will move it to a different position in the order. This is the order:

INDEXED_EXTRACTIONS -> SEDCMD -> TRANSFORMS <---###Transition from Index-Time to Search-Time###---> (sourcetype)RENAME -> EXTRACT -> REPORT -> KV_MODE -> FIELDALIAS -> EVAL -> LOOKUP -> MILLISECONDS -> FILTER -> EVENTTYPING -> TAGGING

Actually, I am not absolutely certain about the order of the first 2.

woodcock
Esteemed Legend
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi mcrawford44, have considered to use the eval further down the search pipe after an automatic lookup? This should work fine.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...