Update lookup file just before running my search?

[the_wolverine]

I'm running a subsearch which updates a lookup file (using outputlookup) but it doesn't seem to work. The subsearch on its own works just fine but I want it to run just before my base search so the lookup is fresh.

Is there some way to tell Splunk to ignore my subsearch results?

My search is something like this:

blah | lookup paper.csv host [ search "stuff | outputlookup paper.csv" ]

Put it another way, how could I get searchA to run immediately before searchB? I'm triggering searchB manually.

[Ayn · Accepted solution]

I think the problem doesn't lie in that your lookup isn't populated before you run the main search, rather the problem is what the subsearch returns to the outer search. When you do

blah | lookup paper.csv host [ search "stuff | outputlookup paper.csv" ]

the outputlookup command will produce output to the search pipeline (a table containing the fields you've written to the lookup). This will, after the subsearch is done and returns its formatted results, make the subsearch expand to something like

blah | lookup paper.csv host ((field1="value1" AND field2="value2") OR (field1="...

Which will result in a syntax error or if you're "lucky" it just won't return anything. You can see for yourself what the subsearch outputs by running it by itself and appending a "| format" at the end.

So, the trick you need to pull off is to either have the subsearch expand to absolutely nothing at all, or expand to something that works syntactically with the lookup command you're calling outside the subsearch.

The problem with the first alternative is that a subsearch with empty output seems to always result in the output string "NOT ()" after format has been run - and format is ALWAYS called implicitly by subsearches.

The second alternative, while pretty ugly, seems to be your best shot. Make the subsearch return, for instance, the word "host". You could achieve that by doing something like

blah | lookup paper.csv [search stuff | outputlookup paper.csv | head 1 | eval search="host" | fields search | format "" "" "" "" "" ""]

This will result in that the paper.csv lookup will be populated by the subsearch before it's used in the outer search, and after the subsearch is done the resulting search should look like

blah | lookup paper.csv host

This definitely isn't pretty, but at least it should work. For other arguments to lookup just modify the "search" field in the subsearch.

[the_wolverine]

Yeap that worked. Thank you Ayn!!!

[rtadams89]

Subsearchs always run after the main search, as subsearchs use the results of the main search as an input.

sourcetype="foo" | [search blah | fields bar]

does work as the main search returns everyhting in the "foo" sourcetype, and the subsearch then takes those results and looks through them for "blah". Of course this is a waste of resorces and you would be better off doing

sourcetype="foo" blah | fields bar

Can you explain what it is you are trying to accomplish in the end, and perhaps I can come up with a better way to get to that end goal.

[Ayn]

This is simply incorrect. What is true is that other commands in the search pipeline BEFORE where the subsearch is used will run before the subsearch, but within the same section of the search pipeline that isn't the case. You're somewhat right when you're talking about

sourcetype="foo" | [search blah | fields bar]

though you will likely get a syntax error because the subsearch will probably expand to something that Splunk doesn't recognize as a command, which is what it expects after the pipe.

Remove that pipe though, and the subsearch will run before the search command outside it.

[rtadams89]

I'm not sure what you are trying to do, but you may be able to do a "join" with a field or values that will never match (so that no results of the subsearch will actually get joined to the primary search). I haven’t tested this but...

index=main keywords to search for | eval joinOn=1 | join type=outer joinOn [search subsearch keywords | outputcsv paper.csv | eval joinOn=2] | ...

The thing to remember is that the subsearch will run after the main search, so the outputcsv command won't happen until the main search has already completed.

What is it you are trying to accomplish this way? There may be a better way to get to the same end results.

[the_wolverine]

Yes, so that's the problem. I'm not trying to perform a join. I'm trying to update a lookup table immediately before my main search runs. Maybe there is a better way to do this?

[Ayn]

subsearches will NOT run after main searches - in that case something simple like

sourcetype="foo" [search blah | fields bar]

wouldn't work, because the main search needs the subsearch's results before it can run.