Splunk Search

How to search recursively without repeating subsearches

hiyer
Explorer

Hi,

I'm trying to search recursively, but it would be nice to avoid duplicate searches.

Concrete example:

  1. Provided search outputs logs, which contain phone numbers.
  2. For each of those phone numbers, search by phone number to find logs with email addresses.
  3. For each of those email addresses, search by email address to find logs with IP info.
  4. Output table with people's phone number, email address, and IP information.

Current Implementation:

PART 1 - Using A to find B to find C, then displaying C by B:

search3 [search2 [search1 | fields A | rename A as query] | fields B | rename B as query] | stats values(field C) by B

*Note that I got lucky with B being re-identified by field discovery in search 3.

PART 2 - Using A to find B to find X, then displaying X by B:

search3 [search2 [search1 | fields A | rename A as query] | fields B | rename B as query] | stats values(field X) by B

PART 3 - Redoing A to find B, to display A by B:

search2 [ search1 | fields A | rename A as query] | stats values(field B) by field A

PART 4 - Joining them all on the common field B and output a table.

Output Table: A B C X

But this requires doing search1 3x and search2 2x. Is there a way to save results in the process? As this needs to be done on real-time results, I can't just save a subsearch for lookup later. Also, if I save more fields than just the one renamed as "query," nothing is returned.

Any better ideas? Thanks so much!

0 Karma

hiyer
Explorer

Provided an example and improved the formatting. Is this better? It's a pretty fundamental operation in the work I do, and I want this to be accessible to others.

0 Karma

asimagu
Builder

wow! I am really overwhelmed and lost by all that nesting, mate

0 Karma

sideview
SplunkTrust
SplunkTrust

I can think of a way to do it in the UI layer with Sideview Utils. This may or may not be your cup of tea, but it feels worth posting so here goes.

The Sideview ResultsValueSetter module, (for which some decent docs and examples exist within Sideview Utils itself), has the general mission of getting one or more field values from the current search results, and pulling them down to the UI so that they can be used for other things.

Notably you can use these $foo$ tokens as arguments to subsequent searches or postProcess searches courtesy of the Sideview modules Search and PostProcess.

So, in general, if you do a search A, use ResultsValueSetter to pull down some field value(s) and then submit those value(s) on search B, ResultsValueSetter becomes an alternate way of doing subsearches. If you learn how to use the format command explicitly in the first search, and you pull down the search or query field as appropriate, it can really be a one for one replacement, albeit a weird one.

You can get a little more flexibility with ResultsValueSetter though, in the sense that

A) You can actually other things or charts or tables from the first search, along with the second, "outer" search. With a subsearch the inner search results are inaccessible.

B) You're suddenly not subject to subsearch limits in time and rowCount, in that each layer of "subsearch" will run as a full search in Splunkd.

C) Notable for your question -- you can actually plug the $foo$ tokens back into a Sideview PostProcess module instead of the Search module, and by so doing you can leverage the search results you already have for search 1.

So I think all of your use cases can be done using ResultsValueSetter and Sideview modules instead of subsearches at all. Granted, this is a pretty advanced topic and you'd have to understand the Sideview modules, the Splunk UI as well as the various pitfalls around postProcess pretty well. (See "Key Techniques > PostProcess > Introduction to postprocess" and "Key Techniques > Overview of the advanced XML" in Sideview Utils)

Remember you can only get the current version of the app (2.5) from the Sideview website.
http://sideviewapps.com/apps/sideview-utils.

hiyer
Explorer

Thanks. I'll try this out and let you know!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...