Splunk Search

Search efficiency question

responsys_cm
Builder

We've got a search that looks for suspicious data from a large number of netblocks. That search looks like: index=product AND src_ip="1.1.1.0/24" OR src_ip="2.2.2.0/24"...

There are dozens of them. That search is pretty fast considering the number of events we have. I would like to manage that search with a lookup table since I've created forms that make it easy to add or remove address ranges from the lookup table.

But if I search on all events from index=product and then do a lookup on that table and filter by events that match the table, the search is incredibly slow.

Is it possible to populate the src_ip=x OR src_ip=y OR src_ip=z for each entry in the lookup table? Or is there another way of implementing this that will be faster?

Thx.

Craig

Tags (1)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

Well, if you have the set of src_ip values in a lookup table already, then (up to a limit) this should be doable with a subsearch and inputlookup. Assuming your lookup is formatted like this:

src_ip
1.1.1.0/24
2.2.2.0/24

then a search like this ought to work:

index=product [ | inputlookup netblocks.csv | fields src_ip ]

The subsearch will populate the outer search with a sequence of AND'ed and OR'ed values coming from the results of the subsearch. There is a limit to the subsearch, however, to around 10,000 values. This is probably a limit than can be increased some, but it may not be possible to make it arbitrarily large.

View solution in original post

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Well, if you have the set of src_ip values in a lookup table already, then (up to a limit) this should be doable with a subsearch and inputlookup. Assuming your lookup is formatted like this:

src_ip
1.1.1.0/24
2.2.2.0/24

then a search like this ought to work:

index=product [ | inputlookup netblocks.csv | fields src_ip ]

The subsearch will populate the outer search with a sequence of AND'ed and OR'ed values coming from the results of the subsearch. There is a limit to the subsearch, however, to around 10,000 values. This is probably a limit than can be increased some, but it may not be possible to make it arbitrarily large.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

It is a function of the lookup you provide and the use of the "format" search command. Basically, each "row" of the lookup is surrounded by an OR, and each "column" within the lookup is surrounded by an AND. So, a lookup with the tuples (foo,green,4),(bar,red,7),(baz,yellow,5) will produce ( ( foo AND green AND 4 ) OR ( bar AND red AND 7 ) OR ( baz AND yellow AND 5 ) )

0 Karma

responsys_cm
Builder

How can it populate the outer search with both AND'ed and OR'ed values? Does it search for both? How can I force it to use one or the other?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...