Splunk Search

[inputlookup] Does inputlookup return logs that have multiple values for the same field?

aakines
Engager

Suppose I performed the following subsearch

index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]

And lookup_file.csv has only id=456. Against thousands of logs, but one of those log events has the following output:

[name="ABC", id=123, name="DEF", id=456]

I saw that when I performed a search similar to this, the log would be returned with the lookup_id of 456 even though both 123 and 456 were present in the log. Is it expected behavior for inputlookup to return this log even if id=123 is found before id=456? Basically, does inputlookup return logs that have multiple values for the same field?

Tags (3)
0 Karma

woodcock
Esteemed Legend

Your event has BOTH id=123 AND id=456 so the result is correct and the results of all of these are the same for this event:

... id=123
... id=456
... id=123 OR id=456
... id=123 AND id=456

This makes perfect sense; to alter, you could add this:

... | where mvcount(id) == 1
0 Karma

somesoni2
SplunkTrust
SplunkTrust

You need to first understand what your inputlookup subsearch is doing here.

Your original search:

index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]

The subsearch will collect first 100 values of field lookup_id in lookup_file.csv and create giant OR condition statement with just the value of the field.

e.g. ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))

Your search will basically be (internally) tranformed like this

 index=whatever "name=" ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))

So you'll basically be doing a text search with those Id values and it'll return all records where that string is present. In your sample log [name="ABC", id=123, name="DEF", id=456], value 456 is present, hence it'll be returned. It's a plain text search so it'll not do any additional checks like only give where id=456 and no other values.

aakines
Engager

Awesome, thanks for that! I thought there was extra logic in place to check no other values. Is it safe to say that since it's a plain text search, any record can be returned if it even partially matches the sequence of characters in the lookup table (i.e. id=4567 will cause the record to be returned)?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...