Splunk Search

Implement a slowly populating lookup file

thomasbader
Engager

Looking for some hints and suggestions about how to implement this:

I have incoming log data that contains EAN barcodes (shown as number). A 3rd party API provides me with further details about what's behind the barcode. That's working so far. Already have an external lookup written in Python. However, the API has a rate limit and only allows to submit 10 queries per minute. It also only allows one barcode per query (so no batch lookups possible). Therefore there's no way to do a "live" lookup when the enduser is searching in the data.

Therefore I need to pre-lookup the barcode as soon as the events come in and cache them locally to allow searching within enriched data. The rate of incoming events might exceed 10 per minute often during the day. Due to the rate limiting the pre-lookups will be in residue during the day and again catch up during the night when there's less load. Therefore I need some way to build something like a fifo buffer (which survives even when Splunk restarts) and have a constantly running job feeding the barcodes to the lookup. Same barcode might also show up multiple time, so it must verify if a barcode is already in the buffer or if it has already been looked up earlier to avoid multiple lookups. Basically I need something like this:

Incoming log data -> verify if barcode is already in the buffer or has already been pre-looked up -> put barcode to fifo buffer -> feed the buffer at a rate of 10/minute to the external lookup command -> write the lookup result to CSV/kvstore so the enduser can search in the data without being rate limited

Any hints?

0 Karma

starcher
SplunkTrust
SplunkTrust

Use kvstore. Do updates based on _key and set the _key field yourself. so if same record it will update, if new it will add it.

In SPL that looks something like the below and make sure your kvstore lookup definition includes _key as an exposed field.

...  | eval _key=MYFIELD | outputlookup append=true mylookup 
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...