kvstore - crud and filtering

[abeeber_merck]

Hi folks,

I have a use case problem and could appreciate a peer review. My use case is to use a kvstore lookup as a journal to track events that have a specific rule that triggers an addition action/search.

The logic workflow is as follows;
1. event comes into Splunk a couple of times a day
2. a splunk search runs 4 times a day and identifies the event for handling.
3. for the first instance of event
4. splunk is used to normalize/creating fields for event so that the value of is mapped to Field3
5. a table command is used to verify that all required fields for follow up processing (eg field1, field2, field3, field4) are formatted correctly
6. a lookup is then used to query the kvstore based on Field3 for values in field5 (eg. | lookup kvstore field3 output field5)
7. a filter is then used to identify event ABC if it has a value in field5 that is null. (eg. I where where isnull(field5)
8. a custommand command is run; which generates a value for field5
10. the results are sent back to the kvstore using a outputlookup (| outputlookup | outputlookup append=t kvstore)

question 1
When the second splunk search runs and picks up event again, how do I tweak the above logic so as to prevent splunk from picking up that event and processing it with the customcommand? Meaning how do I prevent the duplication?

How do I use lookup to identify an existing match of a kvstore entry on top of the one that I have already defined? Or is there a better way?

question 2
How do I an outputlookup where I can drop kvstore entries after a period of time eg, like after 2 weeks?
(note all fields in my collections.conf are currently strings). Do I need to setup my kvstore definition as a temporal lookup, taking one of my fields in my collections.conf and make it a time/numeric value?

[abeeber_merck]

HI Starcher,

Thanks. One point of clarification;

For #1, that is using | eval key = _key, right? If so, how do I insert that into my logic?

Is that lookup kvstore key as Field3? (where field3 is the field of the unique value of my event)?

[starcher]

1. leverage the ability to set the _key field to what you want to control updating same row
2. add a last updated column in epoch, do date math and drop rows older than your desired age limit

[abeeber_merck]

Hi Starcher,

(let me try this again, was replying in the wrong place).

For #1; that means using | eval key = _key; right?
If so, how do I include that in my logic?
Is that through a lookup to match the key to my field that I have identified as unique?

[starcher]

Review the dev docs on kvstore. if you expose _key field you can set it to whatever you want. if you set it same each time then you are updating the row. Such as an IP address.

eval _key=ipaddress

https://dev.splunk.com/enterprise/docs/developapps/kvstore/uselookupswithkvstore/