How to update events with new key/value pairs on a per event basis?

[splunker1981]

Hi Splunkers,

I was wondering if someone could shed some insight on whether this is even possible with Splunk, if so, what is the right approach to accomplish the following.

I'm using dummy data as an example, shown below. We have data which we import and for the most part only has one or two fields populated at ingest time. What I'm trying to figure out is how to go about updating the key/value pairs for specific fields on a per event basis.

Using the example below the first column will always be populated when we ingest the data. What I'd like to be able to do is as we discover additional values I'd like to be able to update the specific event with the new key/value. Let's say for example I have an event Milk and the only thing I know at ingest is the price. This would be the last event. I'd like to be able to go in and have the ability to populate the event with the quantity, vendor and date. All of which in the example below are blank initially. Is this something that is supported by Splunk? Thanks for the help in advanced.

Type         Vendor         Count       Price       Date
Muffin       abc             7.0        4.99        2016-09-01          
tart         xyz            10.0        1.99        2016-09-01
eggs         xyy            12.0        0.99        2016-09-01
milk                                    1.99       

[somesoni2 · Accepted solution]

An alternative is to setup KV Store (assuming you don't have ton of data like in millions) for this data and update the KV Store data as and when you receive new field values. They are basically lookups and support updates based on key-value pair. See these links for more information. Keep in mind, you'd need a scheduled search or something to scan new data and update new field values to KV store.

http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZK
https://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/ConfigureKVstorelookups
http://dev.splunk.com/view/webframework-tutorials/SP-CAAAEZT

[splunker1981]

Thanks, I played around with this and seems to do the trick. How it scales is another question...

[sundareshr]

Splunk does not support modifying indexed data. What you could do, is ingest another event of Type=Milk, Vendor=MilkVendor, Date=2016-09-02 and generate a report that merges Price=1.99 for display purposes. In other words, let's assume you have the following two events ingested.

Type=Milk, Price=1.99
Type=Milk, Vendor=MilkVendor, Date=2016-09-02

A search like this, would give you your desired output

base search | stats last(Vendor) as Vendor last(Price) as Price last(Date) as Date by Type