Knowledge Management

Am I able to use my own custom key for KV Store and update only partial values?

goodsellt
Contributor

I'm looking to use the KV store to power a table to track usage and ownership of IP addresses across our environment, as well as a time based metric when when we send userid and ipaddress mappings to our Palo Alto firewalls.

The sticking point I'm having is with the key, as I'd like the list of IP addresses to be the 'primary key' as it were, with one scheduled search I'm running updating the ownership records, while another external script would be syslogging the entries over to the firewalls and inserting an updated timestamp into the KV store.

I'd like to confirm I could do this. My two questions on the KV store are:
1. Am I able to use my own data as the "_key" instead of Splunk's autogenerated keys?
2. Am I able to update records in the KV store partially? As in only updating one of the supporting fields for a record?

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hi goodsellt!

Yes, you can explicitly set the _key. Splunk only auto populates it if no _key is provided in the outputlookup

http://dev.splunk.com/view/SP-CAAAEZH

The key ID field
By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. When you use outputlookup to write to the KV Store, a key ID is autogenerated if you don't specify one explicitly. If you want to modify a specific record, you need to provide its key ID.

To view the key ID field in lookups, add "_key" to the fields_list in the lookup definition.

As for updating records, KVSTORE supports CRUD and allows you to update records. As for updating a single field in the record, you can essentially fudge that by rewriting the record entirely simply updating the fields you require.

http://dev.splunk.com/view/SP-CAAAEZH#examples

- MattyMo

View solution in original post

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi goodsellt!

Yes, you can explicitly set the _key. Splunk only auto populates it if no _key is provided in the outputlookup

http://dev.splunk.com/view/SP-CAAAEZH

The key ID field
By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. When you use outputlookup to write to the KV Store, a key ID is autogenerated if you don't specify one explicitly. If you want to modify a specific record, you need to provide its key ID.

To view the key ID field in lookups, add "_key" to the fields_list in the lookup definition.

As for updating records, KVSTORE supports CRUD and allows you to update records. As for updating a single field in the record, you can essentially fudge that by rewriting the record entirely simply updating the fields you require.

http://dev.splunk.com/view/SP-CAAAEZH#examples

- MattyMo
0 Karma

goodsellt
Contributor

Thanks! I have my custom key working successfully!

0 Karma

utkarsh21
New Member

How did you use a manual key? I am trying to do the same. Thanks in advance!

0 Karma

goodsellt
Contributor

When you are loading the data into the KVStore, you want to rename the field you're using as _key, therefore it will be used as the primary key record instead of what Splunk autogenerated for you.

I'm using a Python script to generate the documents I put into KVStore as part of another process, but an example of how I build the document is:
doc = dict(_key=str(item['IP']), User=str(item['User']), Domain=str(item['Domain']),
Last_Update=int(float(item['Last_Update'])), Last_PA_Send=int(0))

consider the item[xxxx] as my fields from my search results, since I wanted my IP field to be the primary key I renamed it to _key prior to sending it to KVStore.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...