Can I use multiple kvstore lookups in a single collection?

[wcooper003]

This is my first time trying out the kvstore, so learning by fire. I set up a collection in myapp/default/collections.conf, and set up a number of lookups within that single collection using stanzas in myapp/local/transforms.conf:

[lookup1]
external_type=kvstore
collection=mycoll
fields_list = _key, value, range

[lookup2]
external_type=kvstore
collection=mycoll
fields_list = _key, value, range

[lookup3]
external_type=kvstore
collection=mycoll
fields_list = _key, maxStat, range

I guess my first question is: can you have multiple lookups within a single kvstore collection, where the field names are repeated among lookups?

Here's the issue i'm having: If I try outputting to a single lookup file, and then see if I can read it back in with another search, it works fine. E.g.:

search=... | outputlookup lookup1

I can then see it with:

| inputlookup lookup1

However, if i try writing out to all of the lookups, they will all write fine, but only the last one has data, and the rest are blank. It's as if an outputlookup command will erase all other lookups in the collection except the one it's writing out to.

Maybe i'm not using these as intended - i'm actually doing it as a way to store data from costly searches on data sources that change once daily, where I have a saved search that I run on a schedule and writes to the kvstore lookup. Alternatively I guess I could use a summary index but not sure which is better.

Thanks

[hunters_splunk]

Hi wcooper003,

If you want to use kv store collections, you need to define collections in collections.conf, NOT transforms.conf.

Specify the name of the collection and schema, which requires at least two fields, matching lookup field and an output field. Optionally, you can enforce data types so that any input does not match the type will be silently dropped.

Usage
[collection_name]
endorceTypes = [true|false]
field. = [number|string|bool|time]
field. = [number|string|bool|time]
accelerated_fields. =

Example
[mykv]
enforceTypes = true
field.x = number
field.y = string
accelerated_fields.xl2 = {"x": 1, "y": 1}

Hope it helps. Thanks!
Hunter Shen

[wcooper003]

Hi Hunter,

I did define my collection in collections.conf, and I defined the lookups in transforms.conf.

I think my problem is that i was treating the collection as a unique database, and multiple lookups as individual tables within the collection database, which doesn't seem to be the case. I guess the easiest thing to do if I was to go this route for my use case (which i'm not, i've switched to saved searches), would be to set up three collections, and have a single lookup table within each, that way if the field names are the same it wouldn't matter. Not sure if that is efficient or not though.

Thanks

[s2_splunk]

My question would be: What are you trying to achieve? Why not have a single collection that contains the fields you need and selectively use fields as needed?

If you run a saved scheduled search daily over your events, the results of that search are stored on the search head for 2 days (by default). If you are fine with losing todays results come midnight, you really don't need to store those results elsewhere; Splunk will get it from the dispatch directory on the SH if the same saved search is run again over the same timeframe. If you want to keep history of aggregated results from the search, a summary index is probably your better choice here. Or use | collect into a different index.

But maybe I am not understanding your use case correctly.

[wcooper003]

Ah ok well that's just my lack of understanding of saved searches - it looks like I just need a saved search without having to store the results manually (I was double saving apparently). Not only am I learning kvstore, i'm also relatively new to Splunk 🙂

Out of curiosity though - despite my poorly chosen use case for a kvstore: why would I still see this behavior? Can one not have multiple lookups within a single kvstore location?

[Treep]

The way I understand KV-Stores you can have multiple lookups for a single KV-Store collections. The problem here might be the outputlookup command.
E.g. when you outputlookup to "lookup1", only the fields "_key", "value" and "range" will be set for all the rows/entries contained in your data, because those are the fields defined for "lookup1".
If you then outputlookup to "lookup3" without using the "append=true" parameter, you a) overwrite everything written previously to that KV-Store collection and b) you again only set fields "_key", "maxStat" and "range" because those are the only ones defined for "lookup3".
Edited to add: and of course, since "lookup1" doesn't have "maxStat" or "range" as defined field, it won't see those values.

I'd suggest defining a lookup containing all fields used in your KV-Store collection and only using that for outputlookup (and remember using "append=true" if you don't want to overwrite). You can still use the other lookups to actually lookup data but you won't accidentally overwrite any of it.

[wcooper003]

Ok thanks for the response, I think my problem is that I was assuming a collection is like a regular DB, and lookups would be individual tables within the DB, or at least the output would be logically separated among lookups. Clearly that is not the case from what I can tell.

[sundareshr]

How do you intend to use these KVStores? How much data are we talking about? Think you may be better off with either accelerated data model or summary index.

[wcooper003]

They're small - a couple of rows, a couple of fields. It's a goofy search I hacked together with a lot of 'append' commands to build up a composite health index from multiple data sources.

I wasn't sure if I could actually use a summary index because of the search - it's not using an aggregating search (stats, timechart, etc), but just some appends for head rows of different datasets.