Copying a kvstore between an HF and a SH

[jwhughes58]

The Bloodhound Enterprise TA is run on the HF and generates an updated KV file every 4 hours.  I wrote a script that runs and turns the kvstore entries into alerts.  Due to some weirdness in the data, the question has come up, can the kvstore on the HF be copied to a SH.  I haven't found a suggestion that I think will work.  We are running Splunk Enterprise 9.1.1 on prem on servers running RHEL 8.8.

TIA,

Joe

[gcusello]

Hi @jwhughes58,

sorry but there's a strange thing in your question:

you are speaking of Bloodhound Enterprise TA app, but this isn't a TA, it's an App that should be located in Search Heads, also because usually KV-Store is disabled on Heavy Forwarders.

So you should use the HF for inputs and then use the App in your Search Heads.

In other words, usually KV-Store isn't created in an HF and usually this feature is disabled on HFs.

Ciao.

Giuseppe

[PickleRick]

Quick glance over this app (haven't downloaded it and looked into internals, just relying on the docs) suggest that this app is simply badly written. It aims at downloading some data from the Bloodhound service (whatever that is) and putting that data into an index. The problem is that the authors of this app probably have never seen something more complicated than a single-server Splunk installations. Also even the destination index seems to be hardcoded (or at least provided by default and not docummented as configurable). So I wouldn't be surprised at all if thw app used kvstore for whatever it needs to use it for.

[jwhughes58]

@PickleRickYou are correct.  It is poorly written.  I have already made three suggestions to them of which one is to split it into an ingest piece and a search piece.

[jwhughes58]

Hi Giuseppe,

Greetings from another Joseph.

We have a distributed environment with multiple SHs, 8 I think in 2 different groups.  For that reason any app/TA that have API calls or binaries go onto an HF that we use for this purpose.  We actually don't care about the UI for the app since all we want to do is pull in the events using the binaries.  I tried splitting the App into an input and a search piece but that didn't work.  We have suggested to Bloodhound that they do this.  For now the app runs on the HF which is where the kvstores are created.  The teams use the SHs to look at the events in the Search app.  I've had a request to copy the kvstores to one of the SH groups so they can be examined instead of using the alerts that get created from the kvstores.

Regards,

Joe

[gcusello]

Hi @jwhughes58,

I'm not sure that a kv-store, created in a HF, is usebla from a Search Head: kv-store must be located in SH.

On HF, you should locate the inputs.conf, the props.conf and the transforms.conf, not the other parts of the app.

Ciao.

Giuseppe

[jwhughes58]

Hi @gcusello ,

The kv-store isn't usable in the SHs.  The original ask was to take the items in the kv-store and create an alert for each item.  Since the kv-store gets recreated every 4 hours, this would cause alert duplication which we wanted to avoid.  I added another kv-store on the HF that contains a hash of the values in the items and then check to see if an item is new or already exists in the kv-store.  If new, an alert.  If not, it gets dropped.  The analysts decided they wanted to have the kv-store copied to the SHs they use.  Thus the original question since they don't have access to the HF.  Currently the best suggestion is to output the kv-store as a csv, scp it from the HF to the SHs, and load it into a local kv-store.

Regards,

Joe