Deployment Architecture

Is there any way to get Splunk to replicate search history in a search head cluster?

peter_holmes_an
Path Finder

I've recently noticed that the search history function on Splunk does not replicate over a search head cluster. This means if I go to the search page and look at my search history on one search head, it will be different to the history presented on another search head. It also means that if I forget a search I ran a few days ago, I have to play hunt the search head to find it again.

Is there any way to get Splunk to replicate this in the bundle?

Labels (1)

woodcock
Esteemed Legend

This has finally been addressed in a useable way that seems to not have any downside/impact in 9.1 (search for "Preserve search history across search heads"):
https://docs.splunk.com/Documentation/Splunk/9.1.1/ReleaseNotes/MeetSplunk

Scarily enough, it appears to be enabled by default.

0 Karma

the_wolverinie
Engager

Apparently this is now being treated as an "Enhancement" or "Idea"

https://ideas.splunk.com/ideas/E-I-47

joesrepsolc
Communicator

So essentially an extremely useful feature is no longer functioning when you move to a search head cluster.

That is a bummer, Splunk. Using search history is a major thing for most of my users. Not having that in a search head cluster configuration is a bummer.

bandit
Motivator

If you are willing to grant the user role access to the _audit index, the following query will display search history as a workaround.

index=_audit action=search user="*"
0 Karma

the_wolverine
Champion

No.

According to Splunk support and documentation, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

http://docs.splunk.com/Documentation/Splunk/6.5.3/DistSearch/HowconfrepoworksinSHC

"Note: The cluster does not replicate user search history. This is reflected in the default server.conf file, which includes the line, conf_replication_include.history = false. Changing that value to "true" has no effect and does not cause the cluster to replicate search history."

jplumsdaine22
Influencer

I raised this as a support case for 6.3.2 and was told by Splunk:

we do not recommend changing the value of conf_replication_include.history to true as this could have a significant impact on performance.

I haven't done any performance testing so I'm only taking their word for it - @somesoni2 have you observed any issues?

vasanthmss
Motivator

Is this feature affecting the performance ?

V
0 Karma

cramasta
Builder

I have been told by support as well that it will negatively impact performance and not to turn it on. This is unfortunate as we now have 10 members in a cluster and the chances of a users seeing what searches they ran yesterday are pretty slim when they log in the next day.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It may be a workaround but, instead of making part of SHC member replication (happens more frequently), you can write a script which can sync those search history CSV nightly.

In our 7 member SH cluster, we've sticky session so users are mostly logged onto same server. Plus regular users don't care about search history and most Power users are so amazing they document queries they run.

0 Karma

cramasta
Builder

Yeah the nightly sync approach may be something we should look at doing.

Users can see what they ran a hour ago since they are not changing search heads that quickly, but being able to come in the next day and look at history would be very useful. We have thousands of users and just upgraded to the SHC so all these small changes from the search head pool are being brought to our attention. Still better than being on the pool 🙂

0 Karma

jplumsdaine22
Influencer

@cramasta any tips on cluster host sizing?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

By default the history doesn't gets replicated. To enable the replication of search history, update the server.conf [shclustering] stanza's following property to true (default to false)

[shclustering] 
....other attributes
conf_replication_include.history = true

You have to make the changes in all SH cluster members, either in etc/system/local OR under any app.

the_wolverine
Champion

According to Splunk support today, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

peter_holmes_an
Path Finder

Thanks for the update, we'd decided to not try it as it was not a very big problem for us. I'll mark this as unaccepted though.

0 Karma

peter_holmes_an
Path Finder

Thanks very much that is exactly what I was looking for

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...