Search Head Cluster: "WARN...single_action=PUSH_TO took wallclock_ms=1553! Consider a lower value of conf_replication_max_push_count in server.conf on all members"

[sat94541]

We have three Node Search Head Cluster member and we are seeing the following messages in our splunkd.log:

12-08-2015 10:00:25.468 -0800 WARN  ConfMetrics - single_action=PUSH_TO took wallclock_ms=1553! Consider a lower value of conf_replication_max_push_count in server.conf on all members
12-08-2015 10:00:25.468 -0800 WARN  ConfMetrics - single_action=PUSH_TO took wallclock_ms=1553! Consider a lower value of conf_replication_max_push_count in server.conf on all members

What does this error message mean?

[rbal_splunk · Accepted solution]

Here is some information on this error message

in summary, Configuration replication within the SHC is slower than the ideal standards (of 1 second), possibly because the SHC needs to frequently replicate the config updates with large lookup files.

We currently don't support blacklisting of the lookup files in the SHC.If you want to confirm this, on each of the Search Heads, please take a backup of the latest bundle file under var/run/splunk/snapshot to a temporary directory, and make a note of their sizes. untar the bundle file (tar xvf ...) and make a note of the largest files in the bundle directory. The expectation is the largest files in the bundle would be the lookup csv files, which will confirm this.