with search-head pooling my jobs get canceled or expired

[mataharry]

I have a search-head pooling and shared storage.
Sometimes things get crazy and the clocks are completely drifting.
This causes my searches to fail immediately "cancelled or expired"

see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Troubleshootdistributedsearch#Clock_skew_be...

How to measure my clock drift quickly ?

[dshakespeare_sp]

Comment from Shaky
Not all system support stat -LF
An alternative is to use

ls -l --time-style=full-iso clock.test

[jreuter_splunk]

istat is another alternative to stat, available on aix.

[yannK · Accepted solution]

for SH pooling you need a clock perfectly sync... (more than a few seconds and it's bad)

Here a quick and dirty method to check the clock difference, from each search-head run

cd /my/share/storage

echo `date` "test from" `hostname` > clock.test

echo " my server clock (date)"

cat clock.test

echo "my filesystem clock (modtime)"

stat -LF clock.test

example of result, we can see a 1:09 minutes difference.

my server clock (date)
Tue May 7 17:32:19 PDT 2013 test from splunk.banania.com
my filesystem clock (modtime)
-rw-r--r-- 1 banania admin 69 May 7 17:31:10 2013 clock.test


The long term solution is to fix the clocks and NFS.
Prefer the ntpd daemon to smoothly keep it sync (npdate is too harsh sometimes)