I have a search-head pooling and shared storage.
Sometimes things get crazy and the clocks are completely drifting.
This causes my searches to fail immediately "cancelled or expired"
How to measure my clock drift quickly ?
for SH pooling you need a clock perfectly sync... (more than a few seconds and it's bad)
Here a quick and dirty method to check the clock difference, from each search-head run
cd /my/share/storage
echo `date` "test from" `hostname` > clock.test
echo " my server clock (date)"
cat clock.test
echo "my filesystem clock (modtime)"
stat -LF clock.test
example of result, we can see a 1:09 minutes difference.
my server clock (date)
Tue May 7 17:32:19 PDT 2013 test from splunk.banania.com
my filesystem clock (modtime)
-rw-r--r-- 1 banania admin 69 May 7 17:31:10 2013 clock.test
The long term solution is to fix the clocks and NFS.
Prefer the ntpd daemon to smoothly keep it sync (npdate is too harsh sometimes)
Comment from Shaky
Not all system support stat -LF
An alternative is to use
ls -l --time-style=full-iso clock.test
istat is another alternative to stat, available on aix.
for SH pooling you need a clock perfectly sync... (more than a few seconds and it's bad)
Here a quick and dirty method to check the clock difference, from each search-head run
cd /my/share/storage
echo `date` "test from" `hostname` > clock.test
echo " my server clock (date)"
cat clock.test
echo "my filesystem clock (modtime)"
stat -LF clock.test
example of result, we can see a 1:09 minutes difference.
my server clock (date)
Tue May 7 17:32:19 PDT 2013 test from splunk.banania.com
my filesystem clock (modtime)
-rw-r--r-- 1 banania admin 69 May 7 17:31:10 2013 clock.test
The long term solution is to fix the clocks and NFS.
Prefer the ntpd daemon to smoothly keep it sync (npdate is too harsh sometimes)