Deployment Architecture

Where to check the clock skew?

vrmandadi
Builder

I have issues with clustering .I tried running the resync command but it says "Downloaded an old snapshot created 10018 seconds ago; Check for clock skew on this member or the captain; If no clock skew is found, check the captain for possible snapshot creation failures
"

Where can I see the clock skew and where can I change it

Labels (1)
0 Karma

amitm05
Builder

Hi @vrmandadi

You can run this to check for time skew among your instances -

| rest /services/server/info | eval updated_t=round(strptime(updated, "%Y-%m-%dT%H:%M:%S%z"), 2) | eval delta=now()-updated_t | table serverName, updated, delta | convert ctime(updated_t)| rename updated as "Local Time on Server on Request" delta as "Offset - Time in Seconds"

And if you find some gap here, you might want to set your server clocks
Also check -
https://docs.splunk.com/Documentation/Splunk/7.3.0/DistSearch/Troubleshootdistributedsearch

fredclown
Contributor

I'm not sure this works any more. The update field that is returned from the REST API is always set to"1969-12-31T16:00:00-08:00" for all my servers. Maybe it's just my environment. Anyway, here is an alternate solution.

| rest splunk_server=* /services/search/jobs/export search="| makeresults count=1 | rename _time as time" output_mode=csv
| makemv tokenizer="([^\n]+)" value
| eval local_time=now()
| eval remote_time=mvindex(value,1)
| eval local_time_formatted=strftime(local_time, "%Y-%m-%d %H:%M:%S")
| eval remote_time_formatted=strftime(remote_time, "%Y-%m-%d %H:%M:%S")
| eval delta_secs=abs(local_time-remote_time)
| fields splunk_server, local_time_formatted, remote_time_formatted, delta_secs
| sort - delta_secs

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

To be honest, I know that if you have Splunk you probably want to do everything in Splunk but monitoring the time sync state is something I'd rather do at OS level using your favourite monitoring software (zabbix, nagios, whatever).

0 Karma

amitm05
Builder

Let me know by accepting the answer if it responds your query. Thanks

0 Karma

lmvmandadi
Engager

I checked the time but the offset-time in seconds did not show anything .But still the error persists

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...