Getting Data In

"ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

sympatiko
Communicator

Hi Splunkers,

I'm getting the following error on my search head's splunkd.log:

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1

I have a RF = 3 and SF = 3 setup.

Thanks

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Cl... for more details.

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Cl... for more details.

cheers, MuS

dawid_schulz
Explorer

Hello due to same problem I don't want to create new topic.

In log I found

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1396 seconds exists between this search head and peer1
ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1909 seconds exists between this search head and peer2

But. When I check date I found that date is correct. 00:25:07 CEST on peer and 00:25:08 CEST on cluster master

I have All Data is Searchable Search Factor is Met Replication Factor is Not Met. Settings in server.conf are:

site_replication_factor = origin:1,site2:2,site4:2,total:4
site_search_factor = origin:1,site2:1,site4:1,total:2

We have 4 peers (showing under Indexer Clustering: Master Node) and 2 search (3 if we counting CM) and as you can see above - search factory is met (after a while) but replication factor is not. I also don't know what can happened if we have replication factor not met?

Ohh and one more importaint thing - this cause after we switch from single-site to multisite. Should I add in [clustering] replication_factor = 2 and search_factor = 1 to aviod replication factor not met ?

Thanks for any advice 😉

0 Karma

MuS
SplunkTrust
SplunkTrust

Your error states A time skew of approximately 1396 seconds exists between this search head and peer1 have you checked the time on the search heads and compared them to the indexers?

dawid_schulz
Explorer

Wow - my mistake. I don't know how I can miss that. Sorry for bothering you !

0 Karma

MuS
SplunkTrust
SplunkTrust

That's why I have a fez 😛 Feel free to upvote my hints 😉

ppablo
Retired

*hint upvoted* Precisely why you have a fez ;D Cheers!

0 Karma

akhil4mdev
Explorer

So how did you solve this ?

0 Karma

akhil4mdev
Explorer

I am facing the same problem could u please help me , I am only getting "WARN" not an error and the skew is "76" seconds and I can search the data from the indexer though

0 Karma

MuS
SplunkTrust
SplunkTrust

Make sure all your Splunk instances are time synced using NTP or any similar service.

cheers, MuS

0 Karma

sympatiko
Communicator

Thank you so much MuS.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...