Deployment Architecture

How do we analyze indexers/search head log?

makincerdas
New Member

I have a working environment using index discovery.

While doing command 'tail -f' on idx01 (Indexer - master box), I notice the following logs.

idx01 (tail -f /opt/splunk/var/log/splunk/splunkd.log)

10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx03 at uri https://10.200.2.35:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_HTTP_REPLY_READ_FAILURE
10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx04 at uri https://10.200.2.38:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_HTTP_CONNECTION_FAILURE
10-19-2016 20:59:38.186 +0000 INFO TcpOutputProc - Connected to idx=10.200.2.36:9997
10-19-2016 20:59:46.316 +0000 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 3 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=19129, tar_elapsed_ms=3726, bundle_file_size=75900KB, replication_id=1476910767, replication_reason="async replication allowed"

I did search using this string 'Failed failure info' on splunk, but found nothing.

I am not sure if logs on indexers are parsed and indexed.
What is the best practice to monitor/analyze logs on indexers/search head boxes?

Thank you.

Tags (1)
0 Karma
1 Solution

TStrauch
Communicator

Hi,

the splunkd.log on your indexers should be indexed automatically. You can find them by searching index=_internal. There you will find all internal splunk logdata.

For getting the internal logs of your searchhead to the indexer tier look at this. This works similar for your master node.

http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

kind regards

View solution in original post

makincerdas
New Member

I did not use 'index=_internal' on earlier attempt.
It works by entering the following search, using 'index=_internal'

index="_internal" Failed failure info source="/opt/splunk/var/log/splunk/splunkd.log"

Thank you @TStrauch

0 Karma

TStrauch
Communicator

Hi,

the splunkd.log on your indexers should be indexed automatically. You can find them by searching index=_internal. There you will find all internal splunk logdata.

For getting the internal logs of your searchhead to the indexer tier look at this. This works similar for your master node.

http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

kind regards

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...