Splunk_Server is showing different server than the one in outputs.conf

[jordanking1992]

Hello,

We currently in the process of moving to indexer clustering with 3 new servers. The 3 old servers are standalone. After setting up the 3 new cluster peers and verify that the command ran for this are correct, i decided to send data to them. In my outputs.conf, i have only specified ONE peer from the new indexer cluster. However, when data is sent and indexed, the search head shows the splunk_server field with a value of our old server There are no references to that server name in any of the conf files either.. My machine has never sent data to the old servers so i am not sure whats going....

[woodcock]

Login to the host that has that event and run this command:

/opt/splunk*/bin/splunk btool list outputs --debug

You will see that there is a file somewhere (probably /opt/splunk/etc/system/local/outputs.conf) that is pointing to the old indexer.

[somesoni2]

In what time range you're searching and is the old indexer not receiving any data in that time frame? Is your search head searching to both old and new indexers?

[jordanking1992]

Timerange was in the last 15 minutes that i sent demo data. The searchheads are tied to the 3 old servers and the master_node that searches the three new. I am sending data to only one of new (cluster) nodes.

[somesoni2]

Can you login to Splunk web of that indexer and run the search? Just confirm if the data is indeed ingested on that old indexer or not.

[jordanking1992]

Weird. Yes, it is in fact indexed on that server and NOT the new cluster node specified in my outputs.conf. There is no reference to that server when using btool to check outputs.conf....

Could this be a bug or some type of network routing issue?

[jordanking1992]

root@ubuntu:/opt/splunk/etc/system/local# splunk btool outputs list --debug
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf defaultGroup = qa_cluster
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf [tcpout:qa_cluster]
/opt/splunk/etc/system/local/outputs.conf forceTimebasedAutoLB = true
/opt/splunk/etc/system/local/outputs.conf server = susplkidxqa001.homeoffice.anfcorp.com:9997