Why is the Universal Forwarder reporting the wrong IP to Deployment Server?

[JDukeSplunk]

I have about 6 hosts that are reporting their IP address to my deployment server incorrectly.
They are running Universal Forwarder 6.5.2.

They all show up as the same 172.22.254.250 address.

I have checked the local /etc/hosts on the forwarder systems themselves. I have checked ifconfig on those hosts to confirm that they don't have some weird binding. I checked the deployment server /etc/hosts and can ping them all correctly by their actual 10.214.3.X IP address. I also checked the /SPLUNKHOME/etc/system/local/server.conf file. Nada.

Any ideas?

[JDukeSplunk · Accepted solution]

Thank you everyone for your feedback. On a whim I deleted the record for one host from the deployment server, and when it phoned home again it had the correct IP. Keep it simple. Odd that it did that in the first place.

[jkat54]

What is in the forwarders deploymentclient.conf?

 /opt/splunk/bin/splunk btool deploymentclient list --debug

My hunch is that you've got a load balancer in front of the deployment server and this is the "backend" ip of the VIP that the forwarders are using to commicate with your deployment server.

[mattymo]

My hunch would be that these forwarders all traverse a NAT interface.

Can you try a traceroute from the UF to the DS?

Like wrangler said, I believe the DS is mapping the host to the IP that their phone home was received from, and if the traffic is natted behind a router or firewall (or VIP like jkat suggrsted) then they would all be calling from the same IP

[wrangler2x]

Let's find out if the hostname is consistent on the forwarder's environment. To my knowledge, Splunk does not store the IP address anywhere in its .config files.

Look first in $SPLUNK_HOME/etc/system/local/server.conf, for the [general] stanza. There may be a line like this:

serverName = hostname

You'd expect that to be correct if it is there.

Next take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log (copy it off somewhere or edit it with vi -R)

Searching from the bottom up, find the line INFO loader - Splunkd starting. The next line down will be INFO loader - System info: and the hostname that Splunk thinks it is will be there.

INFO  ServerConfig - My GUID is 5AEF640C-24F3-4A8A-AD4C-08227E9C4FE5
INFO  ServerConfig - My server name is "hostname".
INFO  ServerConfig - Found no site defined in server.conf
INFO  ServerConfig - My hostname is "hostname".
[snippage]
INFO  ServerConfig - Using REMOTE_SERVER_NAME=hostname

All these hostnames should be what you'd expect on that forwarder. If not, that's a clue something is wrong somewhere.

And then look for lines like this:

INFO  HttpPubSubConnection - SSL connection with id: connection_xxx.xxx.xxx.xxx_8089_hostname_5AEF640C-24F3-4A8A-AD4C-08227E9C4FE5

The 5AEF640C-24F3-4A8A-AD4C-08227E9C4FE5 string is the GUID you saw earlier in the logs (it will be different on your system. And again here you should see the hostname you expect, and the IP that you expect.

[wrangler2x]

Does this give you the same results, or differerent (IP address-wise)?

index=_internal source=*metrics.log* group=tcpin_connections  NOT eventType=connect_close NOT eventType=connect_done
| table hostname sourceIp arch fwdType os version
| dedup sourceIp
| where NOT sourceHost=sourceIp

[JDukeSplunk]

Thanks Wrangler
Since all 6 of the hosts are reporting as the same wrong IP (See the picture) the dedup command only shows me one now. What I really need help with isn't the search so much as why these forwarders are reporting the wrong address. I edited the question to better reflect that.

[wrangler2x]

The Splunk forwarder does not store the hosts IP address anywhere in its configs that I am aware of. Take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log and search for

INFO HttpPubSubConnection - SSL connection with id: connection

The whole thing will look something like this:

04-26-2017 16:04:32.212 INFO HttpPubSubConnection - SSL connection with id: connection_xxx.xxx.xxx.xxx_8089_hostname_5AEF640C-24F3-4A8A-AD4C-08227E9C4FE5

where xxx.xxx.xxx.xxx is the IP address the forwarder thinks it is, and hostname is the hostname it thinks it is. How does this IP address compare to what is in your search results?

Now also look for INFO loader - System info. This is in the same log file right after an entry that says Splunkd starting. On that line will be the type of system and the hostname Splunk thinks it is.

A little ways further in the logs you will find a line that says INFO ServerConfig - My GUID is and thee string that follows that is what is used at the end of those HttpPubSubConnection - SSL connection entries like the one I showed, above.

Then you'll see three more lines

04-26-2017 13:11:55.781 -0700 INFO ServerConfig - My server name is "hostname".
04-26-2017 13:11:55.781 -0700 INFO ServerConfig - Found no site defined in server.conf
04-26-2017 13:11:55.781 -0700 INFO ServerConfig - My hostname is "hostname".

Then a bit further look for

04-26-2017 13:11:55.785 -0700 INFO ServerConfig - Using REMOTE_SERVER_NAME=hostname

All of these hostnames should match.

Take a look also at $SPLUNK_HOME/etc/system/local/server.conf

There should be a stanza there called [general] which has
servername = hostname

If your finding something out of line in any of these, that'd be a clue.

[wrangler2x]

I just left a long comment but when I submitted it, it disappeared, so I going to retype it and leave it as an answer, though it really isn't one. About the search, changed the dedup to hostname