We have had our splunk configured for about 2 years and not much has changed recently. All the sudden the other day we thought our firewall stop sending logs because we could not search by hostname (example: hostname="firewall1*"). After futher investigation we found that if we searched by ip (example: hostname="192.168.1.1") we can find all the firewall logs.
Can anyone think of a reason this would happen?
We have checked DNS forward and reverse records. The splunk indexer is capable of doing reverse lookups if needed, etc...
Best Guesses..
I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.
Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.
Or, you might want to add a /etc/hosts entry for that host on the Splunk box.
Best Guesses..
I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.
Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.
Or, you might want to add a /etc/hosts entry for that host on the Splunk box.
I checked the data inputs as you suggested and it was set to DNS, which tells splunk to do a reverse lookup. I am not sure why, but when I restarted the service it all started working again and now it is reporting the hostname as firewall1 again.
Now is there anyway to get the other logs reindexed to all match the same hostname?
It's sad, but no I don't think so. Especially not with syslog type entries.
The only thing I can think of is write you search to pickup both hosts names and either do some rename magic or an eval. There might be a way to put in an alias through the GUI as well.
host=firewall1 OR host=192.168.1.1
|eval host=if(host=="192.168.1.1","firewall1",host)
OK Thanks for all your help
Can you include the payload of the message sent (full _raw
) that gets indexed as the wrong host? It's very possible the firewall is sending logs with with the IP address in the payload instead of the hostname now.
I am not sure how to do that, can you provide an example?