Deployment Architecture

Client stopped indexing as hostname, can only be found by IP now.

vonas
Engager

We have had our splunk configured for about 2 years and not much has changed recently. All the sudden the other day we thought our firewall stop sending logs because we could not search by hostname (example: hostname="firewall1*"). After futher investigation we found that if we searched by ip (example: hostname="192.168.1.1") we can find all the firewall logs.

Can anyone think of a reason this would happen?

We have checked DNS forward and reverse records. The splunk indexer is capable of doing reverse lookups if needed, etc...

Tags (1)
0 Karma
1 Solution

JDukeSplunk
Builder

Best Guesses..

I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.

Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.

Or, you might want to add a /etc/hosts entry for that host on the Splunk box.

View solution in original post

JDukeSplunk
Builder

Best Guesses..

I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.

Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.

Or, you might want to add a /etc/hosts entry for that host on the Splunk box.

vonas
Engager

I checked the data inputs as you suggested and it was set to DNS, which tells splunk to do a reverse lookup. I am not sure why, but when I restarted the service it all started working again and now it is reporting the hostname as firewall1 again.

Now is there anyway to get the other logs reindexed to all match the same hostname?

0 Karma

JDukeSplunk
Builder

It's sad, but no I don't think so. Especially not with syslog type entries.

The only thing I can think of is write you search to pickup both hosts names and either do some rename magic or an eval. There might be a way to put in an alias through the GUI as well.

host=firewall1 OR host=192.168.1.1 
|eval host=if(host=="192.168.1.1","firewall1",host)
0 Karma

vonas
Engager

OK Thanks for all your help

0 Karma

micahkemp
Champion

Can you include the payload of the message sent (full _raw) that gets indexed as the wrong host? It's very possible the firewall is sending logs with with the IP address in the payload instead of the hostname now.

0 Karma

vonas
Engager

I am not sure how to do that, can you provide an example?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...