Deployment Architecture

Client stopped indexing as hostname, can only be found by IP now.

vonas
Engager

We have had our splunk configured for about 2 years and not much has changed recently. All the sudden the other day we thought our firewall stop sending logs because we could not search by hostname (example: hostname="firewall1*"). After futher investigation we found that if we searched by ip (example: hostname="192.168.1.1") we can find all the firewall logs.

Can anyone think of a reason this would happen?

We have checked DNS forward and reverse records. The splunk indexer is capable of doing reverse lookups if needed, etc...

Tags (1)
0 Karma
1 Solution

JDukeSplunk
Builder

Best Guesses..

I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.

Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.

Or, you might want to add a /etc/hosts entry for that host on the Splunk box.

View solution in original post

JDukeSplunk
Builder

Best Guesses..

I would look at the inputs.conf on the box that reads this firewall, assuming that a UF is not present on the firewall itself. Find the stanza with "192.168.1.1" and possibly force a host= firewall1.

Or in the GUI under Settings,Data,Data Inputs and see if you can place a host name for the entry there.

Or, you might want to add a /etc/hosts entry for that host on the Splunk box.

vonas
Engager

I checked the data inputs as you suggested and it was set to DNS, which tells splunk to do a reverse lookup. I am not sure why, but when I restarted the service it all started working again and now it is reporting the hostname as firewall1 again.

Now is there anyway to get the other logs reindexed to all match the same hostname?

0 Karma

JDukeSplunk
Builder

It's sad, but no I don't think so. Especially not with syslog type entries.

The only thing I can think of is write you search to pickup both hosts names and either do some rename magic or an eval. There might be a way to put in an alias through the GUI as well.

host=firewall1 OR host=192.168.1.1 
|eval host=if(host=="192.168.1.1","firewall1",host)
0 Karma

vonas
Engager

OK Thanks for all your help

0 Karma

micahkemp
Champion

Can you include the payload of the message sent (full _raw) that gets indexed as the wrong host? It's very possible the firewall is sending logs with with the IP address in the payload instead of the hostname now.

0 Karma

vonas
Engager

I am not sure how to do that, can you provide an example?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...