Splunk Search

How to fix my props.conf configuration to extract headers from a log file with garbage data on top?

normangoh
Explorer

Hi guys,

I need to extract headers from a log file, so that when it is pushed to the Indexer, those headers will be displayed.

The log file looks like this.

#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2014-11-11 00:00:00.210
#Fields:  date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes TimeTakenMS
2014-11-11 00:00:03.283 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:03.736 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:08.291 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:08.728 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:13.299 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:13.751 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:18.306 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:18.759 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:23.064 172.18.10.88 GET /announce/6mBill-result.html - 80 - "69.191.211.202" "BLP_bbot/0.1" - 301 0 0 257 270 0
2014-11-11 00:00:23.314 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:23.579 172.18.10.88 GET /News/T-Bill-Announcements.aspx - 80 - "69.191.211.202" "BLP_bbot/0.1" "http://www.sgs.gov.sg/announce/6mBill-result.html" 200 0 0 24830 493 62
2014-11-11 00:00:23.766 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:28.337 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:28.665 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:33.329 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:33.673 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:38.384 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62

As you see, the header line starts at line 4 and line 1-3 contains garbage with some time stamp. I tried putting the props.conf file in my universal forwarder's app, but it does not seem to be extracting the header. My props.conf looks like this.

[demozxc]
FIELD_DELIMITER = \s
FIELD_HEADER_REGEX = #Fields:\s+(.*)

May I know if there is anything I have done incorrectly? I tried putting "HEADER_FIELD_LINE_NUMBER = 4" in the props.conf, but it did not work as well.

0 Karma

woodcock
Esteemed Legend

Skip the FIELD_HEADER_REGEX and use only HEADER_FIELD_LINE_NUMBER like this:

[demozxc]
FIELD_DELIMITER = \s
HEADER_FIELD_LINE_NUMBER = 4

Also, because this is an index-time configuration, you need to make sure that you deploy it to your Indexers (or to your Forwarders if you are using a Heavy Forwarder configuration). If it still doesn't work, then the only thing it can be is a mismatch of the sourcetype so triple-check that and be aware that there may be precedence problems if you are overriding it.

0 Karma

normangoh
Explorer

Thank you for your reply!

I was also wondering if I should put this props.conf on my universal forwarder or my indexer? Currently I am placing this props.conf at my universal forwarder and it is not working.

0 Karma

woodcock
Esteemed Legend

That is why; it has to go on your Indexers and the Splunk instance will have to be restarted on each Indexer for it to take effect.

0 Karma

normangoh
Explorer

I tried putting the props.conf on my indexer but it still did not work. When I run the search there are no fields as specified in the log file header.

My code in props.conf looks like this.

[sourcetypename]
FIELD_DELIMITER = \s
HEADER_FIELD_LINE_NUMBER = 4

I have double checked and the sourcetype name is correct but still the fields are not extracted at index time. I shifted the props.conf to my indexer as well. Did I miss out anything?

0 Karma

woodcock
Esteemed Legend

Did you bounce Splunk on each Indexer like I said like rhis?

$SPLUNK_HOME/bin/splunk restart
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...