Splunk Search

How to write the regex for the removal of a header from a log file?

Bliide
Path Finder

I am trying to remove the header from a log file. I know that I need to put a stanza in props.conf on the forwarder and then create a transforms.conf like the following:

[skip_header_logfile]
REGEX = << 20-30 characters of your header line >>
DEST_KEY = queue
FORMAT = nullQueue

My problem is that my REGEX skills are horrible. My log file looks like this:

<Header>
  <Product>Microsoft SQL Server Reporting Services Version 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )</Product>
  <Locale>English (United States)</Locale>
  <TimeZone>Central Standard Time</TimeZone>
  <Path>C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\Logfiles\ReportServerService__01_06_2015_00_01_11.log</Path>
  <SystemName>SERVER</SystemName>
  <OSName>Microsoft Windows NT 6.1.7601 Service Pack 1</OSName>
  <OSVersion>6.1.7601</OSVersion>
  <ProcessID>10653</ProcessID>
  <Virtualization>None</Virtualization>
</Header>
library!WindowsService_112!195c!01/06/2015-00:01:11:: i INFO: Call to CleanBatch()

All that I need is REGEX that will select everything from to and of course include the header text, but I can not get it to work. Any help would be greatly appreciated.

0 Karma

sk314
Builder

AFAIK, having regex expression go to nullQueue will discard entire event, not just the matched pattern. You should be looking for a SEDCMD.

Try this:

props.conf
[your_sourcetype]
SEDCMD-null = s/(?s)<Header>.*(?=<\/Header>)<\/Header>//

Having said so, proceed with caution as this works directly at index time and there is no way to get back lost data. I suggest you try the command on sample data before putting it into production.

0 Karma

Bliide
Path Finder

Thanks for the quick response, I will give it a shot and I am working with sample logs until I get it sorted.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Converted this to an answer. Bliide, if you are still around could you check if this works and accept this answer, or otherwise comment back on what's still needed?

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...