Splunk Search

How to write the regex for the removal of a header from a log file?

Bliide
Path Finder

I am trying to remove the header from a log file. I know that I need to put a stanza in props.conf on the forwarder and then create a transforms.conf like the following:

[skip_header_logfile]
REGEX = << 20-30 characters of your header line >>
DEST_KEY = queue
FORMAT = nullQueue

My problem is that my REGEX skills are horrible. My log file looks like this:

<Header>
  <Product>Microsoft SQL Server Reporting Services Version 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )</Product>
  <Locale>English (United States)</Locale>
  <TimeZone>Central Standard Time</TimeZone>
  <Path>C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\Logfiles\ReportServerService__01_06_2015_00_01_11.log</Path>
  <SystemName>SERVER</SystemName>
  <OSName>Microsoft Windows NT 6.1.7601 Service Pack 1</OSName>
  <OSVersion>6.1.7601</OSVersion>
  <ProcessID>10653</ProcessID>
  <Virtualization>None</Virtualization>
</Header>
library!WindowsService_112!195c!01/06/2015-00:01:11:: i INFO: Call to CleanBatch()

All that I need is REGEX that will select everything from to and of course include the header text, but I can not get it to work. Any help would be greatly appreciated.

0 Karma

sk314
Builder

AFAIK, having regex expression go to nullQueue will discard entire event, not just the matched pattern. You should be looking for a SEDCMD.

Try this:

props.conf
[your_sourcetype]
SEDCMD-null = s/(?s)<Header>.*(?=<\/Header>)<\/Header>//

Having said so, proceed with caution as this works directly at index time and there is no way to get back lost data. I suggest you try the command on sample data before putting it into production.

0 Karma

Bliide
Path Finder

Thanks for the quick response, I will give it a shot and I am working with sample logs until I get it sorted.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Converted this to an answer. Bliide, if you are still around could you check if this works and accept this answer, or otherwise comment back on what's still needed?

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...