Getting Data In

How to remove header of a log?

sabaKhadivi
Path Finder

as I edit props.conf & transforms.conf to remove header of log , but it didn't work
here is my config:

props.conf

[sourcetype]
TRANSFORMS-skiphdr= setnull

transforms.conf

[setnull]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

Is there any idea or suggestion?

0 Karma

whrg
Motivator

I'm assuming you put the correct regex in REGEX. See @nickhillscpl answer.

Here are some more ideas:

Remember to restart Splunk after making changes to configuration fies.

Also, you must put these settings on your Heavy Forwarder / Indexer. I will not work on a Universal Forwarder.

0 Karma

sabaKhadivi
Path Finder

@whrg yes,It's heavy forwarder , and I restart splunk service after changes.

0 Karma

nickhills
Ultra Champion

Your REGEX = does not contain anything.

If there is a header string you can identify, add this to the regex.

For example, if the first line of your log was:
-------Start of Log------
you might set REGEX = \-+Start of Log\-+

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

yes ,I add the regex of unused part of log

0 Karma

nickhills
Ultra Champion

Can you post a copy of the log header and your regex - please use the code formatter which looks like 101010

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

REGEX = ^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d 10.10.10.5\s1\s

useless part of my log is:
Mar 12 15:11:57 10.10.10.5 1

0 Karma

whrg
Motivator

Your REGEX looks too complicated. Try to simplify/shorten it.
Use regex101.com for testing. I noticed that your regex does not match because of the \s at the end.

0 Karma

nickhills
Ultra Champion

Try this regex: ^\w{3}\s\d+\s\d{2}:\d{2}:\d{2}\s\d+\.\d+\.\d+\.\d+\s\d
https://regex101.com/r/TwH2pp/1

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

@nickhillscpl
tnx for your answer, I give the result with SEDCMD in props.conf

0 Karma

sabaKhadivi
Path Finder

@nickhillscpl

0 Karma

sabaKhadivi
Path Finder

Mar 12 14:52:42 x.x.x.x 1 2019-03-12T14:52:42Z x.x.x.x s1 ;

this is the header that I need to remove from Mar to 1 and this is my regex (x are octet of IP Add)

^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d x.x.x.x\s1\s

0 Karma

whrg
Motivator

The REGEX line does not show anything. Is this correct? If not, use the Code Sample formatting for displaying special characters.
You will need a proper regular expression.
It will help us if you post the log header (anonymized).

0 Karma

sabaKhadivi
Path Finder

@whrg Mar 12 13:44:04 10.10.10.5 1
this is the useless part of my log which I want to remove, I put regex of it infront of Regex =

0 Karma

nickhills
Ultra Champion

When you post code (or regex) use the code tool to make sure it’s is formatted/displayed.

The code tool is the icon which looks like 101010

If my comment helps, please give it a thumbs up!
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...