Knowledge Management

How to ignore lines in a text file

gltplus
New Member

I'm struggling with a data source creating daily log files of the following format

01:06:15.558 Server 1.1.1.1: no new files
01:06:35.211 Positive leap second is reported by NTP
Some configuration text
config = 12.12
bootloader = 01.01.2012
01:06:35.267 Positive leap second is reported by NTP
01:07:15.558 Server 1.1.1.1: no new files

After adjusting the props.conf with
SHOULD_LINEMERGE = False
These lines are pared alright in Splunk.

The time is derived from log lines and date from the filename or date. (I don;t know but it just works)

The problem however is that each logfiles starts with +/- 500 lines of start-up config. To make it worse, it is possible that these lines could also be in the middle of these files as well. Because these lines do not contain any time information but other numbers instead Splunk can't interpret these lines. They are showing up with all kind of different weird time stamps. As I don't need these lines I'd like to remove them.

Is there a way to read every line from a text file, parse it as single line=singe event EXCEPT every line that does not start with a timestamp like 01:06:15.558 ?

I tried the following by adding the TRANSFORMS.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{3}\s
DEST_KEY = queue
FORMAT = indexQueue

and in the props.conf:
TRANSFORMS-set = setnull, setparsing

but that does not seems to work.

How can I solve this?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this for your transforms.conf entry (on indexer OR heavy forwarder, needs restart)

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\d{2}:\d{2}:\d{2}\.\d{3}\s 
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

gltplus
New Member

Great!! It works!

Thanx alot somesoni2.
This rexeg seems to do the trick!

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...