Hello,
I would like to filter out the log entries (before indexing) which are created in the debugging mode. They can be identified by the letter "d" in my logfile:
[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N
In the above there is a header line, which has always a similar structure and the 3 additional lines of the event. The header line includes always the pattern:
11:24:29.210542 d
where the letter can be "i" for info, "e" for error, "d" for debug, etc.
Now, I would like to filter out all the events (header line plus belonging lines) where the loglevel is "d".
How would I achieve this?
On the example of the below:
[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[293672]{-1}[-1/-1] 2019-01-08 11:24:29.210542 d PITRestart LogReplayCoordinator.cpp(02658) : replayStepFinished(), logPos=0x1d2e2bab65b8
Line 1
Line 2
Line N
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799
I would like to get rid of the second event and have only the following after indexing:
[259451]{-1}[-1/-1] 2019-01-08 11:24:29.213984 a STATS_CTRL Schedule.cpp(00106) : Enter void StatisticsService::ScheduleReloader::updateExecutionQueue(StatisticsService::ExecutionQueue&)
Arg this = 0x00007e382849b0a8
[265685]{242705}[1340/-1] 2019-01-08 11:24:29.144534 e StatementResourc StatementResourceTracking.cc(00217) : statistics collection is not finished: stmt=0x00007e8643473400, stmtid=1042411823155799
Could you help me with the props.conf, transforms.conf and especially the corresponding REGEX for that?
Kind Regards,
Kamil
Hi @damucka,
Please try below config on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.
props.conf
[yoursourcetype]
TRANSFORMS-eliminatedebug = setnull
transforms.conf
[setnull]
REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd
DEST_KEY=queue
FORMAT=nullQueue
EDIT: If Line 1, Line 2 ... Line N
are events with Debug line then you can try below transforms.conf
[setnull]
REGEX=(?s)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd.*\n
DEST_KEY=queue
FORMAT=nullQueue
Hi @damucka,
Please try below config on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.
props.conf
[yoursourcetype]
TRANSFORMS-eliminatedebug = setnull
transforms.conf
[setnull]
REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd
DEST_KEY=queue
FORMAT=nullQueue
EDIT: If Line 1, Line 2 ... Line N
are events with Debug line then you can try below transforms.conf
[setnull]
REGEX=(?s)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\sd.*\n
DEST_KEY=queue
FORMAT=nullQueue
You would also need to ensure that the line with debug, "d" and the following line 1, 2 ... should be configured/treated as multi-line, so as to remove all of them together with nullQueue
I am guessing that 1,2 ... those are same type of events and not multiline values.
At the moment we have 3 events there, each starting with the header line. Not sure about the multiline values ... I did nothing with the configuration, Splunk recognizes it itself. So, to make it clear - I would like to get rid of the second event.
I have provided 2 different regex for transforms.conf, you can try that and let us know if it will not work.
Thank you. I will test and let you know.
Is it possible to match several sourcetypes in props.conf to the same entry in transforms.conf? Like below:
[yoursourcetype1]
TRANSFORMS-eliminatedebug = setnull
[yoursourcetype2]
TRANSFORMS-eliminatedebug = setnull
[yoursourcetype3]
TRANSFORMS-eliminatedebug = setnull
Yes you can
can we simply use "210542 d"
REGEX=210542\sd
Hi @inventsekar,
I guess datetime value which is provided is for sample only so 210542\sd
will not work because it is subsecond and every event will have different values for subsecond.