Hi all,
I was hoping someone might be able to point me in the right direction for where to set this and how exactly to set it:
I'm consuming some logs from SCCM (log sample below for reference) and they have a distinct pattern for events and event breaks. I know that the regex that i would come up with is supposed to go into either props.conf or transforms.conf but I don't know which.
Sample events:
<![LOG[Active enforcement task with MTC Id a1c5be10-73fe-4d8a-b745-862c63a44faf not found]LOG]!><time="18:31:43.760+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="37360" file="ccitaskmgr.cpp:1054">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/DeploymentType_bbb6820e-f332-4550-91a6-12344b4ed081.3 for target: , consumer {3FC6019A-45DB-4D07-9F1D-E19A7F509482}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/Application_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {E6682089-51BE-4F27-A576-753A8D2B5EF8}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
<![LOG[Release task Enforce for CI ScopeId_7FFCE0E9-58B7-4E0C-8322-5704CF6E0B26/RequiredApplication_63222d92-85cb-40bd-850a-1f1264d1302c.3 for target: , consumer {901FBA32-A63D-4172-94FA-5C31054BF474}]LOG]!><time="18:31:43.838+00" date="11-12-2017" component="CITaskMgr" context="" type="1" thread="30416" file="ccitaskmgr.cpp:405">
As you can see from the sample above:
Each event "header" starts with <![LOG[
and ends with ]LOG]!>
Each event "detail" starts with <
and ends with >
and is always preceded by the "header" before it.
So effectively, it's <![LOG[headerhere]LOG]!><detailhere>
My thought was to do a single event as <![LOG[headerhere]LOG]!><detailhere>
and from there I could do field extractions for the "header text" and each part of the detail (where something=somethingelse).
Assuming I do the field extractions within splunk instead of props.conf/transforms.conf (best practice now), how should I do the event break?
Should I do something where "the end of every event is ">
" and set that as the props.conf LINE_BREAKER
?
Are there any other considerations that I should be aware of, such as SHOULD_LINEMERGE
or other things?
Many thanks
Give this a try as well
props.conf on Indexer/Heavy Forwarder, whichever comes first in data flow.
[YourSourceTypeHere]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)(?=\<\!\[LOG\[)
TIME_PREFIX=time=\"
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
MAX_TIMESTAMP_LOOKAHEAD=33
Same issue as above when I try this config 😞
I wish it were possible to specify the beginning and the end of the event instead
Does your log entries appear in separate lines or they are merged into big lines?
https://i.imgur.com/ufsAGkE.png
The common denominator is the fact that the "event break" as opposed to the "line break" (one would assume that they mean the same thing in this case) should always begin with <![LOG
In the screenshot you will see:
An event being correctly identified
An event being split before ]LOG]!>
Two events in the same event
I've modified the regex slightly to remove the CRLF (technically it may miss the first event as it doesn't preceed a CRLF) and the pattern in a regex tester is matching correctly, but still exhibits the same behaviour:
Hi @Lazarix,
Based on your sample data, it looks like you need break event when <!
comes in data In that case you can below configuration in props.conf on your Indexer or Heavy Forwarder whichever comes first.
[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
If you want to capture timestamp then you can use below config in props.conf
[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=12
Thanks!!
I'll give this a try now, wait for some events and let you know
It's not quite working 😕
my props.conf is:
[sccm]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=<!
Ok, I have modified my config slightly, can you please give this try
[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\<\!
TIME_FORMAT=%H:%M:%S.%3N%:::z" date="%m-%d-%Y
TIME_PREFIX=<time="
MAX_TIMESTAMP_LOOKAHEAD=33
Because I have tried your sample data using Setting->Add Data and while configuring above configuration splunk breaking events perfectly fine.
For reference, the REGEX that I had in mind was:
">