Hi!
I am currently having some problems breaking certain events from an Oracle log correctly. The log is being onboarded via a HF, and so the props.conf file containing the sourcetype of the log is located there. This is my current props.conf configuration for the sourcetype:
[application:env:oracle:alert]
TRUNCATE = 0
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{4}-\d{2}-\d{2}[T]\d{2}:\d{2}:\d{2}.\d{6}\+\d{2}:\d{2}
A sample of the log file:
2019-04-23T02:00:00.234792+02:00
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
2019-04-23T04:07:57.219629+02:00
ALTER SYSTEM ARCHIVE LOG
2019-04-23T04:07:57.258800+02:00
Thread 1 advanced to log sequence 867 (LGWR switch)
Current log# 3 seq# 867 mem# 0: +DATA/FWEB1/ONLINELOG/group_3.260.979816057
Current log# 3 seq# 867 mem# 1: +FRA/FWEB1/ONLINELOG/group_3.259.979816057
2019-04-23T04:07:57.426428+02:00
TT02: Standby redo logfile selected for thread 1 sequence 867 for destination LOG_ARCHIVE_DEST_2
2019-04-23T04:07:58.682079+02:00
Archived Log entry 1382 added for T-1.S-866 ID 0x49172f7c LAD:1
2019-04-23T04:57:18.076346+02:00
Incremental restore complete of datafile 4 to datafile copy +FRA/FWEB1/DATAFILE/users.553.999450151
checkpoint is 213563471108
last deallocation scn is 202352337426
Incremental restore complete of datafile 3 to datafile copy +FRA/FWEB1/DATAFILE/undotbs1.552.999450151
checkpoint is 213563471108
last deallocation scn is 213563368804
Incremental restore complete of datafile 1 to datafile copy +FRA/FWEB1/DATAFILE/system.269.999450149
checkpoint is 213563471108
last deallocation scn is 201963629285
Incremental restore complete of datafile 2 to datafile copy +FRA/FWEB1/DATAFILE/sysaux.432.999450147
checkpoint is 213563471108
last deallocation scn is 213563209794
How it is being parsed in "Add Data" preview on the HF:
How it is being parsed when "live" (host, source and sourcetype have been anonymized):
As you can see the last log message is not being parsed correctly according to the props.conf specifications. Any ideas on how to fix this so that the log is onboarded correctly?
PS: I know I could probably fix this by installing the Splunk TA for Oracle, but in this case that is not an option.
How are you ingesting this data? Through a network input or reading a file? As @whrg mentions, it could be that the line that is shown separately comes in with a small delay, causing splunk to fail in including it in the previous multiline event. Depending on the ingest method, there might be some tricks to resolve that.
This is ingested via reading a file, so there shouldn't be any delay problems =/
I don't know why your live environment is showing different results. Perhaps parts of the events arrive with a delay and thus form a new event.
Instead of modifying LINE_BREAKER and disabling line merging, you can also leave LINE_BREAKER as it is (by default every newline) and set SHOULD_LINEMERGE=true with BREAK_ONLY_BEFORE_DATE=true:
[oracle]
category = Custom
pulldown_type = true
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z
(I left out SHOULD_LINEMERGE=true and BREAK_ONLY_BEFORE_DATE=true because these are the default settings.)
You can ignore the orange warnings, since not all lines contain a timestamp. The lines which do not contain a timestamp are merged.
I also noticed my TIME_FORMAT is slightly different than yours.
Check out https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables
Hi, and thanks for your response! I tried your variant of the props.conf configuration, but with little success. So unfortunately I am stuck with the same problem.