Getting Data In

Why is line breaking on certain events?

tsomod
Path Finder

Hi!

I am currently having some problems breaking certain events from an Oracle log correctly. The log is being onboarded via a HF, and so the props.conf file containing the sourcetype of the log is located there. This is my current props.conf configuration for the sourcetype:

[application:env:oracle:alert]
TRUNCATE = 0
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{4}-\d{2}-\d{2}[T]\d{2}:\d{2}:\d{2}.\d{6}\+\d{2}:\d{2}

A sample of the log file:

2019-04-23T02:00:00.234792+02:00
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
2019-04-23T04:07:57.219629+02:00
ALTER SYSTEM ARCHIVE LOG
2019-04-23T04:07:57.258800+02:00
Thread 1 advanced to log sequence 867 (LGWR switch)
  Current log# 3 seq# 867 mem# 0: +DATA/FWEB1/ONLINELOG/group_3.260.979816057
  Current log# 3 seq# 867 mem# 1: +FRA/FWEB1/ONLINELOG/group_3.259.979816057
2019-04-23T04:07:57.426428+02:00
TT02: Standby redo logfile selected for thread 1 sequence 867 for destination LOG_ARCHIVE_DEST_2
2019-04-23T04:07:58.682079+02:00
Archived Log entry 1382 added for T-1.S-866 ID 0x49172f7c LAD:1
2019-04-23T04:57:18.076346+02:00
Incremental restore complete of datafile 4 to datafile copy +FRA/FWEB1/DATAFILE/users.553.999450151
  checkpoint is 213563471108
  last deallocation scn is 202352337426
Incremental restore complete of datafile 3 to datafile copy +FRA/FWEB1/DATAFILE/undotbs1.552.999450151
  checkpoint is 213563471108
  last deallocation scn is 213563368804
Incremental restore complete of datafile 1 to datafile copy +FRA/FWEB1/DATAFILE/system.269.999450149
  checkpoint is 213563471108
  last deallocation scn is 201963629285
Incremental restore complete of datafile 2 to datafile copy +FRA/FWEB1/DATAFILE/sysaux.432.999450147
  checkpoint is 213563471108
  last deallocation scn is 213563209794

How it is being parsed in "Add Data" preview on the HF:

alt text

How it is being parsed when "live" (host, source and sourcetype have been anonymized):

alt text

As you can see the last log message is not being parsed correctly according to the props.conf specifications. Any ideas on how to fix this so that the log is onboarded correctly?

PS: I know I could probably fix this by installing the Splunk TA for Oracle, but in this case that is not an option.

0 Karma

FrankVl
Ultra Champion

How are you ingesting this data? Through a network input or reading a file? As @whrg mentions, it could be that the line that is shown separately comes in with a small delay, causing splunk to fail in including it in the previous multiline event. Depending on the ingest method, there might be some tricks to resolve that.

0 Karma

tsomod
Path Finder

This is ingested via reading a file, so there shouldn't be any delay problems =/

0 Karma

whrg
Motivator

I don't know why your live environment is showing different results. Perhaps parts of the events arrive with a delay and thus form a new event.

Instead of modifying LINE_BREAKER and disabling line merging, you can also leave LINE_BREAKER as it is (by default every newline) and set SHOULD_LINEMERGE=true with BREAK_ONLY_BEFORE_DATE=true:

[oracle]
category = Custom
pulldown_type = true
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z

(I left out SHOULD_LINEMERGE=true and BREAK_ONLY_BEFORE_DATE=true because these are the default settings.)

alt text

You can ignore the orange warnings, since not all lines contain a timestamp. The lines which do not contain a timestamp are merged.

I also noticed my TIME_FORMAT is slightly different than yours.
Check out https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables

0 Karma

tsomod
Path Finder

Hi, and thanks for your response! I tried your variant of the props.conf configuration, but with little success. So unfortunately I am stuck with the same problem.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...