Getting Data In

How to set a large log to ingest as one single event?

joesrepsolc
Communicator

Been working on this for a week... hence my question now. I have a log that can be anywhere between 3,000 lines or 20,000_ lines. It's an output of a script that takes around 1 minute to complete. It rights to the log as the script progresses, and I want that entire log (start to finish) in a single event into Splunk.

You can see that it's ingesting that logs as multiple events (1st event at the bottom, last event at the top). Tried many combinations of props.conf (see below for current) as well as "interval" in my inputs.conf (hence removed).

Known log formatting:
Always starts with "Job ammolock submitted by teamA123 starting"
Always ends with "+ exit" as the last line of the log file.

6:03:33.000 AM      + ls -l j3483458_*
            ... 
            ...
            + exit
            Show all 381 lines

6:03:31.000 AM      + cd /aa/6/prod/something/custom/spool
            ... 
            ...
            Show all 27 lines

6:03:29.000 AM      + export RETURN_CODE=0
            ... 
            ...
            Show all 357 lines

6:03:23.000 AM      **************************************************
            TOTAL RECORDS READ FROM TABLE = 000120882
            **************************************************
            ... 
            ...
            Show all 60 lines

6:02:45.000 AM      Job ammolock submitted by teamA123 starting 2019/12/19 06:002:45 AM
            ... 
            ...
            Show all 2269 lines

INPUTS.CONF
[monitor:///blah/blah/blah/blah.log]
index = blah
sourcetype = blah:logs
disabled = 0

PROPS.CONF
[blah:logs]
MAX_EVENTS=40000
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]++\sexit)
TIME_PREFIX=Job\s\w+\ssubmitted\sby\s\w+\sstarting\s
MAX_TIMESTAMP_LOOKAHEAD = 22
TRUNCATE=99999999

What am I missing folks?

Joe

0 Karma

bandit
Motivator

Check out these other answers. Most solutions involve a BREAK_ONLY_BEFORE or LINE_BREAKER rule that will never match vs the default carriage return/newline ([\r\n]+) and would expect that the data is written all at once vs having pauses in time between the records being written as I believe a Splunk monitor will only wait 3 seconds by default before breaking up an event (time_before_close parameter) once it reaches the end of a file.

Answers:
https://answers.splunk.com/answers/5426/entire-file-contents-as-a-single-event.html
https://answers.splunk.com/answers/106075/each-file-as-one-single-splunk-event.html

time_before_close parameter and multiline_event_extra_waittime parameter

https://answers.splunk.com/answers/523368/inputsconf-wait-time-t-monitor-file.html

0 Karma

adonio
Ultra Champion

try scripted inputs, or add oneshot attached to a script of your own

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...