Getting Data In

Why are my props.conf configurations not merging lines into one event in Splunk 5?

secuc2r83
Path Finder

Hello,

I have a problem with merging events: I search in this forum's posts and documentation and tried a lot of combinations, but never worked!

My config: Test environment = Splunk v5 on a single machine (indexer and search head are on the same machine)

My log:
alt text

My problem: Splunk treats each line as an event
What I want: Event just for line which have "...New SMTP ICID...", so with the 6 lines above, I should have 2 events
alt text

PROPS.CONF tested:

Tried with different LINE_BREAKER (All tested on regex101 with success)

    [iron_log]
    TIME_PREFIX = ^<\d\d>
    TIME_FORMAT = %b %d %H:%M:%S
    MAX_TIMESTAMP_LOOKAHEAD = 15
    LINE_BREAKER = ^.*New SMTP ICID.*
    #LINE_BREAKER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    #LINE_BREAKER = ([\r\n]+)<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    SHOULD_LINEMERGE = false
    TRUNCATE = 999999 

Try with SHOULD_LINEMERGE and MUST_BREAK_AFTER

    [iron_log]
    MUST_BREAK_AFTER = ^.*New
    #MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    BREAK_ONLY_BEFORE_DATE = false
    MAX_TIMESTAMP_LOOKAHEAD = 15
    TIME_PREFIX = ^<\d\d>

Tried without SHOULD_LINEMERGE

    [iron_log]
    SHOULD_LINEMERGE = false
    MUST_BREAK_AFTER = ^.*New
    #MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
    BREAK_ONLY_BEFORE_DATE = false
    MAX_TIMESTAMP_LOOKAHEAD = 15
    TIME_PREFIX = ^<\d\d>

I always have one event per line. (I restarted Splunk for each modification and there is no error during the boot check)

If someone has an idea, it would be great!

Thanks by advance

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.

The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID.

View solution in original post

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.

The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID.

0 Karma

secuc2r83
Path Finder

Hi Martin,
The problem is solved by your last sentence.
I was sure that when you restart, it always reindex....
(Then all my props's change never be applied because restart splunk didn't reindex!)
I just change my log file and launch a search to see the same result as yours.

Thanks a lot for your time Martin
Regards

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

It'd be terrible if Splunk reindexed Petabytes of data every restart for some of the larger installs...

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Setting only BREAK_ONLY_BEFORE = New SMTP ICID seems to work fine for me:

alt text

Make sure you configure this on the correct machine (any heavy forwarder involved?), there's no other configuration involved (props.conf on the source::, host::?), and you actually reindex that data after the restart.

secuc2r83
Path Finder

Hi Martin,

Thanks for your help.
- For SplunkV6.3 => I understand but for now we can't improve to V6 then i have to find a solution with V5
- I did a basic config and try your solution but the same (6lines, 6 events):

indexes.conf
[iron_idx]
homePath = $SPLUNK_DB/iron_idx/db
coldPath = $SPLUNK_DB/iron_idx/colddb
thawedPath = $SPLUNK_DB/iron_idx/thaweddb

inputs.conf
[monitor:///var/log/iron.log]
index=iron_idx
sourcetype=iron_log

props.conf (I try several solutions but still the same...)
[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = New SMTP ICID

[iron_log]
BREAK_ONLY_BEFORE = ^.+New

[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ^.+New
BREAK_ONLY_BEFORE_DATE = false

If you have other idea you're welcome
thks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...