Hello,
I have a problem with merging events: I search in this forum's posts and documentation and tried a lot of combinations, but never worked!
My config: Test environment = Splunk v5 on a single machine (indexer and search head are on the same machine)
My log:
My problem: Splunk treats each line as an event
What I want: Event just for line which have "...New SMTP ICID...", so with the 6 lines above, I should have 2 events
PROPS.CONF tested:
Tried with different LINE_BREAKER (All tested on regex101 with success)
[iron_log]
TIME_PREFIX = ^<\d\d>
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
LINE_BREAKER = ^.*New SMTP ICID.*
#LINE_BREAKER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
#LINE_BREAKER = ([\r\n]+)<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
SHOULD_LINEMERGE = false
TRUNCATE = 999999
Try with SHOULD_LINEMERGE and MUST_BREAK_AFTER
[iron_log]
MUST_BREAK_AFTER = ^.*New
#MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
BREAK_ONLY_BEFORE_DATE = false
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_PREFIX = ^<\d\d>
Tried without SHOULD_LINEMERGE
[iron_log]
SHOULD_LINEMERGE = false
MUST_BREAK_AFTER = ^.*New
#MUST_BREAK_AFTER = ^<\d\d>.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s[^\:]+:\s[^\:]+:\sNew SMTP ICID
BREAK_ONLY_BEFORE_DATE = false
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_PREFIX = ^<\d\d>
I always have one event per line. (I restarted Splunk for each modification and there is no error during the boot check)
If someone has an idea, it would be great!
Thanks by advance
Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.
The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID
.
Unrelated to the actual question: Splunk 5 is quite old, consider upgrading to 6.3 - many great things await.
The props.conf setting you're looking for should be BREAK_ONLY_BEFORE = New SMTP ICID
.
Hi Martin,
The problem is solved by your last sentence.
I was sure that when you restart, it always reindex....
(Then all my props's change never be applied because restart splunk didn't reindex!)
I just change my log file and launch a search to see the same result as yours.
Thanks a lot for your time Martin
Regards
It'd be terrible if Splunk reindexed Petabytes of data every restart for some of the larger installs...
Setting only BREAK_ONLY_BEFORE = New SMTP ICID
seems to work fine for me:
Make sure you configure this on the correct machine (any heavy forwarder involved?), there's no other configuration involved (props.conf on the source::, host::?), and you actually reindex that data after the restart.
Hi Martin,
Thanks for your help.
- For SplunkV6.3 => I understand but for now we can't improve to V6 then i have to find a solution with V5
- I did a basic config and try your solution but the same (6lines, 6 events):
indexes.conf
[iron_idx]
homePath = $SPLUNK_DB/iron_idx/db
coldPath = $SPLUNK_DB/iron_idx/colddb
thawedPath = $SPLUNK_DB/iron_idx/thaweddb
inputs.conf
[monitor:///var/log/iron.log]
index=iron_idx
sourcetype=iron_log
props.conf (I try several solutions but still the same...)
[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = New SMTP ICID
[iron_log]
BREAK_ONLY_BEFORE = ^.+New
[iron_log]
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ^.+New
BREAK_ONLY_BEFORE_DATE = false
If you have other idea you're welcome
thks