Getting Data In

How to merge messages into a single event using one timestamp and remove the other timestamps?

_smp_
Builder

Hi,

Relative newbie here. I have a host that is splitting large messages into multiple syslog messages. The beginning of the message contains two timestamps, but the trailing messages only contain one. The messages arrive like this:

May  5 08:12:16 myhost.company.com 05/05/16 08:12:16.392 [part1]
May  5 08:12:16 myhost.company.com [part2]
May  5 08:12:16 myhost.company.com [part3]

Miraculously, I somehow managed to get the messages combined into a single by editing the props.conf:

NO_BINARY_CHECK = true
MAX_EVENTS=50000
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_PREFIX = ^
TRUNCATE = 0
LINE_BREAKER = ([\r\n]+)\w+\s+\d+\s+\d+:\d+:\d+\s+\w+\.\w+\.\w+\.\w+\s\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+

And this seems to work fine, but what I noticed is that the timestamps from the two trailing messages are also included in the combined event, kind of like this:

May  5 08:12:16 myhost.company.com 05/05/16 08:12:16.392 [part1]May  5 08:12:16 myhost.company.com [part2]May  5 08:12:16 myhost.company.com [part3]

Is there a way to break the events based on the second timestamp in the first message, but strip out the timestamps from the following two messages?

0 Karma

woodcock
Esteemed Legend

You should be able to use this:

SEDCMD-strip_last_two_timestamps = s/\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+//g

Put this on your indexers and restart Splunk and it should strip them out.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...