Getting Data In

hot to merge multiple lines into a single event?

0xlc
Path Finder

hi guys,

i am trying to merge these lines into a event

so far i tried

[cycledata]
EVENT_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

i got block of lines starting with CycleDataTask started and finishing with CycleDataTask finished and i want to group them into a single event for each started finished.

and MUST BREAK AFTER same regex

these is an example:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS:-> =
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

thank you

0 Karma

sanjeev543
Communicator

I took below sample data and verified the config that I mentioned

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-10 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-08 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - CycleDataTask finished

Config I have tried

LINE_BREAKER = CycleDataTask\sfinished([\r\n]*)
 MUST_BREAK_AFTER  = CycleDataTask\sfinished
 SHOULD_LINEMERGE = true
 TIME_PREFIX = ^
 MAX_TIMESTAMP_LOOKAHEAD = 26
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N

This is parsing as you are expecting, hopefully this helps

0 Karma

0xlc
Path Finder

it's not working for me, as i said below, i believe there is something wrong going on with my cluster and config files.

as soon i find a solution and this work i'll mark it as working

thank you

this is how it looks like

link text

0 Karma

0xlc
Path Finder

i think there is something going on with my cluster, if i upload a txt sample, and i add the regex (finished) it merges it almost fine, but then, when i add to props.conf does not work at all.

0 Karma

koshyk
Super Champion

For sample data

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished

Please find solution

[cycledata]
LINE_BREAKER = CycleDataTask finished([\r\n]+)
SHOULD_LINEMERGE = false

cheers

0 Karma

0xlc
Path Finder

this does not work

0 Karma

koshyk
Super Champion

eh? Can you please verify if the sample data is like above?
Your example had all lines merged already, so no settings required. I have split that into individual lines

Please put your sample data again in a formatted way as it exactly occurs in your file (not in Splunk)

0 Karma

0xlc
Path Finder

check now please

reading again my message i was not very clear, i edited it.

0 Karma

PowerPacked
Builder

Hi

Just check if you gave (EVENT_BREAKER) instead of LINE_BREAKER attribute

[cycledata]
LINE_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

the above should work

Thanks

0 Karma

0xlc
Path Finder

no it does not work

i almost find a solution adding the example in data file but now it cuts off the word 'finished' which i use as regex. and of course i need it in.

(finished)

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is the example a single event you want to break into multiple events or multiple events you want to make into a single event?

---
If this reply helps you, Karma would be appreciated.
0 Karma

0xlc
Path Finder

the example has been edited, and like that seems just one long line, instead i have multiple lines which i want to merge in a single event

it should start here:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________

and finish here:

2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...