Getting Data In

Lines Must break ONLY when they see regex -----#-----

desi-indian
Path Finder

What I am trying to achive is to ignore any carriage returns , new lines ,special symbols and Break the log into lines/ events when It sees regex "-----#-----" . I tried the g on following entries in my props.conf on full forwarder ..I am running splunk 4.2.2 on redhat .

Props.conf entries that I tried are :

[work_request]

TZ = US/Eastern

TIME_FORMAT = %Y-%m-%d %H:%M:%S

MAX_EVENTS=1


SHOULD_LINEMERGE = false


LINE_BREAKER = (?m)([-]{5,}#[-]{5,})

LINE_BREAKER = (\-----#-----*)
LINE_BREAKER_LOOKBEHIND = 1000

#BREAK_ONLY_AFTER = -----#-----*
#BREAK_ONLY_AFTER = ([\r]*[-----#-----]{1}[\r]*)
#BREAK_ONLY_AFTER = ([\r\n]+[-----#-----]{1}[\r\n]+)**

sample log :

1 Not Yet Requested ROCC Phone [code]Dear Ops,



xxxx would like to set registry lock on about 550 domains, they are asking how long it will take to proceed this.
the customer had asked this twice now, could you please investigate?


Thank you.
[/code] Normal 0 1 WREQ0002372 yyyyyy 0 Bulk update for 550 domain name Open Work Request sfdc 2011-10-14 05:10:14 global 00d676a30a0a3c4e01fcd527a37bbca9 4 kaddada 2011-10-14 14:27:01 ROCC 2011-10-14 14:27:01 2011-10-14 05:10:16 CORE SRS Verification T-00002893: Bulk update for 550 domain name Request for Information Low 1 2011-10-14 14:27:00 Proceed to Next Task Cancel all future Tasks -----#-----
1 Not Yet Requested ROCC Phone [code]





Dear OPS,



Please find the attached file which has list of domanis, which need to be put on Registry Lock.|

Below are Registrar Details


Registrar Name:| xxxxxxxx INTERNET NAMES WORLDWIDE

GURID: 22

NCC ID: 33

Please note : Registrar is requesting this to be done ASAP.


Regards

asvvvvr

[/code] Normal 0 1 WREQ0002373 Nitin Asher 0 Registry Lock Open Work Request sfdc 2011-10-14 06:17:33 global 0114162f0a0a3c4e01124621ff497c0c 4 kaddada 2011-10-14 14:26:35 ROCC 2011-10-14 14:26:35 2011-10-14 06:17:35 CORE SRS Verification T-00002894: Registry Lock Request for Information Low 1 2011-10-14 14:26:34 Proceed to Next Task Cancel all future Tasks -----#-----
1 Not Yet Requested SYMC Colo User CORP.MTV1 Phone Details: Hi^M
We are changing the IP address for authconnect-mtv.verisign.net to 216.168.241.251. The details as follows^M ^M

0 Karma
1 Solution

desi-indian
Path Finder

Ayn ,
Thanks for the answer but line merging is not a option . The reason I have line merging off is, its a very high volume system where lots of customers data come at the same milli second , So If I do not set it to false, splunk groups the whole event which will have data from multi users..So even when querying for a specific event from a client you will see others data in the event

      The "-----#-----" does not always stay on its ownline 

BUT
I am able to get this work ..I got most of the props.conf but missed a (.) as pointed by support(Yann) which did the trick ..My final working props.conf is as follows

TZ = US/Eastern
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_EVENTS=1
SHOULD_LINEMERGE = false
LINE_BREAKER = (?m)([-]{5,}#[-]{5,})
LINE_BREAKER_LOOKBEHIND = 1000
BREAK_ONLY_AFTER = (.)-----#-----

View solution in original post

0 Karma

desi-indian
Path Finder

Ayn ,
Thanks for the answer but line merging is not a option . The reason I have line merging off is, its a very high volume system where lots of customers data come at the same milli second , So If I do not set it to false, splunk groups the whole event which will have data from multi users..So even when querying for a specific event from a client you will see others data in the event

      The "-----#-----" does not always stay on its ownline 

BUT
I am able to get this work ..I got most of the props.conf but missed a (.) as pointed by support(Yann) which did the trick ..My final working props.conf is as follows

TZ = US/Eastern
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_EVENTS=1
SHOULD_LINEMERGE = false
LINE_BREAKER = (?m)([-]{5,}#[-]{5,})
LINE_BREAKER_LOOKBEHIND = 1000
BREAK_ONLY_AFTER = (.)-----#-----
0 Karma

Ayn
Legend

Is there a specific reason for not wanting to use line merging? I find that messing with LINE_BREAKER often gets unnecessarily complex compared to using line merging settings to define when to create a new event. The sample log events you pasted seem to have been a bit messed up in the paste, so it's hard to know whether that is all on one line or if it's separate lines - is the -----#----- on its own line? If it isn't you should consider turning line merging on and use MUST_BREAK_AFTER. I see you've been trying something similar previously, however the directive BREAK_ONLY_AFTER does not exist.

0 Karma

_d_
Splunk Employee
Splunk Employee

See if this works:

[work_request]
TZ = your_own
TIME_FORMAT= your_own
LINE_BREAKER = (\-{5}\#\-{5})
SHOULD_LINEMERGE = false

  • please upvote if you find this answer useful
0 Karma

desi-indian
Path Finder

Thanks for the help but It did not work ..still not breaking the lines properly

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...