Getting Data In

Need a help on Line Breaking and Time Prefix, Time_Format on props.conf ?

Hemnaath
Motivator

Hi All, Need a help on Line Break Regex and TIME_FORMAT on props.conf, I am ingesting sonarqube logs in to splunk for the below log details with the following source type, but got stuck with the Regex part.

Ce.log details:
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:05:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:15:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:25:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:35:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Deleting any worn out task
2019.07.12 11:45:15 DEBUG ce[][o.s.c.c.CeCleaningSchedulerImpl] Resetting state of tasks with unknown worker UUIDs

Source type and stanza details:

[sonarqube:ce]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
TIME_FORMAT=%Y.%m.%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19

Access.log details:

x.x.x.x- - [11/Jul/2019:17:54:12 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc71kFTVuzYd0BsSB&p=1&ps=500 HTTP/1.1" 200 288 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACB"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc74DFTVuzYd0BsTO&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACC"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWqYLPh9Yaosnfiy-EtA&p=1&ps=500 HTTP/1.1" 200 3989 "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACD"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWkV46fXeS_Bw5qUD5wC&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACE"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8AHFTVuzYd0BsWp&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACF"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWbCc8LZFTVuzYd0Bsd5&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACG"
x.x.x.x- - [11/Jul/2019:17:54:13 +0100] "GET /api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWoCWw4hDVsFMZYP1Pzs&p=1&ps=500 HTTP/1.1" 200 - "-" "ScannerMaven/X.X.X.XXXX/3.0.5" "XXXXXXs0y7TAACH"

[sonarqube:access]
SHOULD_LINEMERGE=false
TIME_FORMAT= %d/%b/%Y:%H:%M:%S %Z
TIME_PREFIX= \d+.\d+.\d+.\d+\s+-\s+-\s+[
MAX_TIMESTAMP_LOOKAHEAD=26
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d

Need help on the above stanza's to parse the sonarqube data in splunk. Kindly guide me on this.

0 Karma

woodcock
Esteemed Legend

Try this (you had some problems with RegEx):

[sonarqube:ce]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^
TIME_FORMAT = %Y.%m.%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

[sonarqube:access]
SHOULD_LINEMERGE = false
# LINE_BREAKER = default is fine: do not change
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s*-\s+-\s+\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 26
0 Karma

gcusello
SplunkTrust
SplunkTrust

HI Hemnaath,
at first you don't need LINE_BREAKER if you have SHOULD_LINEMERGE=false because each row is an event.

About the first one, TIME_PREFIX and TIME_FORMAT are correct.

About the second one, I hint to modify TIME_PREFIX adding "^" at the beginning and escaping special chars (e.g. "." and "["), in other words:

TIME_PREFIX=^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[

Then TIME_FORMAT is correct.

The best way to check props.conf is to extract some logs in a file and follow the guided procedure for ingestion, in this way you can immediately check your settings.

Bye.
Giuseppe

0 Karma

Hemnaath
Motivator

thanks gcusello, for responding on this. but in our organisation we are following some standard recommendation when you are going to ingest a feed into splunk based on that, it is mandatory to add this stanza's. in this case my LINE_BREAKER regex and Max_time_look-ahead are correct.

[sonarqube:ce]
LINE_BREAKER = ([\r\n]+)\d+.\d+.\d+\s+\d+:\d+:\d+\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=19

[sonarqube:access]
LINE_BREAKER =([\r\n]+)\d+.\d+.\d+.\d
MAX_TIMESTAMP_LOOKAHEAD=26

Please guide me on this.

0 Karma

gcusello
SplunkTrust
SplunkTrust

HI Hemnaath,
ok, I don't understand but rules are rules!
Anyway, probably the problem are special chars to escape:

[sonarqube:ce]
LINE_BREAKER = ^\d+\.\d+\.\d+\s+\d+:\d+:\d+\s+\w+

[sonarqube:access]
TIME_PREFIX = ^\d+\.\d+\.\d+\.\d+\s+-\s+-\s+\[
LINE_BREAKER = ^\d+\.\d+\.\d+\.\d

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...