Splunk Search

How to extract fields delimited by semicolons and the last field with variable values from my sample data?

smudge797
Path Finder

Using Splunk Enterpise 6.2, I'm trying to get the fields extracted using search-time props.conf / transforms.conf and the issue is a variable value in the last field. Can anyone tell me what the settings would be ensure these values are taken care of? The message that logs are delimited by semicolon, but unfortunately, the different messages have different amount of fields and almost constant in size 40 characters except the last field that is variable, but also not more than 300 characters.

Is there a way to take care of this situation, at search-time?

Log sample:

[1445004534] SERVICE ALERT: madmsc03j1;Status_Host_ID-0104;OK;HARD;2;PING OK - Packet loss = 0%, RTA = 47.89 ms
[1445004568] SERVICE ALERT: de2secimpmx001;Resource_System_Load_ID-0108;OK;HARD;1;4 CPU, average load 70.8% < 75% : OK
[1445004576] SERVICE ALERT: pt1secimpmx001;Resource_System_Load_ID-0108;CRITICAL;HARD;1;2 CPU, average load 91.0% > 90% : CRITICAL
[1445004576] SERVICE FLAPPING ALERT: pt1secimpmx001;Resource_System_Load_ID-0108;STARTED; Service appears to have started flapping (12.4% change >= 10.0% threshold)
[1445004586] SERVICE ALERT: it1sec02mibi-002;Resource_Memory_ID-0011;OK;SOFT;2;OK - Physical memory: 98.0% used (7.5GiB/7.7GiB)
[1445004592] SERVICE ALERT: it1sec02mibi-002;Status_Interfaces_ID-0017;OK;SOFT;2;OK - No valid historical dataset... <a href="https://mydomain.com/interfacetable_v3t/tables/it1sec02mibiQ2D002--Interfacetable.html" target="_self">[details]</a>
[1445004634] SERVICE FLAPPING ALERT: it1sec04mibi-001;Resource_System_Load_ID-0109;STOPPED; Service appears to have stopped flapping (0.0% change < 1.0% threshold)
[1445004673] SERVICE ALERT: nl2secsou3d001;Status_Process_SS_Min2_ID-0101;UNKNOWN;SOFT;1;ERROR: Alarm signal (Nagios time-out)

Currently I have props.conf:

TIME_FORMAT=%s
TIME_PREFIX=\[
TZ=GMT
MAX_TIMESTAMP_LOOKAHEAD=10
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)
TRUNCATE=10000

Cheers!

0 Karma

santiagoaloi
Path Finder

So! I've done most of the field extractions. You cannot combine delimited data extractions with regex, so is either one or the other.

Regex is a little tricky to tune the way you want it to work, the regex based field extractions in Splunk not always works as we expect, so most of the times www.regex101.com becomes your best friend.

You will have to change the names of the fields because since I don't know where this dataset comes from I couldn't guess much 🙂

EXTRACT-description = \s-|\s(?[^;]+$) is the one you needed help with.

props.conf

TIME_FORMAT=%s
TIME_PREFIX=\[
TZ=GMT
MAX_TIMESTAMP_LOOKAHEAD=10
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)
TRUNCATE=10000

[yoursourcetype]
EXTRACT-service_type    = ^\[\d+\]\sSERVICE\s(?<service_type>\w+)
EXTRACT-service         = ^[^:\n]*:\s+(?P<service>\w+)
EXTRACT-objet_id        = _ID-(?<objet_id>\d+)
EXTRACT-status          = ^(?:[^;\n]*;){2}(?P<status>\w+)
EXTRACT-description     = \s\-|\s(?<description>[^;]+$)
EXTRACT-layer           = ^(?:[^;\n]*;){3}(?P<layer>\w+)
EXTRACT-other_status    = ^\[\d+\]\s+\w+\s+\w+:\s+\w+\d+\w+\d+\w+\d+;\w+_\w+_\w+_\w+\d+_\w+\-\d+;\w+;\w+;\d+;(?P<other_status>\w+)
0 Karma

smudge797
Path Finder

So this is all in props.conf at search time on the search head, correct? I will try it today.

Thanks!

0 Karma

santiagoaloi
Path Finder

Hi!

I need to understand what is that you want to achieve. If I understood you correctly , you want to extract all the key fields that are separated by the semicolon, and then the values at the end of the event such as "4 CPU, average load 70.8% < 75% : OK".
I will help you to do this, although I need to understand better what you need 🙂

0 Karma

smudge797
Path Finder

Yes Im trying to extract the key fields separated by ; and the values. But the problem is the last field is variable value but no more than 300 characters. Hope that helps clarify 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...