Splunk Search

After building a regex for transforms.conf and props.conf, why am I unable to see correctly extracted fields?

PCIIT
New Member

I need help.

I am unable to see the correct value after extracting a field with this regex. Why is the parser not extracting the correct value for the field we are using with the below log format :

  1. Mon Sep 24 10:40:03 2018 Info: Retrospective verdict received.

    SHA256: 3137893bc260c014974de84a
    Timestamp: 1537778403.1
    Verdict: MALICIOUS
    Reputation Score: 0
    Spyname: W32.3137893BC2-96.SBX.VIOC

    1. fields in our data model dvc_time log_level amp_verdict_type amp_sha_value _time verdict_type amp_score amp_Malware

3) wrote Regex in transforms.conf
[acc_log]

REGEX = ^\w{3}\s\w{3}\s\d{2}\s(\d{2}:\d{2}:\d{2})\s\d{4}\s(?:[^\s]+)\s{1,2}(Retrospective verdict received)\.\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^:]+):\s([^\s]+)
FORMAT =  dvc_time::$1 log_level::$2 amp_verdict_type::$3 amp_sha_value::$4 _time::$5 verdict_type::$6 amp_score::$7 amp_Malware::$8

4) props.conf
[sourcetype]
TRANSFORMS-set= setnull,setparsing
REPORT-log = acc_log

5) i am not able to see the correct value for extracting a field. My regex is not working properly.

0 Karma

FrankVl
Ultra Champion

You regex is working fine, except that you made the group for the loglevel a non-capturing group, causing that piece not to get extracted and all other extractions to therefor shift 1 position.

Just remove the ?: to make it a capturing group and it will work fine.
^\w{3}\s\w{3}\s\d{2}\s(\d{2}:\d{2}:\d{2})\s\d{4}\s([^\s]+)\s{1,2}(Retrospective verdict received).\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^:]+):\s([^\s]+)
https://regex101.com/r/7xlXzS/1

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi PCIIT,
I tried your regex in regex.101 but it seems to be correct, probably in Splunk there's a different behaviour dor multiline.
So try with:

(?ms)^\w{3}\s\w{3}\s\d{2}\s(\d{2}:\d{2}:\d{2})\s\d{4}\s(?:[^\s]+)\s{1,2}(Retrospective verdict received).\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^\,\:]+)\:\s([^\,\"]+)\n(?:[^:]+):\s([^\s]+)

If it doesn't run, try to manually ingest your log in a test index and then extract fields at search time, so you can test your regex in Splunk.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...