Splunk Dev

Someone please help me translating Regular expression for Field extractions.

superhm
Explorer

Splunk collecting security solusion logs that is DefensPro.

I want to extract fields.

Log samples are..

Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies "TTL Less Than or Equal to 1" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A medium forward

Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies "Invalid TCP Flags" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular "Packet Anomalies" sampled 1 11 N/A 0 N/A low drop

Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies "L4 Source or Dest Port Zero" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A low drop
Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies "Invalid IP Header or Total Length" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A low drop

And my select fields that
alt text

In regular order
1. Date : 26-12-2017 16:20:58
2. Severity : Warning
3. Category : Anomalies
4. AttackName : "TTL Less Than or Equal to 1"
5. Protocol : UDP
6. SrcIP : 59.150.19.252
7. DstIP : 224.0.0.2
8. DstPort : 1985
9. PolicyName : "Packet Anomalies"
10. AttackStatus : sampled
11. Risk : medium
12. Action : forward

Automatic log extraction by Splunk is invalid.
Can you create a regular expression to extract fields from the log above?

Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

HI @superhm,

Please use below regex

DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^"]*"[^"]*")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^"]*"[^"]*")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)

Based on above regex I have created sample query with dummy data you have provided (First 4 lines are used to generate dummy data)

|makeresults | eval _raw="Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies \"TTL Less Than or Equal to 1\" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A medium forward"
|append [makeresults | eval _raw="Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies \"Invalid TCP Flags\" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular \"Packet Anomalies\" sampled 1 11 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies \"L4 Source or Dest Port Zero\" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies \"Invalid IP Header or Total Length\" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
| rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

So your query will be

<yourBasesearch> | rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

I hope this helps.

Thanks,
Harshil

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

HI @superhm,

Please use below regex

DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^"]*"[^"]*")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^"]*"[^"]*")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)

Based on above regex I have created sample query with dummy data you have provided (First 4 lines are used to generate dummy data)

|makeresults | eval _raw="Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies \"TTL Less Than or Equal to 1\" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A medium forward"
|append [makeresults | eval _raw="Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies \"Invalid TCP Flags\" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular \"Packet Anomalies\" sampled 1 11 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies \"L4 Source or Dest Port Zero\" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies \"Invalid IP Header or Total Length\" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
| rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

So your query will be

<yourBasesearch> | rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

I hope this helps.

Thanks,
Harshil

superhm
Explorer

Thank you so much harsmarvania57!
You were a great help to me.
I will study more by consulting carefully. : )

0 Karma

niketn
Legend

@harsmarvania57, you beat me to it. Almost same regex. I have extracted src port and dest port both and kept provision for some other capturing groups to be extracted. Following is the run anywhere search:

| makeresults
| eval data="Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies \"TTL Less Than or Equal to 1\" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A medium forward;Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies \"Invalid TCP Flags\" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular \"Packet Anomalies\" sampled 1 11 N/A 0 N/A low drop;Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies \"L4 Source or Dest Port Zero\" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop;Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies \"Invalid IP Header or Total Length\" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"
| makemv data delim=";" 
| mvexpand data
| rename data as _raw
| rex "DefensePro: (?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]+)\s\d+\s(?<Category>[^\s]+)\s\"(?<AttackName>[^\"]+)\"\s(?<Protocol>[^\s]+)\s(?<SrcIP>[^\s]+)\s(?<SrcPort>[^\s]+)\s(?<DstIP>[^\s]+)\s(?<DstPort>[^\s]+)\s\d+\s([^\s]+)\s\"(?<PolicyName>[^\"]+)\"\s(?<AttackStatus>[^\s]+)\s\d+\s\d+\s([^\s]+)\s\d+\s([^\s]+)\s(?<Risk>[^\s]+)\s(?<Action>.*)"
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

superhm
Explorer

Thank you for your opinion. : )
And it helped me a lot.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I thought to extract SrcPort but in original requirement it was not given so I skipped it. 😛

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...