Splunk Search

Possible to retrofit props.conf file

jwalzerpitt
Influencer

I have a situation in which Cisco Sourcefire files are being ingested into Splunk (v6.0.1) under different sourcetypes into one index (sourcefire), and whose fields are being extracted at search time.

Here's what the props.conf file looks like for one of the sourcetypes:

[sourcefire_si]
EXTRACT-Access_Policy = [^]\n]](?P<Access_Policy>[^ ]+)
EXTRACT-Access_Policy-Connection_Type = [^]\n]
](?P<Access_Policy>[^ ]+)[^:\n]:\s+(?P<Connection_Type>[^,]+)
EXTRACT-User = (?i) URL: (?P<User>[^,]+)
EXTRACT-Client = (?i) URL: (?P<Client>[^,]+)
EXTRACT-Application_Protocol = (?i) URL: (?P<Application_Protocol>[^,]+)
EXTRACT-Web_App = (?i) URL: (?P<Web_App>[^,]+)
EXTRACT-Access_Control_Rule_Name = (?i) Name: (?P<Access_Control_Rule_Name>[^,]+)
EXTRACT-Access_Control_Rule_Action = (?i) Action: (?P<Access_Control_Rule_Action>[^,]+)
EXTRACT-Access_Control_Rule_Reason = (?i) Reasons: (?P<Access_Control_Rule_Reason>[^,]+)
EXTRACT-URL_Category = (?i) URL: (?P<URL_Category>[^,]+)
EXTRACT-URL_Reputation = (?i) Reputation: (?P<URL_Reputation>[^,]+)
EXTRACT-URL = (?i) URL: (?P<URL>[^,]+)
EXTRACT-Interface_Ingress = (?i) Ingress: (?P<Interface_Ingress>[^,]+)
EXTRACT-Interface_Egress = (?i) Egress: (?P<Interface_Egress>[^,]+)
EXTRACT-Security_Zone_Ingress = (?i) URL: (?P<Security_Zone_Ingress>[^,]+)
EXTRACT-Security_Zone_Egress = (?i) .
?: (?P<Security_Zone_Egress>\w+/\w+)(?=,)
EXTRACT-SI_Matching_IP = (?i) IP: (?P<SI_Matching_IP>[^,]+)
EXTRACT-SI_Category = (?i) Name: (?P<SI_Category>[^,]+)
EXTRACT-srcip = (?i){.?} (?P<srcip>\d+.\d+.\d+.\d+)(?=:)
EXTRACT-src_port = (?i)^(?:[^.]
.){6}\d+:(?P<src_port>[^ ]+)
EXTRACT-dst_ip = (?i)^[^>]>\s+(?P<dst_ip>[^:]+)
EXTRACT-dst_port = (?i)^(?:[^.]
.){9}\d+:(?P<dst_port>.+)

Reviewing the Sourcefire log files I see they are delimited via a comma with headers, some with key/values. Here is a sample log file:

2015-07-25T15:28:41-04:00
[hostip of sourcefire] SFIMS:
[Primary Detection Engine (d9fd69ee-b1c9-11e4-ade9-c9e0fcb0c479)]
[PolicyName]
Connection Type: Start,
User: Unknown,
Client: Unknown,
Application Protocol: Unknown,
Web App: Unknown,
Access Control Rule Name: Malware,
Access Control Rule Action: Block,
Access Control Rule Reasons: IP Block,
URL Category: Unknown,
URL Reputation: Risk unknown,
URL: Unknown,
Interface Ingress: [NAME-NAME]/[NAME-NAME],
Interface Egress: [NAME-NAME]/[NAME-NAME],
Security Zone Ingress: Unknown,
Security Zone Egress: N/A,
Security Intelligence Matching IP: Source,
Security Intelligence Category: Malware,
Client Version: (null),
Number of File Events: 0,
Number of IPS Events: 0,
TCP Flags: 0x0,
NetBIOS Domain: (null),
Initiator Packets: 1,
Responder Packets: 0,
Initiator Bytes: 66,
Responder Bytes: 0,
Context: [NAME-NAME],
SSL Rule Name: N/A,
SSL Flow Status: N/A,
SSL Cipher Suite: N/A,
SSL Certificate: 0000000000000000000000000000000000000000,
SSL Subject CN: N/A,
SSL Subject Country: N/A,
SSL Subject OU: N/A,
SSL Subject Org: N/A,
SSL Issuer CN: N/A,
SSL Issuer Country: N/A,
SSL Issuer OU: N/A,
SSL Issuer Org: N/A,
SSL Valid Start Date: N/A,
SSL Valid End Date: N/A,
SSL Version: N/A,
SSL Server Certificate Status: N/A,
SSL Actual Action: N/A,
SSL Expected Action: N/A,
SSL Server Name: (null),
SSL URL Category: N/A,
SSL Session ID: 0000000000000000000000000000000000000000000000000000000000000000,
SSL Ticket Id: 0000000000000000000000000000000000000000,
{TCP} x.x.x.x:51645 -> x.x.x.x:443

Is there a way I can retrofit the props.conf file (and I'm confused as I thought I would need to configure data transformations in transforms.conf) to extract fields based on the conf file during search time?

Looking at the transforms.conf examples I see this:

Extract comma-delimited values into fields:

[extract_csv]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

I would appreciate any help for this - thx

0 Karma

acharlieh
Influencer

From your example result... I'm guessing that

... | extract kvdelim=":" pairdelim="," 

does what you want on a search. Now if I'm reading the transforms doc correctly, then what you would want on your search head is:

props.conf:

[sourcefire_si]
REPORT-kv = sourcefire_kv 

transforms.conf:

[sourcefire_kv]
DELIMS = ",", ":"

This should get you most of the way there but there may be other fiddling that you'd want to do as well.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

"Is there a way I can retrofit the props.conf file to extract fields based on the conf file during search time?"

That's exactly what those EXTRACT-foo lines in props.conf are supposed to do, so I'm not sure what the question is...?

jwalzerpitt
Influencer

I was under the assumption I could enter the following:

[sourcefire]
DELIMS = ","
FIELDS = "Connection Type", "User", "Client" - etc

and the fields will automatically be extracted without the need for regex?

Thx

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...