Splunk Add-on for Symantec Endpoint Protection: How to replace REGEX with DELIMS in transforms.conf?

[jwalzerpitt]

I'm working on creating search macros and while reviewing the Symantec transforms.conf file, I see the following for one of the source types:

[field_extraction_for_traffic]
REGEX = (\s*'[^']*'|\s*"[^"]*"|\s*[^,]*)(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*
'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[
^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}
FORMAT = Enforce_Type::$2 Enforce_ID::$3 Local_IP::$4 Local_Port::$5 Remote_IP::$6 Remote_Port::$7 Network_Protocol::$8 Traffic_Direction::$9 Begin_Time::$10 End_Time::$11 Occurrences::$12

When I look at the actual log file – Symantec:ep:traffic:file – I see it’s comma delimited as follows:

Event Time,Severity,Host Name,Local Host IP,Local Port,Local Host MAC,Remote Host IP,Remote Host Name,Remote Port,Remote Host MAC,Network Protocol,Traffic Direction,Begin Time,End Time,Occurrences,Application Name,Rule Name,Location,User Name,Domain Name,Action

How would I go about replacing the REGEX in the transforms.conf file with the comma delimited information? Would I replace REGEX with DELIMS = “, ” and replace FORMAT with FIELDS = Event Time,Severity,Host Name,Local Host IP,Local Port,Local Host MAC,Remote Host IP,Remote Host Name,Remote Port,Remote Host MAC,Network Protocol,Traffic Direction,Begin Time,End Time,Occurrences,Application Name,Rule Name,Location,User Name,Domain Name,Action

I assume if I can replace the REGEX with the DELIMS and FORMAT for the field_extraction_for_traffic, I can do it for the other Symantec sources as well.

Thx,
Jeff

[woodcock]

Usually, when it looks like a simpler configuration is "just as good", it isn't. In this case, I see that the REGEX handles cases where commas occur inside the fields and makes sure that Splunk breaks the event up without mis-interpreting those commas as field delimiters. I would DEFINITELY NOT change it.

[jwalzerpitt]

Thx for the info.

With that, is there a way to edit the regex so the field is not listed as a value in the field extraction?

Separately, while reviewing the sourcetype events, I found that the Symantec:ep:traffic:file [field_extraction_for_traffic] sourcetype does not extract all fields with its current regex settings, yet if I replace it with the delims and fields values, I see all of the extracted fields. That's what originally led me to replacing the regex with delims at least for the Symantec:ep:traffic:file sourcetype. Wondering if modifying the regex for this sourcetype would both extract all fields and make sure the field names aren't extracted as values as well.

Thx

[woodcock]

Yes, use the REGEX/FORMAT construct (like originally) and not the DELIMS/FIELDS format. Take this part:

(\s*'[^']*'|\s*"[^"]*"|\s*[^,]*)

This will correctly handle a first field that looks any of these:

'This,is,the,first,field,and,has,commas',
"This,first,field,does,too",
This is normal for a first field,

[jwalzerpitt]

Thx - greatly appreciated

[sundareshr · Accepted solution]

Try this

transforms.conf
[field_extraction_for_traffic]
DELIMS = ","
FIELDS = "Event Time","Severity","Host Name".....,

props.conf
REPORT-traffic_fields= field_extraction_for_traffic

[jwalzerpitt]

Thx for the reply and recommendation as that worked as I now see all of the fields per the comma delim parsing.

I do have another question as the field name is being extracted as a value as well, such as:

Values Count
Action: Blocked 1,695
Action: Allowed 122
Action 1

How would I go about removing the field name from the extraction so I only see the values (i.e., Blocked, Allowed)?

Here's an actual log (redacted):

2015-12-07 09:17:16,Critical,,Local: 224.0.0.252,Local: 5355,Local: 01005E0000FC,Remote: ,Remote: ,Remote: 58102,Remote: 989096D843F1,UDP,Inbound,Begin: 2015-12-07 09:17:00,End: 2015-12-07 09:17:00,Occurrences: 2,Application: C:/Windows/System32/svchost.exe,Rule: Block Ipv4 LLMNR,Location: Default,User: NETWORK SERVICE,Domain: NT AUTHORITY,Action: Blocked

I'm seeing the field sometimes - Action, Application Name, Domain Name, Local Host IP, etc., but not for others, such as Host Name, Severity

Thx

[sundareshr]

You cannot use delims then. Change you transforms to

REGEX = ,(\w+):.([^,]+)?
FORMAT = $1::$2    

In addition, you will have to add the following

props.conf

REPORT-traffic_fields= field_extraction_for_traffic, field_extraction_protoanddirection

transforms.conf

[field_extraction_protoanddirection]
REGEX = (?UDP|TCP),(?Inbound|Outbound)

[jwalzerpitt]

Looking at the transforms.conf file for the app I see:

[field_extraction_for_traffic]
REGEX = (\s*'[^']*'|\s*"[^"]*"|\s*[^,]*)(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*
'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[
^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}(?:,(\s*[^,']*'[^']*'|\s*[^,"]*"[^"]*"|\s*[^,]*)){1}
FORMAT = Enforce_Type::$2 Enforce_ID::$3 Local_IP::$4 Local_Port::$5 Remote_IP::$6 Remote_Port::$7 Network_Protocol::$8 Traffic_Direction::$9 Begin_Time::$10 End_Time::$11 Occurrences::$12

Would I need to modify their REGEX somewhat to remove the field being listed as a value?

Thx