Field Extractions not working with Splunk_TA_symantec_ep 2.3 and some weirdness

[abeeber_3]

Hi Folks,

I'm using SEP 14+ or latest version, the Splunk supported TA version 2.3 and SEP dump/tmp files.

Field extraction were not working right out of the box. After reviewing the actual files themselves, it turns out that they are "," delimited with a header row.
Event Time,Severity,Host Name,SHA-256,MD-5,Event Description,Local Host IP,Local Host MAC,Remote Host Name,Remote Host IP,Remote Host MAC,Traffic Direction,Network Protocol,Hack Type,Begin Time,End Time,Occurrences,Application Name,Location,User Name,Domain Name

If tune each sourcetype for each actual tmp like the following:
[symantec:ep:behavior:file]
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX =
TIME_FORMAT = %Y-%m-%d %H:%M:%S
INDEXED_EXTRACTIONS = csv
FIELD_DELIMITER = ,
HEADER_FIELD_LINE_NUMBER = 1
TIMESTAMP_FIELDS = field1

For some field extractions like Traffic Direction, I get the correct parsing. But I also have field extractions coming from the App re: Traffic_Direction which is pulling in something completely different.

Does anyone have a handle on this log source so the field extractions are consistent?

[ravinder1k]

Parsing done for [symantec:ep:risk:file]

\,([^\,]+)\,IP Address:\s([^\,]+)\,Computer name:\s([^\,]+)\,Source:\s([^\,]+)\,Risk name:\s([^\,]+)\,Occurrences:([^\,]+)\,([^\,]+)\,\,Actual action:\s([^\,]+)\,Requested action:\s([^\,]+)\,Secondary action:\s([^\,]+)\,Event time:\s([^\,]+)\,Inserted:\s([^\,]+)\,End:\s([^\,]+)\,Last update time:\s([^\,]+)\,Domain:\s([^\,]+)\,Group:\s([^\,]+)\,Server:\s([^\,]+)\,User:\s([^\,]+)\,Source computer:([^\,]+)\,Source IP:([^\,]+)\,Disposition:\s([^\,]+)\,Download site:\s([^\,]+)\,Web domain:\s([^\,]+)\,Downloaded by:\s([^\,]+)\,Prevalence:\s([^\,]+)\,Confidence:\s([^\,]+)\,URL Tracking Status:\s([^\,]+)\,First Seen:\s([^\,]+)\,Sensitivity:([^\,]+)\,Permitted application reason:\s([^\,]+)\,Application hash:\s([^\,]+)\,Hash type:\s([^\,]+)\,Company name:\s([^\,]+)\,Application name:\s([^\,]+)\,Application version:\s([^\,]+)\,Application type:\s([^\,]+)\,File size (bytes):\s([^\,]+)\,Category set:\s([^\,]+)\,Category type:([^\,]+)\,Location:\s([^\,]+)\,Intensive Protection Level:\s([^\,]+)\,Certificate issuer:\s([^\,]+)\,Certificate signer:\s([^\,]+)\,Certificate thumbprint:\s([^\,]+)\,Signing timestamp:\s([^\,]+)\,Certificate serial number:\s([^\,]+)

[abeeber_3]

For the agt_traffic sourcetype, the issue is that the regex was missing an end of string ($)

Here is the regex without the modular regex. Note User and Rule where not set correctly.
(?i)
(?:\s*'[^']'|\s"[^"]"|\s[^,]),
\s(?[^,']'[^']'|[^,"]"[^"]"|[^,]),
\s(?[^,']'[^']'|[^,"]"[^"]"|[^,]),
\s(?:SHA-256:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:MD-5:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Local:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Local:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Local:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Remote:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Remote:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Remote:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Remote:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?[^,']'[^']'|[^,"]"[^"]"|[^,]),
\s(?[^,']'[^']'|[^,"]"[^"]"|[^,]),
\s(?:Begin:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:End:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Occurrences:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Application:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Rule:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Location:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:User:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Domain:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),
\s(?:Action:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,]*))$

[muralikoppula]

We recently started ingesting Sep14 logs into Splunk and deployed add-on(Splunk Add-on for Symantec Endpoint Protection) directly to indexers, search heads and UFs as well and our logs are "," delimited with a header row. The fields are being extracted perfectly and so far we don't see any issues. Here is the props conf for symantec:ep:behavior:file , just for your reference .

Props.conf
[symantec:ep:behavior:file]
TRANSFORMS-nullqueueheader = sep_file_header
KV_MODE = none
pulldown_type = true
category = Network & Security
description = Symantec Endpoint Protection agent behavior events
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = false
REPORT-field_extraction_for_agt_behavior = field_extraction_for_agt_behavior, process_from_caller_process_name, caller_md5_from_description
FIELDALIAS-vendor_action_SEP_behavior_vendor_action = vendor_action as SEP_behavior_vendor_action

[abeeber_3]

That didn't help. The field extractions for agt_traffic_tmp are way off.

[abeeber_3]

turns out the problem was bugs in Splunk's TA and the regex's groups for each.

For the field_extraction_for_agt_behavior; I found 4. Below is my code from regex101.
(?i)(?:\s*'[^']'|\s"[^"]"|\s[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?:Begin:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:End:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:Rule:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?[^,']'[^']'|[^,"]"[^"]"|[^,]),(?:User:\s*(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:Domain:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:Action\sType:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:\s*File\ssize\s(bytes):\s(?[^,']'[^']'|[^,"]"[^"]"|[^,])),(?:Device\sID:\s(?[^,']'[^']'|[^,"]"[^"]"|[^,]*))

[abeeber_3]

More specifics: for the Symantec:ep:traffic:file.

Any two word field extraction that has a "_" does not seem to work like Begin_Time does not work. However if the same field extraction is parsed with a space like "Begin Time"; then it does.

I'm thinking that the field extraction with the underscore is coming from the app, vs my customization above.