Update of Splunk WSA Plugin to support WSA 10.x

[tmayer]

Hi,

The current plugin is supporting WSA with 9.x logs. Cisco has already released 10.0 for some time now and is coming out with 10.5 release in two weeks. Logs from 10.x Appliances are not correctly parsed because there have been some changes in the accesslogs on the WSA. Is there a timeline when the Splunk Plugin for WSA will support WSA with 10.x logs?

[bigjohnmax]

mine worked with this regex:
^(?<timestamp>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?P<field2>[^\s]+)\s+(?P<x_elapsed_time>\w+\s+\d+\s+\d+:\d+:\d+)\s(?P<field4>\w+.\w+):\s+(?P<field5>\w+):\s(?P<field6>\d+.\d+)\s(?P<field7>\d+)\s(?<src_ip>[a-zA-Z0-9:.]*)\s+(?<txn_result_code>[A-Z_]*)\/(?<status>[0-9]*)\s+(?<bytes_in>[0-9]*)\s+(?<http_method>\w*)\s+(?<url>\S*)\s+["|']?(?<user>[^\s"']+)["|']?\s+(?<server_contact_mode>[^\/]+)\/(?<dest>\S*)\s+(?<http_content_type>\S*)\s+(?<acltag>\S*)\s+(?:<|&lt;)(?<x_webcat_code_abbr>[^,]+),(?<wbrs_score>[^,]+),["|']?(?<x_webroot_scanverdict>[0-9]{0,2}|\-|\w+)["|']?,["|']?(?<webroot_threat_name>[^,"']+)["|']?,(?<x_webroot_trr>[^,]+),(?<x_webroot_spyid>[^,]+),(?<x_webroot_trace_id>[^,]+),(?<x_mcafee_scanverdict>[^,]+),["|']?(?<x_mcafee_filename>[^,]+?)["|']?,(?<x_mcafee_scan_error>[^,]+),(?<x_mcafee_detecttype>[^,]+),(?<x_mcafee_av_virustype>[^,]+),["|']?(?<x_mcafee_virus_name>[^,]+?)["|']?,(?<x_sophos_scanverdict>[^,]+),(?<x_sophos_scancode>[^,]+),["|']?(?<x_sophos_file_name>[^,]+?)["|']?,["|']?(?<x_sophos_virus_name>[^,]+?)["|']?,(?<x_ids_verdict>[^,]+),(?<x_icap_verdict>[^,]+),(?<x_webcat_req_code_abbr>[^,]+),["|']?(?<x_webcat_resp_code_abbr>[^,]+?)["|']?,["|']?(?<x_resp_dvs_threat_name>[^,]+?)["|']?,["|']?(?<x_wbrs_threat_type>[^,"']+)["|']?,["|']?(?<x_avc_app>[^,"']+)["|']?,["|']?(?<x_avc_type>[^,"']+)["|']?,["|']?(?<x_avc_behavior>[^,"']+)["|']?,["|']?(?<x_request_rewrite>[^"',]+)["|']?,(?<x_avg_bw>[^,]+),(?<x_bw_throttled>[^,]+),(?<x_user_type>[^,]+),["|']?(?<x_resp_dvs_verdictname>[^,"']+)["|']?,["|']?(?<x_req_dvs_threat_name>[^,"']+)["|']?(,["|']?(?<x_amp_verdict>[^,"']+)["|']?,["|']?(?<x_amp_malware_name>[^"']+)["|']?,(?<x_amp_score>[^,]+),(?<x_amp_upload>[^,]+),["|']?(?<x_amp_filename>[^,]+?)["|']?,["|']?(?<x_amp_sha>[^"',]+)["|']?)?(,["|']?(?<x_file_verdict>[^"',]+)["|']?)?(,(?<x_archive_scan_verdict>[^,]+),["|']?(?<x_archive_scan_verdict_reason>[^"']+)["|']?),(?P<field42>[^,]+)(?:\>)\s(?:\")?(?P<field43>[^,]+)(?:\")?$

[HomeGrownCoder]

Here is a quick edit to hold you over until the application is patched. I added update place holders in for the new fields. I am working with cisco to try and get the correct field names for these values. It does not appear to be documented. The three capture groups before the last needs to added

(?[0-9.]+)\s+(?[0-9]+)\s+(?[a-zA-Z0-9:.]*)\s+(?[A-Z_]*)/(?[0-9]*)\s+(?[0-9]*)\s+(?\w*)\s+(?\S*)\s+"?(?[^\s"]*)"?\s+(?[^/]*)/(?\S*)\s+(?\S*)\s+(?\S*)\s+(?:<|&lt;)(?[^,]+),(?[^,]+),"*(?[0-9]{0,2}|\-|\w+)"*,"(?[^"]+)",(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),"(?[^,]+)",(?[^,]+),(?[^,]+),(?[^,]+),"(?[^"]+)",(?[^,]+),(?[^,]+),"(?[^"]+)","(?[^"]+)",(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),"(?[^"]+)","(?[^,]+)","(?[^,]+)","(?[^,]+)","(?[^,]+)","(?[^,]+)",(?[^,]+),(?[^,]+),(?[^,]+),"(?[^,]+)","(?[^,]+)"(,(?[^,]+),"(?[^"]+)",(?[^,]+),(?[^,]+),"(?[^"]+)","(?[^"]+)"),(?[^"]+),(?[^"]+),"(?[^"]+)"?(?:>|&gt;)\s+(?"[^"]*"|-)?\s*.*$

[Alliant]

Any update on this? I'm not sure how or where to apply what was posted above.

Thanks

[Yorokobi]

I removed the last capture group from the scan_verdict_info_extract in transforms.conf in order to get that extraction to work properly (and the related automatic lookups to work), leaving the new(?) fields unaccounted for.

It looks like the x-result-code for WSA 10.x has three additional fields and the capture group at the end of the giant regex (?:>|>) breaks the entire extraction.

<IW_infr,3.9,0,"-",0,0,0,1,"-",-,-,-,"-",1,-,"-","-",-,-,IW_infr,-,"Unknown","-","Unknown","Unknown","-","-",288.91,0,Local,"Unknown","-",-,"-",-,-,"-","-",-,-,"-">

Our Ironports are currently running on 10.1.0-204 for Web

[knalla]

we are having the same issue, Can you please paste the stanza from transforms.conf.

[mplaksin]

Hi, we are having some problem.
Where wich transforms.conf you edited:

./etc/system/local/transforms.conf
./etc/system/default/transforms.conf
./etc/apps/Splunk_TA_cisco-wsa/default.old.20170421-151119/transforms.conf
./etc/apps/Splunk_TA_cisco-wsa/default/transforms.conf
./etc/apps/maps/default/transforms.conf
./etc/apps/search/default/transforms.conf
./etc/apps/SplunkforCiscoIronportWSA/default/transforms.conf
./etc/apps/splunk_monitoring_console/default/transforms.conf
./etc/apps/cisco_wsa_reporting/etc/system/default/transforms.conf
./etc/apps/cisco_wsa_reporting/etc/apps/cisco_cws_addon/default/transforms.conf
./etc/apps/cisco_wsa_reporting/etc/apps/cisco_wsa_reporting/default/transforms.conf
./etc/apps/cisco_wsa_reporting/etc/apps/splunk_management_console/default/transforms.conf
./etc/apps/Splunk_TA_cisco-asa/default/transforms.conf

Can you give one example before and after?

Thanks!

[Yorokobi]

$SPLUNK_HOME/etc/apps/Splunk_TA_cisco-wsa/local/transforms.conf

You may have to create the local/ directory then copy default/transforms.conf to local/ to edit. Afterwards, restart Splunk.

[Yorokobi]

[scan_verdict_info_extract]                                                                                                                                           
SOURCE_KEY=scan_verdict_info                                                                                                                                          
REGEX= (?:<|&lt;)(?<x_webcat_code_abbr>[^,]+),(?<wbrs_score>[^,]+),"*(?<x_webroot_scanverdict>[0-9]{0,2}|\-|\w+)"*,"(?<webroot_threat_name>[^"]+)",(?<x_webroot_trr>[^
,]+),(?<x_webroot_spyid>[^,]+),(?<x_webroot_trace_id>[^,]+),(?<x_mcafee_scanverdict>[^,]+),"(?<x_mcafee_filename>[^,]+)",(?<x_mcafee_scan_error>[^,]+),(?<x_mcafee_det
ecttype>[^,]+),(?<x_mcafee_av_virustype>[^,]+),"(?<x_mcafee_virus_name>[^"]+)",(?<x_sophos_scanverdict>[^,]+),(?<x_sophos_scancode>[^,]+),"(?<x_sophos_file_name>[^"]+
)","(?<x_sophos_virus_name>[^"]+)",(?<x_ids_verdict>[^,]+),(?<x_icap_verdict>[^,]+),(?<x_webcat_req_code_abbr>[^,]+),(?<x_webcat_resp_code_abbr>[^,]+),"(?<x_resp_dvs_
threat_name>[^"]+)","(?<x_wbrs_threat_type>[^,]+)","(?<x_avc_app>[^,]+)","(?<x_avc_type>[^,]+)","(?<x_avc_behavior>[^,]+)","(?<x_request_rewrite>[^,]+)",(?<x_avg_bw>[
^,]+),(?<x_bw_throttled>[^,]+),(?<x_user_type>[^,]+),"(?<x_resp_dvs_verdictname>[^,]+)","(?<x_req_dvs_threat_name>[^,]+)"(,(?<x_amp_verdict>[^,]+),"(?<x_amp_malware_n
ame>[^"]+)",(?<x_amp_score>[^,]+),(?<x_amp_upload>[^,]+),"(?<x_amp_filename>[^"]+)","(?<x_amp_sha>[^"]+)")?(?:>|&gt;)

[tmayer]

bumping up the question... Anyone has any insights?