How do I successfully populate Cisco Security Suite with syslog data from an ESA and WSA?

[tleaton]

I have configured two TA applications - the Cisco ESA and Cisco WSA add-on. I have enabled these add-ons within the initial setup of Cisco Security Suite and am using splunk version 6.2 and the new version of Cisco Security Suite (3.1) . Also the latest versions of the TAs.

I have copied the 'Splunk_TA_Cisco-wsa' and 'Splunk_TA_Cisco-esa' folder contents across to 'SA-cisco-wsa' and 'SA-cisco-esa' folders, respectively (within the 'SPLUNK_HOME/etc/apps' directory).

I have then customised the necessary files within the 'local' folder inside the 'SA-cisco-wsa' and 'SA-cisco-esa' folders, respectively - 'inputs.conf' to point to the local directory that my FTP server points to (and where syslog files in are pushed from the ESA and WSA respectively, using the recommended squid formatting). For the ESA I have also customised the 'props.conf' and the 'eventtypes.conf'.

The customisations I have for the WSA are detailed below:

'inputs.conf'

[monitor://C:/Program Files/Splunk/var/log/cisco-wsa/squid/]
source = cisco:wsa
sourcetype = cisco:wsa:squid
disabled = false

host = 127.0.0.1

'props.conf'

#access logs in squid format

[source::...wsa.access]
sourcetype = cisco:wsa:squid

[cisco:wsa:squid]
KV_MODE = none
SHOULD_LINEMERGE = false
REPORT-extract = kv_for_cisco_wsa_squid cs_url_host

FIELDALIAS-src = src_ip AS src
FIELDALIAS-signature = mcafee_virus_name AS signature
FIELDALIAS-signature = webroot_threat_name AS signature
FIELDALIAS-vendor_action = txn_result_code AS vendor_action
FIELDALIAS-bytes = bytes_in AS bytes
FIELDALIAS-CSS_compatibility = wbrs_score AS x_wbrs_score user AS cs_username txn_result_code AS http_result
LOOKUP-vendor_info_for_cisco_wsa = cisco_wsa_vendor_info_lookup sourcetype OUTPUT vendor,product,ids_type
LOOKUP-code_info = cisco_wsa_category_lookup x_webcat_code_abbr OUTPUT webcat_code_full AS vendor_category, webcat_code_full AS x_webcat_code_full,usage,severity
LOOKUP-malware_action = cisco_wsa_malware_action_lookup x_webroot_scanverdict OUTPUT malware_action
LOOKUP-proxy_action = cisco_wsa_proxy_action_lookup vendor_action OUTPUT action
EVAL-malware_action = case(wbrs_score>=6 AND wbrs_score<=10, "allowed", wbrs_score>=-10 AND wbrs_score<=-6, "blocked", wbrs_score = "-", "allowed")
EVAL-http_user_agent=coalesce(http_user_agent,vendor_suspect_user_agent)

#L4TM logs

[source::...wsa.l4tm]
sourcetype = cisco:wsa:l4tm

[cisco:wsa:l4tm]
KV_MODE = none
SHOULD_LINEMERGE = false
REPORT-extract = kv_for_cisco_wsa_Firewall_l4tm,kv_for_cisco_wsa_Address_l4tm,kv_for_cisco_wsa_removed_l4tm
LOOKUP-vendor_info_for_cisco_wsa = cisco_wsa_vendor_info_lookup sourcetype OUTPUT vendor,product,ids_type
LOOKUP-vendor_traffic_action = cisco_wsa_traffic_action_lookup vendor_action OUTPUT action

#access logs in w3c format

[cisco:wsa:w3c]
KV_MODE = none
SHOULD_LINEMERGE = false
REPORT-extract = auto_kv_for_cisco_wsa_w3c
FIELDALIAS-src = c_ip AS src
FIELDALIAS-signature = x_mcafee_virus_name AS signature
FIELDALIAS-signature = x_webroot_threat_name AS signature
FIELDALIAS-vendor_action = sc_result_code AS vendor_action
FIELDALIAS-bytes = cs_bytes AS bytes
FIELDALIAS-status = sc_http_status AS status
FIELDALIAS-http_method = cs_method AS http_method
FIELDALIAS-url = cs_url AS url
FIELDALIAS-user = cs_username AS user
FIELDALIAS-dest = s_ip AS dest
FIELDALIAS-http_content_type = cs_mime_type AS http_content_type
LOOKUP-vendor_info_for_cisco_wsa = cisco_wsa_vendor_info_lookup sourcetype OUTPUT vendor,product,ids_type
LOOKUP-code_info = cisco_wsa_category_lookup x_webcat_code_abbr OUTPUT webcat_code_full AS x_webcat_code_full,usage,severity
LOOKUP-malware_action = cisco_wsa_malware_action_lookup x_webroot_scanverdict OUTPUT malware_action
LOOKUP-proxy_action = cisco_wsa_proxy_action_lookup vendor_action OUTPUT action

EVAL-malware_action = case(x_wbrs_score>=6 AND x_wbrs_score<=10, "allowed", x_wbrs_score>=-10 AND x_wbrs_score<=-6, "blocked", x_wbrs_score = "-", "allowed")

'transforms.conf'

Access logs in squid format

[kv_for_cisco_wsa_squid]
REGEX = ([0-9.]) *[0-9] ([0-9.]) ([A-Z_])/([0-9]) ([0-9]) ([A-Z]) ([^ ]) "?([^ "])"? ([^/])/([^ ]) ([^ ]) ([^ ]+) <([^,]+),([^,]+),"([0-9]{0,2}|-|\w+)","([^"]+)",[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,"([^"]+)",[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,"([^"]+)",[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,[^>]+>\s*-\s*"?([^"]+)"?$
FORMAT = src_ip::$2 txn_result_code::$3 status::$4 bytes_in::$5 http_method::$6 url::$7 user::$8 server_contact_mode::$9 dest::$10 http_content_type::$11 acltag::$12 x_webcat_code_abbr::$13 wbrs_score::$14 x_webroot_scanverdict::$15 webroot_threat_name::$16 mcafee_virus_name::$17 malware_category::$18 vendor_suspect_user_agent::$19

[cisco_wsa_category_lookup]
filename = cisco_wsa_category_map_lookup.csv

[cisco_wsa_vendor_info_lookup]
filename = cisco_wsa_vendor_lookup.csv

[cisco_wsa_malware_action_lookup]
filename = cisco_wsa_malware_action_lookup.csv

[cisco_wsa_proxy_action_lookup]
filename = cisco_wsa_proxy_action_lookup.csv

L4TM logs

[kv_for_cisco_wsa_Firewall_l4tm]
REGEX = [A-Za-z]* ([A-Za-z]* +[0-9]* [0-9:]* [0-9]) [A-Za-z]: Firewall ([A-Za-z]) ([A-Z]+). data from ([0-9a-z.])(:([0-9a-z])){0,1} to ([0-9a-z.])((([A-Za-z0-9 -_]))){0,1}(:([^.]+)){0,1}.
FORMAT = vendor_action::$2 transport::$3 src::$4 src_port::$6 dest::$7 dest_domain::$9 dest_port::$11

[kv_for_cisco_wsa_Address_l4tm]
REGEX = [A-Za-z]* ([A-Za-z]* +[0-9]* [0-9:]* [0-9]) [A-Za-z]: Address ([0-9.:]) [A-Za-z] [A-Za-z]* ([A-Za-z0-9._-])( ([A-Za-z0-9 ._-])){0,1} [A-Za-z]* [A-Za-z]* firewall ([A-Za-z ]*)
FORMAT = dest::$2 dest_domain::$3 vendor_action::$5

[kv_for_cisco_wsa_removed_l4tm]
REGEX = [A-Za-z]* ([A-Za-z]* +[0-9]* [0-9:]* [0-9]) [A-Za-z]: Address ([0-9.:]) [A-Za-z] ([A-Za-z0-9.-_])( ([A-Za-z0-9 .-_])){0,1} ([A-Za-z]) [A-Za-z ]
FORMAT = dest::$2 dest_domain::$3 vendor_action::$5

[cisco_wsa_traffic_action_lookup]
filename = cisco_wsa_traffic_action_lookup.csv

[cs_url_host]
SOURCE_KEY=url

REGEX=\w+://(?[^/:]+)[:/]

The customisations I have for the ESA are detailed below:

'eventtypes.conf'

[cisco_esa]
search = sourcetype = cisco_esa

tags = cisco e-mail security

'inputs.conf'

[monitor://C:/Program Files/Splunk/var/log/cisco-esa/squid/]
disabled = false
followTrail = 0
sourcetype = cisco_esa

host = 127.0.0.1

'props.conf'

[cisco_esa]

REPORT-ironport = get_mid, get_to, get_from, get_icid, get_dcid, get_attach_name, get_attach_size, get_subject1, get_subject2, get_subject3

Log files are being received succesfully - I can see them in the FTP directory being pushed across from the WSA and ESA. I can also perform searches of the sourcetypes 'cisco:wsa:squid' within the WSA TA and 'cisco:esa:squid' within the ESA TA and these both return expected logs which correspond to test traffic pushed through and modifications made on both gateways.

The problem is, however, that nothing in the Cisco Security suite populates apart from 2 panes on the summary page: under ‘security events statistics by sourcetype’ and ‘security event statistics by host’ – This shows, respectively, the sourcetype ‘cisco:wsa:squid’ and the local host 127.0.0.1

If anyone has any ideas why this might be the case or is able to offer suggestions or point out errors in my configurations, I would be greatly appreciative.

[jconger · Accepted solution]

Why did you copy the 'Splunk_TA_Cisco-wsa' and 'Splunk_TA_Cisco-esa' folder contents across to 'SA-cisco-wsa' and 'SA-cisco-esa' folders? Do you have both 'SA-cisco-esa' and 'Splunk_TA_Cisco-esa' folders in etc/apps?

[tleaton]

Thank you very much for this post - this got me thinking - I copied the 'Splunk_TA_Cisco-' folders across to the 'Splunk_CiscoSecuritySuite/appserver/addons' as per one of your earlier posts here 'http://answers.splunk.com/answers/125863/splunk-6-cisco-security-suite-3-0-app-config-files-needed.h...'. I then renamed them to 'TA-cisco-'. Just for good measure I then renamed them to the same within the apps directory although presumably they are not required in this directory? Are only 'SA-cisco-esa' and 'SA-cisco-wsa' are required within etc/apps ?

Anyhow this is now working and I can see data for the WSA on Security Suite! However, I still can't see data for the ESA - presumably this is something wrong with my configuration in the 'props.conf', the 'inputs.conf' or the 'eventtypes.conf' within the local directory inside the 'Splunk_CiscoSecuritySuite/appserver/addons/TA-cisco-' or within 'etc/apps/SA-cisco-' ?

My final question is, does the configuration need to be the same in both of these locations (for me the local folder within both of these locations contains the same configuration files) ?

[jconger]

With Cisco Security Suite 3.1, you no longer need the SA-cisco-* folders. Your setup should look like this:

$SPLUNK_HOME/etc/apps/Splunk_CiscoSecuirtySuite
$SPLUNK_HOME/etc/apps/Splunk_TA_Cisco-wsa
$SPLUNK_HOME/etc/apps/Splunk_TA_Cisco-esa

[tleaton]

Thank you. Do you have an idea of what the issue might be with the ESA configuration as I'm still to have this view populating with data.

[jconger]

Looks like your sourcetype is wrong for ESA. Check out the documentation here -> http://docs.splunk.com/Documentation/AddOns/released/CiscoESA/Configureinputsonasingleinstance

The Cisco Security Suite relies on certain eventtypes - which are based on the Splunk_TA_Cisco-esa eventtypes.conf - which are defined by the inputs.conf sourcetypes.

[ssackrider]

The "single instance " doc was removed-- 17 Apr 2015

[tleaton]

Thank you very much, this is now working. I set the source correctly as per the link you send me - needed to be cisco:esa:mailtext; also I needed to change the source type inside the data inputs -> files and directories, and finally I copied the 'eventtypes.conf', 'transforms.conf', 'props.conf' and 'tags.conf' from default into local (once they had been generated with the corect source type).

I will now close of this question - however one last quick query - do you know whether the Cisco Security Suite has a built in view for monitoring syslog data pertaining to changes that have been made to policies on the WSA and ESA? - because the all the views I have founds are regarding web/email transactions, blocked messages/traffic, website categories etc, but I would like to have a panel which lists events concerning any alteration to the policies on the gateways.

[TonyLeeVT]

I think you may mean: cisco:esa:textmail
Instead of what is written: cisco:esa:mailtext

Unless this has changed in the past year.