Splunk Add-on for Blue Coat ProxySG: cs_uri field is missing

[mikaelbje]

The field cs_uri is missing which creates problem for the Splunk App for Enterprise Security. I suggest adding the following to the code:

[bluecoat:proxysg:access:syslog]
EVAL-cs_uri = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

And the same for the file input.

[jcoates_splunk]

Hi Mikael, can you tell us a bit more about the Blue Coat configuration you're working with? I'm seeing cs_uri extracting fine on our test systems. I've added a backlog story for this more robust option, but I'm curious how prevalent that configuration is.

[mikaelbje]

Thanks. I'll check this in a week when I'm back on-site, but I don't believe we've changed anything from the defaults. We're receiving the proxy logs through syslog. The auto_kv_for_bluecoat_v6_5_x transform which is shipped in the app does not contain the cs_uri field extraction, but the auto_kv_for_bluecoat_v5_3_3 transform does, which suggests that one must eval this field for Bluecoat 6.5.x.

The FIELDALIAS-url = cs_uri as url must perhaps also be added after the EVAL. Not sure if order matters.

[mreynov_splunk]

Mikael, thanks for pointing out this problem. We are working with bcreportermain_v1 format, which does not include cs_uri field in SGOS 6.5.

Since evals happen after fieldalias commands, just populate url directly in the eval.

EVAL-url = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

[mikaelbje]

Thanks, that seems to have done the trick. We also had to make the following additions after migrating away from the old Bluecoat app:

local/props.conf

[bluecoat:proxysg:access:syslog]
TRANSFORMS-main=nullPound
TIME_FORMAT = %Y-%m-%d %T
SHOULD_LINEMERGE = false
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = UTC

EVAL-url = coalesce(cs_uri, cs_host + cs_uri_path + if(cs_uri_query == "-", "", cs_uri_query))

local/transforms.conf

#[auto_kv_for_bluecoat_v6_5_x]
#REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$
#FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

[nullPound]
REGEX = ^\#
DEST_KEY=queue
FORMAT=nullQueue

[auto_kv_for_bluecoat_v6_5_x]
DELIMS=" "
FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"

We are getting the logs from syslog, not files.

I'm also wondering if the cs_uri_port and cs_uri_scheme fields should be used in the url EVAL, or does that mess it up for ESS?

[jcoates_splunk]

Hi Mikael, I think we've got this one fixed in the latest release. http://docs.splunk.com/Documentation/AddOns/latest/BlueCoatProxySG/Releasenotes Please let us know if you still see issues.

Thanks!

[mikaelbje]

Thanks. Version 3.4.1 seems to be working good! I still had to keep the following in local/transforms.conf, probably because we're logging in an unstandard format:

 [auto_kv_for_bluecoat_v6_5_x]
 DELIMS=" "
 FIELDS="date","time","time_taken","c_ip","cs_username","cs_auth_group","x_exception_id","sc_filter_result","cs_categories","cs_Referer","sc_status","s_action","cs_method","rs_Content_Type","cs_uri_scheme","cs_host","cs_uri_port","cs_uri_path","cs_uri_query","cs_uri_extension","cs_User_Agent","s_ip","sc_bytes","cs_bytes","x_virus_id"

[jcoates_splunk]

Yeah, I would expect that sort of thing to be required for the foreseeable future when working with W3C logs because they are so wonderfully great.