How to configure blue coat proxysg on heavy forwarder?

[damonmanni]

Problem
I am trying to index Blue Coat proxySG log file (BCPSG) with no success. I suspect something wrong with the mapping in transforms.conf & props.conf - but I don't understand how to correlate it to the *.conf entries. I see that the BCPSG log file has a "Fields" definition (shown below) but I don't know how to match up the fields. Also, looks like BC changes its formatting on each version.

• So far, I am successful in getting rsyslog to bring in the data from BC, b/c I have it config'd to write to disk on the HF (See config file below). But I can't get it to get indexed from there.
• Below shows how i use tcp but i've also tried the file path in inputs.conf and still no luck.

The BC device and log file

• The BC device is sending data out port: 514 to the Splunk HF successfully and I can store it to disk on the HF
• This is the header of the BC log file #Software: SGOS 6.7.3.6#015 #Version: 1.0#015 #Fields: date time time-taken c-ip cs-username cs-auth-group s-supplier-name s-supplier-ip s-supplier-country s-supplier-failures x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bluecoat-application-name x-bluecoat-application-operation x-bluecoat-application-groups cs-threat-risk x-bluecoat-transaction-uuid x-icap-reqmod-header(X-ICAP-Metadata) x-icap-respmod-header(X-ICAP-Metadata)#015

On the Heavy Forwarder (HF)
/etc/rsyslog.conf
$ModLoad imuxsock
$ModLoad imjournal
$ModLoad imtcp
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
$template splunklog, "/data/splunk_rsyslog/%HOSTNAME%/%PROGRAMNAME%.log"
$template splunkmsg, "%rawmsg%\n"
$template bluecoatlog, "/data/splunk_rsyslog/%fromhost%/%$year%-%$month%-%$day%-%$hour%-syslog.log"
$RuleSet bluecoat
$DirOwner root
$DirGroup splunk
$FileOwner root
$FileGroup splunk
$DirCreateMode 0750
$FileCreateMode 0640
*.* ?bluecoatlog;splunkmsg
$InputTCPServerBindRuleset bluecoat
$InputTCPServerRun 514


/opt/splunk/etc/apps/Splunk_TA_bluecoat-proxysg/local
inputs.conf
[tcp://514]
source = tcp.bluecoat
sourcetype = bluecoat:proxysg:access:syslog
disabled = false

transforms.conf
I added this stanza to support 6.7.x format
`[auto_kv_for_bluecoat_v6_7_x]
REGEX = (?:"([^"]+)"|(\S+))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"|(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s+(?:"([^"]+)"|(\S+))\s*$

FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 x_exception_id::$13 x_exception_id::$14 sc_filter_result::$15 sc_filter_result::$16 cs_categories::$17 cs_categories::$18 cs_Referer::$19 cs_Referer::$20 sc_status::$21 sc_status::$22 s_action::$23 s_action::$24 cs_method::$25 cs_method::$26 rs_Content_Type::$27 rs_Content_Type::$28 cs_uri_scheme::$29 cs_uri_scheme::$30 cs_host::$31 cs_host::$32 cs_uri_port::$33 cs_uri_port::$34 cs_uri_path::$35 cs_uri_path::$36 cs_uri_query::$37 cs_uri_query::$38 cs_uri_extension::$39 cs_uri_extension::$40 cs_User_Agent::$41 cs_User_Agent::$42 s_ip::$43 s_ip::$44 sc_bytes::$45 sc_bytes::$46 cs_bytes::$47 cs_bytes::$48 x_virus_id::$49 x_virus_id::$50 x_bluecoat_application_name::$51 x_bluecoat_application_name::$52 x_bluecoat_application_operation::$53 x_bluecoat_application_operation::$54

FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 s-supplier-name::$13 s-supplier-name::$14 s-supplier-ip::$15 s-supplier-ip::$16 s-supplier-country::$17 s-supplier-country::$18 s-supplier-failures::$19 s-supplier-failures::$20 x-exception-id::$21 x-exception-id::$22 sc-filter-result::$23 sc-filter-result::$24 cs-categories::$25 cs-categories::$26 cs_Referer::$27 cs_Referer::$28 sc_status::$29 sc_status::$30 s-action::$31 s-action::$32 cs-method::$33 cs-method::$34 rs_Content_Type::$35 rs_Content_Type::$36 cs_uri_scheme::$37 cs_uri_scheme::$38 cs_host::$39 cs_host::$40 cs_uri_port::$41 cs_uri_port::$42 cs_uri_path::$43 cs_uri_path::$44 cs_uri_query::$45 cs_uri_query::$46 cs_uri_extension::$47 cs_uri_extension::$48 cs_User_Agent::$49 cs_User_Agent::$50 s_ip::$51 s_ip::$52 sc_bytes::$53 sc_bytes::$54 cs_bytes::$55 cs_bytes::$56 x_virus_id::$57 x_virus_id::$58 x_bluecoat_application_name::$59 x_bluecoat_application_name::$60 x_bluecoat_application_operation::$61 x_bluecoat_application_operation::$62 cs_threat_risk::$63 cs_threat_risk::$64`

props.conf

[bluecoat:proxysg:access:syslog]
pulldown_type = true
category = Network & Security
description = Data from Blue Coat ProxySG in W3C ELFF format thru syslog
KV_MODE = none
SHOULD_LINEMERGE = false
MAX_DAYS_AGO = 10951
TRUNCATE = 20000
REPORT-auto_kv_for_bluecoat_v5 = auto_kv_for_bluecoat_v5_3_3
REPORT-auto_kv_for_bluecoat_v6 = auto_kv_for_bluecoat_v6_7_x,auto_kv_for_bluecoat_v6_5_x


[teunlaan]

Not sure I understand what you are doing.
Are you collecting the BlueCoat log with Rsyslog (tcp 514) OR with Splunk (TCP 514)??
You can't run both at the same time.

there are 2 options:
1) collect your BC data with Rsyslog >> write it too disk > let splunk monitor the file
OR
2))collect your BC data with splunk, with a tcp input. But keep in mind that port 514 is only available if splunk is running as ROOT, other wise you must pick a port higher than 1024

( we are running option 2, Rsyslog wasn't collection the "very long " messages in the right way

[damonmanni]

Excellent explanation and advice! I went with #2 also running splunk as ROOT. I am now getting data indexed using a tcp input via /opt/splunk/etc/apps/Splunk_TA_bluecoat-proxysg/local/inputs.conf

But, I cannot get the pre-built panels included with the add-on to show the data on a dashboard. They are not picking up the newly indexed data. I suspect the transforms.conf and props.conf

• I tried to use the [auto_kv_for_bluecoat_v6_5_x] stanza but no change.
• Then added a new stanza called: [auto_kv_for_bluecoat_v6_7_x]. Tried my best to match up the fields as shown above but I am not sure how to do this.

Any other advice on how to create and/or match up the fields in the log file to a newly defined trans/props.conf files?

[damonmanni]

Also, I found a .../lookups/bluecoat_proxy_actions.csv file. What is this file for? Do I make changes to it also? What would that be?