All Apps and Add-ons

Is is possible to disable or remove Blue Coat comments before they are indexed?

TonyLeeVT
Builder

The current Blue Coat app uses an event type called: bcproxysg_search

The content of the event type is the following:

sourcetype = bluecoat:proxysg:* NOT "#Fields:" NOT "#Version:" NOT "#Software:" NOT "#Date:"

This event type hides comments that are sent from the Blue Coat and do not appear to be used in the app.

Comment Examples:

#Date: ...
#Fields: ...
#Version: ...
#Software: ...

Unfortunately, the comments still count against the Splunk license and potentially misrepresent statistics generated from commands such as tstats. Is it possible to either prevent Blue Coat from sending these fields or have the TA prevent this data from being indexed? Thanks.

0 Karma
1 Solution

TonyLeeVT
Builder

Even though Splunk can adapt, that may not be the most graceful solution. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format (which includes the headers mentioned above). Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair which appears to get rid of those fields.

Have the Blue Coat admins define a custom log format... for example:

 Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

This should get rid of the headers and make ingesting data faster. See the following page for the rest of the solution:
https://answers.splunk.com/answers/376872/blue-coat-field-extractor-namecustom-client-events.html

View solution in original post

0 Karma

TonyLeeVT
Builder

Even though Splunk can adapt, that may not be the most graceful solution. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format (which includes the headers mentioned above). Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair which appears to get rid of those fields.

Have the Blue Coat admins define a custom log format... for example:

 Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

This should get rid of the headers and make ingesting data faster. See the following page for the rest of the solution:
https://answers.splunk.com/answers/376872/blue-coat-field-extractor-namecustom-client-events.html

0 Karma

maciep
Champion

I don't know anything about Blue Coat, so i'm not sure if they can stop being sent. But you should be able to drop them before indexing. Just update/create props/transforms conf in the local folder for your app. This should be done on whatever box does parsing in your infrastructure - typically an indexer or a heavy forwarder.

Something along these lines (regex not tested)

props.conf

[your_bluecoat_sourcetype]
TRANSFORMS-drop_comments = drop_bluecoat_comments

transforms.conf

[drop_bluecoat_comments]
REGEX = (#Fields|#Versions|#Software|#Date)
DEST_KEY = queue
FORMAT = nullQueue

There are a lot of examples on how to do this out here as well.

0 Karma

TonyLeeVT
Builder

Thank you for the solution. This is great information for any appliance in which you are stuck receiving junk data. Fortunately, it appears Blue Coat can be told to send data in a different format. See answer below.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...