Field Extration Slow

[hartfoml]

I have the BlueCoat Field Extractions running on my three indexers. One of the indexers shows this error.

What does this mean and is it important?

[MySplunkIndexer.local] Field extractor name=bcreportermain_v1 is unusually slow (max single event time=1036ms, probes=422 warning max=1000ms)

[TonyLeeVT]

Figured out not just an answer, but a solution since the one provided does not scale to large enterprises. The solution is not to make Splunk adapt, but instead change the way data is sent to it. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format. Then the app and TA try to parse this space separated data using the complex regex seen above. Instead of doing that, you can instruct Blue Coat to send the data in a different format such as key value pair--which Splunk likes and natively parses.

Have the Blue Coat admins define a custom log format with the following fields:

Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

Since this data comes into Splunk as key=value pair now, Splunk parses it natively.

Remove the TAs from the indexer and replace it with a simpler props.conf file of this:

[bluecoat:proxysg:customclient]
SHOULD_LINEMERGE = false

This just turns off line merging which is on by default and makes the parsing even faster. Also remember to rename the props.conf and transforms.conf (ex: .bak files) included in the app if you have it installed on your search head--that contains the same complicated regex which will slow down data ingestion. By the way, by defining your own format, you can add other fields you care about--such as the target IP (cs-ip) which is not included in the default bcreportermain_v1 format for some reason. Hope this helps others than run into this situation.

[tiny3001]

Thank you. This makes more sense to me than trying to wrangle with a regex to make things faster. Just fix the format, make it CIM compliant and index that, let auto-kv take care of the rest.

[TonyLeeVT]

I have the same issue. Will post back if I find anything. Environment is distributed with three indexers and a fair amount of traffic. Thanks.

[martin_mueller · Accepted solution]

Splunk encountered an event where applying the field extraction bcreportermain_v1 took 1036ms. This usually has one of two reasons (or both):

• The field extraction is slow/complicated
• The event is unusually large

This is important for the speed of your search.

[hartfoml]

Thanks Martin for the help