Job inspector: How to identify search time extraction is kicking in

[splunk_zen]

We may be having performance issues as newly saved search time extractions are not working even after being successfully tested via the Field Extractor

Sample example:

"faQUF","2.3.7","False","2","4","9","1","N-281","PF","19800","India Standard Time","3.8.0.5","2016-11-03T07:19:17.000Z","2016-11-03T10:49:35.000Z","3.8.0.8","/x/api/v2/hosts/fUF","","None","Windows 7 Enterprise","Service Pack 1","64-bit","7x-5x-fx-0x-xx-xx","dcfb"

the following props.conf on were set on the SH

[fireye:hx:asset_inventory]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
EXTRACT-agentId,agentVersion,excluded_from_containment,stats_acqs,stats_alerting_conditions,stats_alerts,stats_exploit_alerts,hostname,domain,gmt_offset_seconds,timezone,src_ip,last_audit_timestamp,last_poll_timestamp,last_poll_ip,url,last_alert_id,last_alert_timstamp,os_product_name,os_patch_level,os_bitness,src_mac,md5 = \"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\"
EXTRACT-agentId = ^"(?P<agentId>[^"]*)

NOTES:

** Search was run on Verbose
** the extraction was tested first as belonging to its owner, and then shared globally
** Both the single EXTRACT-agentId as well as the composed fields one were tested separately, just kept the single one to exemplify even such a simple extraction is not working

Using the job inspector I'm seeing a very quick key value extraction (the 6 invocations may be the 6 default interesting fields Splunk extracts)

Duration (seconds)          Component       Invocations
0.01          command.search.kv     6 

I can only see the expected fields when I use the very same regex as a | rex command

sourcetype = fireye:hx:asset_inventory
| rex field=_raw "\"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\""

and as expected we get the rex command kicking in

Duration (seconds)          Component       Invocations
1.40        command.rex     5,501 

Anything which can point me to why this is broken?

[vasanthmss]

Hope you noticed or missed, while restarting splunk should through some exception on the props.conf malformed or something because of Extract class name? Since your first extract failed after the confs will be ignored.

I would suggest keep the Extract class name as small and meaningful. class name is a unique literal string. So simple change the below,

from

EXTRACT-agentId,agentVersion,excluded_from_containment,stats_acqs,stats_alerting_conditions,stats_alerts,stats_exploit_alerts,hostname,domain,gmt_offset_seconds,timezone,src_ip,last_audit_timestamp,last_poll_timestamp,last_poll_ip,url,last_alert_id,last_alert_timstamp,os_product_name,os_patch_level,os_bitness,src_mac,md5=

To

EXTRACT-sourcefields

Hope this will helps you!!!!

[splunk_zen]

Thanks vasanthmss
After I restarted with the shorted name it did work
Though not totally convinced it was that as when I rolled back to the previous long EXTRACT- class name it kept working
May have been related to a colleague having suspended the knowledge bundle sync as it had reached a huge size