Why are REPORT field extractions not being applied in my distributed search environment?

[gustavomichels]

Hey all,

Using Splunk 6.0.2 across the board, I'm trying to extract key="value" pairs from WinEventLog entries present in the Message field using REPORT because there will be multiple pairs. I'm using the following:

props.conf

[source::WinEventLog...]
REPORT-MESSAGE1-extrafields= wel-extrafields-kv

transforms.conf

[wel-extrafields-kv]
SOURCE_KEY = Message
REGEX = ([\w-]+)="*([\w-]+)"*
FORMAT = $1::$2
MV_ADD = true

This works well on a single server acting as the indexer/search head, however I can't make it work on a distributed environment (events come from UF, indexed at the indexer tier and searched at the search head tier - using search head pooling). According to http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F I should add those files to the search head tier only. I already tried adding then to the indexing as well, to no avail.

What am I missing?

Thank you

Some debug data:

# /opt/splunk/bin/splunk btool props list "source::WinEventLog" --debug
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/props.conf                    [source::WinEventLog...]
/opt/splunk/etc/system/default/props.conf                                                        ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                                                        BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                                                        BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                                                        CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                                                        DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                                                        HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                                                        KV_MODE = none
/opt/splunk/etc/system/default/props.conf                                                        LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                                                        LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
/opt/splunk/etc/system/default/props.conf                                                        LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-action-for_fs_notification = nix_endpoint_change_action_lookup vendor_action OUTPUT action
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-object_category-for_fs_notification = nix_endpoint_change_fs_notification_object_category_lookup vendor_object_category OUTPUTNEW object_category
/opt/splunk/etc/system/default/props.conf                                                        MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                                                        MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                                                        MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                                                        MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                                                        MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                                                        MAX_TIMESTAMP_LOOKAHEAD = 30
/opt/splunk/etc/system/default/props.conf                                                        MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                                                        MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                                                        MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                                                        REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/props.conf                    REPORT-MESSAGE1-extrafields = wel-extrafields-kv
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                                                        SHOULD_LINEMERGE = false
/opt/splunk/etc/system/default/props.conf                                                        TRANSFORMS = 
/opt/splunk/etc/system/default/props.conf                                                        TRANSFORMS-FIELDS = strip-winevt-linebreaker
/opt/splunk/etc/system/default/props.conf                                                        TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                                                        detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                                                        maxDist = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/splunk_app_windows_infrastructure/default/props.conf [source::WinEventLog:System]
/opt/splunk/etc/system/default/props.conf                                                        ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                                                        BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                                                        BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                                                        CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                                                        DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                                                        HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                                                        LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                                                        LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-action-for_fs_notification = nix_endpoint_change_action_lookup vendor_action OUTPUT action
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf                            LOOKUP-object_category-for_fs_notification = nix_endpoint_change_fs_notification_object_category_lookup vendor_object_category OUTPUTNEW object_category
/opt/splunk/etc/system/default/props.conf                                                        MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                                                        MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                                                        MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                                                        MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                                                        MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                                                        MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf                                                        MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                                                        MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                                                        MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                                                        SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                                                        SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                                                        TRANSFORMS = 
/opt/splunk_shared_storage/sh_pool/etc/apps/splunk_app_windows_infrastructure/default/props.conf TRANSFORMS-force_sourcetype_system_ias_for_wineventlog = force_sourcetype_system_ias_for_wineventlog
/opt/splunk/etc/system/default/props.conf                                                        TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                                                        detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                                                        maxDist = 100


# /opt/splunk/bin/splunk btool transforms list wel-extrafields-kv --debug       
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf   [wel-extrafields-kv]
/opt/splunk/etc/system/default/transforms.conf                                       CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                                       CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                                       DEFAULT_VALUE = 
/opt/splunk/etc/system/default/transforms.conf                                       DEST_KEY = 
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf   FORMAT = $1::$2
/opt/splunk/etc/system/default/transforms.conf                                       KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                                       LOOKAHEAD = 4096
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf   MV_ADD = true
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf   REGEX = ([\w-]+)="*([\w-]+)"*
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf   SOURCE_KEY = Message
/opt/splunk/etc/system/default/transforms.conf                                       WRITE_META = False

[JScordo]

@gustavomichels Just a little heads up regarding your initial question. I just found this out myself, but if you turn on XML logging for wineventlog all of the values that come in the "message" field gets extracted. You can check it out here: http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/

[gustavomichels · Accepted solution]

Managed to get it working, the problem was the objects were not available to the search app. Basically needed to add to the app:

metadata/default.meta

[]
export = system

App is distributed to the search head tier only, so the information on the wiki page is validated.