Hey all,
Using Splunk 6.0.2 across the board, I'm trying to extract key="value"
pairs from WinEventLog
entries present in the Message
field using REPORT
because there will be multiple pairs. I'm using the following:
props.conf
[source::WinEventLog...]
REPORT-MESSAGE1-extrafields= wel-extrafields-kv
transforms.conf
[wel-extrafields-kv]
SOURCE_KEY = Message
REGEX = ([\w-]+)="*([\w-]+)"*
FORMAT = $1::$2
MV_ADD = true
This works well on a single server acting as the indexer/search head, however I can't make it work on a distributed environment (events come from UF, indexed at the indexer tier and searched at the search head tier - using search head pooling). According to http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F I should add those files to the search head tier only. I already tried adding then to the indexing as well, to no avail.
What am I missing?
Thank you
Some debug data:
# /opt/splunk/bin/splunk btool props list "source::WinEventLog" --debug
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/props.conf [source::WinEventLog...]
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf KV_MODE = none
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-action-for_fs_notification = nix_endpoint_change_action_lookup vendor_action OUTPUT action
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-object_category-for_fs_notification = nix_endpoint_change_fs_notification_object_category_lookup vendor_object_category OUTPUTNEW object_category
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 30
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/props.conf REPORT-MESSAGE1-extrafields = wel-extrafields-kv
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRANSFORMS-FIELDS = strip-winevt-linebreaker
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/splunk_app_windows_infrastructure/default/props.conf [source::WinEventLog:System]
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-action-for_fs_notification = nix_endpoint_change_action_lookup vendor_action OUTPUT action
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-nix/default/props.conf LOOKUP-object_category-for_fs_notification = nix_endpoint_change_fs_notification_object_category_lookup vendor_object_category OUTPUTNEW object_category
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk_shared_storage/sh_pool/etc/apps/splunk_app_windows_infrastructure/default/props.conf TRANSFORMS-force_sourcetype_system_ias_for_wineventlog = force_sourcetype_system_ias_for_wineventlog
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
# /opt/splunk/bin/splunk btool transforms list wel-extrafields-kv --debug
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf [wel-extrafields-kv]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEST_KEY =
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf FORMAT = $1::$2
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf MV_ADD = true
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf REGEX = ([\w-]+)="*([\w-]+)"*
/opt/splunk_shared_storage/sh_pool/etc/apps/SA-extrafields/default/transforms.conf SOURCE_KEY = Message
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
Managed to get it working, the problem was the objects were not available to the search app. Basically needed to add to the app:
metadata/default.meta
[]
export = system
App is distributed to the search head tier only, so the information on the wiki page is validated.
@gustavomichels Just a little heads up regarding your initial question. I just found this out myself, but if you turn on XML logging for wineventlog all of the values that come in the "message" field gets extracted. You can check it out here: http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/
Managed to get it working, the problem was the objects were not available to the search app. Basically needed to add to the app:
metadata/default.meta
[]
export = system
App is distributed to the search head tier only, so the information on the wiki page is validated.