I am trying to use the Splunk for AWS app to monitor cisco web logs files but am having a very hard time extracting the fields. The files are tab separated values and are created approximately every half hour, so I should theoretically be able to use the header as the field values, but cannot do so while monitoring, only when manually uploading in a test environment. The problem I am finding with manual extraction is not every field is filled for all events. I attempted to use the following settings in the props.conf file but they are only pulling the header line.
Sample logs here : http://pastebin.com/5t7xjt41
[aws_s3]
FIELD_DELIMITER = tab
HEADER_FIELD_DELIMITER = tab
FIELD_NAMES = "datatime","c-ip","cs(X-Forwarded-For)","cs-username","cs-method","cs-uri-scheme","cs-host","cs-uri-port","cs-uri-path","cs-uri-query","cs(User-Agent)","cs(Content-Type)","cs-bytes","sc-bytes","sc-status","sc(Content-Type)","s-ip","x-ss-category","x-ss-last-rule-name","x-ss-last-rule-action","x-ss-block-type","x-ss-block-value","x-ss-external-ip","x-ss-referer-host"
INDEXED_EXTRACTIONS = tsv
KV_MODE = none
sourcetype = cws:proxy
TZ = EST
NO_BINARY_CHECK = true
disabled = false
BREAK_ONLY_BEFORE_DATE = true
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Tab-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
... View more