Hi,
I have a csv file with headers that needs processing. I want to 1) filter out the header and 2) have the fields recognized in the indexer. I tried following the steps listed in this link: http://splunk-base.splunk.com/answers/41551/how-do-i-get-auto-field-detection-on-forwarded-csv ,but it's not working.
My csv file:
"Time","IOPS","Latency(ms)","BW (MBps)"
"2013-04-03 22:06:00","9715","3.0","353.0"
"2013-04-03 22:07:00","8308","2.0","179.0"
"2013-04-03 22:08:00","6436","3.0","244.0"
"2013-04-03 22:09:00","4894","4.0","223.0"
"2013-04-03 22:10:00","4730","4.0","246.0"
input.conf:
[monitor:///fisc/dasd/xiv/perfstats/*.csv]
index=perfstats
sourcetype=xiv:perf:arrayStats
followTail = 0
props.conf:
[xiv:perf:arrayStats]
CHECK_FOR_HEADER = true
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:$S
SHOULD_LINEMERGE = false
TIME_PREFIX = ^"
TZ = US/Eastern
TRANSFORMS-xiv:perf:arrayStats = NoHeader
TRANSFORMS-xiv:perf:arrayStats = csv-fieldextraction
transforms.conf:
[NoHeader]
REGEX = "Time","IOPS","Latency(ms)","BW (MBps)"
DEST_KEY = queue
FORMAT = nullQueue
[csv-fieldextraction]
DELIMS=","
FIELDS="Time","IOPS","Latency","BW (MBps)"
props file:
Have you considered bringing this all in, as a single event and then just piping it to multikv at search time? This will utilize the header as the field name and eliminate the overhead of processing this pre-index.
Can't be a single event - they are different timestamps associated with performance stats.