Getting Data In

csv with headers

a212830
Champion

Hi,

I have a csv file with headers that needs processing. I want to 1) filter out the header and 2) have the fields recognized in the indexer. I tried following the steps listed in this link: http://splunk-base.splunk.com/answers/41551/how-do-i-get-auto-field-detection-on-forwarded-csv ,but it's not working.

My csv file:

"Time","IOPS","Latency(ms)","BW (MBps)"
"2013-04-03 22:06:00","9715","3.0","353.0"
"2013-04-03 22:07:00","8308","2.0","179.0"
"2013-04-03 22:08:00","6436","3.0","244.0"
"2013-04-03 22:09:00","4894","4.0","223.0"
"2013-04-03 22:10:00","4730","4.0","246.0"

input.conf:
[monitor:///fisc/dasd/xiv/perfstats/*.csv]
index=perfstats
sourcetype=xiv:perf:arrayStats
followTail = 0

props.conf:

[xiv:perf:arrayStats]
CHECK_FOR_HEADER = true
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:$S
SHOULD_LINEMERGE = false
TIME_PREFIX = ^"
TZ = US/Eastern

TRANSFORMS-xiv:perf:arrayStats = NoHeader
TRANSFORMS-xiv:perf:arrayStats = csv-fieldextraction

transforms.conf:
[NoHeader]
REGEX = "Time","IOPS","Latency(ms)","BW (MBps)"
DEST_KEY = queue
FORMAT = nullQueue

[csv-fieldextraction]
DELIMS=","
FIELDS="Time","IOPS","Latency","BW (MBps)"
props file:

Tags (2)
0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

Have you considered bringing this all in, as a single event and then just piping it to multikv at search time? This will utilize the header as the field name and eliminate the overhead of processing this pre-index.

0 Karma

a212830
Champion

Can't be a single event - they are different timestamps associated with performance stats.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...