Splunk Search

Select only some fields from csv to index

fabrizioalleva
Path Finder

Hi all,
I'm in enviroment so configured:

1 uf, 1 hf, 4 indexers, 1 search head, 1 master cluster.

I've to index a large CSV, read from the universal forwarder, which forwards data to the HF which pass the data to the indexer.

The CSV has 150 fields and I want to index only 10 of these. So I've configured these things:

on universal forwarder:

------------------

inputs.conf

------------------

[monitor:///myfolder/Interface*]
disabled = 0
index = interface_metrics
sourcetype = if_csv

on heavy forwarder

------------------

inputs.conf

------------------

[splunktcp://9996]
index=interface_metrics
sourcetype = if_csv

------------------

props.conf

------------------

[if_csv]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
HEADER_FIELD_DELIMITER =,
FIELD_DELIMITER=,
HEADER_FIELD_LINE_NUMBER = 0
TRANSFORMS-set=setnull, setparsing, nullhead

------------------

transforms.conf

------------------

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[nullhead]
REGEX = ifInDiscardsDelta
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^([^,]),([^,]),(?:[^,]+,\s*)([^,]),([^,])(?:[^,]+,\s*){5}([^,]),([^,])(?:[^,]+,\s*){3}([^,]),([^,]),(?:[^,]+,\s*){2}([^,]),([^,])(?:[^,]+,\s*){7}([^,])(?:(?:[^,]+,?\s)|(?:[,,])){123}([^,]),([^,])
DEST_KEY = queue
FORMAT = indexQueue

example CSV row :
0ef1fa5f-586c-48a4-a902-827aef967f47,1569309580446,300.0,100,9,0,0,0,0,6.6107712E7,5.0463189E7,151356.0,150857.0,0.176,0.135,0.0,0.0,0.0,0.0,0,0,0,0,0,0,4b16e13e-c391-4626-b364-2890fe5a009a,0,0,0,0,,,151351,149267,0,0,451,5,1139,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,039550ed-1d39-487f-9b12-276ad9472771,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,3.0,3.0,,,0.056,3.4E-4,0.2,3.0,3.0,,,0.056,3.4E-4,0.2,1569309300000,300

I want to keep the fields:
1,2,4,5,10,11,14,15,26,149,150

I don't succeed in indexing only the fields that I choose, but the whole row.

What I'm wrong ?

Thanks
Fabrizio

0 Karma
1 Solution

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

View solution in original post

0 Karma

fabrizioalleva
Path Finder

Ok,
I've solved with a SED command as

https://answers.splunk.com/answers/564940/remove-fields-at-index-time-from-a-csv-file.html

But is there another way to solve it ?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...