Getting Data In

how to import some columns from csv

splunk6161
Path Finder

I have a csv file to import by app data ->monitor
i would to import some columns (not all) before to index.
It's possible?
Thanks

Tags (1)
0 Karma

splunk6161
Path Finder

I've created a new csv to do a test:

alt text

did this in props.conf:

[csv_n402_rex]
BREAK_ONLY_BEFORE_DATE = 
DATETIME_CONFIG = 
FIELD_DELIMITER = ,
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SEDCMD-rex = s/([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).([^,]+).*/\4,\12,\14,\17\n/
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = 1

In search works, when I add *| table ** in the search, it shows me all fields. Why?
I suppose regex is just a view, so I'm indexing all the fields.
alt text

0 Karma

to4kawa
Ultra Champion

because INDEXED_EXTRACTIONS = csv is before SEDCMD-rex
|table * display all extracted fields.

0 Karma

splunk6161
Path Finder

so I'm indexing all the fields?

0 Karma

to4kawa
Ultra Champion

yes , I guess

0 Karma

splunk6161
Path Finder

I tried to reverse as you said but the sorting of the fields would seem to be automatic and so like in the props.conf file above

0 Karma

to4kawa
Ultra Champion

How about transforms.conf ?

0 Karma

splunk6161
Path Finder

not present, i'll try to configure it.

0 Karma

woodcock
Esteemed Legend

Do this in props.conf:

[YourSourcetypeHere]
SEDCMD-trim_raw = s/([^,]+),(?:[^,]+,){2}(.*$)/\1\2/

For proof try this:

| makeresults 
| fields - _time
| eval _raw="_time,f1,f2,f3,f4,f5,f6,f7,f8,f9,f10" 
| rex mode=sed "s/([^,]+),(?:[^,]+,){2}(.*$)/\1\2/"

https://answers.splunk.com/answers/530547/filter-data-and-extract-field-before-indexed.html#comment-...

0 Karma

woodcock
Esteemed Legend

My rule on CSVs is this: If the file does NOT contain a timestamp, it should NOT be indexed (do not use Add data. Instead, it should be uploaded as a lookup. If you must index this data, then use SEDCMD to skip (erase) columns in your data as it is indexed:

https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Data/Anonymizedata

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi splunk6161,
I didn't tried to do this, but you could delete columns that you don't want to index using a SEDCMD command.
e.g. if you have a csv like this

field1,field2,field3,field4,field5,field6
aaa,bbb,ccc,ddd,eee,fff

and you don't want to index field4

you could insert in props.conf stanza:

[your_sourcetype]
 SED-alter=s/[^,],[^,],[^,],[^,],[^,],[^,]/[^,],[^,],[^,],[^,],,[^,]/g

Try it

Bye.
Giuseppe

0 Karma

splunk6161
Path Finder

It doesn't work
I have 10columns plus 1column "_time" as first column.
I would keep the first column, skip the second and the third, keep the rest.
Is correct this scenario?
SEDCMD-alter=s/[^,],[^,],[^,],[^,],[^,],[^,],[^,],[^,],[^,],[^,],[^,]/[^,],,,[^,],[^,],[^,],[^,],[^,],[^,],[^,],[^,]/g

thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...