Getting Data In

How to omit columns from CSV-style event input?

ziq
Engager

I intend to import a CSV-style file into Splunk. The file has about 30 columns, about 120 million lines and is about 150GB of size. Of the file, I only require a subset of columns.

The file's contents shall be imported as events and not as a CSV lookup file.

For sake of simplicity, assume the structure below:

src,dest_ip,dest_port,dest_user,dest_zone
10.50.60.80,192.0.2.92,443,emily,Internet
10.50.60.53,203.0.113.12,389,brian,Intranet
10.33.118.40,198.51.100.65,80,john,Internet
...

Is there any way to exclude the "dest_user" column from the import?

(Running the data through a sed/awk/perl script beforehand is certainly possible, but given the size of the file this would be computationally expensive. And as Splunk already extracts the field headers, it appears to me that excluding columns from import would be the cleaner and more efficient approach. Furthermore, it is likely that I will have to deal with similar files (that have different fieldsets or column orders) in the future.)

p_gurav
Champion

Hi ziq,

You can use script to create new csv with selected fields and then index into splunk.

0 Karma

ziq
Engager

I'm aware and mentioned the script option in my question. But running the data through a script beforehand would be twice as computationally expensive (or more). I also anticipate similar files with different field sets in the future, hence I would need to touch/modify the script for each of these files.

(Running the data through a
sed/awk/perl script beforehand is
certainly possible, but given the size
of the file this would be
computationally expensive. And as
Splunk already extracts the field
headers, it appears to me that
excluding columns from import would be
the cleaner and more efficient
approach. Furthermore, it is likely
that I will have to deal with similar
files (that have different fieldsets
or column orders) in the future.)

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...