Splunk Search

Can Splunk handle indexing a CSV file containing more than 6000 fields/columns?

splunknewbie05
Explorer

I have a heavily nested structured/dynamic XML event. I converted it to CSV and it generated more than 6000 unique fields (Columns). I noticed splunk is able to detect only a few columns. From the Splunk documentation, the default is set to 512 columns.
I added "maxcols" to limits.conf and still Splunk uses only few columns.
Can splunk handle this many large no of columns?

0 Karma

ebaileytu
Communicator

sure - I have a csv file I push into splunk every day with 10K columns. I had to set truncation=0 in props to get it work. One catch is that Splunk will not display all the columns as interesting fields. Otherwise works fine.

DalJeanis
Legend

To show up in the "Interesting fields" list, values have to be populated on 20% of the events. If many or most of your 10K columns are optional data, then they would not appear in the list.

0 Karma

chimell
Motivator

Please splunknewbie05 can you explicit well your problem ?What do you want to do ?

0 Karma

Yasaswy
Contributor

There are other limitations.. how long are your lines? Are you truncating them?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...