Getting Data In

Why does Splunk Web limit the total of number of rows in a result that can be exported to a CSV file?

reed_kelly
Contributor

Does anyone know the technical or another reason that the Splunk Web interface limits the total number of rows of the result set that can be exported to a CSV file? I know that unlimited sizes can be achieved by the REST API or SDK, but I have users who want to extract huge amounts of data. Since they are not handy with the command line, I end up using curl on their behalf.

Tags (3)
0 Karma
1 Solution

abhijit_mhatre
Path Finder

The default limit for csv export from a saved search is 10000. However, if you want to change this:

1) You can go to savedsearches.conf & change it.

savedsearches.conf
action.email.maxresults = <integer>
* Set the maximum number of results which you want to export
* Currently defaults to 10000

2) You can export data using outputlookup command which will create a csv file in Splunk. You can export unlimited number of results from the lookup file now. Use this at the end of your query.

For example:
index=abc | search host=* | stats count by b | outputlookup hosts.csv

This will create a lookup file hosts.csv & you can export it by running | inputlookup hosts.csv

Let me know if it helps.

View solution in original post

0 Karma

nabeel652
Builder

As far as I have seen it doesn't limit the rows, or maybe I am getting you wrong. However, I think the problem may be in the roles. If you assign a user role to someone his search size is limited to 100MB and therefore it stops once the search occupies more than 100MB. Change "Limit Total Jobs Disk Quota" in the roles settings and this may fix the problem

0 Karma

abhijit_mhatre
Path Finder

The default limit for csv export from a saved search is 10000. However, if you want to change this:

1) You can go to savedsearches.conf & change it.

savedsearches.conf
action.email.maxresults = <integer>
* Set the maximum number of results which you want to export
* Currently defaults to 10000

2) You can export data using outputlookup command which will create a csv file in Splunk. You can export unlimited number of results from the lookup file now. Use this at the end of your query.

For example:
index=abc | search host=* | stats count by b | outputlookup hosts.csv

This will create a lookup file hosts.csv & you can export it by running | inputlookup hosts.csv

Let me know if it helps.

0 Karma

reed_kelly
Contributor

I should have clarified. I am talking about the export to file option on the main search interface. This allows the end user to export directly to their desktop. I suppose exporting to a lookup file would allow this by giving the users access to the Lookup Editor, but that could also increase our search bundle and cause issues with distributed search bundle replication.

There is a maxresultrows=nnn in limits.conf, but the documentation explicitly says not to set this higher than 50000.

My question is Why? And if there is a technical reason, then is there an app that would run the search in the REST API and then let the user download the result >100000000 rows and/or >100GB, etc...

0 Karma

abhijit_mhatre
Path Finder

There is app "Splunk for Excel Export" which will help you to download 1 million events per workbook. Please check the app compatibility with your Splunk version.

App link : https://splunkbase.splunk.com/app/760/#/details
Splunk for Excel Export

Thanks & Regards,
Abhijit Mhatre

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...