Getting Data In

How to export > 10000 Events to a .csv via CLI with good performance?

tpaulsen
Contributor

Hello,

in Splunk 3 we were exporting during night time via cronjob 1-hour chunks of data from the previous day via CLI. All together more than 800000 events of data. We need the export because Splunk can´t visualize certain aspects of the data in a report.

In Splunk 3 each 1-hour chunk export took about 8 - 10 minutes to export.

In Splunk 4 the same 1-hour export takes literally HOURS to export, though one export is only about 120 - 250,000 events. Within the Splunk GUI the search takes about 1 minute to run.

Here´s the CLI command for Splunk 4:

./splunk search 'index="idx_prod_online" host="blade504" source="/var/opt/noa/prod/current/online/log/online1.http.log" | regex _raw!="^#"' -earliest_time -1h@h -latest_time @h -maxout 0 -auth username:password >> /tmp/LogEventsRaw/splunkexport.log

What can we do to speed up the command?

The |outputcsv option from here http://blogs.splunk.com/2009/08/07/help-i-cant-export-more-than-10000-events/ doesn´t work in our case, since the diskspace for the searchhead is very limited.

Please help.

Tags (4)
0 Karma

tpaulsen
Contributor

We are importing the splunk-export into "CIC tool" a special "Intershop" tool to visualize certain aspects of our business. We already consulted the Splunk support about it, and they admitted that Splunk can´t do this special kind of visualization.

The problem is simply that with Splunk 3 the bulk export worked fine, but with Splunk 4 we have problems to get the data out.

0 Karma

Johnvey
Contributor

Would you mind elaborating on what kind of reporting you are attempting to do? In general, bulk exporting raw events from Splunk is a method of last resort.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...