I want to deal with big data uising Splunk.
To reduce time for searching data, I want to select specific data from original data, pre-process it, and save the output data as csv format. Also I want to make dashboard using out data.
Please let me know about example of query or helpful article.
Hi @cross521,
your question id very vague.
Anyway, in general you have to index data in Splunk to analyze and use them.
The steps to do this are (in general) these:
To save the search results in csv forma theres the outputcsv command (https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Outputcsv), but anyway you have to index data in Splunk.
If you want to pre-process them, you have to use a script (done in the language you like) to prepare data before ingestion but I'am not an expert in scripting and this isn't a Splunk issue so I cannot help you.
Ciao.
Giuseppe
Thank you for answer.
Here is an example where I would like to process data:
1. There are 3 years of data accumulated every 2 seconds.
2. The value of a particular point is always 0 and only becomes 1 or more when a failure occurs.
3. I would like to retrieve the records of any failures over a period of 3 years, i.e. spikes in the data, and save them as csv format.
Can you help me one more time?
Hi @cross521,
yes the Use Case you describe it's possible and easy to create.
I suppose that you already ingested data and stored them in an index using a sourcetype (item 1).
I suppose also that you already extracted fields associated to that sourcetype (item 2), if not please share a sample of your logs.
For the item 3, I need to know how to identify failures, in the following example I use the rule that if there's a failure, "status" field has the value "failure", and you have to define the fields to add in the results
A the end, you can download the csv from the GUI or use the outputcsv command (at the end of the search) that saves the csv in $SPLUNK_HOME/var/run/splunk/csv, it isn't possible to use a different location for te saving folder, if you want a different one, you have to create a custom script to move this file.
index=your_index status =failure
| table _time host field1 field2
| outputcsv your_csv.cv
if there are different conditions you can modify my search.
Ciao.
Giuseppe