Using Splunk central repository (the indexed DB) to costum needs

[avital]

Hi,

We would like to leverage Splunk repository (the indexed DB?) for bringing parts of the data collected by Splunk to our processing engine to get statistics and trends.

Is it possible to get data collected by Splunk as files or streams and leverage them for our own custom purposes?

Thanks,
Avital

[barakreeves]

There are a few approaches you can take.

1- Try to do in Splunk what your other distributed system is doing; such search commands such as 'transaction' and others can follow an ID from start to finish
2- Run a search and export the results to JSON, CSV, raw text, etc. and import into your system
3- Take advantage of the SDK/API to pull the data out and send the data to wherever you want it to go (including processing)

Try option 1 fist. The search language provided by Splunk is quite rich and powerful.

Here are a few links:
SDK: http://dev.splunk.com/view/sdks/SP-CAAADP7
Create a save search and export via REST: http://docs.splunk.com/Documentation/Splunk/5.0.2/RESTAPI/RESTsearch
Identify and group events based upon transaction: http://docs.splunk.com/Documentation/Splunk/5.0.2/Search/Identifyandgroupeventsintotransactions

Hope this helps.

[avital]

Thanks for your answer!

We have a system designed for distributed processing of data that from which you can get valuable business intelligence.
One simple scenario is to track our data flow according a specific ID. Meaning to find all stations this ID visited and also to get statistics on which station most of the data is stuck. We have this ability in our own engine and we would like to pass all records with a specific ID(s) to it for processing. The engine can get the data as flat files or as streams.

Could you please elaborate on the different approaches to get data out of Splunk?

[Ayn]

Could you explain a bit more what you're trying to do? You can definitely get data out of Splunk and send it to third party systems - which approach you should take depends very much on what your use-case is.