Splunk Search

How can I use splunk like spark or storm ?

perlish
Communicator

I want to analysis 100k targets using the same search command in the realtime,splunk will create 100k search jobs in the same time.
I thought the mission is impossilble.
How can I use splunk like spark or storm ?
Thank you very much!

Tags (3)
0 Karma

mikebd
Path Finder

Could you add a bit more context? I'm specifically wondering about your use of the word target and whether that is implying a lot of context that I am missing. I've strictly worked with Splunk in the last couple of years and am at a disadvantage in recognizing some of the terminology assumed by users of the other platforms.

Splunk can create a single search job that returns many events across multiple indexes. When it is reasonable to create custom fields at index time, these are very efficient to search at very high scale but come with lots of advisory notices since poor decisions on deploying these can be very troublesome to recover from.

The most efficient such statistical search would be based on tstats, which works great for statistics assuming you do not require the search output to contain the full event text but just a summary of counts of events matching various conditions and the like. tstats can not be run using a Splunk real-time search so you would likely use a scheduled job with a historic search across a recent time frame that is within your tolerance level for processing latency.

Configure index time field extractions: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction
tstats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats

Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...