Splunk Search

How can I use splunk like spark or storm ?

perlish
Communicator

I want to analysis 100k targets using the same search command in the realtime,splunk will create 100k search jobs in the same time.
I thought the mission is impossilble.
How can I use splunk like spark or storm ?
Thank you very much!

Tags (3)
0 Karma

mikebd
Path Finder

Could you add a bit more context? I'm specifically wondering about your use of the word target and whether that is implying a lot of context that I am missing. I've strictly worked with Splunk in the last couple of years and am at a disadvantage in recognizing some of the terminology assumed by users of the other platforms.

Splunk can create a single search job that returns many events across multiple indexes. When it is reasonable to create custom fields at index time, these are very efficient to search at very high scale but come with lots of advisory notices since poor decisions on deploying these can be very troublesome to recover from.

The most efficient such statistical search would be based on tstats, which works great for statistics assuming you do not require the search output to contain the full event text but just a summary of counts of events matching various conditions and the like. tstats can not be run using a Splunk real-time search so you would likely use a scheduled job with a historic search across a recent time frame that is within your tolerance level for processing latency.

Configure index time field extractions: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction
tstats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...