Splunk Search

What Hunk search language command generates a map reduce job in hadoop cluster?

splunkIT
Splunk Employee
Splunk Employee

What search commands in Hunk kick off reducers vs. trying to collection data via a streaming session? I ask, since I looked at the search log for a query of search index=vir-test minutesago=60.

06-17-2015 21:58:42.616 INFO ERP.isa-prod - SplunkMR$SearchHandler - Reduce search: null 
06-17-2015 21:58:42.617 INFO ERP.isa-prod - SplunkMR$SearchHandler - Search mode: stream 
06-17-2015 21:58:42.617 INFO ERP.isa-prod - SplunkMR$SearchHandler - setting requiredFields=* 

Based on the data, it appears that a streaming job was kicked off (not too fast). I have looked at
http://docs.splunk.com/Documentation/Hunk/6.2.3/Hunk/distributableandnondistributablesearchcommands , but it isn't clear as to which commands kick of a reducer.

splunkIT
Splunk Employee
Splunk Employee

A map-only MR job will be submitted to Hadoop when the search a) contains any reporting / transforming commands (assuming verbose mode is not in use) or b) the search contains filtering predicates

http://docs.splunk.com/Documentation/Splunk/6.2.3/Search/Aboutreportingcommands

For example:

index=vir-test <<-- that's just streaming data

index=vir-test error OR warn <<-- this should kick off a MR job

index=vir-test | stats count by my_field <<-- this should kick off a MR job

ddrillic
Ultra Champion
index=vir-test error OR warn

is intriguing.

I tried -

index=xxxx  source = "*part-m-00078*" OR source = "*part-m-00079*"

I see the MapR job running, but the query runs at a very slow speed.
Weird thing.

0 Karma

tsunamii
Path Finder

RE: A map-only MR job will be submitted to Hadoop...
So Hunk will never kick-off a reducer job on the hadoop side?

0 Karma

pburgu_splunk
Splunk Employee
Splunk Employee

That's correct. The reduce function happens on Hunk search head.

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...