Monitoring Splunk

Can searches with joins be accelerated ?

nibinabr
Communicator

I was trying to accelerate a search with join command(trying to accelerate two searches). When I look at the job inspector, it says the search is using summaries for the search. But I think only the first search is accelerated and the second search is executes normally because the job inspector doesn't show the subsearch using any summaries.

I also tried accelerating the searches separately, with the expectation that each search in the combined search (search with JOIN) would match with the individual accelerated search and use the summary for searching. However the combined search only uses the summary created by the first search.

Lets say if this is the structure of the search

index=myindex sourcetype=sourcetype1 | eval something =1| stats count(something) as count by somethingelse |JOIN somethingelse [search index=myindex sourcetype=sourcetype2|stats max(splunk_is_awesome) as max by somethingelse] | table somethingelse max count

In the above search

index=myindex sourcetype=sourcetype1 | eval something =1| stats count(something) as count by somethingelse

gets accelerated but the subsearch doesn't get accelerated

index=myindex sourcetype=sourcetype2|stats max(splunk_is_awesome) as max by somethingelse

Any thoughts ??

Tags (1)

acharlieh
Influencer

I know you're anonymizing your search here, but would restructuring so you don't need the join and accelerating the one combined search be an option?

e.g. given your search, what about this one?

index=myindex (sourcetype=sourcetype1 OR sourcetype=sourcetype2) 
| eval something = if(sourcetype=="sourcetype1",1,null())
| eval splunk_is_awesome = if(sourcetype=="sourcetype2",splunk_is_awesome,null())
| stats count(something) as count, max(splunk_is_awesome) as max by somethingelse

acharlieh
Influencer

Something I realized... assuming you only in fact only looking at sourcetypes and indexes, you could actually do a metasearch to get your results. (it wouldn't work with anything parsed at search time, but if you're only using default index-time fields like index, _time, sourcetype, source it's fair game)

0 Karma

nibinabr
Communicator

That is an option and I'm trying it out now. However I was expecting that there would be a way to accelerate searches with JOINS.

0 Karma

jacobwilkins
Communicator

Friends don't let friends use JOIN.

martin_mueller
SplunkTrust
SplunkTrust

Technically, some searches with join such as yours can be accelerated. However, that doesn't happen in the way you were looking for.

Splunk looks at your search and scans it for the first reporting command, that'd be your stats count(something) before the join. This prefix of your search is sent off for acceleration. Whatever comes after that is irrelevant to the accelerator, any search beginning with this prefix is now using this acceleration.

In order to let your subsearch be accelerated as well you would need to save the subsearch as a report and accelerate that. To avoid code duplication you can use the savedsearch command to load the saved subsearch from the saved main search: http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/Savedsearch

martin_mueller
SplunkTrust
SplunkTrust

To tell if it is accelerated, are you looking at the debug output at the top of the job inspector? That might just apply to the main search. Is there a huge time difference to notice?

0 Karma

nibinabr
Communicator

I was only looking at the job inspector. Since there is a section in the debug output that starts with "[subsearch]" which didn't show anything like "using summaries for search", my assumption is that the subsearch is not using summaries for search.

The search is certainly faster than the normal search as the first search uses summaries for searching. I can do a quick test to see if the subsearch is really using the summaries by removing the summaries for the subsearch and check if search becomes slower.

0 Karma

nibinabr
Communicator

I did a quick test by slightly modifying my subsearch so that it doesn't use the summary (if it is using) anymore.

Comparing the runduration field in the job inspector the modified search (which doesn't use the summary for the subsearch) took almost the same time as the search that used the summary for subsearch. So the subsearch was not using the summaries for acceleration.

0 Karma

nibinabr
Communicator

>In order to let your subsearch be accelerated as well you would need to save the subsearch as a report and accelerate that.

I think I wasn't clear. I also tried accelerating each search separately(main search and subsearch). However when I try to run the whole search only the main search is accelerated (uses the summary created by the accelerated main search) and not the sub search (doesn't use the summary created by the accelerated subsearch).

When both the searches are executed individually, it uses the summaries for the search.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...