Monitoring Splunk

Improve speed of "append"

therealdpk
Path Finder

I am trying to use append to combine the results of two searches, but when I use append the search time explodes. Individually, the two searches take a few seconds each. Together, the search takes over five minutes. The full search:

index=foo sourcetype=foo_st | append [search index=bar sourcetype=bar_st] | timechart count by index

over a 24 hour period.

Individually, the searches find a small set of results (336k and 42k respectively). Together, with the above append command, the Search Job Inspector reports that the append component has an input count of 57,000,000,000 and an output count of 58,000,000, which seems completely insane and arbitrary. timechart ends up with an input count of 58,000,042,474. I suspect that append is somehow multiplying the result rows or something.

FWIW, the "matching event" counter on the UI says 379,453 events. And this is with Splunk 4.3.2.

Has anyone else seen this issue and, if so, what did you do to resolve it? What can I view using the Splunk UI to try to diagnose the problem?

I've left this question open in case someone knows how to diagnose the 57-58 billion result problem, or how to use the UI to diagnose the problem.

Tags (2)

chris
Motivator

You could just try to avoid using append and the subsearch by doing trying the following:

 (index=foo sourcetype=foo_st) OR ( index=bar  sourcetype=bar_st) | timechart count by index

I haven't run accross the exploding number of results though.

Chris

therealdpk
Path Finder

Ahh, excellent. That makes sense.

FWIW, I got the idea to use append from http://blogs.splunk.com/2012/02/19/compare-two-time-ranges-in-one-report/ , where the same search is performed against two dates in order to compare the results. I was using it to compare Monday to last Monday and two Mondays ago and so on, and had the same multiple-billions result. I gave up on a fix for that, but then I found this much simpler way to demonstrate the problem.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...