Splunk Search

How can I optimize the performance of my search?

m_vivek
Path Finder

I am doing a simple search:

index=pqr host=xyz* NOT TYPE="*ABCDE*" | fields X, Y | timechart limit=0 span=10m count, avg(X) by Y

on a two week period.
The search has been running for more than two hours now and is still only at about 60%. Is there a way to make it run faster?

Also sometimes I get

'search has been remotely
cancelled or expired'

What does that mean? how do I prevent a search from getting cancelled?

0 Karma

woodcock
Esteemed Legend

To answer your question directly, you can prevent it from happening by EITHER speeding up your search OR clicking on the Jobs menu click Send Job to Background and your job will NOT timeout and you can opt to receive an email when the job completes.

muebel
SplunkTrust
SplunkTrust

This looks like a good case for a summary index. Populate the index every 10 minutes with your stats, and then you search the summaries when you want to build a trend.

0 Karma

m_vivek
Path Finder

hi muebel, could you elaborate some. i'm pretty new to splunk .

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Some reading about summary indexing available here.
http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Usesummaryindexing

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try

index=pqr host=xyz* TYPE!="*ABCDE*" | table _time, X, Y | timechart limit=0 span=10m count, avg(X) by Y

Also, if the data span is to a limited sourcetypes, please add them as well as filter.

m_vivek
Path Finder

when i tried above, its running but all fields are staying blank. So i replaced 'table _time' with 'fields' and it's working better.
Is there anyway I can measure/compare the performance of the two search queries in terms of total approx. run time?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

YOu can check the job inspector (there is a job menu just below the search bar towards the right side) which shows the amount of time taken by the search.

0 Karma

walkerhound
Path Finder

According to this link: http://docs.splunk.com/Documentation/Splunk/6.3.0/Search/Writebettersearches
you should avoid using NOT wherever possible

MuS
SplunkTrust
SplunkTrust

Also, do not use * searches. Since Splunk will have to read in your example, all events containing host fields from disk and compare if the values starts with xyz.

Take a good lock at the slide here http://conf.splunk.com/session/2015/conf2015_JHarty_DuncanTurnbull_Splunk_UsingSplunkSearchLanguage_... to learn more about Search Efficiency Optimisation.

cheers, MuS

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...