Splunk Search

Reduce number of searches

angersleek
Path Finder

I have about 20 searches going on in my dashboard which seems to have really slowed down the dashboard.
I am trying to make the search more efficient.

These are a few current searches and results are each displayed as a single number chart in each panel.

service=Service1 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service2 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service3 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service4 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service5 locale=$locale_token$ | chart avg(TIME_TAKEN)

Another search which is slightly different on the dashboard.

service=Service1 NOT "API=*_CHECK" locale=$locale_token$ (INDEX=0 OR REQ_SIZE=1) | timechart span=5m avg(TIME_TAKEN) as average | fillnull | sort average

Found that I could use a base search and go on with further searches for each panel using base thus tried following which is placed at the top of the dashboard after the dropdown field sets (the token values are coming from these field sets):

<search id="base">
    <query>service=Service* locale=*</query>
    <earliest>$datetime_field.earliest$</earliest>
    <latest>$datetime_field.latest$</latest>
 </search>

One of the panel search currently looks like the following.

<search>
    <query>service=Service1 locale=$locale_token$ | chart avg(TIME_TAKEN)</query>
    <earliest>$datetime_field.earliest$</earliest>
    <latest>$datetime_field.latest$</latest>
    <sampleRatio>1</sampleRatio>
    <refresh>10m</refresh>
    <refreshType>delay</refreshType>
</search>

But I am getting an error "unknown node" from (earliest - refreshtype) when I add base to search.
Thus I have resorted to using the following search now:

<search base="base">
    <query>service=Service1 | chart avg(TIME_TAKEN)</query>
</search>

When I save this dashboard, I am getting the following error:

 Search Factory: Unknown search command 'service'.

Seems base search is restricted in what I can search thus looking for a way to make my dashboard more efficient instead of making 20+ searches when the search is somewhat similar with slight variations.

Tags (1)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

Okay, the first chunk is easy. Do this as your base search...

<search id="base1">
   <query>
     index=foo service=* locale=$locale_token$ 
     | stats avg(TIME_TAKEN) as avg_Time_Taken by service
   </query>
   <earliest>$datetime_field.earliest$</earliest>
   <latest>$datetime_field.latest$</latest>
 </search>

...then use this as post processing

 <search base="base1">
    <query>| where service="Service1" </query>
 </search>

Notes -

1) The complaint was about the naked service=Service1 in the post processing, with no verb.

2) When doing base + postprocessing, you should aim to chew up the data as much as practical in the base query. Otherwise, you are transferring all that base query data around the electronic universe and not saving anything.

3) Depending on your use case, it might be useful to have the base search by service locale and have the postprocessing searches filter both the service and the locale. You could try it both ways and see which one provided the better user experience.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

Okay, the first chunk is easy. Do this as your base search...

<search id="base1">
   <query>
     index=foo service=* locale=$locale_token$ 
     | stats avg(TIME_TAKEN) as avg_Time_Taken by service
   </query>
   <earliest>$datetime_field.earliest$</earliest>
   <latest>$datetime_field.latest$</latest>
 </search>

...then use this as post processing

 <search base="base1">
    <query>| where service="Service1" </query>
 </search>

Notes -

1) The complaint was about the naked service=Service1 in the post processing, with no verb.

2) When doing base + postprocessing, you should aim to chew up the data as much as practical in the base query. Otherwise, you are transferring all that base query data around the electronic universe and not saving anything.

3) Depending on your use case, it might be useful to have the base search by service locale and have the postprocessing searches filter both the service and the locale. You could try it both ways and see which one provided the better user experience.

DalJeanis
SplunkTrust
SplunkTrust

Your second search I'm not sure about. Since you are sorting ascending by average after using timechart, I don't understand the data being presented (neither what or why). Presumably you could do something like this...

 <search id="base2">
   <query>
    index=foo service=* NOT "API=*_CHECK" locale=$locale_token$ (INDEX=0 OR REQ_SIZE=1)
    | bin _time span=5m 
    | stats avg(TIME_TAKEN) as avg_Time_Taken by service
    | sort 0  service avg_Time_Taken 
  </query>
  <earliest>$datetime_field.earliest$</earliest>
  <latest>$datetime_field.latest$</latest>
</search>

..and your post processing would be something like this...

<search base="base2">
   <query>| where service="Service1" | head 100 </query>
</search>

The head command is to cut off at 100 records, the way that sort defaults to do.

0 Karma

angersleek
Path Finder

The second search, I am trying to get average values for every 5 mins, and get the the highest value from these values. I am presenting this (highest 5 min average) value on a single digit chart. Question about your suggested query. Is the "index=foo" needed?

0 Karma

angersleek
Path Finder

Hm using your first query returns no results with no errors.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

If you want the highest values, then you need to sort descending.

     | sort 0  service - avg_Time_Taken 

Always tell splunk which indexes it is supposed to look in. Otherwise, it has to look at literally everything. "foo" is a nonsense value that we use to demo things. "foo" "bar" and "baz" are the first three nonsense values we usually use, although people also type yourindexnamehere and things like that.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...