Monitoring Splunk

deployment monitor MB received SLOOOOOOWWWW

mcbradford
Contributor

If I run the All Sourcetypes dashboard, the MB received panel for the past 24 hours, the panel takes just over nine minutes to complete. I studied the search and it is made up of three macros that form this search:

index="_internal" source="*license_usage.lo*" type!=*Summary | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server source | bin _time span=10m | stats sum(bytes) as bytes max(lastReceived) as lastReceived by mysourcetype _time pool host | eval kb = bytes/1024 | eval mb = kb/1024 |timechart minspan=10m bins=200 sum(mb) as mbytes by mysourcetype

If I run this search manually the results are returned within 1 minute.

Any idea about what is going on????

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

The macro powering the search in the "MB Recevied" Panel is:

sourcetype_metrics_timechart

You can see what is consuming all the time in your search by inspecting the job. Here is what you'll want to do:

  1. Reload the "All Sourcetypes" panel and go on a small coffee break ( not too long of a break or the job details will get cleaned up).

  2. Click on the "Jobs" link in the upper right corner

  3. Click "Inspect" on the entry for "sourcetype_metrics_timechart"

It sounds like the search might not be using Report Acceleration correctly. Do you see a message indicating that sumaries are being used?

It would look something like the following:

DEBUG: [my.host.name] Using summaries for search, 

If Search summaries are being used, then you may have run into a bug in the core product with search acceleration. In that case, I would recommend opening a support case.

araitz
Splunk Employee
Splunk Employee

Not enough information. Can you open a support case please?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...