Monitoring Splunk

Splunk Performance Problems in Dashboard

mpepem
New Member

Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.

I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.

I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.

I pasted the Base Search and one of the Panel configuration.

Thanks in advance,
Marcelo

<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query>
<earliest>$time_tok.earliest$</earliest>
<latest>$time_tok.latest$</latest>




<panel>
  <chart>
    <title>Http status by time</title>
    <search base="baseSearch" id="subsearch1">
      <query>
        timechart sum(count) by http_status
      </query>
    </search>
    <selection>
      <set token="selection_earliest_tok">$start$</set>
      <set token="selection_latest_tok">$end$</set>
    </selection>
    <drilldown>
      <set token="http_status_tok">$click.name2$</set>
    </drilldown>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="refresh.display">preview</option>
  </chart>
  <html>
          <input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/>

  <!-- Modal -->
  <div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
      <div class="modal-dialog modal-lg" role="document">
        <div class="modal-content">
          <div class="modal-header">
            <h4 class="modal-title" id="myModalLabel">
              Http status detail view</h4>
            </div>
            <div class="modal-body">
              <div id="table1"></div>

            </div>
            <div class="modal-footer">
              <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
            </div>
          </div>
        </div>
  </div>
</html>
</panel>
0 Karma

mpepem
New Member

Thanks ehudb for your answer, I'll try it and keep you update.

0 Karma

ehudb
Contributor

It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.

I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.

Maybe an accelerated report or a summary index could help more to achieve better performance.

0 Karma

mpepem
New Member

I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?

Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.

Thanks,
Marcelo

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...