Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.
I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.
I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.
I pasted the Base Search and one of the Panel configuration.
Thanks in advance,
Marcelo
<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query>
<earliest>$time_tok.earliest$</earliest>
<latest>$time_tok.latest$</latest>
<panel>
<chart>
<title>Http status by time</title>
<search base="baseSearch" id="subsearch1">
<query>
timechart sum(count) by http_status
</query>
</search>
<selection>
<set token="selection_earliest_tok">$start$</set>
<set token="selection_latest_tok">$end$</set>
</selection>
<drilldown>
<set token="http_status_tok">$click.name2$</set>
</drilldown>
<option name="charting.chart.stackMode">stacked</option>
<option name="refresh.display">preview</option>
</chart>
<html>
<input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/>
<!-- Modal -->
<div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
<div class="modal-dialog modal-lg" role="document">
<div class="modal-content">
<div class="modal-header">
<h4 class="modal-title" id="myModalLabel">
Http status detail view</h4>
</div>
<div class="modal-body">
<div id="table1"></div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
</html>
</panel>
Thanks ehudb for your answer, I'll try it and keep you update.
It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.
I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.
Maybe an accelerated report or a summary index could help more to achieve better performance.
I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?
Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.
Thanks,
Marcelo