Dashboards & Visualizations

How to troubleshoot why Splunk is not rendering all data on graphs in random dashboard panels in Splunk 6.1.2?

HattrickNZ
Motivator

I have a dashboard with 4 panels, 1 is shown below. my search is not that intense i don't think something like ... | timechart span=h max(c10099) by measObjLdn. the underlying rawdata is in hourly chunks. I am using the timepicker and selecting realtime 2 days ago to now

Image and video hosting by TinyPic

What is happening intermittantly but quite alot, is the graph is not showing all the data.
This happens on all panels and can vary which one it happens to on loading the dashboard.
I never see it when i run the search in its own search window.
The dashboard just sits there and I have no idea if it is finished or not.

Is this my computer/browser that does not have the resources? I am using firefox
Or is it the splunk server?

Can anyone advise how I can debug this?

EDIT1

I just tried it there again on Firefox and chrome. And it happened again.
1st time openened in firefox, it happened on 2 of the pannels
2nd time on firefox(after a refresh), it did not happen
3rd time on firefox, I am now getting waiting for data, this is another related question I have asked here.

1st time on chrome it happened on 2 panels
2nd time on chrome it happened on 1 panel
3rd time on chrome, I am now getting waiting for data on 2 panels

I am thinking it might be my PC resources,Browser resources, or splunk resources.
It may also be my maximum numbe of concurrent realtime searches has been reached, something i will look into.

below is my xml code

<dashboard>
  <label>MGW_CPU Occupation Rate Statistics</label>
  <description>MGW_CPU Occupation Rate Statistics</description>
  <row>
    <panel>
      <chart>
        <title>MGWSJH-FN-SN-BP</title>
        <searchString>index=core host="snzclakl598" elementType=MGW measInfoId=100666003 duration=PT3600S userLabel=MGWSJH | strcat userLabel "-" FN "-" SN "-" BP  measObjLdn_formatted |timechart  span=h max(c100991561) by measObjLdn_formatted</searchString>
        <earliestTime>rt-2d</earliestTime>
        <latestTime>rtnow</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisStart</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <title>top 10 for yesterday - MGWSJH-FN-SN-BP</title>
        <searchString>index=core host="snzclakl598" elementType=MGW measInfoId=100666003 duration=PT3600S userLabel=MGWSJH earliest=-1d@d latest=@d |strcat userLabel "-" FN "-" SN "-" BP  measObjLdn_formatted |stats max(c100991561) as max by measObjLdn_formatted | sort 10 -max</searchString>
        <earliestTime>-1d@d</earliestTime>
        <latestTime>@d</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">bar</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>MGWKPR</title>
        <searchString>index=core host="snzclakl598" elementType=MGW measInfoId=100666003 duration=PT3600S userLabel=MGWKPR | timechart  span=h max(c100991561) by measObjLdn</searchString>
        <earliestTime>rt-2d</earliestTime>
        <latestTime>rtnow</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.axisY.maximumNumber">100</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>MGW02KPR</title>
        <searchString>index=core host="snzclakl598" elementType=MGW measInfoId=100666003 duration=PT3600S userLabel=MGW02KPR | timechart  span=h max(c100991561) by measObjLdn</searchString>
        <earliestTime>rt-2d</earliestTime>
        <latestTime>rtnow</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>MGW01HAM</title>
        <searchString>index=core host="snzclakl598" elementType=MGW measInfoId=100666003 duration=PT3600S userLabel=MGW01HAM | timechart  span=h max(c100991561) by measObjLdn</searchString>
        <earliestTime>rt-2d</earliestTime>
        <latestTime>rtnow</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
  </row>
</dashboard>
0 Karma

juvetm
Communicator

Hi hattrickN,
i have two alternative ideals
i will like you to check on maximum number of concurrent realtime searches has been reached as you just said above or change the browers that you are using it may be the problem of the browsers for more information on the type of browsers to use in splunk i will like you to check on the documentation Splunk-6.1.1-Installation on Supported browsers that you need this may either be the solution to your problem.
for the xml code is okay.
thanks

0 Karma

juvetm
Communicator

Hi hattrickN,
i have use the search request that you gave me about with my own personal data from my machine it is working .This the request that i use it is working very well with personal data from my machine
.............| timechart span=h max(date_hour) count by sourcetype
Are you sure that you are have a problem with your Xml code i will like you to paste the xml code on answser so that i help you verify on the xml code .
Another alternative ideal maybe is the browser that you are using can not be support by splunk this may cause problem like that

thanks

0 Karma

HattrickNZ
Motivator

see my EDIT1 above. If you could check my XML that would be great as I do not think that is it.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...