Calculate totals for disk space over time and show Top 20 disk hogs

[gnovak]

I posted this question in the past here: http://splunk-base.splunk.com/answers/35859/timechart-command-to-calculate-totals-for-7-days

However I feel that perhaps my question was buried.

I have a log indexed in splunk which contains the disk space usage for home directories. Below is an example of what the text in the log looks like.

1351424 /home/crazysue
36 /home/support
3001388 /home/hhogan
179624 /home/hpotter
2612 /home/lluther

I made a search to extract the size and username into fields and make a table of the results for me:

host="ynfs1" sourcetype=userdiskusage | rex field=_raw "(?[\d]+)\s*\/home\/(?\S+)" max_match=1000 | search NOT UserName="shares" | table UserName Space | sort -Space | head 20

I saved the search and gave it a name.

AT this point I decided to graph this search. Here's what I have for my dashboard:

<view template="dashboard.html">
  <label>YNFS1 Space Usage</label>
  <module name="AccountBar" layoutPanel="navigationHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>

  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>

  <module name="TimeRangePicker" layoutPanel="panel_row1_col1">
    <param name="searchWhenChanged">True</param>
    <param name="default">Last 24 hours</param>
      <module name="HiddenSavedSearch" layoutPanel="panel_row2_col1" group="Space Usage by user over a given time" autoRun="True">
        <param name="savedSearch">Andrew ynfs1 search</param>
        <module name="HiddenChartFormatter">
          <param name="charting.chart">bar</param>
          <param name="chart.stackMode">stacked</param>
          <param name="secondaryAxisTitle.text">Space</param>
          <module name="JobProgressIndicator"></module>
            <module name="FlashChart">
              <param name="width">100%</param>
              <param name="height">400px</param>
            </module>
          </module>
          <module name="ViewRedirectorLink">
            <param name="viewTarget">flashtimeline</param>
            <param name="popup">True</param>      
          </module> 
      </module>
     <module name="HiddenSavedSearch" layoutPanel="panel_row3_col1" group="Space Usage by user over a given time" autoRun="True">
       <param name="savedSearch">Andrew ynfs1 search</param>
       <module name="SimpleResultsTable"></module>
       <module name="ViewRedirectorLink">
         <param name="viewTarget">flashtimeline</param>
         <param name="popup">True</param>      
       </module> 
    </module>
  </module> 
</view>

This produces a nice bar graph and a small non clickable results table at the bottom. However the SimpleResultsTable at the bottom is only displaying 10 results and not 20.

Also when i change the time in the time range picker, the results are not what I am looking for. I tried using timechart by changing my search to this:

host="ynfs1" sourcetype=userdiskusage | rex field=_raw "(?[\d]+)\s*\/home\/(?\S+)" max_match=1000 | search NOT UserName="shares" | timechart sum(Space) by UserName | sort -Space | head 20

My results looked good again for 24 hours but as soon as I picked 7 days or more the results were not correct.

I'd like to always show the top 20 usernames who are taking up the most disk space for a given period of time. If I was to select 7 days from the time range picker, I'd like splunk to look at 7 days worth of data, calculate who the top 20 disk hogs were for a 7 day stretch and then present them to me in a chart and simple results table.

I tried putting in paginator for the table but it was complaining about how having any "events". I changed events to "results" and it still complained. Can anyone give me some advice on how to make this tables and charts work how I want? I've been reading so much my brain is so fried! 😞

[Ant1D · Accepted solution]

Hi,

In order to see 20 results in your table, inside your SimpleResultsTable module, you need to add the following:
<param name="count">20</param>

Maybe you should change your search by replacing your timechart command with the following:
stats sum(space) by username

From looking at your search, I would imagine that the UserName and Space fields should really be written in lowercase because your regular expressions <space> and <username> are lowercase.

[Ant1D]

Hi Gnovak, I am glad I was able to help 🙂

[gnovak]

This search eventually works out well: host="ynfs1" sourcetype=userdiskusage | rex field=_raw "(?\d+)\s*\/home\/(?\S+)" | search NOT UserName="shares" | eval SpaceGB = Space/1048576| timechart Max(SpaceGB) span=1d by UserName where max in top20. Thanks for your input!

[gnovak]

This command doesn't seem to be sorting the results in size order though. I tried even doing host="ynfs1" sourcetype=userdiskusage | rex field=_raw "(?\d+)\s*\/home\/(?\S+)" | search NOT UserName="shares" | timechart Max(Space) span=1d by UserName | sort -Max(Space) | head 20 or using the old command and putting | | sort -Max(Space) at the end of the old command and it didn't work it seems

[gnovak]

host="ynfs1" sourcetype=userdiskusage | rex field=_raw "(?\d+)\s*\/home\/(?\S+)" | search NOT UserName="shares" | timechart Max(Space) span=1d by UserName where max in top20

This seems to work a lot better it seems. although i liked the look of stats better...

[gnovak]

however i am noticing something is weird with the space. Splunk reports a user bwilson as having 2771158076k in their home dir over a 30 day period and being #2 on the list of the user having the most disk space. However bwilson only has 168K /home/bwilson/. Also bwilson no longer works here and hasn't for about 6 months. ????? perhaps multivalue fields are messing up the counts?

[gnovak]

I think that did it...numbers are looking good. I'm sort of surprised timechart didn't work w/ this situation.

[gnovak]

I think that I should change the simpleresulttable back to savedsearch...i will try that

[gnovak]

UserName Space
1 bruce 43326964
2 bruce 43237676
3 bruce 43237676
4 bruce 43236904
5 bruce 43236824
6 bruce 43229092
7 bruce 43227868
8

fhu
fhuang
file

41706920
108
4
9 fhu 41706908
10 fhu 41706908
11 fhu 41706908
12 fhu 41706904
13 fhu 41706900
14 fhu 41706896
15 fhu 41706896
16 lfan 39745952
17 lfan 39742372
18 lfan 39659892
19 lfan 39633424
20 lfan 39632344

[gnovak]

That worked. Also using stats in the search also works, however I still have the same problem where if I change the time range, my results are not what I'm expecting. For example, here are the results from the results table for 7 days:

[gnovak]

ah yeah...let me fix that...ugh alway some minor detail I miss!

[Ant1D]

That's wrong. It should be

[gnovak]

also this editing window is not showing that the search has the special characters in the search to replace the <>....

[gnovak]

[Ant1D]

When you changed it to HiddenSearch, did you change the param underneath it from savedSearch to search?

[gnovak]

Is this perhaps a bug?????

[gnovak]

I even tried changing HiddenSavedSearch to just HiddenSearch and putting the search directly into the dashboard XML. I still go thee same exact error. Misconfigured view 'My View' - Unknown parameter 'count' is defined for module HiddenSearch. Make sure the parameter is specified in HiddenSearch.conf.

[gnovak]

Also i might add when looked at this post and trying to edit it based on the uppercase issue you pointed out, in the editing window it IS uppercase throughout the entire search. However when viewing it here, it is not. May have something to do with the <>?

[gnovak]

When I try to add the Count feature for the results table, I got the error "Misconfigured view - Unknown parameter 'count' is defined for module HiddenSavedSearch. Make sure the parameter is specified in HiddenSavedSearch.conf.". I'm going ot look into this. I added it directly under the results table like this:

[gnovak]

oh actually i fixed that. I made them all uppercase. It was just a typo. Thanks for pointing that out. I'll try these suggestions and see if this resolves the issue.