If I have a number of storage devices, each with a number of volumes, and every hour I am querying the used capacity of each volume, and I want to graph the total used capacity for all storage devices by day over the last two weeks, how do I construct a search string for that?
_time=1am filer=bob volume=source usedGB=35
_time=1am filer=jill volume=build usedGB=10
_time=1am filer=jill volume=test usedGB=50
_time=2am filer=bob volume=source usedGB=35
_time=2am filer=jill volume=build usedGB=11
_time=2am filer=jill volume=test usedGB=49
To get the current usage I do something like
sourcetype=<blah> | dedup filer, volume | stats sum(usedGB)
But if I want to chart over time, I get confused. I could do something like
sourcetype=<blah> | timechart sum(usedGB)
but if the timechart span is larger than the filer polling period, I'm going to end up adding the volume sizes into the total multiple times for each time bucket, which isn't correct.
Then I wanted to do something like
sourcetype=<blah> | timechart last(usedGB) as usedGB by filer, volume | timechart sum(usedGB)
but you can't timechart by two fields.
Then I tried to construct a single unique field
sourcetype=<blah> | eval unique=filer.":".volume | timechart last(usedGB) as usedGB by unique| timechart sum(usedGB)
which seemed both clunky and promising, but didn't generate any data. I don't think the nested timecharts are doing what I want.
Finally I started brainstorming around combining bucket and stats last(), but had no luck.
This feels like a problem with an obvious answer that I'm just not seeing. Any help appreciated.
Lee
I came back to needing to do this again, dug around, and found a trivial solution (https://answers.splunk.com/answers/47101/complete-a-timechart-with-a-total-column.html) -- do the timechart, then just use | addtotals
to get what I needed. Sigh.
Based on ramdaspr's comment, I fumbled my way to this solution:
sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time
This solves my immediate need.
Wouldn't per_day solve the problem? It would total the storage for a day for graphing.
timechart per_day(usedGB) usenull=f
As I understand it, per_day()
is kind of like span=1d
-- if I poll a volume 24 times in 1 day, and have 24 corresponding events that say volume=foo usedGB=35
, per_day(usedGB)
isn't going to return '35', it will return 35*24, which isn't what I need.
Conceptually what I want is "for each day, do a dedup filer,volume | stats sum(usedGB)
'.
After your comment I spent some time digging into timechart and per_day documentation, and then on bucket and stats. I think I have arrived at a working (but probably not optimal) solution, which looks like
sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time
The 'bucket' forces all the events to have discrete daily times, the stats max serves to deduplicate identical filer/volume events that occur in the same time bucket, and stats sum gets me the total I want per day.
I end up with nested "stats | stats
" where before I was attempting "timechart | timechart
". It looks like the timechart wanted to convert all the component fields into columns (and beyond a certain number of columns it starts lumping everything into 'OTHER'), but stats puts everything into discrete events, which ended up being a little more flexible. One of the takeaways for me is that "timechart foo" is not identical to/short for "stats foo by _time".
per_day() didn't solve my issue, but started me towards a resolution - thanks!