Splunk Search

How to graph a (non-cumulative) total of fields?

lee_melvin
Path Finder

If I have a number of storage devices, each with a number of volumes, and every hour I am querying the used capacity of each volume, and I want to graph the total used capacity for all storage devices by day over the last two weeks, how do I construct a search string for that?

_time=1am filer=bob volume=source usedGB=35
_time=1am filer=jill volume=build usedGB=10
_time=1am filer=jill volume=test usedGB=50
_time=2am filer=bob volume=source usedGB=35
_time=2am filer=jill volume=build usedGB=11
_time=2am filer=jill volume=test usedGB=49

To get the current usage I do something like

sourcetype=<blah> | dedup filer, volume | stats sum(usedGB)

But if I want to chart over time, I get confused. I could do something like

sourcetype=<blah> | timechart sum(usedGB)

but if the timechart span is larger than the filer polling period, I'm going to end up adding the volume sizes into the total multiple times for each time bucket, which isn't correct.

Then I wanted to do something like

sourcetype=<blah> | timechart last(usedGB) as usedGB by filer, volume | timechart sum(usedGB)

but you can't timechart by two fields.

Then I tried to construct a single unique field

sourcetype=<blah> | eval unique=filer.":".volume | timechart last(usedGB) as usedGB by unique| timechart sum(usedGB)

which seemed both clunky and promising, but didn't generate any data. I don't think the nested timecharts are doing what I want.

Finally I started brainstorming around combining bucket and stats last(), but had no luck.

This feels like a problem with an obvious answer that I'm just not seeing. Any help appreciated.

Lee

Tags (3)
0 Karma

lee_melvin
Path Finder

I came back to needing to do this again, dug around, and found a trivial solution (https://answers.splunk.com/answers/47101/complete-a-timechart-with-a-total-column.html) -- do the timechart, then just use | addtotals to get what I needed. Sigh.

0 Karma

lee_melvin
Path Finder

Based on ramdaspr's comment, I fumbled my way to this solution:

sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time

This solves my immediate need.

0 Karma

ramdaspr
Contributor

Wouldn't per_day solve the problem? It would total the storage for a day for graphing.

timechart per_day(usedGB) usenull=f
0 Karma

lee_melvin
Path Finder

As I understand it, per_day() is kind of like span=1d -- if I poll a volume 24 times in 1 day, and have 24 corresponding events that say volume=foo usedGB=35, per_day(usedGB) isn't going to return '35', it will return 35*24, which isn't what I need.

Conceptually what I want is "for each day, do a dedup filer,volume | stats sum(usedGB)'.

After your comment I spent some time digging into timechart and per_day documentation, and then on bucket and stats. I think I have arrived at a working (but probably not optimal) solution, which looks like

sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time

The 'bucket' forces all the events to have discrete daily times, the stats max serves to deduplicate identical filer/volume events that occur in the same time bucket, and stats sum gets me the total I want per day.

I end up with nested "stats | stats" where before I was attempting "timechart | timechart". It looks like the timechart wanted to convert all the component fields into columns (and beyond a certain number of columns it starts lumping everything into 'OTHER'), but stats puts everything into discrete events, which ended up being a little more flexible. One of the takeaways for me is that "timechart foo" is not identical to/short for "stats foo by _time".

per_day() didn't solve my issue, but started me towards a resolution - thanks!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...