How to find delta of cumulative values by day?

[campbellwarren]

I am storing some performance counters in splunk. The data gets written one or more times per day (though not necessarily at the exactly the same time each day) and the totals are cumulative for multiple days.

For example, the data may include "total_worker_time" for various stored procedures which we can assume will be an ever-increasing value. I would like to chart the change in this value over time per stored procedure to show how much work time was spent on a particular day for each stored procedure. For each day, I want to find the maximum value by subtracting the maximum previous day's value to find the delta.

I know I can do:

 <...mysearch...>| timechart max(total_worker_time)  span=1d  by procname 

to return the cumulative total for each procname like this:

_time   Proc1   Proc2
2018-01-16  29710092875 4354587351
2018-01-17  54315798685 5977664529
2018-01-18  78055137053 7739773570

but I'd rather return the difference like this:

DATE    Proc1Delta  Proc2Delta
1/16/2018   no prev value   no prev value
1/17/2018   24605705810 1623077178
1/18/2018   23739338368 1762109041

Is this possible?

[elliotproebstel · Accepted solution]

Well, drat. If you only have two procname values, then it's not too bad to do it like this, I guess:

<...mysearch...>
| timechart max(total_worker_time) span=1d procname
| delta Proc1 AS Proc1Delta p=1
| delta Proc2 AS Proc2Delta p=1

Does that do it?

[elliotproebstel]

Looks like something the Splunk command delta would be good for doing. How about this:

<...mysearch...>
| timechart max(total_worker_time)  span=1d  by procname 
| foreach Proc* 
  [ |delta <<FIELD>> AS <<FIELD>>Delta p=1 ]

I don't have access to my system at the moment to test this, so I'm air-coding. Let me know if this works; if not, I'll try to help tweak it.

[campbellwarren]

Thanks for your help. When I run it exactly as you have above I get an error:

Error in 'foreach' command: Search pipeline may not contain non-streaming commands

[campbellwarren]

That will do it! Would be nice to be able to pivot over the names, but I can work with this. Thanks again!

[elliotproebstel]

Yeah, I didn't think about the fact that delta isn't a steaming command. It looks so much nicer all bundled up inside a foreach loop! 🙂