Splunk Search

How does splunk determine timechart bucketing options automatically?

anjafischer
Path Finder

Hello,

I am trying to build a dashboard that shows a chart of cpu consumption over time. This alone, is quite easy, however, I also need to show peak lines in the same chart...

I get a log file per program that contains the program's cpu consumption within a 5 minute interval. So, If I chart cpu consumption with a span=5m everything is fine. I simply chart the metric and run a subsearch that determines the maximum cpu consumption within a 5 minute interval of the - let's say - last 7 days.

Like this:
system=cics dc=RZ2 type=trxPerf | timechart minspan=5m sum(cputot) as "CPU Consumption" | eval MaxConsumptionThisWeek = [search system=cics dc=RZ2 type=trxPerf daysago=7 | bucket _time span=5m | stats sum(cputot) as SumCPU by _time | stats max(SumCPU) as MaxCPU | fields + MaxCPU | rename MaxCPU as search]

I can add more subsearches to get the monthly peak, the half-year-peak and so on.
I do, however have a problem when th euser selects a time range of 4 weeks for this chart. Then splunk will not be able to chart the data in 5 minute intervals. It will aggregate data and merge it into bigger chunks, which means a single datapoint for the main search will represent a bigger interval, probably a whole day.

This means, that I would calculate the peaks based on day-buckets as well. Unfortunately I do not know what splunks charting span-length is...

Does anyone have any idea on how to solve this problem?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Concerning your question from the title, Splunk looks for a "neat" bucket length that produces 100 buckets or less. For example, with a timerange of an hour Splunk will choose minutes to get 60 buckets. If you double the timerange to two hours minutes would require 120 buckets, so Splunk bumps the span up to 5 minutes. Going from eight to nine hours, 5 minute spans would increase the number of buckets from 96 to 108, so Splunk will use 30 minutes for nine-hour-timeranges.
You can set the span explicitly, set the maximum number of bins, or set minimum spans.

I'm not quite sure whether that helps you solve the issue though. Here's an idea:

system=cics dc=RZ2 type=trxPerf | timechart span=5m sum(cputot) as CPU | eval MaxConsumptionThisWeek = [search system=cics dc=RZ2 type=trxPerf daysago=7 | timechart span=5m sum(cputot) as SumCPU | stats max(SumCPU) as MaxCPU | fields + MaxCPU | rename MaxCPU as search] | timechart minspan=5m avg(CPU) as "CPU Consumption" avg(MaxConsumptionThisWeek) as MaxConsumptionThisWeek

I've split up the search into three parts instead of two.
The first part calculates your consumption in 5-minute intervals for the selected timerange, regardless of whether that's chartable or not.
The second part calculates your weekly maximum, as you already have. (Note: It may be a good idea to schedule a search that calculates these weekly, monthly, whatever maximums and stores them in a lookup.)
The third part takes these 5-minute spans and may or may not turn them into longer spans to keep everything chartable, depending on the selected timerange.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Concerning your question from the title, Splunk looks for a "neat" bucket length that produces 100 buckets or less. For example, with a timerange of an hour Splunk will choose minutes to get 60 buckets. If you double the timerange to two hours minutes would require 120 buckets, so Splunk bumps the span up to 5 minutes. Going from eight to nine hours, 5 minute spans would increase the number of buckets from 96 to 108, so Splunk will use 30 minutes for nine-hour-timeranges.
You can set the span explicitly, set the maximum number of bins, or set minimum spans.

I'm not quite sure whether that helps you solve the issue though. Here's an idea:

system=cics dc=RZ2 type=trxPerf | timechart span=5m sum(cputot) as CPU | eval MaxConsumptionThisWeek = [search system=cics dc=RZ2 type=trxPerf daysago=7 | timechart span=5m sum(cputot) as SumCPU | stats max(SumCPU) as MaxCPU | fields + MaxCPU | rename MaxCPU as search] | timechart minspan=5m avg(CPU) as "CPU Consumption" avg(MaxConsumptionThisWeek) as MaxConsumptionThisWeek

I've split up the search into three parts instead of two.
The first part calculates your consumption in 5-minute intervals for the selected timerange, regardless of whether that's chartable or not.
The second part calculates your weekly maximum, as you already have. (Note: It may be a good idea to schedule a search that calculates these weekly, monthly, whatever maximums and stores them in a lookup.)
The third part takes these 5-minute spans and may or may not turn them into longer spans to keep everything chartable, depending on the selected timerange.

anjafischer
Path Finder

Thanks a lot! 🙂
This really works like a charm, I only changed the first two timechart spans to minspan=5m but that is just a detail..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...