Splunk for k8s - incorrect search syntax will produce incorrect results

[guilmxm]

Hi,

The embedded search:

[Pod CPU/Memeory Usage Trend]
search = | mstats avg(_value) as cpu  WHERE `k8s-metrics-index` AND metric_name="kube.pod.cpu.usage" span=10s  BY "pod" | join type=left _time [ | mstats avg(_value) as memory WHERE `k8s-metrics-index` AND metric_name="kube.pod.memory.usage" BY span=10s "pod"] | stats sparkline(avg(cpu)) as "CPU Usage", sparkline(avg(memory)) as "Memory Usage" BY "pod"

Produces incorrect results to the (ugly) usage of "| joint type=left _time", as the following can demonstrate:

replaced with an append / appendcols:

To avoid ugly subsearches, I would kindly suggest something in replacement like:

| mstats avg(_value) as value WHERE index = kubernetes-metrics AND (metric_name="kube.pod.cpu.usage" OR metric_name="kube.pod.memory.usage") span=10s BY metric_name "pod"
| rex field=metric_name "[^\.]*\.[^\.]*\.(?<metric>[^\.]*)\."
| eval {metric}=value
| stats sparkline(avg(cpu)) as "CPU Usage", sparkline(avg(memory)) as "Memory Usage" BY "pod"

Which would avoid running subsearches that can result in bad performances and truncation.

Kind regards,

Guilhem

[mattymo · Accepted solution]

Hey guilmxm!

Will pass along the feedback and work with the team. Will let you know once cleaned up!

Matt