Splunk Search

Stats sum command experiencing strange behavior after 7.2.0 upgrade

paimonsoror
Builder

Hi Folks;

So getting a very bizaare issue here after our upgrade to 7.2

index="app_rocket_dxs" sourcetype="fluentd_json" source="vbs-dxs-int*" 
| where message like "%Summary%" 
| eval temp=split(substr(message,64,250),":")  
| eval DomainName=mvindex(temp,1)  
| eval StartTime=mvindex(temp,3) 
| eval EndTime=mvindex(temp,5) 
| eval TopicName=mvindex(temp,7) 
| eval MsgCount=mvindex(temp,9) 
| convert num(MsgCount) as MsgCounts |convert timeformat="%Y-%m-%d" ctime(_time) AS date
| table StartTime,EndTime,MsgCounts,DomainName,TopicName,date
| stats sum(MsgCounts) as PublishedCount by date,TopicName
| sort date desc

Here is the way the data looks as a table
alt text

However after applying the stats command the 'PublishedCount' is blank:
alt text

Sample Event (scrubbed)

{"docker":{"container_id":"8203837773d4f65d9a3382d381c97f64af01209f865463239e7d59e6ed2972ec"},"kubernetes":{"container_name":"coverageemclntbenplan","namespace_name":"vbs-dxs-int","pod_name":"covplan-1-m9g6x","pod_id":"a8004109-e37d-11e8-b28e-fa163e193d33","labels":{"app":"covenplan","appname":"Rocket","deployment":"covernplan-1","deploymentconfig":"coveeplan"},"host":"cilver.com","master_url":"https://kubernetes.default.svc.cluster.local","namespace_id":"23eecb03-7947-11e8-9035-fa163ee5bb62"},"message":"11-12 16:37 oraclepool.oraclekafka INFO     Publisher Summary - Domain:coverage:Start_Bound:2018-11-12-11.33.26.421532 :End_Bound:2018-11-12-11.35.26.532198 :Topic Name:ibis.cif_em_o.em_clnt_ben_plan:count:0\n","level":"info","pipeline_metadata":{"collector":{"ipaddr4":"100.00.00.00","ipaddr6":"fe80::0000:0000:0000:a728","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2018-11-12T16:37:21.820821+00:00","version":"0.12.43 1.6.0"}},"@timestamp":"2018-11-12T16:37:21.767889+00:00","viaq_index_name":"project.vbs-dxs-int.23eecb03-7947-11e8-9035-fa163ee5bb62.2018.11.12","viaq_msg_id":"NzM0OWEzZGEtMmJiNy00MDQ3LWI4ZjAtZTdkMGU1MzY0MzZj","kubernetes_node":"cilver.com"}
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Look at your message string in detail - it has a tailing line breaker \n. Your approach of splitting at colons will include the line breaker in your field value, breaking the sum. Sanitize your value using trim() or replace(), or use rex instead of splitting:

... | rex field=message ":count:(?<MsgCounts>\d+)"

Side note, add the word Summary to your initial search to reduce the number of events loaded off disk (scanCount in the job inspector).

View solution in original post

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Look at your message string in detail - it has a tailing line breaker \n. Your approach of splitting at colons will include the line breaker in your field value, breaking the sum. Sanitize your value using trim() or replace(), or use rex instead of splitting:

... | rex field=message ":count:(?<MsgCounts>\d+)"

Side note, add the word Summary to your initial search to reduce the number of events loaded off disk (scanCount in the job inspector).

0 Karma

paimonsoror
Builder

you sir, are a scholar. thanks!

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do post a sample event.

0 Karma

paimonsoror
Builder

Updated with sample!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...