Getting Data In

How to report on the top 10 fields when logs have a variable number of key value pairs that are numeric?

plynch52
Explorer

I have log records with a variable number of KV (key value) pairs. Both the field and the values are numeric. The following search parses all of the log records correctly and builds a very long row of results given that there are thousands of potential key values to sum.
I have a transforms.conf & props.conf setup

[root@splunk local]# cat transforms.conf 
[get_rule_stat]
REGEX = ([0-9]+)=([0-9]+)
FORMAT = $1::$2
MV_ADD =true
CLEAN_KEYS = false

[get_rule_id]
REGEX =(?<_KEY_1>[0-9]+)=(?<_VAL_1>[0-9]+)
MV_ADD = true
CLEAN_KEYS = false
--------
[root@splunk local]# cat props.conf 
[ScoutShield]
KV_MODE = multi
REPORT-Stats = get_rule_stat
REPORT-R-ID = get_rule_id
---




source="qa_debug.log" host="splunk.localdomain" sourcetype="ScoutShield" 
| rex field=_raw "(?ms)(?=[^N]*(?:NetDefender Rule Hits Digest|N.*NetDefender Rule Hits Digest))^(?P[^\\[]+)[^\\]\\n]*\\]\\[(?P[^\\]]+)\\]\\[(?P\\d+)\\]\\[(?P\\d+)[^\\]\\n]*\\]\\[(?P[^\\]]+)" offset_field=_extracted_fields_bounds 
| rex field=stats max_match=100 "(?\d*=\d*)" 
|stats sum(*)  
| addtotals   fieldname=TotalHits 
| rename sum(*) as * 

What I would like to do is report on the top fields (as in top 10 with largest count) or to do a timechart with only those fields where the fields belong to the top 10.

It seems as though I can only use * for the field names, and there isn't a way to sub set the names.
Any suggestions would be appreciated.

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues
0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...