Getting Data In

How to report on the top 10 fields when logs have a variable number of key value pairs that are numeric?

plynch52
Explorer

I have log records with a variable number of KV (key value) pairs. Both the field and the values are numeric. The following search parses all of the log records correctly and builds a very long row of results given that there are thousands of potential key values to sum.
I have a transforms.conf & props.conf setup

[root@splunk local]# cat transforms.conf 
[get_rule_stat]
REGEX = ([0-9]+)=([0-9]+)
FORMAT = $1::$2
MV_ADD =true
CLEAN_KEYS = false

[get_rule_id]
REGEX =(?<_KEY_1>[0-9]+)=(?<_VAL_1>[0-9]+)
MV_ADD = true
CLEAN_KEYS = false
--------
[root@splunk local]# cat props.conf 
[ScoutShield]
KV_MODE = multi
REPORT-Stats = get_rule_stat
REPORT-R-ID = get_rule_id
---




source="qa_debug.log" host="splunk.localdomain" sourcetype="ScoutShield" 
| rex field=_raw "(?ms)(?=[^N]*(?:NetDefender Rule Hits Digest|N.*NetDefender Rule Hits Digest))^(?P[^\\[]+)[^\\]\\n]*\\]\\[(?P[^\\]]+)\\]\\[(?P\\d+)\\]\\[(?P\\d+)[^\\]\\n]*\\]\\[(?P[^\\]]+)" offset_field=_extracted_fields_bounds 
| rex field=stats max_match=100 "(?\d*=\d*)" 
|stats sum(*)  
| addtotals   fieldname=TotalHits 
| rename sum(*) as * 

What I would like to do is report on the top fields (as in top 10 with largest count) or to do a timechart with only those fields where the fields belong to the top 10.

It seems as though I can only use * for the field names, and there isn't a way to sub set the names.
Any suggestions would be appreciated.

0 Karma
1 Solution

somesoni2
Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues

View solution in original post

0 Karma

somesoni2
Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...