Getting Data In

How to report on the top 10 fields when logs have a variable number of key value pairs that are numeric?

plynch52
Explorer

I have log records with a variable number of KV (key value) pairs. Both the field and the values are numeric. The following search parses all of the log records correctly and builds a very long row of results given that there are thousands of potential key values to sum.
I have a transforms.conf & props.conf setup

[root@splunk local]# cat transforms.conf 
[get_rule_stat]
REGEX = ([0-9]+)=([0-9]+)
FORMAT = $1::$2
MV_ADD =true
CLEAN_KEYS = false

[get_rule_id]
REGEX =(?<_KEY_1>[0-9]+)=(?<_VAL_1>[0-9]+)
MV_ADD = true
CLEAN_KEYS = false
--------
[root@splunk local]# cat props.conf 
[ScoutShield]
KV_MODE = multi
REPORT-Stats = get_rule_stat
REPORT-R-ID = get_rule_id
---




source="qa_debug.log" host="splunk.localdomain" sourcetype="ScoutShield" 
| rex field=_raw "(?ms)(?=[^N]*(?:NetDefender Rule Hits Digest|N.*NetDefender Rule Hits Digest))^(?P[^\\[]+)[^\\]\\n]*\\]\\[(?P[^\\]]+)\\]\\[(?P\\d+)\\]\\[(?P\\d+)[^\\]\\n]*\\]\\[(?P[^\\]]+)" offset_field=_extracted_fields_bounds 
| rex field=stats max_match=100 "(?\d*=\d*)" 
|stats sum(*)  
| addtotals   fieldname=TotalHits 
| rename sum(*) as * 

What I would like to do is report on the top fields (as in top 10 with largest count) or to do a timechart with only those fields where the fields belong to the top 10.

It seems as though I can only use * for the field names, and there isn't a way to sub set the names.
Any suggestions would be appreciated.

0 Karma
1 Solution

somesoni2
Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues

View solution in original post

0 Karma

somesoni2
Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...