Splunk Search

How can I identify duplicates in a multivalue field based on the value of another field?

jedatt01
Builder

I need to be able to identify duplicates in a multivalue field. The difficulty is that I want to identify duplicates that match the value of another field.

for example
field1 = "something"
(MV field) field2 = "something, nothing, everything, something"

I need to be able to count how many times field1 is seen.

eval test = mvfilter(match(field2,field1)) <------------- how can I do the equivalent of this since mvfilter doesn't support tokens?

0 Karma

micahkemp
Champion

This would likely not be a great search if there are a lot of events that meet your criteria (mvexpand can be a memory hog), but this may work well enough for your use case:

<your base search>
| where field2==field1
| streamstats count AS row
| mvexpand field2
| eval matches=if(field2==field1, field1, dummy)
| stats list(field2) AS field2, values(field1) AS field1, list(matches) AS matches BY row
| table field1 field2 matches
0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can try this workaround. (runanywhere search, first two lines are used to generate sample data)

| gentimes start=-1 | eval field1="something" | table field1 | eval field2="something,nothing,everything,something,anything,something,something,something" | makemv field2 delim="," 
| eval test=field2 | nomv test| eval test=mvcount(split(replace("start".test."end",field1,"|"),"|"))-1
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...