Splunk Search

how to display unique values only from one particular multi-value field

Umesh_Vedicsoft
Path Finder

Hi
i have a field like msg="this is from: 101,102,103,101,104,102,103,105,106" but i would like to display that field with unique numbers only, without duplicates, like this msg="this is from: 101,102,103,104,105,106" using a search query
Please help!

0 Karma

gcusello
SplunkTrust
SplunkTrust

insert in your search

| rex field=msg "(?<a>\d+)" max_match=10 
| mvexpand a 
| dedup a 
| mvcombine delim=", " a 
| nomv a 
| eval msg="this is from: "+a 
| table msg

Bye.
Giuseppe

0 Karma

javiergn
SplunkTrust
SplunkTrust

Option with stats values (Assuming there is _raw field in your data 😞

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Another option with mvdedup:

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = mvjoin(mvdedup(split(msg_numbers, ",")), ",")
| eval msg = msg_text . msg_numbers

Example:

| stats count | fields - count
| eval _raw = "_time=2016-12-18 00:00:00 this is from: 101,102,103,101,104,102,103,105,106"
| eval msg = "this is from: 101,102,103,101,104,102,103,105,106"
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Output (see picture below):

alt text

0 Karma

hgrow
Communicator

Hi there,

maybe the eval-statement mvdedup is what you are looking for.

You can just execute the query below as an example:

| stats count | eval msg="this is from: 101,102,103,101,104,102,103,105,106" | rex max_match=99 field=msg "(?<msg_from>\d+" | eval u_msg_from=mvdedup(msg_from)

Greetings
hgrow

arkadyz1
Builder

Or, if you are certain the original field has only this format, something like:

| rex field=msg "^this is from: (?P<msg_from>.*)$" | eval msg_from=split(msg_from, ",") | eval msg_from=mvdedup(msg_from) | eval msg_from=mvjoin(msg_from, ",")

I'm not sure which way is more efficient - rex with max_match or split, but it is always nice to have more than one way of doing things.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...