Splunk Search

how to display unique values only from one particular multi-value field

Umesh_Vedicsoft
Path Finder

Hi
i have a field like msg="this is from: 101,102,103,101,104,102,103,105,106" but i would like to display that field with unique numbers only, without duplicates, like this msg="this is from: 101,102,103,104,105,106" using a search query
Please help!

0 Karma

gcusello
SplunkTrust
SplunkTrust

insert in your search

| rex field=msg "(?<a>\d+)" max_match=10 
| mvexpand a 
| dedup a 
| mvcombine delim=", " a 
| nomv a 
| eval msg="this is from: "+a 
| table msg

Bye.
Giuseppe

0 Karma

javiergn
SplunkTrust
SplunkTrust

Option with stats values (Assuming there is _raw field in your data 😞

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Another option with mvdedup:

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = mvjoin(mvdedup(split(msg_numbers, ",")), ",")
| eval msg = msg_text . msg_numbers

Example:

| stats count | fields - count
| eval _raw = "_time=2016-12-18 00:00:00 this is from: 101,102,103,101,104,102,103,105,106"
| eval msg = "this is from: 101,102,103,101,104,102,103,105,106"
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Output (see picture below):

alt text

0 Karma

hgrow
Communicator

Hi there,

maybe the eval-statement mvdedup is what you are looking for.

You can just execute the query below as an example:

| stats count | eval msg="this is from: 101,102,103,101,104,102,103,105,106" | rex max_match=99 field=msg "(?<msg_from>\d+" | eval u_msg_from=mvdedup(msg_from)

Greetings
hgrow

arkadyz1
Builder

Or, if you are certain the original field has only this format, something like:

| rex field=msg "^this is from: (?P<msg_from>.*)$" | eval msg_from=split(msg_from, ",") | eval msg_from=mvdedup(msg_from) | eval msg_from=mvjoin(msg_from, ",")

I'm not sure which way is more efficient - rex with max_match or split, but it is always nice to have more than one way of doing things.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...