Splunk Search

Duplicate rex field extraction results

funghorn
Explorer

In my log files there is a field (path = info.message) that has a certain string. I want to extract a part of that string by using "rex". Here is my query:

rex field="info.message" "- (?<pcc>\S{3,4}) "

Rex is doing a great job and extracts the value the way I want, but when I try to view results in a table, I get two values in one row, like that (these results are for three log files):
alt text

Even if I use "dedup pcc" before "table pcc", I still get those duplicate value rows. In logs there is another field called "msg" (path = info.msg) that has the same string as "message", so that can be the cause of this duplication, but my query says that the field should be "info.message", not "info.msg".

Does anyone know what is the issue here and what am I missing out on?
Thanks!!

UPD:
After I removed "field="info.message"" from rex command, the consequent search results came without any duplicates, so this issue has something to do with "field" option in rex.

UPD2:
I have used a workaround with spath and it worked. If you use spath before rex and insert spath's "output" value to rex'es "field", there are no duplicates in results. However, this means that in order to successfully use rex'es "field" option, a user should be aware of spath, which is kind of poor usability, I think.
woodcock, is there a chance a feature request could be added to enable rex'es field value to accept full paths (like in my example - "info.message")? Thank you a lot!

UPD3:
Below is a screenshot of the "info.message" field. I erased confidential info, but that is not important, because I am trying to extract the visible 4 symbol code. As you can see, there are 3 unique records but each of them has a count of 2. Also, there is another field that has a path "info.msg" and contains the exact same information as "info.message".
DalJeanis, do you know maybe whether rex'es field treats its input not as a path but rather a kind of alias or something? If yes, then how can I know that "info.message" is an alias?
alt text

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Check to see if info.message is an alias for another field as well as an extracted field in its own right.

That is, do a search that returns one of the records in question, and look at the record's fields to see if info.message has two values.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

1) Please post an example of the entire _raw that causes the issue, minus any confidential data, plus any parts of your search that extract the info.message

2) try mvdedup after the field="info.message" option on the rex and see what happens.

0 Karma

woodcock
Esteemed Legend

Remember he said then when he operates on _raw, it works fine. If that's true, then it has to be a bug.

0 Karma

woodcock
Esteemed Legend

If your description is accurate then this is definitely a bug so I would certainly open a support case with Splunk.

0 Karma

woodcock
Esteemed Legend

I would add a bug tag to this question, too.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...