Getting Data In

Why does INGEST_EVAL duplicate my events if I assign a value to _raw?

peter_krammer
Communicator

When I configure INGEST_EVAL to replace _raw with something else, it duplicates the event. 

Splunk Enterprise Version 8.2.1

props.conf:

Spoiler
[source::http:splunk_hec_token]
TRUNCATE = 500000
SHOULD_LINEMERGE = false
KV_MODE = json
TRANSFORMS-fdz_event = fdz_event

transforms.conf

Spoiler
[fdz_event]
INGEST_EVAL = _raw="Test"

Output:

splunk_duplicate.png 

Labels (2)
Tags (3)
0 Karma
1 Solution

peter_krammer
Communicator

I found a workaround to circumvent this bug. 

tranforms.conf

Spoiler
[fdz_event]
INGEST_EVAL = _raw="Test", queue=if(isnull(timestamp),"nullQueue", queue)

Because I notices that one of the duplicates has an indexed field "timestamp::none" and the other does not. With this I am routing only one of the to the nullQueue and keep the other.

View solution in original post

0 Karma

peter_krammer
Communicator

I found a workaround to circumvent this bug. 

tranforms.conf

Spoiler
[fdz_event]
INGEST_EVAL = _raw="Test", queue=if(isnull(timestamp),"nullQueue", queue)

Because I notices that one of the duplicates has an indexed field "timestamp::none" and the other does not. With this I am routing only one of the to the nullQueue and keep the other.

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...