Getting Data In

Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer

gn694
Communicator

I have some TSV files that I am forwarding with a Universal Forwarder.
I have props.conf configured on the UF with the following for the sourcetype:
FIELD_DELIMITER = \t
HEADER_FIELD_LINE_NUMBER = 1

That has worked great. But now I have a need to drop some events so they do not get indexed.
On the Indexer I have configured the following for the sourcetype in props.conf:
[]
TRANSFORMS-null = drop_batchrequests
...and in transforms.conf:
[drop_batchrequests]
REGEX = batchRequest
DEST_KEY = queue
FORMAT = nullQueue

At first it was not working, I was still getting events that contain batchRequest. So I temporarily removed the structured data configuration on the Universal Forwarder (shown above) and the transform worked as desired - batchRequest events were no longer indexed.... But now the tsv format and field recognition was not there...

So I tried to configure everything in one place. On the Indexer I specified the structured data config in props.conf using FIELD_DELIMITER and FIELD_NAMES (since I can't use HEADER_FIELD_LINE_NUMBER on the Inedxer.) The result of that was the batchRequests events were not indexed, but the fields (from the header row) still were not extracted.

Am I doing something wrong? Or is there some reason why these configurations (TSV/structured data field recognition and dropping certain events to the nullQueue) on the same sourcetype will not work together? I can get each to work independently - but not together.

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hi gn694,

Try configuring the INDEXED_EXTRACTIONS props and the filtering on the UF. Structured data is the only data that a UF can complete filtering on.

This should be helpful in this scenario:

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Caveats_for_routin...

- MattyMo

View solution in original post

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi gn694,

Try configuring the INDEXED_EXTRACTIONS props and the filtering on the UF. Structured data is the only data that a UF can complete filtering on.

This should be helpful in this scenario:

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Caveats_for_routin...

- MattyMo
0 Karma

gn694
Communicator

I added the transforms to the Universal Forwarder to send the unwanted stuff to the nullQueue and it is now working as I need it to. I didn't think that would work (even on structured data) but it seems that it does.

thank you!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...