Getting Data In

Filtering fields in log before forward to indexing

SplunkCSIT
Communicator

Hi,
If i need to filtering some data in the log before forward to indexing, how to go abt doing it? thks

Tags (1)

thesteve
Path Finder

There's a page of documentation dedicated to just this question: http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Routeandfilterdatad

If you want to filter before forwarding, you have to use a heavy forwarder. The reason for this is that a light forwarder doesn't use transforms.conf and therefore does very limited parsing of the source data before sending it on.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

  1. In props.conf, set the TRANSFORMS-null attribute:

    [source::/var/log/messages]
    TRANSFORMS-null= setnull

  2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

    [setnull]
    REGEX = [sshd]
    DEST_KEY = queue
    FORMAT = nullQueue

That does it.

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

  1. In props.conf:

    [source::/var/log/messages]
    TRANSFORMS-set= setnull,setparsing

  2. In transforms.conf:

    [setnull]
    REGEX = .
    DEST_KEY = queue
    FORMAT = nullQueue

    [setparsing]
    REGEX = [sshd]
    DEST_KEY = queue
    FORMAT = indexQueue

Alternatively, you can use SED command processing to replace all text in a given line with nothing (s/something//). Splunk doesn't support SED delete line commands, but it shouldn't index blank lines, so replacing a line of text with nothing should do it.

0 Karma

mdzmuran
Observer

Hi. Are these features also available in Splunk Light (version 6.5.1)?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Good answer, but I'm not sure it fits the use case. He doesn't want to filter entire events (nullQueue) but wants to strip out most of the event, and leave parts.

dshpritz
SplunkTrust
SplunkTrust

If you want to avoid indexing part of an event, you should look into using a SED entry in your props.conf file. You would need a regular expression matching the portion of the data you want to remove, then in your props.conf, you have an entry like:

[mysourcetype]
SED-remove_data = s/<your regex here>//g

If you want that to occur before the data is sent over the network, that would need to be done on a heavy forwarder, or another parsing system. Note that the SED entry is done at index time, so it would need to be on your indexers or other parsing systems.

HTH,

Dave

dshpritz
SplunkTrust
SplunkTrust

The SED command goes into the props.conf file on the first parsing system (indexer or heavy forwarder).

0 Karma

MuS
SplunkTrust
SplunkTrust

read Dave's answer and you find every information you need 😉

0 Karma

SplunkCSIT
Communicator

thks for the valuable info, where will SED command be implement, at forwarder or the indexer and at which file? thks

0 Karma

dshpritz
SplunkTrust
SplunkTrust

In this case, you can run two SED commands, one to stip away what is in front of and one for after :

SED-remove_before = s/(?s).*(?=)//g
SED-remove_after = s/(?s)(?<=
).*//g

mmsull4
Engager

Sorry for these silly question, I'm new to SED modification. 1) This doesn't seem to be working for me (I'm using splunk 6.2 on Windows) which leads me to 2) where do I find doc on remove_data, remove_before or remove_after?

0 Karma

SplunkCSIT
Communicator

If i want to forward the below xml file to index, but before forward to index, i want the data field and values (this is the data i want only)to be forwarded and the rest to be filter off, how to configure in the props.conf? thks

file.xml testing focusing on the inputs this is the data i want only this problem

0 Karma

SplunkCSIT
Communicator

If i want to forward the below xml file to index, but before forward to index, i want the data field and values (this is the data i want only)to be forwarded and the rest to be filter off, how to configure in the props.conf? thks

file.xml
testing
focusing on the inputs
this is the data i want only
this problem

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...