How do you exclude all lines with INFO or WARN from being indexed?

[nls7010]

I have been reading through a lot of the previous answers to exclusion, but none match what I need. I need to exclude all INFO and WARN lines from one of my indexes, so that they are never processed. Only the ERROR lines should be processed.

I have this so far, but I'm not certain of a couple of things. One what should I have in the props file to complete it and two, do I need the in the transforms.conf file:

PROPS.CONF

TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

<code>[setnull]
REGEX = INFO, WARN
DEST_KEY = queue
FORMAT = nullQueue
</code>

[nls7010]

we have clustered indexers and I pushed it our via the cluster master. I believe it's supposed to restart them all

[lakshman239]

You need configs in both props.conf and transforms.conf. The following sends events with only ERROR to your indexer.

PROPS.CONF
[your_sourcetype]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull] 
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue 

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue 

[nls7010]

So, If I'm understanding this correctly, I would do something like the following:
PROPS.CONF

[value1-logs]
TRANSFORMS-set = setnull, setparsing

[value2-logs]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

So that only lines with ERROR will show?

[lakshman239]

yes correct

[nls7010]

quick question. the FORMAT key, should that be the name of the actual index in the transforms.conf file? Or it that literal? Asking because I still see the WARN and INFO logs coming in after pushing out the following:
[my1-logs]
TRANSFORMS-set = setnull, setparsing

[my2-logs]
TRANSFORMS-set = setnull, setparsing

TRANSFORMS.CONF

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

[nls7010]

I just realized I asked that comment on the wrong value. It's the REGEX= . Is that supposed to have a value of INFO,WARN?

[lakshman239]

what do you want? do you want INFO and WARN OR ERROR? you can adjust them as per your needs.

# This sends all events to be ignored
[setnull] 
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

# this says ignore all events, except the ones containing ERROR
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

[nls7010]

ok, I understand, so now I'll need to work with the client to see if everything is getting to the forwarders as needed. I did verify that the props and transform files did make it to all the indexers and the indexer cluster members were all restarted.

[nls7010]

I just want the ERROR lines. So I would want to eliminate INFO, WARN, etc that are not ERROR related.

[lakshman239]

its a literal. Did you restart your indexer after the changes?

[nls7010]

Yes, I did a rolling restart, but am still seeing INFO and WARN which I wanted to block out and only get ERROR.

[lakshman239]

can you test the config in your dev with the above config?

[nls7010]

I am testing in my development environment and I get the same issue. Nothing gets parched out, it's all still coming into the logs. I will be working with my client in a little while to ensure the props and transforms made it to the servers (they are containers).

[nls7010]

They have all been restarted, but it's still sending the INFO and WARN messages 😞

[nls7010]

There were some code lines surrounding the transforms lines, those were what I was asking about

[nls7010]

What if there are multiple source types? I wanted to exclude those lines for the entire index which was created for this customer and they have multiple sourcetypes.

[lakshman239]

for each sourcetype, you need to have the same settings

[your_sourcetype1]
TRANSFORMS-set = setnull, setparsing

[your_souretype2]
TRANSFORMS-set = setnull, setparsing

However, you just need one setting for transforms, as provided earlier.

After updating, you need to restart your indexer, so any new data will have these filtering applied.