I have been reading through a lot of the previous answers to exclusion, but none match what I need. I need to exclude all INFO and WARN lines from one of my indexes, so that they are never processed. Only the ERROR lines should be processed.
I have this so far, but I'm not certain of a couple of things. One what should I have in the props file to complete it and two, do I need the in the transforms.conf file:
PROPS.CONF
TRANSFORMS-set = setnull, setparsing
TRANSFORMS.CONF
<code>[setnull]
REGEX = INFO, WARN
DEST_KEY = queue
FORMAT = nullQueue
</code>
we have clustered indexers and I pushed it our via the cluster master. I believe it's supposed to restart them all
You need configs in both props.conf and transforms.conf. The following sends events with only ERROR to your indexer.
PROPS.CONF
[your_sourcetype]
TRANSFORMS-set = setnull, setparsing
TRANSFORMS.CONF
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue
So, If I'm understanding this correctly, I would do something like the following:
PROPS.CONF
[value1-logs]
TRANSFORMS-set = setnull, setparsing
[value2-logs]
TRANSFORMS-set = setnull, setparsing
TRANSFORMS.CONF
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue
So that only lines with ERROR will show?
yes correct
quick question. the FORMAT key, should that be the name of the actual index in the transforms.conf file? Or it that literal? Asking because I still see the WARN and INFO logs coming in after pushing out the following:
[my1-logs]
TRANSFORMS-set = setnull, setparsing
[my2-logs]
TRANSFORMS-set = setnull, setparsing
TRANSFORMS.CONF
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue
I just realized I asked that comment on the wrong value. It's the REGEX= . Is that supposed to have a value of INFO,WARN?
what do you want? do you want INFO and WARN OR ERROR? you can adjust them as per your needs.
# This sends all events to be ignored
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
# this says ignore all events, except the ones containing ERROR
[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue
ok, I understand, so now I'll need to work with the client to see if everything is getting to the forwarders as needed. I did verify that the props and transform files did make it to all the indexers and the indexer cluster members were all restarted.
I just want the ERROR lines. So I would want to eliminate INFO, WARN, etc that are not ERROR related.
its a literal. Did you restart your indexer after the changes?
Yes, I did a rolling restart, but am still seeing INFO and WARN which I wanted to block out and only get ERROR.
can you test the config in your dev with the above config?
I am testing in my development environment and I get the same issue. Nothing gets parched out, it's all still coming into the logs. I will be working with my client in a little while to ensure the props and transforms made it to the servers (they are containers).
They have all been restarted, but it's still sending the INFO and WARN messages 😞
There were some code lines surrounding the transforms lines, those were what I was asking about
What if there are multiple source types? I wanted to exclude those lines for the entire index which was created for this customer and they have multiple sourcetypes.
for each sourcetype, you need to have the same settings
[your_sourcetype1]
TRANSFORMS-set = setnull, setparsing
[your_souretype2]
TRANSFORMS-set = setnull, setparsing
However, you just need one setting for transforms, as provided earlier.
After updating, you need to restart your indexer, so any new data will have these filtering applied.