Hi
We have a very volatile log source which we today control by sending unwanted events to the nullQueue. This is good, but not 100% waterproof and reliable due to the nature of the log source.
Is there a way to achieve the same behavior, but instead whitelist events? Basically, we only want to index events that starts with a timestamp and has a LogLevel, examples below:
2015-10-21 12:10:05,786 +0200 INFO .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 WARNING .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 ERROR .... "the rest of the event..."
2015-10-21 12:10:05,786 +0200 SEVERE .... "the rest of the event..."
cheers
Magnus
Yes, you do this by first sending ALL events to nullQueue
and then pull back just the events that you like. NOTE: I did not test this RegEx.
In props.conf
:
[YourSourceTypeHere]
TRANSFORMS-set= setnull,setparsing
In transforms.conf
:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ^\d{4}-\d{2}-\d{2} \d{1-2}:\d{2}:\d{2},\d{3}\s+(?:[+\-0-9]*\s+)?(INFO|WARNING|ERROR|SEVERE)
DEST_KEY = queue
FORMAT = indexQueue