How to eliminate events with ">Debug"

[joesrepsolc]

Trying to eliminate logs that start with ">Debug". Must be missing something with my logic.
All the data has a sourcetype=diplomat:server which I believe I've done correctly in the props, and trying to utilize the nullqueue.
I did an apply cluster-bundle after creating the new apps, see it on the indexers, and even did a rolling restart of the indexers to make sure this was in effect. Still getting the logs that start with ">Debug" though,
what did I miss???

props.conf

[diplomat:server]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = ^>Debug
DEST_KEY = queue
FORMAT = nullQueue

[MuS · Accepted solution]

Move the app from $SPLUNK_HOME/etc/master-apps/_cluster/ to $SPLUNK_HOME/etc/master-apps/ and apply the bundle again. The directory $SPLUNK_HOME/etc/master-apps/_cluster/ is a special one and should only be used to deploy config files, not apps - see https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Updatepeerconfigurations#On_the_master

cheers, MuS

[joesrepsolc]

Appreciate all the feedback everyone. Awesome help on the Splunk Answers site.

[joesrepsolc]

You are 100% correct. I missed this and must be an old habit.

[MuS]

Hi joesrepsolc,

Good to hear this solved the issue. Can you please accept this as the answer? Thanks

cheers, MuS

[joesrepsolc]

UPDATE:

The regex assistance may have helped but the issue was resolve with a Splunk support call. Evidently the props.conf/transforms.conf in my app needed to be in the /opt/splunk/etc/master-apps/ folder... not one level lower in the /opt/splunk/etc/master-apps/_cluster folder. I moved the app I had built "up" one level, and instantly started to work.

WORKED:
/opt/splunk/etc/master-apps/newApp/local/props.conf

NOT WORKED:
/opt/splunk/etc/master-apps/_cluster/newApp/local/props.conf

[gcusello]

Hi joesrepsolc,
where do you located you props.conf and transforms.conf?
they must be located on Indexers or Heavy Forwarders (when present), not on UniversalForwarders.

In addition use backslash when you use special chars (like >) in regexes, in other words in transforms.conf:

[setnull]
REGEX = ^\>Debug
DEST_KEY = queue
FORMAT = nullQueue

Bye.
Giuseppe

[joesrepsolc]

Trying this now... was not aware of the special character issue with REGEX line. Thank you.

[VatsalJagani]

@joesrepsolc - Could you please accept this answer by @gcusello so other users can find approved answer easily.

[MuS]

Hi joesrepsolc,

two things that are important in this case:

1. The sourcetype in props.conf is case sensitive, so make sure it really matches.
2. The regex uses > which is a special character in regex and needs escaping like this ^\>Debug

Further more use $SPLUNK_HOME/bin/splunk btool props list diplomat:server --debug on one of the indexers to validate your props.conf is being applied and not overwritten by some other app taking precedence over your app.

Hope this helps ...

cheers, MuS

[MuS]

Update:

The issue was not the regex, but the config files were at the wrong path. See this answer for the correct solution https://answers.splunk.com/answers/750050/how-to-eliminate-events-with-debug.html#answer-751623

cheers, MuS

[joesrepsolc]

So i put the props.conf and transforms.conf in the /opt/splunk/etc/master-apps/_cluster/diplomat/local folder. and did the apply-cluster-bundle command. I see the "app" was pushed out to the indexers in the cluster (under /opt/splunk/etc/slave-apps/_cluster/diplomat/local). And even restarted the indexers manually.

Still nothing when i run this command:
/opt/splunk/bin/splunk btool props list diplomat:server --debug

And still getting the unwanted ">Debug" events ingested. What am I missing??? thanks!

Joe

[VatsalJagani]

Is it still your question? If yes then do not make it accepted answer so it keeps in someone's eyesight.

[joesrepsolc]

sourcetype is case-correct. Tried that btool command, and got nothing back. I see the props/transforms on the indexers now after running apply cluster-bundle command. Maybe not restarted though... doing a rolling-restart cluster-peers now. I'll report back on outcome. Thanks!