Suppress alert email due to splunk internal error

[shangshin]

Hi, Is it possible to suppress alert email from the saved searches due to splunk internal error.

For example, I received alert email from the saved search due to this splunk internal error.

-- Search generated the following messages -- 
Message Level: WARN 
1. [subsearch]: Failed to start search on peer 'mysplunkindexer'. 

[dchang]

Correction: Should be sendemail.py and not sendmail.py

[linu1988 · Accepted solution]

Hello,
If you get the desired alert and don't want to have this custom message in the alert mail, then comment out the following line in your sendmail.py file for the app. But do investigate the reason of the error.

# if (searchid != None) and (type(searchid) is not dict) and (len(searchid) > 0):
    # intro += renderJobMessages(getJobMessages(searchid, sessionKey), plainText)

Thanks

[linu1988]

Yeah It's fine. So i suppose you are setting the alert for the stopped things.You can set the alert per result and make the throttling condition. It will send you individual results or vise versa. The no of results should be set accordingly to satisfy the alert condition.

I had the same reason to comment out the definition in py file. It is awkward for a end user to see such message and panic.

[shangshin]

Thank you very much. I feel awkward to ask you again cause this saved search will always return 5 events with a column "Status". The column value can "Running" or "Stop".

The monitor is set up on a search head so when the indexer is in the middle of restart, I will get false alarm email. The only current option is to modify sendmail.py.

[linu1988]

Now i got you.

Please use the

Alert condition "if number of events" > 0 in the Splunk UI savedsearch section.

You can also add the below condition to the savedsearch.conf as well:

quantity = 0
relation = greater than

[shangshin]

yes, I tried the option to comment those 2 lines out. I still received the alert email even though the warning message is not included.

The goal is to suppress alert email when there is a warning message....

[linu1988]

May be for the time being or may be not if it's a realtime job till it expires. You don't need to modify anything else.

If your alert condition is met the result will come in the mail or you won't get the alert. I had suggested the option to comment out as it looks awkward in a mail,that's it.

[shangshin]

I want to send out the email only if
getJobMessages(searchid, sessionKey) == ""

Basically, this means the search is completed.

Can you shed some light on this?
Thanks!

[linu1988]

sendmail.py is the default python program that sends mail. You don't have to terminate the process. Instead you can write your own script for sending alert.

The message level WARN is just a default statement in the script file which included the message to the mail body. You need to check for the indexer i.e. your search peer status why it didn't initiate the search. Other than that everything works fine as long as you get the result. This gives errors to me when any real-time alerts are set and they keep the error result with it till it's fired.

Thanks

[shangshin]

How can I exit the program sendmail.py if the message level is WARN?

Thank you!

[shangshin]

Thanks. I will give it a try and let you know.

[linu1988]

Yes that's the purpose of the alerts. Please mark it as answer if it solved the problem.

[shangshin]

Thank you very much for the help.
I want to receive the alert email when the search is completely done.

If the search fails before it completes, I do NOT want to receive the alert email.

[shangshin]

yes, this message comes with email alert. Basically, I don't want to be notified when the search fails due to splunk internal error. e.g. when it's in the middle of restarting...

[linu1988]

Does it come along with your email alert?