Whenever we set up a Slack alert to send to a #channel in our Slack instance (other than default for the web hook), we get a 500 in Splunk.
07-08-2016 18:51:17.286 +0000 ERROR sendmodalert - action=slack STDERR - Error sending message: HTTP Error 500: Server Error
Sending the alert destination to @user works fine, and leaving blank works fine for hitting the default channel
I can not find what is causing this behavior
This appears to be a bug in splunk, as changing setting there seem to "trick" this into working.
If I set up a new slack alert to a #channel - it throws the 500's.
If I set up a slack alert with default (blank) channel, it triggers properly, and ~50% of the time, once a successful message was triggered to default endpoint, the alert can be set to a different #channel and it will trigger correctly.
I have not tried setting this up specifically with full admin permissions, however it seems a fairly large gap to have to grant full admin rights to something that simply need to post messages.
The Slack user owning the webhook may need elevated privileges in Slack (not Splunk.) Setting the account to 'admin' in Slack seems to work, but I would look into whether some kind of lesser permissions will also work.