How to promote a scheduled dashboard to test and Prod, but only send emails from Prod?

[kennethehughes]

I have a frustrating problem.

At my organization, we are required to build everything in a DEV environment, and then push our entire app to a QA and a UAT environment for testing, and then finally to a Prod environment. We are not allowed to bundle the app ourself. A second team bundles the app from DEV, and a THIRD team deploys the app.

My team is in charge of building an inventory of scheduled searches/dashboards/field aliases, but we do not have admin rights of any kind. We can only access Splunk through the UI (no access to the config files). We have limited power user rights in DEV, but standard user rights (can't even schedule searches) in the test/prod environments.

Here's the issue:
A subset of our saved searches/dashboards must be scheduled to deliver via email. Since we only have schedule rights in DEV, we must schedule there and promote the already-scheduled searches/dashboards up the environment chain.

We found a way to prevent our standard scheduled searches from emailing in the lower environments - but this does not work on the dashboards. Basically we just brought in a field called "environment" to every report and added a custom condition to the alert that only sends the email if ENV=PROD. This does not work with the multi-panel dashboards we've created.

Assuming that scheduling directly in production is not an option - can anyone think of a possible solution? Kind of at my wit's end here.

Oh, and disabling emails in the QA/UAT environment altogether is not an option either - it's a shared environment used by other teams as well.

Thanks!

[koshyk]

Multiple options
- You can have an email app for SMTP settings. This is pushed ONLY in PROD but not in TEST. So all the apps are the same, but TEST won't send mails out
- In your search, you can put a filter just for prod. Hope your splunkServers in prod have a naming convention? | search splunk_server=prod*

[woodcock]

Just make sure that you move the threshold logic from the Alert into your search (so that the Alert threshold is number of events and greather than 0) and then stop using the built-in Send email function inside of the alert and instead use sendemali inside the SPL of your search string (positioned after the | where threshold). You can then parameterize whether to call | sendemail (if ENV=PROD) or call |noop otherwise as explained here:

https://answers.splunk.com/answers/511308/how-abort-a-search-based-on-a-condition-1.html

[kennethehughes]

In this case our dashboard is made up of 10 panels, all linked to reports (as opposed to in-line searches).

How would I add this logic into the SPL in this case? Would it be in the source XML for the dashboard...or maybe the individual search for each panel?

Here's an example of the source code for the dashboard:

KPCC_CO_CLAIM_DASHBOARD

<panel>
  <title>KPCC_CO_INB_Recon_Summary_Claim_REPORT</title>
  <table>
    <title>CO Summary counts of inbound external claims at each life cycle point</title>
    <search ref="KPCC_CO_INB_Recon_Summary_Claim_REPORT"></search>
  </table>
</panel>
<panel>
  <title>KPCC_CO_OTB_Recon_Summary_Claim_REPORT</title>
  <table>
    <title>CO Summary counts of outbound external claim remits at each life cycle point</title>
    <search ref="KPCC_CO_OTB_Recon_Summary_Claim_REPORT"></search>
  </table>
</panel>

etc
etc

[woodcock]

They key is to disable the emailing function. What is it that generates the email (I thought that it was the Alert send email action)?