Dashboards & Visualizations

How to promote a scheduled dashboard to test and Prod, but only send emails from Prod?

kennethehughes
New Member

I have a frustrating problem.

At my organization, we are required to build everything in a DEV environment, and then push our entire app to a QA and a UAT environment for testing, and then finally to a Prod environment. We are not allowed to bundle the app ourself. A second team bundles the app from DEV, and a THIRD team deploys the app.

My team is in charge of building an inventory of scheduled searches/dashboards/field aliases, but we do not have admin rights of any kind. We can only access Splunk through the UI (no access to the config files). We have limited power user rights in DEV, but standard user rights (can't even schedule searches) in the test/prod environments.

Here's the issue:
A subset of our saved searches/dashboards must be scheduled to deliver via email. Since we only have schedule rights in DEV, we must schedule there and promote the already-scheduled searches/dashboards up the environment chain.

We found a way to prevent our standard scheduled searches from emailing in the lower environments - but this does not work on the dashboards. Basically we just brought in a field called "environment" to every report and added a custom condition to the alert that only sends the email if ENV=PROD. This does not work with the multi-panel dashboards we've created.

Assuming that scheduling directly in production is not an option - can anyone think of a possible solution? Kind of at my wit's end here.

Oh, and disabling emails in the QA/UAT environment altogether is not an option either - it's a shared environment used by other teams as well.

Thanks!

0 Karma

koshyk
Super Champion

Multiple options
- You can have an email app for SMTP settings. This is pushed ONLY in PROD but not in TEST. So all the apps are the same, but TEST won't send mails out
- In your search, you can put a filter just for prod. Hope your splunkServers in prod have a naming convention? | search splunk_server=prod*

0 Karma

woodcock
Esteemed Legend

Just make sure that you move the threshold logic from the Alert into your search (so that the Alert threshold is number of events and greather than 0) and then stop using the built-in Send email function inside of the alert and instead use sendemali inside the SPL of your search string (positioned after the | where threshold). You can then parameterize whether to call | sendemail (if ENV=PROD) or call |noop otherwise as explained here:

https://answers.splunk.com/answers/511308/how-abort-a-search-based-on-a-condition-1.html

0 Karma

kennethehughes
New Member

In this case our dashboard is made up of 10 panels, all linked to reports (as opposed to in-line searches).

How would I add this logic into the SPL in this case? Would it be in the source XML for the dashboard...or maybe the individual search for each panel?

Here's an example of the source code for the dashboard:

KPCC_CO_CLAIM_DASHBOARD

<panel>
  <title>KPCC_CO_INB_Recon_Summary_Claim_REPORT</title>
  <table>
    <title>CO Summary counts of inbound external claims at each life cycle point</title>
    <search ref="KPCC_CO_INB_Recon_Summary_Claim_REPORT"></search>
  </table>
</panel>
<panel>
  <title>KPCC_CO_OTB_Recon_Summary_Claim_REPORT</title>
  <table>
    <title>CO Summary counts of outbound external claim remits at each life cycle point</title>
    <search ref="KPCC_CO_OTB_Recon_Summary_Claim_REPORT"></search>
  </table>
</panel>

etc
etc

0 Karma

woodcock
Esteemed Legend

They key is to disable the emailing function. What is it that generates the email (I thought that it was the Alert send email action)?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...