Setting up alert to send email when an web service is down

[snchow]

Hi: I am trying to set up an alert at Splunk to send me an email when one of our web services is down. We use Java Spring Boot web service. Using Docker image at Rancher servers. I am new to Splunk. I need help creating the search string to create the alert. Say, abc is the web service name
We use search string:

index=xyz_nonprod sourcetype=abc-wsqa-logger 

The docker-compose.yml has the following info , ->

splunk-token: ${SPLUNK_TOKEN}
splunk-url: ${SPLUNK_URL}
splunk-index: ${SPLUNK_INDEX}
splunk-sourcetype: "abc-ws${APPLICATION_ENV }-logger"
splunk-source: "HttpEventCollectorLogbackAppender"
splunk-insecureskipverify: "true"
tag: "{{.Name}}/{{.FullID}}"

I found sample example to create the a search as

index=_internal " error " NOT debug source=*splunkd.log*

Need help to fit it in my case to set up an alert when service abc is down. What would be the search string?
Thanks
Nahid Chow

[woodcock]

Why not keep a list of sourcetypes that are logging as a lookup? You can create one like this:

index=xyz_nonprod earliest=0 latest=now
| dedup sourcetype
| sort 0 - sourcetype
| outputlookup logger_sourcetypes.csv

Then keep this up-to-date manually or by topping it off with a scheduled search.
Then you can run a search like this every hour:

index=xyz_nonprod earliest=-1h latest=now
| stats count by sourcetype
| lookup logger_sourcetypes.csv OUTPUT sourcetype AS MATCHED
| appendpipe [|inputlookup logger_sourcetypes.csv]
| stats values(*) AS * BY sourcetype
| where isnull(MATCHED)

[snchow]

Hi: Thanks for your response. I am new to Splunk- taking time to conceptualize different aspects! When I try to create a new Look up Definition at Splunk "Settings" -> choose a destination App, Give a Name, choose type (file-based as default) and Choose a Lookup file from a drop-down. When I go to create a new look up table file-> it asks to upload a (.csv) lookup table file.

1. As you have mentioned logger_sourcetypes.csv, is this the one to create first and then upload? What will be the content of this file?
2. Where does the content you specified as
index=xyz_nonprod earliest=0 latest=now
| dedup sourcetype
| sort 0 - sourcetype
| outputlookup logger_sourcetypes.csv
actually stay?
Thanks for the support.