Getting Data In

Is it possible to include log contents from one sourcetype in an email alert triggered by a search on another sourcetype?

cacarpenter89
New Member

I've found a few different answers that approximate, but nothing yet that I can synthesize into a new solution for my environment.

Essentially, I have one log file that keeps a running log of the application's jobs (sourcetype 1). I have a second directory where technical logs of each job are kept and grouped in a subdirectory according to job group (sourcetype 2).

I'm alerting on the first sourcetype and want to include the contents of the second sourcetype in an email based on that alert condition (I can pull the filename from the first sourcetype). Though a failure of the job occurs, it is not certain that there will be any consistency in the messages within the technical logs that I can use to conduct a reliable second search.

Can I include the contents of a file of the second sourcetype in an email alert triggered by a search on the first sourcetype?

EDIT: Adding some detail.

Current search: sourcetype=sourcetype_1 "failure alert" | convert ctime(_time) as time | ...several regex field extractions... | table time,failed_group,count_failed,clients_total,total_status,count_succ,count_dis | sort -time

This gives me a table to monitor the jobs and work with stats. The technical log for any failed job is located at /app/logs/log_type/failed_group/filename. This tree from log_type down is sourcetype_2; filename (unique by job) can be pulled from a different line in sourcetype_1 than the one being matched with failed_group as the connector between sourcetypes.

I want to alert on the matches in the table built from sourcetype_1and include the contents of the relevant technical log file in an email.

Use case is non-technical monitoring people calling on-call technical people. Working on various stats to trigger phone calls from the monitoring folks (they don't take complex instruction well at all) and trying to reduce the need to come on-site in the event of a job failure.

0 Karma

inventsekar
Ultra Champion

If you give some more details, we can write the query.
But, currently, this one will give you some hints.
https://answers.splunk.com/answers/114015/running-one-part-of-search-if-first-part-is-true.html

0 Karma

cacarpenter89
New Member

Added some detail to the original question. I think what will end up having to happen is an alert search separate from the dashboard search that includes the results of a second search as you detailed in your answer. PDF is probably going to be the best bet for this.

I'll get to work on altering this with the info you provided and see what I can come up with. Thanks!

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...