Getting Data In

Is it possible to include log contents from one sourcetype in an email alert triggered by a search on another sourcetype?

cacarpenter89
New Member

I've found a few different answers that approximate, but nothing yet that I can synthesize into a new solution for my environment.

Essentially, I have one log file that keeps a running log of the application's jobs (sourcetype 1). I have a second directory where technical logs of each job are kept and grouped in a subdirectory according to job group (sourcetype 2).

I'm alerting on the first sourcetype and want to include the contents of the second sourcetype in an email based on that alert condition (I can pull the filename from the first sourcetype). Though a failure of the job occurs, it is not certain that there will be any consistency in the messages within the technical logs that I can use to conduct a reliable second search.

Can I include the contents of a file of the second sourcetype in an email alert triggered by a search on the first sourcetype?

EDIT: Adding some detail.

Current search: sourcetype=sourcetype_1 "failure alert" | convert ctime(_time) as time | ...several regex field extractions... | table time,failed_group,count_failed,clients_total,total_status,count_succ,count_dis | sort -time

This gives me a table to monitor the jobs and work with stats. The technical log for any failed job is located at /app/logs/log_type/failed_group/filename. This tree from log_type down is sourcetype_2; filename (unique by job) can be pulled from a different line in sourcetype_1 than the one being matched with failed_group as the connector between sourcetypes.

I want to alert on the matches in the table built from sourcetype_1and include the contents of the relevant technical log file in an email.

Use case is non-technical monitoring people calling on-call technical people. Working on various stats to trigger phone calls from the monitoring folks (they don't take complex instruction well at all) and trying to reduce the need to come on-site in the event of a job failure.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

If you give some more details, we can write the query.
But, currently, this one will give you some hints.
https://answers.splunk.com/answers/114015/running-one-part-of-search-if-first-part-is-true.html

0 Karma

cacarpenter89
New Member

Added some detail to the original question. I think what will end up having to happen is an alert search separate from the dashboard search that includes the results of a second search as you detailed in your answer. PDF is probably going to be the best bet for this.

I'll get to work on altering this with the info you provided and see what I can come up with. Thanks!

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...