Splunk Search

How can I pull and alert on a value found from a search?

larryleeroberts
Path Finder

I am trying to pull data from Splunk via a search and send it to Netcool OMNIbus. Right now I am just sending it via an Alert Action to my email to figure this out. In doing so, I cannot seem to find a way to lock on to the actual message in the recorded log event itself. I hope this makes sense. It seems like it is difficult to actually pull and send out the actual result of a search. Passing all the information used for the search seems easy. Am I missing something here? I am really new to Splunk.

For example, if you look at the screen below from my search in Splunk, it finds and returns the log event I was looking for but within the Alert Trigger I send out from Splunk via email, I want to actually send the log event which is...

 "[2016-10-14T13:14:57]:WARNING:HEMDP0173W:[WebContainer : 3]:No translation for severity 'P3-Low' could be found. Using the data source conversion instead."

Is this possible?
alt text

I do see that you can pass the following arguments...
Arg Environment Variable Value
0 SPLUNK_ARG_0 Script name
1 SPLUNK_ARG_1 Number of events returned
2 SPLUNK_ARG_2 Search terms
3 SPLUNK_ARG_3 Fully qualified query string
4 SPLUNK_ARG_4 Name of report
5 SPLUNK_ARG_5 Trigger reason
For example, "The number of events was greater than 1."
6 SPLUNK_ARG_6 Browser URL to view the report.
7 SPLUNK_ARG_7 Not used for historical reasons.
8 SPLUNK_ARG_8 File in which the results for the search are stored.

But none of these contain the actual value of the search result. The log entry which is what I want to send from Splunk via an Alert. So basically I guess I am looking for a way to actually send returned data of the search result.

0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

When you save or edit a search in recent versions of Splunk you are given an option to include the results. You add an action to send an email then put a checkmark beside the "Include" item "Inline". I can't remember, but I think this is in Splunk 6.3 and above. What version are you on?

Screen shot of including results in alert

View solution in original post

Richfez
SplunkTrust
SplunkTrust

When you save or edit a search in recent versions of Splunk you are given an option to include the results. You add an action to send an email then put a checkmark beside the "Include" item "Inline". I can't remember, but I think this is in Splunk 6.3 and above. What version are you on?

Screen shot of including results in alert

somesoni2
SplunkTrust
SplunkTrust

It's available in all Splunk versions.

0 Karma

larryleeroberts
Path Finder

Yes, I do get that and have played around with it today. It is true that it does pass the information. It would be nice though if I could actually assigned the entry to a field.

The main problem here is that if your trying to forward these events to say another monitoring product, you need to be able to map the fields. So for example, if I want to forward an Alert to something like Netcool OMNIbus, I need to be able to map it to fields. OMNIbus has tons of ways to receive events including an email probe but even that is an issue when it comes to Splunk. In order for that to even work, I would need total control over the email body and subject line that are sent. When setting up an Alert Action in Splunk, they do not appear to offer this option. I LOVE Splunk so far but not thrilled with the abilities to send data out.

0 Karma

Yasaswy
Contributor

hi,
check out tokens which gives the options to include fields from your search result. That said it is usually easier just to get it from SPLUNK_ARG_8 as is being suggested if you are using a script anyway.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You do get the reference to gzip file (results.csv.gz) which contains the raw result. You could decompress the file and see the content.

8   SPLUNK_ARG_8    File in which the results for the search are stored.
Contains raw results in gzip file format.
0 Karma

larryleeroberts
Path Finder

Thanks! True, but it really seems that there should be a much simpler method of doing this. I seems like it could be pulled directly as a field. Is this not possible then?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I just realized that you're looking to include the search result in Email Alert (the argument list that you provided is for Alert script and confused me). See @rich7177's answer for standard way of including results inline OR as attachment. You can also see this link to see what all other attributes (tokens) are available for use in the email alert
https://docs.splunk.com/Documentation/Splunk/6.5.0/AdvancedDev/ModAlertsLog#Pass_search_result_value...

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...