Alerting

How to send entire search result payload and send an email with entire search results?

bestSplunker
Contributor

I have an alert that sets 2 actions, sending emails and webhook.

if I set Trigger Condition as once,webhook alert sends only the first row of events in the table.( User unknown: ccc),The following screenshot is the payload sent by webhook alert, which you can see captures only the first line of events
alt text

The following screenshot is my e-mail content,my mailbox received an e-mail containing four lines events.
alt text

This is not my ideal result.

if I set Trigger Condition as For each result
When the alert trigger returns the multi-line event. webhook will send each search result in turn and I will get a lot of emails,It's like an email bomb. I wish I could get an email containing all the search results.
This is also not my ideal result.

so. I hope it can send the entire search results and an email with all the search results.

My alert

Enabled: Yes

App: search

Permissions: Shared in App

Alert Type: Scheduled (Run on cron schedule)

Time Range : Last 5 minutes

Cron Expression: */5 * * * *

Trigger Condition: Number of Results is > 0

Actions: Send Email + Webhook

So I want to solve this problem. I hope webhook alert can send payload containing the entire search results (4 events in a table). But I don't want to merge them into one line through stats or any other command.

in addition. I can't set Trigger Condition as For each result,otherwise, it will be an email bomb, which constantly sends me emails, each containing the result of one line of events, that is, when the alert triggers, sendemail will be split into four emails and sent to me.

Do custom alerts require development ability? Do I need to do a lot of development work?

Labels (1)
0 Karma

suresh401
Engager

We have faced similar issue. We have change the alert output to Json format as one entry and send it to Jenkins. it works for us. Please try to convert alert output to Json. it works. 

0 Karma

danny1
Loves-to-Learn

Hi @suresh401 , I need to send the entity/host information like "kubernetes_node" which we get from the correlation search as a payload to my jenkins to trigger a job that can restart that perticular node. How can I get those payload automatically populated with 'kubernetes node address' and get passed to the Jenkins?

0 Karma

ysposada
Loves-to-Learn Lots

Hello !! Can someone solve this problem? Context, presented the same problem example, a search that runs every 5 minutes, the result is mailed (all 3 events are sent), a notable event is generated (3 notable events are generated), but when reviewing the webhook only receives the first event.

SH internal sendalert

for this example every 5 minutes more than 2 events are generated and only one POST is sent that refers to the first event, but a second request to the webhook is not evidenced

2022-11-18_18-19.png

Capture on webhook server only 1 packet received (POST)

2022-11-18_18-16.png

 

Does anyone know how to send multiple events?

Thank you

 

0 Karma

Sukisen1981
Champion

hi @bestSplunker
You can only get a json paylod by using a custom alert, why don't you try something like below?
I have used _audit index so you can run the queries as it is.
Now, assuming I am running an alert every 15 minutes that checks for action = search and should trigger an webhook alert with the alert results, see screen shot1

index="_audit" 
| where  action="search"
| table _time,action,date_wday

alt text

our aim is that the webhook should send all results BUT as we know it is only sending the first row.
Now, what if I combine all results into 1 single row? webhook WILL send 1 single row.
so the above query becomes something like this, I am using dummy json like formatting,

index="_audit" 
| where  action="search"
| table _time,action,date_wday 
| eval _time=strftime(_time,"%Y-%m-%d %H:%M:%S") |  eval result="{\""+"time"+"\""+":"+_time+"}"+","+"{\""+"action"+"\""+":"+action+"}"+"{\""+"day"+"\""+":"+date_wday+"}"
| fields result
| fields - _time
| mvcombine result

Now, this becomes 1 single row in the result, see second snap
alt text

Now when you use webhook this will return you all results in a json like format , since now we have just 1 row.
For the moment forget about the formatting of the result string,rather try to see if the webhook fires with 1 single multi valued field, which is nothing but the combination of all alert results, made into a single field.
You will have to retrofit your results a bit or if you can use the _audit index and test the webhook alert, that also should be fine

Sukisen1981
Champion

hi @bestSplunker
Can you please check and confirm on your issue? Please accept the answer if it significantly helped resolve your issue. Do not forget to add/modify the answer if you did some modifications and then accept the answer.

0 Karma

bestSplunker
Contributor

@Sukisen1981 thank you for your reply ,but It doesn't seem like a very easy way to do that. I have more than 100 alerts. Do I need to merge each one alert into a dummy json? In addition, this format looks very ugly in mail messages.

0 Karma

AnnaVakulyuk
Engager

You could save the alert twice and have one trigger on each result for the webhook (or use @Sukissen 's formatting to send in one row) and the other to trigger once for the email.

0 Karma

Sukisen1981
Champion

hi @bestSplunker
I can feel your pain, the issue here is webhook will only take the first result row by default.
Now, you have 100 alerts or more, so whether you go the custom alert route or the ugly multivalue route, you have to sort of 're-author' the 100 alerts.
the webhook alert documentation does not explicitly say that webhook takes only the first result row BUT all the examples point towards that.
You also can not use per result since you will get too many mails. See if you can reach out to splunk support directly on this issue.

0 Karma

Sukisen1981
Champion

hi @bestSplunker it comes like this for me 😞 😞
alt text

0 Karma

bestSplunker
Contributor

@Sukisen1981 I added pictures to the post and updated it.

0 Karma

Sukisen1981
Champion

quote -"it seems the webbhook sends the first result from the search results. but the mailbox received a message containing multiple search results."
are you sure on this? The mail will contain multiple events from the same search which is used for alert.
For example, if you are counting alerts of sat https status 404. Now, you run it once every 5 minutes and your search is something like index=xxxx status=404|table host,source,status
You need to run this search FOR the last 5 minutes providing earliest / latest AND run it once every 5 minutes, which will give you a table of all the 404 errors in the last 5 minutes in your email
something like -
_index_earliest=-5m _index_latest=now index=your index | rest of the stuff...

0 Karma

bestSplunker
Contributor

yes, I am sure, don't know if you understand what I mean? sorry, my English is not good. I mean is that when an alert is triggered, the alert result returns a multi-line event (e.g., three results), and the webhook sends only the first line event result. but my email received a complete list of events(i.e., mail containing three results)

index=xxxx status=404|table host,source,status

 1.1.1.1   /www/logs/nginx.log  404
 2.2.2.2  /www/logs/nginx.log  404
 3.3.3.3 /www/logs/nginx.log  404

webhook send only the first line of results 1.1.1.1 /www/logs/nginx.log 404, But the received alert mail message contain three lines of event record

0 Karma

Sukisen1981
Champion

hi @bestSplunker
I understand now, you are saying that if you trigger the webhook alert it only triggers for the first line, that is how a webhook alert behaves. There is a way to sort of do this through custom alert but before that.
When you say multiline do you mean 3 separate lines in a table? For example, the table above does not give you multiline - it gives you 3 separate events. How does your result look like?
The reason I am asking is - what if we can combine all events (like the 3 separate lines in the above 404 example) into 1 line containing all 3 events? Then when the webhook fires it will pick one line (the first line) BUT we all have information for all events if we can combine your results into 1 line.
Can you share a screen snap of your query and how the alert output looks in statistics tab?

0 Karma

bestSplunker
Contributor

ok,

if I set Trigger Condition as once,webhook alert sends only the first row of events in the table.( User unknown: ccc),The following screenshot is the payload sent by webhook alert, which you can see captures only the first line of events

alt text

The following screenshot is my e-mail content,my mailbox received an e-mail containing four lines events.

alt text

So I want to solve this problem. I hope webhook alert can send payload containing the entire search results (4 events in a table). But I don't want to merge them into one line through stats or any other command.

in addition. I can't set Trigger Condition as For each result,otherwise, it will be an email bomb, which constantly sends me emails, each containing the result of one line of events, that is, when the alert triggers, sendemail will be split into four emails and sent to me.

Do custom alerts require development ability? Do I need to do a lot of development work?

0 Karma

Sukisen1981
Champion

hi @bestSplunker can you upload the images to imgur or something and share, they are not visible here.
Firstly, webhook alert will not send consolidated results refer here - https://docs.splunk.com/Documentation/Splunk/7.3.1/Alert/Webhooks
see this section in the above document
Webhook data payload
The webhook POST request's JSON data payload includes the following details.

Search ID or SID for the saved search that triggered the alert
Link to search results
Search owner and app
First result row from the triggering search results
can you please upload the images

0 Karma

bestSplunker
Contributor

@Sukisen1981 I have updated the picture link, you could see it.

Thank you, so how should I solve this problem?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...