I have an alert that sets 2 actions, sending emails and webhook.
if I set Trigger Condition as once,webhook alert sends only the first row of events in the table.( User unknown: ccc),The following screenshot is the payload sent by webhook alert, which you can see captures only the first line of events
The following screenshot is my e-mail content,my mailbox received an e-mail containing four lines events.
This is not my ideal result.
if I set Trigger Condition
as For each result
When the alert trigger returns the multi-line event. webhook will send each search result in turn and I will get a lot of emails,It's like an email bomb. I wish I could get an email containing all the search results.
This is also not my ideal result.
so. I hope it can send the entire search results and an email with all the search results.
My alert
Enabled: Yes
App: search
Permissions: Shared in App
Alert Type: Scheduled (Run on cron schedule)
Time Range : Last 5 minutes
Cron Expression: */5 * * * *
Trigger Condition: Number of Results is > 0
Actions: Send Email + Webhook
So I want to solve this problem. I hope webhook alert can send payload containing the entire search results (4 events in a table). But I don't want to merge them into one line through stats or any other command.
in addition. I can't set Trigger Condition as For each result,otherwise, it will be an email bomb, which constantly sends me emails, each containing the result of one line of events, that is, when the alert triggers, sendemail will be split into four emails and sent to me.
Do custom alerts require development ability? Do I need to do a lot of development work?
We have faced similar issue. We have change the alert output to Json format as one entry and send it to Jenkins. it works for us. Please try to convert alert output to Json. it works.
Hi @suresh401 , I need to send the entity/host information like "kubernetes_node" which we get from the correlation search as a payload to my jenkins to trigger a job that can restart that perticular node. How can I get those payload automatically populated with 'kubernetes node address' and get passed to the Jenkins?
Hello !! Can someone solve this problem? Context, presented the same problem example, a search that runs every 5 minutes, the result is mailed (all 3 events are sent), a notable event is generated (3 notable events are generated), but when reviewing the webhook only receives the first event.
SH internal sendalert
for this example every 5 minutes more than 2 events are generated and only one POST is sent that refers to the first event, but a second request to the webhook is not evidenced
Capture on webhook server only 1 packet received (POST)
Does anyone know how to send multiple events?
Thank you
hi @bestSplunker
You can only get a json paylod by using a custom alert, why don't you try something like below?
I have used _audit index so you can run the queries as it is.
Now, assuming I am running an alert every 15 minutes that checks for action = search and should trigger an webhook alert with the alert results, see screen shot1
index="_audit"
| where action="search"
| table _time,action,date_wday
our aim is that the webhook should send all results BUT as we know it is only sending the first row.
Now, what if I combine all results into 1 single row? webhook WILL send 1 single row.
so the above query becomes something like this, I am using dummy json like formatting,
index="_audit"
| where action="search"
| table _time,action,date_wday
| eval _time=strftime(_time,"%Y-%m-%d %H:%M:%S") | eval result="{\""+"time"+"\""+":"+_time+"}"+","+"{\""+"action"+"\""+":"+action+"}"+"{\""+"day"+"\""+":"+date_wday+"}"
| fields result
| fields - _time
| mvcombine result
Now, this becomes 1 single row in the result, see second snap
Now when you use webhook this will return you all results in a json like format , since now we have just 1 row.
For the moment forget about the formatting of the result string,rather try to see if the webhook fires with 1 single multi valued field, which is nothing but the combination of all alert results, made into a single field.
You will have to retrofit your results a bit or if you can use the _audit index and test the webhook alert, that also should be fine
hi @bestSplunker
Can you please check and confirm on your issue? Please accept the answer if it significantly helped resolve your issue. Do not forget to add/modify the answer if you did some modifications and then accept the answer.
@Sukisen1981 thank you for your reply ,but It doesn't seem like a very easy way to do that. I have more than 100 alerts. Do I need to merge each one alert into a dummy json? In addition, this format looks very ugly in mail messages.
You could save the alert twice and have one trigger on each result for the webhook (or use @Sukissen 's formatting to send in one row) and the other to trigger once for the email.
hi @bestSplunker
I can feel your pain, the issue here is webhook will only take the first result row by default.
Now, you have 100 alerts or more, so whether you go the custom alert route or the ugly multivalue route, you have to sort of 're-author' the 100 alerts.
the webhook alert documentation does not explicitly say that webhook takes only the first result row BUT all the examples point towards that.
You also can not use per result since you will get too many mails. See if you can reach out to splunk support directly on this issue.
@Sukisen1981 I added pictures to the post and updated it.
quote -"it seems the webbhook sends the first result from the search results. but the mailbox received a message containing multiple search results."
are you sure on this? The mail will contain multiple events from the same search which is used for alert.
For example, if you are counting alerts of sat https status 404. Now, you run it once every 5 minutes and your search is something like index=xxxx status=404|table host,source,status
You need to run this search FOR the last 5 minutes providing earliest / latest AND run it once every 5 minutes, which will give you a table of all the 404 errors in the last 5 minutes in your email
something like -
_index_earliest=-5m _index_latest=now index=your index | rest of the stuff...
yes, I am sure, don't know if you understand what I mean? sorry, my English is not good. I mean is that when an alert is triggered, the alert result returns a multi-line event (e.g., three results), and the webhook sends only the first line event result. but my email received a complete list of events(i.e., mail containing three results)
index=xxxx status=404|table host,source,status
1.1.1.1 /www/logs/nginx.log 404
2.2.2.2 /www/logs/nginx.log 404
3.3.3.3 /www/logs/nginx.log 404
webhook send only the first line of results 1.1.1.1 /www/logs/nginx.log 404
, But the received alert mail message contain three lines of event record
hi @bestSplunker
I understand now, you are saying that if you trigger the webhook alert it only triggers for the first line, that is how a webhook alert behaves. There is a way to sort of do this through custom alert but before that.
When you say multiline do you mean 3 separate lines in a table? For example, the table above does not give you multiline - it gives you 3 separate events. How does your result look like?
The reason I am asking is - what if we can combine all events (like the 3 separate lines in the above 404 example) into 1 line containing all 3 events? Then when the webhook fires it will pick one line (the first line) BUT we all have information for all events if we can combine your results into 1 line.
Can you share a screen snap of your query and how the alert output looks in statistics tab?
ok,
if I set Trigger
Condition as once
,webhook alert sends only the first row of events in the table.( User unknown: ccc),The following screenshot is the payload sent by webhook alert, which you can see captures only the first line of events
The following screenshot is my e-mail content,my mailbox received an e-mail containing four lines events.
So I want to solve this problem. I hope webhook alert can send payload containing the entire search results (4 events in a table). But I don't want to merge them into one line through stats
or any other command.
in addition. I can't set Trigger Condition
as For each result
,otherwise, it will be an email bomb, which constantly sends me emails, each containing the result of one line of events, that is, when the alert triggers, sendemail
will be split into four emails and sent to me.
Do custom alerts require development ability? Do I need to do a lot of development work?
hi @bestSplunker can you upload the images to imgur or something and share, they are not visible here.
Firstly, webhook alert will not send consolidated results refer here - https://docs.splunk.com/Documentation/Splunk/7.3.1/Alert/Webhooks
see this section in the above document
Webhook data payload
The webhook POST request's JSON data payload includes the following details.
Search ID or SID for the saved search that triggered the alert
Link to search results
Search owner and app
First result row from the triggering search results
can you please upload the images
@Sukisen1981 I have updated the picture link, you could see it.
Thank you, so how should I solve this problem?