Alerting

key word in trigger text

jclemons7
Path Finder

Hello all,

I'm writing some trigger automation which will send trigger details to a WCF service for further processing, alerting and data enrichment. I'm pretty much done at this point.. I've got the Splunk trigger kicking off a .sh script, which then kicks off some Perl code (borrowed from this site) that reads the .csv file created by the trigger and passes the 9 environment variables along with the entire output of the .csv file to the WCF service successfully.

However, on the WCF side I'm writing some logic that can do different things with different triggers (e.g.. if the term = a do this if the term = b then do this etc..). My problem is that we've written a single realtime trigger which checks for multiple conditions (instead of writing a single trigger per condition) and I'm not really sure how to tell which condition was met to trigger the alert. In my testing it seems simple because I'm manually triggering syslog events with values like (testa, testb, testc etc..) but in our production environment there will be more complex trigger conditions like regex and stuff.

So, long question short, what should I be looking at to determine which trigger condition sent the data without having to replicate / maintain a copy of my Splunk rules in my script logic.

Any help is greatly appreciated.. thanks

0 Karma
1 Solution

lguinn2
Legend

Based on your reply, I would do this

index=ios 
    %L2FM-1-L2FM_LINE_FULL 
 OR "storm control" 
 OR %ISIS_OTV-5-ADJCHANGE 
 OR (sourcetype=cisco:ios facility=OSPF mnemonic=ADJCHG AND "FULL to DOWN") 
 OR (sourcetype=cisco:ios facility=DUAL mnemonic=NBRCHANGE NOT "new adjacency")
| eval triggerCondition=case(searchmatch("sourcetype=cisco:ios facility=DUAL"),"condition5",
                             searchmatch("sourcetype=cisco:ios facility=OSPF"),"condition4",
                             searchmatch("%ISIS_OTV-5-ADJCHANGE"),"condition3",
                             searchmatch("storm control"),"condition2",
                             searchmatch("%L2FM-1-L2FM_LINE_FULL" ),"condition1",
                             1==1,"other")

I don't know that the searchmatch function is terribly efficient, but it should be okay. I tried to put the most specific matches first in the case function, but you might want to order it as most common first. The case function tries each option in succession and takes the first match; once it finds a match, it does not look further. The last condition is a catch-all; you should never see this unless you have made a mistake in the earlier conditions.

View solution in original post

lguinn2
Legend

Based on your reply, I would do this

index=ios 
    %L2FM-1-L2FM_LINE_FULL 
 OR "storm control" 
 OR %ISIS_OTV-5-ADJCHANGE 
 OR (sourcetype=cisco:ios facility=OSPF mnemonic=ADJCHG AND "FULL to DOWN") 
 OR (sourcetype=cisco:ios facility=DUAL mnemonic=NBRCHANGE NOT "new adjacency")
| eval triggerCondition=case(searchmatch("sourcetype=cisco:ios facility=DUAL"),"condition5",
                             searchmatch("sourcetype=cisco:ios facility=OSPF"),"condition4",
                             searchmatch("%ISIS_OTV-5-ADJCHANGE"),"condition3",
                             searchmatch("storm control"),"condition2",
                             searchmatch("%L2FM-1-L2FM_LINE_FULL" ),"condition1",
                             1==1,"other")

I don't know that the searchmatch function is terribly efficient, but it should be okay. I tried to put the most specific matches first in the case function, but you might want to order it as most common first. The case function tries each option in succession and takes the first match; once it finds a match, it does not look further. The last condition is a catch-all; you should never see this unless you have made a mistake in the earlier conditions.

jclemons7
Path Finder

This worked perfectly.. value comes through CSV triggerCondition which my WCF can then use for separation logic. Thanks a ton.

0 Karma

jclemons7
Path Finder

I definitely can change the trigger, as I have some control over it.. and this does look like the answer I'm looking for but can you help me with the query itself? I have this currently, but have no idea how to get it formatted as you have above.

index=ios 
   %L2FM-1-L2FM_LINE_FULL 
OR "storm control" 
OR %ISIS_OTV-5-ADJCHANGE 
OR (sourcetype=cisco:ios facility=OSPF mnemonic=ADJCHG AND "FULL to DOWN") 
OR (sourcetype=cisco:ios facility=DUAL mnemonic=NBRCHANGE NOT "new adjacency")
0 Karma

lguinn2
Legend

Could you create a field as part of your realtime search that would give you this information? For example

yoursearchhere
| eval triggerCondition=case(field1>field2,"field1 is greater than field2",
                             count>0,"count greater than zero"
                             1==1,"Other")
| where field1>field2 OR count>0

The field should show up in the CSV file of results.

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...