I have a requirement to check if a employee shift roster(lookup in Splunk) covers 24 hours in a day for each team. If it doesn't cover, I need to send out an alert to the respective team notifying them that their respective shift roster is not configured properly. Can anybody help me out as to how I can proceed in this. The employee_shift_roster.csv looks something like this: Start time End time Team Employee Name Available 8:00 5:30 Team A Roger Y 5:30 8:00 Team A Federer Y 8:00 5:30 Team B Novak Y 5:30 7:00 Team B Djokovic Y   Now the alert should go out to Team B stating that their shift roster is not configured properly because 24 hours are not cover in shift. Thanks in advance for the help 🙂 ... View more

We have different levels of data flow coordinated by a set of saved searches. We divide them into three tiers where the data after 3rd tier is displayed on the dashboard. Everyday for 30-40 minutes, this data flow is interrupted. The internal logs show nothing out of the ordinary and data flow is stuck at tier 2 saved searches. There are no skipped saved searches as well during this time and the server health is at top notch as well.  The trend of time range of the issue increases daily i.e. first day issue happened between 4-4:45, next day 5-5:45 and so on. Yesterday this happened between 10PM -10:45PM IST. There is no network connectivity issue as well as data is coming in tier 1 index. Please check the attached image.  Thanks in advance.   ... View more

Hi all, I made an alert which sends out mail to the respective teams whenever a high priority task has not been updated for more than an hour. The query is as follows:- index="abc" INC* main_metric, state="New" OR state="In Progress" OR state="Awaiting Third Party" OR state="Pending" priority = "1 - Critical" OR priority = "2 - High" | rex field=_raw "main_metric=\"(?<main_metric>\S+\s\d+\:\d+\:\d+)\"" | dedup main_metric | dedup number | eval main_metric = upper(main_metric) | lookup lookup_inactivity_alert_distribution_list.csv assignment_group OUTPUT "Email_To" "Email_Cc" "Email_Bcc" "Enabled" | fillnull value=0 | search number != 0 AND Enabled = "Y" AND main_metric != 0 | eval end=strptime(main_metric, "%Y-%m-%d %H:%M:%S.%N") | eval start=now() | eval diff = start - end | lookup lookup_frequency_impact.csv impact output "Frequency1" "Frequency2" "Frequency3" "Frequency4" | eval freqdiff1 = Frequency1 + 600 | eval freqdiff2 = Frequency2 + 600 | eval freqdiff3 = Frequency3 + 600 | eval freqdiff4 = Frequency4 + 600 |eval result = case('caller_id' = "SCOM System" AND 'diff' >= 'Frequency3' AND 'diff' <= 'freqdiff3',"outcome1",'caller_id' != "SCOM System" AND 'diff' >= 'Frequency1' AND 'diff' <= 'freqdiff1',"ouitcome2",'caller_id' != "SCOM System" AND 'diff' >= 'Frequency2' AND 'diff' <= 'freqdiff2',"outcome3",'caller_id' != "SCOM System" AND 'diff' >= 'Frequency3' AND 'diff' <= 'freqdiff3',"outcome4",'caller_id' != "SCOM System" AND 'diff' >= 'Frequency4' AND 'diff' <= 'freqdiff4',"outcome5",1==1,"no outcome") | search result="outcome1" OR result="outcome2" OR result="outcome3" OR result="outcome4" OR result="outcome5" AND state!="Closed" | table main_metric priority caller_id result assignment_group u_updated_on "Email_To" Email_Cc Email_Bcc number start end diff Frequency1 Frequency2 Frequency3 Frequency4 | map alert_main_metric_mail assignment_group="$assignment_group$" to="$Email_To$" cc="$Email_cc$" bcc="$Email_Bcc$" The second lookup handles the frequency with which the alert emails are to be sent with respect to the priority of the ticket. Now the problem that I am having is that if the ticket or task is closed within half an hour of it being created, the alert is still generated. Even if the ticket is de-escalated, the alert is still being received. I tried many modifications in the code but nothing seems to work. Could you all help me with this bug? P.S.: The map commands connect just the saved search which sends out the emails with the appropriate subject and description. ... View more