How to setup Splunk Alert when status not changed

[babujlinuz]

Hi Amigo,

I need to setup a Splunk alert when the status is not changed from "status = pending_app_gw." to "status = ACTIVE. with in 5-10 sec" max. if not changed to active it should alert.

08.02.2017 07:25:26.280 | 344 | INFO | 10098094 | GW: session(90200371) status = INITIALIZING.
08.02.2017 07:25:26.280 | 344 | INFO | 10098094 | GW: session(90200371) status = pending_app_gw.

*****after restart of the application *******
08.02.2017 08:20:36.618 | 1752 | INFO | 10098094 | GW: session(90200371) status = INITIALIZING.
08.02.2017 08:20:36.618 | 1752 | INFO | 10098094 | GW: session(90200371) status = pending_app_gw.
08.02.2017 08:20:36.706 | 5344 | INFO | 10098094 | GW: session(90200371) status = ACTIVE.

I tried with the below query. but it is not working as expected.
pending_app_gw sourcetype=sesionlog |rex "^[^(\n]*((?P\d+)"|join ses_id [ search sourcetype=oslog earliest=-10s latest=now "status = ACTIVE."]|table host ses_id

Appreciate your help to setup the alert working properly.

Thanks in Advance.

Regards
Babujlinuz

[niketn]

Since you do not have a unique transaction ID, you can make use of streamstats. First you would need to perform field extractions. Since status is key value pain in your data, it should be automatically extracted as search time.
You can use Pipe delimited Interactive Field Extraction in Splunk
transforms.conf

[sesionlog]
DELIMS = "|"
FIELDS = "field1","thread_id","log_level","trans_id","status_details"

Finally, for Session ID you can use regular expression based field extraction.
props.conf

[sesionlog]
EXTRACT-session_id = ^[^\(\n]*\((?P<session_id>\d+)

Finally try streamstats like following which resets counter every time after finding an Active status(there can be multiple options)

index=<YourIndexName> sourcetype=sesionlog status="pending_app_gw." OR status="ACTIVE."
| table _time session_id trans_id thread_id log_level status 
| reverse
| streamstats count current=f reset_after="("status==\"ACTIVE.\"")" 
| stats count as eventcount min(_time) as MinTime max(_time) as MaxTime values(status) as status values(count) as CountFlag by session_id trans_id
| search (eventcount=3 AND CountFlag=2 AND status="pending_app_gw." AND status="ACTIVE.") OR (eventcount=1 AND CountFlag=0 AND status="pending_app_gw." AND status!="ACTIVE.")
| eval OrphanedPendingStatusDuration=now()-MinTime
| rename MinTime as OrphanedPendingStatusTime
| fieldformat OrphanedPendingStatusTime=strftime(OrphanedPendingStatusTime,"%c")
| fieldformat MaxTime=strftime(MaxTime,"%c")

PS: Based on your data there might be other combinations as well. Like (eventcount=2 AND CountFlag=1 AND status="pending_app_gw." AND status="ACTIVE.") represents all expected Pending Status followed by Active Status.

[DalJeanis]

(your base search) 
| table ses_id status 
| dedup ses_id 
| where status!="ACTIVE" 
| addinfo 
| where _time < info_max_time - 10

Basically, any status that is not ="ACTIVE" as/of 10 seconds before the end of the search needs an alert.

However, it seems like there ought to be identifying information about WHAT is running... the host or service name , so you need to add that into the code as appropriate. Let's assume that it was appname. It could just as well be sourcetype or host or whatever.

(your base search) earliest=-1h latest=-10s 
| table  ses_id appname status 
| dedup ses_id appname 
| where status!="ACTIVE" 
| addinfo 
| where _time < info_max_time - 10

[adonio]

Hi babujlinuz,
Take a look here: https://answers.splunk.com/answers/177265/how-to-configure-an-alert-for-when-a-given-event-i-1.html