To monitor files stuck more than 4hrs

[rchittip]

Dears,
I need to monitor files in a directory which stuck more than 4hrs without any update.

We need alert if files doesn't get updated for more than 4hrs.

Can this be achieved from Splunk. Kindly assist me in addressing this request.

Thanks,
Ramu Chittiprolu

[rchittip]

Hi,
Team has come with the different requirement. PFB.

1.In a directory, files are generated through a script (~25files) every 4hrs and within 10 mins all of them gets deleted by a script
2.Incase if any file doesn't get deleted and stuck in the directory for 4 hrs, alert needs to be triggered.

They don't want to go trigger alert, if the directory is not updated in 4hrs.
Alert needs to be triggered only for files which are stuck in directory for 4hrs.

Thanks,
Ramu Chittiprolu

[mayurr98 · Accepted solution]

hey

I am assuming that if the file is stuck more than 4 hrs without any update means that it will not get an index into Splunk. So one logic is to send an alert if the data is not indexed into Splunk more than 4 hours.
for eg. at 12 pm if u get data into Splunk so don't send an alert and if at 4 pm if the file will not get an update so it won't index any new data so you will get alert
so you need to schedule alert every four hours!

here is a search query you can try

index=<your_index> | stats latest(_indextime) as index_time | eval latest_time=now()-14400 | eval alert=case(latest_time>index_time,1) | fields alert

let me know if this helps !

[rchittip]

Thanks for the reply.

What is the alert condition to trigger the email alerts here. I have setup email alert to "To Trigger alert when Number of Results is greater than Zero. But eventhough log file updated, I received the email alert.

Kindly update the trigger condition of email.

Thanks,
Ramu Chittiprolu

[mayurr98]

Have you ran this query ? See if you are getting events in splunk. Also i have wrote this in general you need to customize it as if you have many files and you are checking for a specific file then it wont run . In that case you need to add source in your base search. Trigger condition should when alert=1..so basically my query is...get the latest event time and a threshold which is current time minus last four hours. So if the threshold is greater than the latest event time then alert will set to 1.
Also specify the source of that file in base search.before creating an alert first see if you getting proper results. See if you do not get data from last four hours then but this query and check if alert is set to 1. And check vice-versa as well if you are getting events in last four hours then alert field should not be created.

[rchittip]

I am able to see data in splunk and with the below query I'm getting the output as well. I need to trigger email alert if "alert=1". I did this for number of results greater than zero and I'm getting alert everytime. Though the below condiation is false and return balnk value for alert.

sourcetype=CommerceHub | stats latest(_indextime) as index_time | eval latest_time=now()-14400 | eval alert=case(latest_time>index_time,1) | eval index_time=strftime(index_time,"%Y-%m-%d %H:%M:%S") | eval latest_time=strftime(latest_time,"%Y-%m-%d %H:%M:%S") | fields alert
when i try triggering alert=1 under custom, value is not taking. Getting the below error.

Cannot parse alert condition. Search Factory: Unknown search command 'alert'.

[mayurr98]

the trigger condition should be where alert=1.
Also in your search query at the end write fillnull
sourcetype=CommerceHub | stats latest(_indextime) as index_time | eval latest_time=now()-14400 | eval alert=case(latest_time>index_time,1) | eval index_time=strftime(index_time,"%Y-%m-%d %H:%M:%S") | eval latest_time=strftime(latest_time,"%Y-%m-%d %H:%M:%S") | fields alert | fillnull
so you will get 0 instead of blank when the condition is false.1 when the condition is true.