The scheduled cron time and trigger time is different

[AbhiTryingAgain]

Hi,

I have business use case of creating an alert wherein it has to search and trigger if the condition is matched, this alert is cron scheduled at 1pm from Monday through Friday.

The query: index=xyz | head 1 | eval month_year=strftime(now(),"%c") | table month_year

I work on IST zone, the splunk server is CST/CDT zone, but from the alert mail we can see that the search was executed on 1pm(13:00), but trigger time is 1:14 am CST, I received the alert mail on 11:44am IST.

Actually I should receive the mail on 11pm IST, Please help me out there.

Thanks

[Richfez]

Can you paste the actual cron entry in here?  From your further description, my guess is that it's just wrong somehow (or at least that's one of a few problems).

Also if this is still happening, have you tried the simple expedient of just *changing* the timings to make it come at the time you expect it to come?  I think if you take a careful and measured approach, changing one thing at a time and seeing what effect it has, you'll a) figure it out and b) also figure out *why* it's doing what it's doing.

[Richfez]

I think I've read this in its entirety 4 times now over the past week.  I am having difficulty understanding what the problem is.  Let me walk through it and see if writing it down helps...

You work in IST which is +10.5 hours from CST/DST.

You have alert, which the cron schedule says to fire at 1 PM (13:00) in CDT.  That's 11:30 PM (23:30) IST.  You maybe mistyped "11:00 PM" for that, and maybe that's the issue?

Disregarding the 11:00/11:30 issue, the second thing I think you mentioned is that the alert didn't actually come until 11:44, which is a 14 minute delay.   The search itself is a lightweight, it should run practically instantly and run-time shouldn't be an issue. 

The most obvious reason for the 14 minute delay is because your server is too busy at 1 PM CDT to get this out any faster.  You should check into that - there's a lot of resources available inside Splunk to see what might be going on, but my guess is just that it's a busy time of the day, coupled with possibly too many "heavy" searches that trigger then. 

You could also increase the priority of that search, though this doesn't address the core problem and may actually make things *worse* and not better.  I mean, maybe better for this one search, and being so fast that's probably OK, but still, it's just trying to hide the bigger problem.

Anyway, hope that helps and happy Splunking!

-Rich

[AbhiTryingAgain]

Hi Rich,

I am sorry for the poorly worded question.

"You have alert, which the cron schedule says to fire at 1 PM (13:00) in CDT.  That's 11:30 PM (23:30) IST. "

The issue is instead of receiving the mail at 11:30 PM (23:30) IST, I receive it on 11:30 am IST.

If you check the mail screenshot, you can see the inline query result returned wed Apr 3 13:00, but trigger time is April 4, 01:19 am CST, and the mail reached my inbox on April 4, 11:49 am IST.

Shouldn't it be actually April 3 13:19 CST and 23:49 IST?