Alerting

Splunk alert no results

sonila
Path Finder

alt text

I have made an email alert. but when i click to view events on triggered alert i see no results. why this happens? how can i show the results?

Tags (3)
0 Karma

woodcock
Esteemed Legend

The only thing that I can think of is that your events are expiring between when the alert hits and when you double-check. This should tell you what the oldest event still in the index is. It should be weeks, if not months old but maybe it is hours or days old.

|metadata type=sourcetypes | search sourcetype=log4net
0 Karma

woodcock
Esteemed Legend

Usually when I have stuff that "tests OK" in an ad-hoc search but fails in a scheduled search it is due to pipeline latency. Check out the values of _indextime - _time for your events. These should be positive and no more than 300ish.

0 Karma

sonila
Path Finder

And what do you recommend after checking the values of _indextime - _time
_indextime - _time is less than 0 to my indexed data. What should i do?
_indextime - _time is around -9.581

0 Karma

woodcock
Esteemed Legend

Since the magnatude is so low, the problem is surely that your forwarders and/or indexers are not using NTP and have drifted from true. To see if it is your indexers, try this:

| rest /services/server/info 
| eval updated_t=round(strptime(updated, "%Y-%m-%dT%H:%M:%S%z")) 
| eval delta_t=now()-updated_t 
| eval delta=tostring(abs(delta_t), "duration") 
| table serverName, updated, updated_t, delta, delta_t

If delta is anything other than about 00:00:01 (which is easy to account for when processing a lot of indexers), you have clock skew and are a naughty boy because you should have setup NTP on your indexers.

NOTE: this IS a problem, but it is not the problem that you were asking about.

0 Karma

sonila
Path Finder

delta is 00:00:00 and _indextime - _time is around 9.581 it is positive

0 Karma

woodcock
Esteemed Legend

In that case never mind this whole answer.

0 Karma

sonila
Path Finder

can you help me about my problem why i dont see results in splunk?

0 Karma

kmaron
Motivator

Your trigger times in the capture show 12:27 to 12:34 but your search shows 1:11 to 1:21. Is it possible that there were no triggered events between 1:11 and 1:21? What if you change your search time frame to the 12:27-12:34?

0 Karma

sonila
Path Finder

I tried but no result again

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...