Splunk Search

Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

wsnyder2
Path Finder

1969 ... wow, what a year!

We have noticed a problem where we find users complaining that they can't search, and in most cases, their work is all queued. They often have one or more running jobs where the Dispatched at date/time is 12/31/69 7:00:00 PM.

This usually means they are trying to search all-time.

Is there are a way to search and alert on this condition?
We would like to find these as they are happening rather than after the user is bogged down.

Also, any tips on debugging this?

andrewcg
Path Finder

After upgrading to 6.4.1, these seem to have disappeared for us.

0 Karma

bk028s
Path Finder

Snoobzilla, did you ever find a reasonable solution to this problem? It seems that it has begun happening to me just within the past day or so.

0 Karma

snoobzilla
Builder

I suspect these are scheduled jobs that have not started yet (and should have) so if your instance is running behind on scheduled jobs these start to stack up... which of course is exactly when you are most likely to be looking at activity, when things are slow. Suspicion is based on feedback from related case we are working through with Splunk.

Take a look at your skipped jobs ratio at the same time you are seeing these in DMC app/splunk_management_console/scheduler_activity_deployment

0 Karma

the_wolverine
Champion

There is a rt search that Splunk decides to launch when a user loads the Home page. So your users do not intentionally run this search, Splunk slid in in by default for silly reasons, IMO.

Luckily, in recent versions this can be disabled:

https://answers.splunk.com/answers/141179/how-to-remove-automatic-real-time-searches-that-run-when-u...

0 Karma

woodcock
Esteemed Legend

The very first thing that I do and suggest that all of my clients do is to COMPLETELY DISABLE realtime by removing it from all roles and deleting it from the TimePicker settings. It is too dangerous and tricky to be let loose except in very VERY specific (and highly uncommon) use cases. Kill realtime dead and all of these problems might go away.

j6white
Path Finder

This is simply not the case. Real-time searches do not coincide with these issues (which we are seeing as well). These problems simply cannot be attributed to real-time searches.

0 Karma

snoobzilla
Builder

Will have admin check, but I think we killed real time.

We do have accelerated data models and reports.

0 Karma

ArthurGautesen
Path Finder

I get jobs dispatched at 12/31/69 7:00:00 PM, but I was not attempting to search all-time, I was using a relative 7 Day search starting at midnight (to be specific earliest=-7d@d latest=now). Once one of these appears, every attempt afterwards has this same 1969 timestamp.

I usually have to Finalize each one, then Delete it afterwards, wait a few minutes and try again.

Oh, by the way, it would not matter if it was -7days at midnight or -7days from this time, the issue appears somewhat randomly and stays for a few minutes to an hour before it goes away. It also occurs when I'm just doing a |inputlookup Tablename.csv and using the default relative 24hour time, that same 1969 timestamp in the Dispatched At field in the Jobs area just appears, and everything is on hold until I can Finalize, Delete, and retry in a few minutes (sometimes repeatedly).

snoobzilla
Builder
0 Karma

andrewcg
Path Finder

Any resolution? We have 2200 of these "jobs".

0 Karma

snoobzilla
Builder

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

0 Karma

snoobzilla
Builder

Walt, I got all excited when this came up in google. 😛

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...