I'm getting a "The running job 'id' was canceled remotely or expired" error

[werye]

The machine the Splunk server was running on crashed. Once we rebooted it and started Splunk, all searches return empty results and display the running job was canceled remotely or expired error. I get the same results even if I use remote desktop and run the browser on the Splunk server. Any idea how to fix this?

[the_wolverine · Accepted solution]

Posting here because this may be useful to someone else. Support was able to reproduce the condition when the system clock was off between indexers/search head (240 seconds offset).

Check your system clocks and make sure they are all in sync. (Use NTP)

[jmaslowski]

it really helped, thank you!

[the_wolverine]

Is this resolved? What version were you running? This happened to us immediately after upgrading the search head to version 4.2.5.6.

[shadinaif]

I have had the same error this month while exporting to csv using outputcsv command. The problem was related actually to free disk space but not logged in Splunk log files! I think that Splunk is using multiphase mechanism to accomplish indexing jobs; that’s why it needs more than double free space of the expected results size.
I have monitored the directory %SPLUNK%\var\run\splunk . There were many .LOCK files eating disk space while indexing job was running.