Monitoring Splunk

I'm getting a "The running job 'id' was canceled remotely or expired" error

werye
Engager

The machine the Splunk server was running on crashed. Once we rebooted it and started Splunk, all searches return empty results and display the running job was canceled remotely or expired error. I get the same results even if I use remote desktop and run the browser on the Splunk server. Any idea how to fix this?

Tags (1)
1 Solution

the_wolverine
Champion

Posting here because this may be useful to someone else. Support was able to reproduce the condition when the system clock was off between indexers/search head (240 seconds offset).

Check your system clocks and make sure they are all in sync. (Use NTP)

View solution in original post

the_wolverine
Champion

Posting here because this may be useful to someone else. Support was able to reproduce the condition when the system clock was off between indexers/search head (240 seconds offset).

Check your system clocks and make sure they are all in sync. (Use NTP)

jmaslowski
Engager

it really helped, thank you!

0 Karma

the_wolverine
Champion

Is this resolved? What version were you running? This happened to us immediately after upgrading the search head to version 4.2.5.6.

0 Karma

shadinaif
New Member

I have had the same error this month while exporting to csv using outputcsv command. The problem was related actually to free disk space but not logged in Splunk log files! I think that Splunk is using multiphase mechanism to accomplish indexing jobs; that’s why it needs more than double free space of the expected results size.
I have monitored the directory %SPLUNK%\var\run\splunk . There were many .LOCK files eating disk space while indexing job was running.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...