Hello,
Splunk 7.1.3, Linux x86_64.
One of my custom (SCPv1) commands errors when the number of events returned exceeds 20,000-30,000 (the value slightly changes between runs; it poses no problem if count(events)<10,000); this is the associated suspicious snippet from search.log:
09-08-2018 17:40:55.446 ERROR ScriptRunner - stderr from 'xxx': INFO Running /opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin/python xxx
09-08-2018 17:40:56.247 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL
09-08-2018 17:40:56.247 INFO DispatchExecutor - User applied action=CANCEL while status=0
09-08-2018 17:40:56.247 ERROR SearchStatusEnforcer - sid:1536453655.14705 Search auto-canceled
09-08-2018 17:40:56.247 INFO SearchStatusEnforcer - State changed to FAILED due to: Search auto-canceled
09-08-2018 17:40:56.255 INFO ReducePhaseExecutor - Ending phase_1
09-08-2018 17:40:56.255 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.255 ERROR SearchOrchestrator - Phase_1 failed due to : DAG Execution Exception: Search has been cancelled
09-08-2018 17:40:56.255 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL
09-08-2018 17:40:56.255 INFO DispatchExecutor - User applied action=CANCEL while status=3
09-08-2018 17:40:56.255 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='1536453655.14705', username='admin')
09-08-2018 17:40:56.256 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.261 WARN SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=0 currentSetStart=0.000000
09-08-2018 17:40:56.261 WARN LocalCollector - Local Collector Orchestrator terminating, writing to the collection manager failed.
09-08-2018 17:40:56.263 INFO UserManager - Unwound user context: NULL -> NULL
09-08-2018 17:40:56.263 WARN ScriptRunner - Killing script, probably timed out, grace=0sec, script="xxx"
09-08-2018 17:40:56.265 INFO UserManager - Unwound user context: NULL -> NULL
Note: I've obfuscated the script name from the log above.
My questions:
— What conditions must arise to have a search auto-canceled?
— What's a DAG Execution Exception?
— What's a known workaround?
thank you.
... View more