Errors from real-time custom search command not shown in SplunkWeb

[dmr195]

I am writing a custom search command, and under some circumstances I want it to report an error to the user.

I have used splunk.Intersplunk.generateErrorResults as suggested here, and for a historical search it works exactly as I'd like.

However, for a real-time search, no red bar containing the error message gets displayed in SplunkWeb unless I click the "Finalize" button. I would like the error message to be reported in SplunkWeb as soon as my custom command generates it. I can see the messages appearing in $SPLUNK_HOME/var/run/splunk/dispatch//search.log as soon as my command outputs them, but they're just not getting through to SplunkWeb.

The problem can be reproduced with this very simple custom search command that does nothing with its input and always outputs an error message (obviously my actual command does more than this):

import splunk.Intersplunk

settings = {}
inputEvents = splunk.Intersplunk.readResults(None, settings, True)

errorResults = splunk.Intersplunk.generateErrorResults('Oops : something went wrong')
splunk.Intersplunk.outputResults(errorResults)

The corresponding section in commands.conf is:

[testerrors]
filename = testerrors.py
streaming = true

If I run this custom search command as a historical search, for example:

sourcetype=mysql | testerrors

then the error is displayed immediately, but if I enter that same search in the search box in SplunkWeb, and then select Real-time -> 1 minute window in the time selection dropdown on the right hand side then no error is displayed even though errors are appearing in $SPLUNK_HOME/var/run/splunk/dispatch//search.log.

Is there anything I can change to have the errors displayed immediately in SplunkWeb for a real-time search?

I'm using Splunk version 4.3.2.

[dmr195]

I've done some more investigation into what is happening here, and have realised that my original post wasn't completely accurate. In fact my entry in commands.conf also had "local = true" set, and this appears to be what triggers the problem.

If I put the following in commands.conf:

[testerrors4]
filename = testerrors.py
streaming = true
local = false

[testerrors7]
filename = testerrors.py
streaming = true
local = true

then testerrors4 will result in the error being displayed in SplunkWeb in a real-time search but testerrors7 will not.

This is the search.log extract from testerrors4 (local = false; real-time error does make it through to SplunkWeb):

10-18-2012 12:17:49.523 INFO  script - Invoked script testerrors4 with 221 input bytes (0 events).  Returned 29 output bytes in 92 ms.
10-18-2012 12:17:49.523 DEBUG SearchResults - Head from csv file read: inds.size() = 1
10-18-2012 12:17:49.523 ERROR script - command="testerrors4", Oops : something went wrong
10-18-2012 12:17:49.523 DEBUG SearchPipelinePerformance - processor=testerrors4 executetime=0.093 secs
10-18-2012 12:17:49.523 DEBUG SearchProcessor - execute_dispatch invoked for 'fields': type = 'SP_STREAM'
10-18-2012 12:17:49.523 DEBUG SearchPipelinePerformance - processor=fields executetime=0.001 secs
10-18-2012 12:17:49.523 DEBUG DispatchCommand - Time Descending ProviderQueue: read 0 events from peer 'linux.prelert.com', info.setStart = 1350559009, et = 1350559009
10-18-2012 12:17:49.525 INFO  DispatchCommand - Generating results preview took 2 ms

This is the search.log extract from testerrors7 (local = true; real-time error does not make it through to SplunkWeb):

10-18-2012 12:15:54.044 INFO  script - Invoked script testerrors7 with 196 input bytes (0 events).  Returned 29 output bytes in 92 ms.
10-18-2012 12:15:54.044 DEBUG SearchResults - Head from csv file read: inds.size() = 1
10-18-2012 12:15:54.044 ERROR script - command="testerrors7", Oops : something went wrong
10-18-2012 12:15:54.044 DEBUG SearchPipelinePerformance - processor=testerrors7 executetime=0.093 secs
10-18-2012 12:15:54.045 INFO  DispatchCommand - Generating results preview took 95 ms

Less debug is being output for the command that's declared to be local, so it looks like the code that propagates custom search command error messages to SplunkWeb in real-time searches is inside a branch that doesn't get executed when the search command is set to run locally.