I've created a custom command in python that needs to view an entire set of events as a single batch, because it's comparing subsequent events. Unfortunately, Splunk is sending events to the custom command in chunks of <= 50,000 events. The commands.conf has streaming = false. Setting run_in_preview = false only changes the way the results are displayed, as expected.
In case it's relevant, the command is running on a search head which receives events from several distributed search nodes.
Here's the basic code -- run() is invoked by a minimal plugin "manager":
class RemoteLogins( SplunkPlug ):
def run( self, events, keywords, options ):
out_events = []
if not events:
intersplunk.outputResults( out_events )
return
now = datetime.now()
with open( "/opt/splunk/var/log/test.log", "a" ) as f:
f.write( "Running at %s with %s events\n" % ( now, len( events ) ) )
for related_events in self.related( events ):
self.find_overlap( related_events, out_events )
with open( "/opt/splunk/var/log/test.log", "a" ) as f:
f.write( "Ending %s with %s results\n" % ( now, len( out_events ) ) )
intersplunk.outputResults( out_events )
When invoked by a single splunk search, these results are generated:
Running at 2011-08-27 16:56:18.619245 with 25 events
Ending 2011-08-27 16:56:18.619245 with 0 results
Running at 2011-08-27 16:56:19.078111 with 2942 events
Ending 2011-08-27 16:56:19.078111 with 0 results
Running at 2011-08-27 16:56:20.900458 with 19980 events
Ending 2011-08-27 16:56:20.900458 with 1 results
Running at 2011-08-27 16:56:31.590848 with 50000 events
Ending 2011-08-27 16:56:31.590848 with 4 results
Running at 2011-08-27 16:56:55.376255 with 50000 events
Ending 2011-08-27 16:56:55.376255 with 3 results
Once the search is complete, only the 3 results from the last batch of events is shown.
For completeness, here's commands.conf:
[py]
type = python
filename = py.py
streaming = false
run_in_preview = false
maxinputs = 0
So, is there any way aside from the settings in commands.conf to really convince Splunk not to stream events into a custom command? Maybe an intermediate command I could insert into the pipeline?
... View more