I'm have a custom command that parses an input field in each given record and emits 0 to N records as its output. I'm doing this to avoid a bunch of mvzip/mvexp and logic in the calling SPL. It does seem to work fine, but I'd like some reasurance that this is really supported in the SDK.
The SDK doc for StreamingCommand of the PythonSDK (http://docs.splunk.com/Documentation/PythonSDK) says (bold highlighting mine),
Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records....
This (and the rest of the article and others like it that I've found) don't really seem to specify how many records can be returned. The above seems to suggest that it's really supposed to be 1:1 and not 1:n.
Here's an edited down version of my code. (I'm also a Python newbie, so apologies for any ugliness there.)
import sys
from mytokeninfo import Info
sys.path.append("splunk_sdk-1.6.5-py2.7.egg")
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration
@Configuration(local=True) # Per doc on "stateful" streaming commands
class ExStatefulCommand(StreamingCommand):
def stream(self, records):
for record in records:
tokens = self.parseRecordForTokens(record)
for token in tokens:
info = self.processToken(token)
record['newField1'] = info.field1 # Application specifics simplified here for clarity (hopefully)
record['newField2'] = info.field2 # ...etc
yield record
So, for each record, I'm augmenting it one or more times and also yield'ing it each time. If so, I'd love to see the doc for it. If not, can I get an explanation as to why and also suggestions for how best to deal with this in a clean and proper manner?