Hi @GindiKhangura , I was able to replicate this to my environment, with the app as attached. It worked great on the standalone node, or if i used a non-streaming command in the search before the streaming command (head 1000), but as soon as I streamed it across the indexers, I received the same error. I was able to look into the search.log from the search on one of the indexers, and found the trace. 10-08-2020 17:53:52.554 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
10-08-2020 17:53:52.554 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/var/run/searchpeers/E138C522-1210-4D6F-AD0B-C91FDB3E95D8-1602178084/apps/testapp/bin/condensefields.py", line 7, in <module>
10-08-2020 17:53:52.554 ERROR ChunkedExternProcessor - stderr: from splunklib.searchcommands import \
10-08-2020 17:53:52.555 ERROR ChunkedExternProcessor - stderr: ImportError: No module named splunklib.searchcommands
10-08-2020 17:53:52.556 ERROR ChunkedExternProcessor - EOF while attempting to read transport header
10-08-2020 17:53:52.556 ERROR ChunkedExternProcessor - Error in 'condensefields' command: External search command exited unexpectedly with non-zero error code 1.
10-08-2020 17:53:52.556 ERROR SearchPipelineExecutor - sid:remote_sh1_1602179632.20_86CF40A2-33DE-4558-8481-6CFF1E8B36D4 Streamed search execute failed because: Error in 'condensefields' command: External search command exited unexpectedly with non-zero error code 1.. Looking at the error "10-08-2020 17:53:52.555 ERROR ChunkedExternProcessor - stderr: ImportError: No module named splunklib.searchcommands". You'll also notice that the app itself is being passed in the knowledge bundle and being ran in /opt/splunk/var/run/searchpeers/E138C522-1210-4D6F-AD0B-C91FDB3E95D8-1602178084/apps/testapp/bin/condensefields.py and NOT /opt/splunk/etc/slave-apps/testapp/bin/condensefields.py I looked into that /var/run/searchpeers/<SH_GUID>-<SID>/apps/testapp/ directory, and /lib/ does not get streamed to the indexer, but /bin does. To resolve this issue, I moved splunklib from /lib to /bin and removed the syspath for it to point into its own directory. After this change, the command was able to stream properly.
... View more