Splunk Dev

Debugging a custom search command

cphair
Builder

Hello,

I'm new to Splunk, and I have some basic questions about custom search commands. I wrote a Python script and put it in $splunk_home\etc\apps\search\bin, and I added a stanza to $splunk_home\etc\local\commands.conf and restarted Splunk. When I try to pipe a search to the custom command, I get a message saying "External search command 'foo' failed with error code 1." I don't know what error code 1 is, and I can't find any useful details in any of the files in var\log\splunk--the search string shows up in searches.log, but nothing else looks relevant. It's possible I have a straightforward problem (like a syntax error or the files in the wrong place), but how do I go about pinpointing it? Can I turn up the error reporting in the web interface? What log files are supposed to be useful here?

1 Solution

Lowell
Super Champion

Unfortunately there isn't a whole lot of logging going on about errors in custom search scripts. Id's suggests setting up your own top-level log handled in your script to capture and log any exceptions yourself. You can throw the logs into the $SPLUNK_HOME/var/log/splunk/ directory and then the events will be available with a index=_internal search, or just put them in /tmp/ if you want something quick and dirty.

An example is posted here:

http://splunk-base.splunk.com/answers/3935/debugging-custom-splunk-search-commands/3942

I did a quick test and if my python search command had any typos in it would report the same "error code 1" like you saw. (I just threw in an extra line with the word "BLAH". This raises a NameError and python returns with an exit code of 1.) Like you saw, I couldn't find anything in the logs that showed me the error specifically.

So catch top-level exceptions, and consider putting all of your logic into a top-level function. This gives the ability to use tools like pychecker or even write unit tests around a search command core logic and do test independently of a splunk search.

View solution in original post

Lowell
Super Champion

Unfortunately there isn't a whole lot of logging going on about errors in custom search scripts. Id's suggests setting up your own top-level log handled in your script to capture and log any exceptions yourself. You can throw the logs into the $SPLUNK_HOME/var/log/splunk/ directory and then the events will be available with a index=_internal search, or just put them in /tmp/ if you want something quick and dirty.

An example is posted here:

http://splunk-base.splunk.com/answers/3935/debugging-custom-splunk-search-commands/3942

I did a quick test and if my python search command had any typos in it would report the same "error code 1" like you saw. (I just threw in an extra line with the word "BLAH". This raises a NameError and python returns with an exit code of 1.) Like you saw, I couldn't find anything in the logs that showed me the error specifically.

So catch top-level exceptions, and consider putting all of your logic into a top-level function. This gives the ability to use tools like pychecker or even write unit tests around a search command core logic and do test independently of a splunk search.

cphair
Builder

Finally came back to this. I ran


splunk cmd python .\myscript.py searchargs

where searchargs was just the plain string, trying to see if anything was going to stdout/stderr. I had failed to escape the backslashes in my Windows pathname, so some of them were treated as special characters and greatly confused the logger. I still think that kind of error ought to be logged somewhere, especially since my python settings were set to debug--splunkd.log, maybe.

Once I fixed the backslash issue, the custom log worked as described above.

0 Karma

cphair
Builder

@Lowell, thanks for the reply. Since I posted this, I figured out how to run the bundled Python interpreter (.\splunk cmd python), and I tested non-Splunk subsections of my script successfully. I tried to set up custom logging but didn't have much luck; I'll give it another shot.

The only parts of my script I couldn't test directly were the calls to Intersplunk. Is there a way to provide input/output from the command line, e.g. a small specially formatted text file as a dummy result?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...