Splunk Dev

Why does my external scripted lookup give me an error code 1?

reswob4
Builder

So I created a way to use Mark Baggett's freq_server script as a lookup and blogged about it here (http://shadowtrackers.net/blog/get-your-freq-on-in-splunk).

But, that is not the point of this forum post.

I had based my blog post on my setup at home. I was running a single Splunk server and making the URL connection to a separate Linux server running in a vm on the same box as the Splunk server. So, when I got to work and tried to implement the same lookup, I was surprised and frustrated when it didn't work right out of the box. On the small Splunk instance where I implemented the lookup, I have a single search head and two Indexers. When I ran the lookup search:

index=dns | rename query as domain | lookup freqserver domain

I received the following error:

    2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.   

        [INDEX01] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect.
        [INDEX02] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect.
0 Karma
1 Solution

reswob4
Builder

My first thought was WTH?

But here are some of the things I tried:

https://answers.splunk.com/answers/369810/python-alert-script-fails-and-i-cant-see-errors-in.html
https://answers.splunk.com/answers/559456/script-for-lookup-table-whois-returned-error-code.html

But ultimately the information in the error clued me in. The linux server at work where I was running the frequency analysis server had iptables running and I had configured it to allow my SH to connect on the exact port the frequency analysis server was listening on. But what I didn't know, and isn't mentioned here (https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups), is that the script is actually run from the search peers (which in my case were the indexers). Since the search peers were not permitted to connect to the frequency analysis server per the iptables rules, the script was blocked from connecting and thus it failed.

Once I opened up iptables to allow the search peers access to the frequency analysis server, everything worked.

Maybe someone from Splunk can better explain, but apparently the script is copied from the SH to one of the following locations (or both) on each search peer and run from there:

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/searchscripts/

or

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/system/bin/

Hopefully this helps someone else troubleshoot their scripted lookups....

View solution in original post

0 Karma

reswob4
Builder

My first thought was WTH?

But here are some of the things I tried:

https://answers.splunk.com/answers/369810/python-alert-script-fails-and-i-cant-see-errors-in.html
https://answers.splunk.com/answers/559456/script-for-lookup-table-whois-returned-error-code.html

But ultimately the information in the error clued me in. The linux server at work where I was running the frequency analysis server had iptables running and I had configured it to allow my SH to connect on the exact port the frequency analysis server was listening on. But what I didn't know, and isn't mentioned here (https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups), is that the script is actually run from the search peers (which in my case were the indexers). Since the search peers were not permitted to connect to the frequency analysis server per the iptables rules, the script was blocked from connecting and thus it failed.

Once I opened up iptables to allow the search peers access to the frequency analysis server, everything worked.

Maybe someone from Splunk can better explain, but apparently the script is copied from the SH to one of the following locations (or both) on each search peer and run from there:

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/searchscripts/

or

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/system/bin/

Hopefully this helps someone else troubleshoot their scripted lookups....

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...