Installation

In Splunk Enterprise, can you help me call an external Python3 via Script?

blangrill
Explorer

Splunk still uses Python 2.7 internally but has the ability to call external scripts to generate data via Scripted Inputs

I would like to pull data using an external API which has a Python 3 library. I have installed Python 3 to a separate place on the file system and written a Windows batch script, which calls this and invokes the Python 3 API. When run from the OS, this generates the data I would like, but if I try to add this batch script as an input to Splunk, I get an error along the lines of the following:

ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" Fatal Python error: Py_Initialize: unable to load the file system codec 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" File "E:\splunk\Python-2.7\Lib\encodings\__init__.py", line 123 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" raise CodecRegistryError,\ 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" ^ 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" SyntaxError: invalid syntax 

I get a similar error even if I minimally set the batch script to only contain a minimal statement.

<path to python3>\python.exe -c 'print("Hello")' 

This implies the problem is something from Python 2 being passed to the Python 3 environment since the Splunk script call is ultimately from that.

I asked Splunk support about this and was told it "isn't supported" and was directed to contact professional services which seems like overkill for what is likely just an environment issue.

Is there a way to wrap the python commands such that this still works?

As of now my workaround is to have the batch file called as a scheduled task in Windows and write the results to a file which is then monitored by Splunk. Additionally, at some point Splunk will make the jump internally to Python 3 and likely people will have the reverse problem with older Python 2.X libraries.

0 Karma
1 Solution

blangrill
Explorer

The answer is quite simple. Adding -E to the Python3 call tells it to ignore other PYTHONPATH variables and it is able to run successfully.

The batch file I am calling as a Splunk scripted input now looks like:

@ECHO off
C:\Python37\python.exe -E "<path to python3 script>"

View solution in original post

0 Karma

badarsebard
Communicator

There's a new app called PyDen which would allow for an alternative approach to this problem. The app allows a developer to create Python virtual environments of any version (2.7, 3.5+) for use by scripts run by Splunk. This would allow you to do simply use the Python 3 script you've written as the input script instead of using a wrapper.

0 Karma

blangrill
Explorer

The answer is quite simple. Adding -E to the Python3 call tells it to ignore other PYTHONPATH variables and it is able to run successfully.

The batch file I am calling as a Splunk scripted input now looks like:

@ECHO off
C:\Python37\python.exe -E "<path to python3 script>"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...