Why is there an error "ModuleNotFoundError: No module named 'syslog'" on running my python script?

[sutom]

Dear All,

I have created a TA to monitor a custom python script named log_parser_v1.py". Here is the configuration from /splunk/etc/apps/TA-logs/default/inputs.conf

[script://./bin/log_parser_v1.py]
python.version = python3.9
interval = 300
disabled = false

But while running TA got failed with the error "ModuleNotFoundError: No module named 'syslog'"

So I am trying to debug with splunk cmd python, and it's throwing "ModuleNotFoundError: No module named 'syslog'" error -

[ss@localhost bin]$ ./splunk cmd python log_parser_v1.py
Traceback (most recent call last):
  File "bin/log_parser_v1.py", line 7, in <module>
    import syslog
ModuleNotFoundError: No module named 'syslog'

But the same script runs fine with the command python3.9 bin/log_parser_v1.py

Here are the few lines from the script with the import statement of the module "syslog" in the line 7-

[ss@localhost bin]$ cat log_parser_v1.py
#!/usr/bin/env python
import os, sys
sys.path.append('/usr/bin/python3.9')
sys.path.append('/usr/lib/python3.9/site-packages')
sys.path.append('/usr/lib64/python3.9/site-packages')
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
import json, logging, syslog, datetime, argparse, shutil, zipfile, tarfile, bz2, socket, sys, errno, time, gzip, hashlib
from logging.handlers import SysLogHandler, SYSLOG_TCP_PORT
from syslog import LOG_USER

To use python3.9. I append the python3.9 package path in script but it still is not picking the syslog module. here is the python3.9 path -

[ss@localhost bin]$ whereis python
python: /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.9 /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib/python3.9 /usr/lib64/python2.7 /usr/lib64/python3.6 /usr/lib64/python3.9 /usr/include/python3.9 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz

I also tried to import syslog package with ./splunk cmd python, but it got failed

[ss@localhost bin]$ ./splunk cmd python
Python 3.7.11 (default, May 25 2022, 12:23:55)
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import syslog
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'syslog'
>>> exit()

And here is imported successfully with python3.9

[ss@localhost bin]$ python3.9
Python 3.9.7 (default, Sep 13 2021, 08:18:39)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import syslog
>>> exit()

Guys, I am looking for your help to understand like what is missing. please help here.

[richgalloway]

Splunk runs Python scripts using its own Python interpreter.  If you develop code using a different interpreter, be sure to test with the Splunk interpreter to find out about errors like ths.

The fix is to add the missing module to the bin/lib or lib directory of your TA.

[sutom]

@richgalloway  Thanks for the response.

But in the past, I had the same issue for another script with BeautifulSoup, and that got resolved by appending the python3.9 path

sys.path.append('/usr/local/bin/python3.9')
sys.path.append('/usr/local/lib/python3.9/site-packages')
sys.path.append('/usr/lib64/python3.9/site-packages')
from bs4 import BeautifulSoup

Here Also, I thought it was a similar one. but not working...

Another analysis is that if I am running ./splunk cmd python3.9 then syslog gets imported..

[ss@localhost bin]$./splunk cmd python3.9
Python 3.9.7 (default, Sep 13 2021, 08:18:39)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import syslog
>>> exit()

also if I run script manually with the command - ./splunk cmd python3.9 log_parser_v1.py, it is running fine.

but It failing under the TA schedule.

[terminaloutcome]

At that point you're either importing modules across python versions or running a non-standard interpreter, both of which are unsupported and asking for trouble. I'd definitely recommend @richgalloway 's suggestion of having the package in the app's path.

[richgalloway]

I don't have a "python3.9" in my Splunk installation, just "python3".

According to the Admin manual, "python3.9" is not a valid value for python.version.

If what you've done before isn't working this time then consider filing a support request.  While you wait for that, try my suggestion.  🙂

[sutom · Accepted solution]

Got a recommendation to use below app- 

splunkbase.splunk.com/app/4322