Python scripts importing modules from other apps

[nvonkorff]

In the process of creating a new app, I found I was getting cross-app module imports if another app has Python modules with the same name.

For example:

app1/bin/test_script.py
app1/bin/test_module.py

app2/bin/test_script.py
app2/bin/test_module.py

If test_script.py does an "import test_module", and I execute "app2/bin/test_script.py", it actually imports/executes "app1/bin/test_module.py"

I am writing a new app to import mail headers from audited Gmail accounts, and I used some of the same modules for the credential auth process from the Gsuite for Splunk app.

I found when I tried to auth credentials in the Gsuite for Splunk app, it imported and executed the "ga_authorize.py" from my app directory, not the GSuiteForSplunk/bin directory.

I added some logging into my copy of the "ga_authorize.py" script to print the sys.path, PYTHONPATH and the path of the script being executed to prove this:

2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="handle_GET" line_number="51" version="GSuiteForSplunk.v1.2.3.b220" operation=build_url
2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="<module>" line_number="32" version="GSuiteForSplunk.v1.2.3.b220" sys.path=['/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle', '/opt/splunk/bin', '/opt/splunk/lib/python2.7/site-packages', '/opt/splunk/lib/python27.zip', '/opt/splunk/lib/python2.7', '/opt/splunk/lib/python2.7/plat-linux2', '/opt/splunk/lib/python2.7/lib-tk', '/opt/splunk/lib/python2.7/lib-old', '/opt/splunk/lib/python2.7/lib-dynload', '/opt/splunk/bin', '/opt/splunk/etc/apps/FitbitAddonforSplunk/bin', '/opt/splunk/etc/apps/GSuiteForSplunk/bin', '/opt/splunk/etc/apps/Splunk_SA_CIM/bin', '/opt/splunk/etc/apps/Splunk_Security_Essentials/bin', '/opt/splunk/etc/apps/Splunk_TA_aws/bin', '/opt/splunk/etc/apps/Splunk_TA_nix/bin', '/opt/splunk/etc/apps/TA-GMail-audit/bin', '/opt/splunk/etc/apps/TA-GMail-audit_PROD/bin', '/opt/splunk/etc/apps/TA-cyberchef/bin', '/opt/splunk/etc/apps/TA-fitbit/bin', '/opt/splunk/etc/apps/TA-gmail-audit/bin', '/opt/splunk/etc/apps/alert_logevent/bin', '/opt/splunk/etc/apps/alert_webhook/bin', '/opt/splunk/etc/apps/base64/bin', '/opt/splunk/etc/apps/cis-controls-app-for-splunk/bin', '/opt/splunk/etc/apps/introspection_generator_addon/bin', '/opt/splunk/etc/apps/kapsch_roadside/bin', '/opt/splunk/etc/apps/lookup_editor/bin', '/opt/splunk/etc/apps/rest-storage-passwords-manager/bin', '/opt/splunk/etc/apps/search/bin', '/opt/splunk/etc/apps/splunk_app_addon-builder/bin', '/opt/splunk/etc/apps/splunk_archiver/bin', '/opt/splunk/etc/apps/splunk_instrumentation/bin', '/opt/splunk/etc/apps/splunk_monitoring_console/bin', '/opt/splunk/etc/system/bin', '/opt/splunk/etc/apps/TA-gmail-audit/bin/lib']

2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="<module>" line_number="31" version="GSuiteForSplunk.v1.2.3.b220" PYTHONPATH=/opt/splunk/lib/python2.7/site-packages

2019-05-27 13:36:07,274 INFO SCRIPTPATH=/opt/splunk/etc/apps/TA-gmail-audit/bin/ga_authorize.py

You can see that the 'app/bin' directory of all apps is included in the sys.path and the name of the script being executed is 'TA-gmail-audit/bin/ga_authorize.py' not 'GSuiteforSplunk/bin/ga_authorize.py'

Is this expected behaviour? Is there any way to force Splunk/Python to only import/execute modules under the current app directory, and not pick up the same named modules in other app directories?

Keen to hear if anyone else has run into this issue, and if so, what was done to fix/workaround the issue.

Thanks.

[nvonkorff]

To get around this issue, you can also specify the path of the module you wish to load, and load it using the 'imp' module:

import imp
modfile, pathname, description = imp.find_module('httplib2', ['/opt/splunk/etc/apps/TA-gmail-audit/bin/'])
httplib2 = imp.load_module('httplib2', modfile, pathname, description)

This is explained in the troubleshooting section of the Gmail Audit add-on

[nvonkorff · Accepted solution]

It's not ideal, bit I decided to simply rename the Python script in my app that was conflicting with the other Python script of the same name in the other app.

i.e. instead of having ga_authorize.py in both apps, I have gmail_authorize.py in my app, so that it doesn't conflict with ga_authorize.py in the GSuite for Splunk app.

It doesn't fix or explain the underlying problem, but works around it enough for my needs.