All Apps and Add-ons

IBM Websphere MQ Modular Input for Splunk: How to test importing libraries for pymqi and ctypes?

jgbricker
Contributor

In following instructions from https://splunkbase.splunk.com/app/3004/#/documentation I am not sure how to test the ability to test importing libraries for pymqi and ctypes. I copied py files from both pymqi-master/code/pymqi/ and ctypes-1.0.2/ctypes/test to mq_ta/bin

Here is my bin folder:

CMQCFC.py    pymqe.c                   test_bitfields.py    test_errcheck.py    test_internals.py     test_objects.py        test_repr.py              test_struct_fields.py
CMQC.py      responsehandlers.py       test_buffers.py      test_find.py        test_keeprefs.py      test_parameters.py     test_returnfuncptrs.py    test_structures.py
CMQXC.py     runtests.py               test_byteswap.py     test_funcptr.py     test_libc.py          test_pointers.py       test_simplesubclasses.py  test_unaligned_structures.py
CMQZC.py     test_anon.py              test_callbacks.py    test_functions.py   test_loading.py       test_prototypes.py     test_sizes.py             test_unicode.py
__init__.py  test_array_in_pointer.py  test_cast.py         test_incomplete.py  test_macholib.py      test_python_api.py     test_slicing.py           test_values.py
mqchs.py     test_arrays.py            test_cfuncs.py       test_init.py        test_memfunctions.py  test_random_things.py  test_stringptr.py         test_varsize_struct.py
mqinput.py   test_as_parameter.py      test_checkretval.py  test_integers.py    test_numbers.py       test_refcounts.py      test_strings.py           test_win32.py

I know that I can access the python interpreter via /opt/splunk/bin/splunk cmd python

I'm not sure what to type to test the libraries from there. I'm a python beginner 🙂

Also, I do not see the add-on in the UI after restart. This may be a different problem or related, I'm not sure.

Any help is appreciated!

jjw420
Loves-to-Learn

Use Splunk's python interpreter to test the library imports. Follow these steps:

1.) Login to the user used to run Splunk.
2.) cd to the /opt/splunk/etc/apps/mq_ta/bin folder.
3.) run "/opt/spunk/bin/splunk cmd python" to start Splunk's python interpreter.
4.) try to import pymqi using: "import pymqi"

If all is well you should be able to import all libraries without error.

I personally prefer not to fiddle with Splunk's python interpreter and to simply copy the pymqi and c_types folders to the /opt/splunk/etc/apps/mq_ta/bin folder. But you can also add the library locations to the PYTHONPATH like suggested for the Splunk Python SDK. See here: http://dev.splunk.com/view/python-sdk/SP-CAAAEDG

0 Karma

GKC_DavidAnso
Path Finder

I'm running on RedHat 6.4

To get it working I had to copy an MQ library to Splunk:

cp /opt/mqm/lib64/libmqic_r.so /opt/splunk/lib/libmqic_r.so

Then the bin folder of the app looked like this:

[splunk@servername apps]$ find mq_ta/bin
mq_ta/bin
mq_ta/bin/mqinput.py
mq_ta/bin/CMQC.py
mq_ta/bin/_ctypes.so
mq_ta/bin/_ctypes_test.so
mq_ta/bin/mqchs.py
mq_ta/bin/responsehandlers.py
mq_ta/bin/ctypes-1.0.2-py2.6.egg-info
mq_ta/bin/CMQC.pyc
mq_ta/bin/pymqi
mq_ta/bin/pymqi/__init__.py
mq_ta/bin/pymqi/CMQZC.py
mq_ta/bin/pymqi/CMQCFC.py
mq_ta/bin/pymqi/CMQXC.py
mq_ta/bin/pymqi/CMQXC.pyc
mq_ta/bin/pymqi/CMQC.py
mq_ta/bin/pymqi/pymqe.so
mq_ta/bin/pymqi/pymqe.py
mq_ta/bin/pymqi/CMQC.pyc
mq_ta/bin/pymqi/CMQCFC.pyc
mq_ta/bin/pymqi/__init__.pyc
mq_ta/bin/ctypes
mq_ta/bin/ctypes/__init__.py
mq_ta/bin/ctypes/macholib
mq_ta/bin/ctypes/macholib/__init__.py
mq_ta/bin/ctypes/macholib/framework.pyc
mq_ta/bin/ctypes/macholib/dyld.pyc
mq_ta/bin/ctypes/macholib/dylib.py
mq_ta/bin/ctypes/macholib/framework.py
mq_ta/bin/ctypes/macholib/dylib.pyc
mq_ta/bin/ctypes/macholib/dyld.py
mq_ta/bin/ctypes/macholib/__init__.pyc
mq_ta/bin/ctypes/test
mq_ta/bin/ctypes/test/test_struct_fields.pyc
mq_ta/bin/ctypes/test/test_as_parameter.pyc
mq_ta/bin/ctypes/test/test_memfunctions.py
mq_ta/bin/ctypes/test/test_repr.pyc
mq_ta/bin/ctypes/test/test_returnfuncptrs.pyc
mq_ta/bin/ctypes/test/__init__.py
mq_ta/bin/ctypes/test/test_objects.py
mq_ta/bin/ctypes/test/test_checkretval.py
mq_ta/bin/ctypes/test/test_win32.pyc
mq_ta/bin/ctypes/test/test_python_api.pyc
mq_ta/bin/ctypes/test/test_functions.py
mq_ta/bin/ctypes/test/test_refcounts.py
mq_ta/bin/ctypes/test/test_incomplete.pyc
mq_ta/bin/ctypes/test/test_arrays.py
mq_ta/bin/ctypes/test/test_python_api.py
mq_ta/bin/ctypes/test/test_bitfields.pyc
mq_ta/bin/ctypes/test/test_loading.py
mq_ta/bin/ctypes/test/test_funcptr.py
mq_ta/bin/ctypes/test/test_refcounts.pyc
mq_ta/bin/ctypes/test/test_anon.pyc
mq_ta/bin/ctypes/test/test_incomplete.py
mq_ta/bin/ctypes/test/test_numbers.pyc
mq_ta/bin/ctypes/test/test_cast.py
mq_ta/bin/ctypes/test/test_memfunctions.pyc
mq_ta/bin/ctypes/test/test_strings.py
mq_ta/bin/ctypes/test/test_simplesubclasses.pyc
mq_ta/bin/ctypes/test/test_unaligned_structures.py
mq_ta/bin/ctypes/test/test_functions.pyc
mq_ta/bin/ctypes/test/test_array_in_pointer.pyc
mq_ta/bin/ctypes/test/test_array_in_pointer.py
mq_ta/bin/ctypes/test/test_cfuncs.py
mq_ta/bin/ctypes/test/runtests.pyc
mq_ta/bin/ctypes/test/test_unicode.pyc
mq_ta/bin/ctypes/test/test_init.py
mq_ta/bin/ctypes/test/test_values.pyc
mq_ta/bin/ctypes/test/test_simplesubclasses.py
mq_ta/bin/ctypes/test/runtests.py
mq_ta/bin/ctypes/test/test_unaligned_structures.pyc
mq_ta/bin/ctypes/test/test_arrays.pyc
mq_ta/bin/ctypes/test/test_callbacks.pyc
mq_ta/bin/ctypes/test/test_returnfuncptrs.py
mq_ta/bin/ctypes/test/test_errcheck.py
mq_ta/bin/ctypes/test/test_repr.py
mq_ta/bin/ctypes/test/test_pointers.py
mq_ta/bin/ctypes/test/test_sizes.pyc
mq_ta/bin/ctypes/test/test_keeprefs.py
mq_ta/bin/ctypes/test/test_slicing.pyc
mq_ta/bin/ctypes/test/test_byteswap.pyc
mq_ta/bin/ctypes/test/test_internals.pyc
mq_ta/bin/ctypes/test/test_errcheck.pyc
mq_ta/bin/ctypes/test/test_init.pyc
mq_ta/bin/ctypes/test/test_unicode.py
mq_ta/bin/ctypes/test/test_objects.pyc
mq_ta/bin/ctypes/test/test_funcptr.pyc
mq_ta/bin/ctypes/test/test_structures.pyc
mq_ta/bin/ctypes/test/test_values.py
mq_ta/bin/ctypes/test/test_as_parameter.py
mq_ta/bin/ctypes/test/test_find.pyc
mq_ta/bin/ctypes/test/test_prototypes.py
mq_ta/bin/ctypes/test/test_buffers.pyc
mq_ta/bin/ctypes/test/test_keeprefs.pyc
mq_ta/bin/ctypes/test/test_random_things.pyc
mq_ta/bin/ctypes/test/test_cast.pyc
mq_ta/bin/ctypes/test/test_win32.py
mq_ta/bin/ctypes/test/test_loading.pyc
mq_ta/bin/ctypes/test/test_stringptr.py
mq_ta/bin/ctypes/test/test_integers.pyc
mq_ta/bin/ctypes/test/test_struct_fields.py
mq_ta/bin/ctypes/test/test_anon.py
mq_ta/bin/ctypes/test/test_integers.py
mq_ta/bin/ctypes/test/test_find.py
mq_ta/bin/ctypes/test/test_macholib.py
mq_ta/bin/ctypes/test/test_varsize_struct.py
mq_ta/bin/ctypes/test/test_pointers.pyc
mq_ta/bin/ctypes/test/test_parameters.pyc
mq_ta/bin/ctypes/test/test_stringptr.pyc
mq_ta/bin/ctypes/test/test_buffers.py
mq_ta/bin/ctypes/test/test_structures.py
mq_ta/bin/ctypes/test/test_libc.py
mq_ta/bin/ctypes/test/test_sizes.py
mq_ta/bin/ctypes/test/test_checkretval.pyc
mq_ta/bin/ctypes/test/test_macholib.pyc
mq_ta/bin/ctypes/test/test_slicing.py
mq_ta/bin/ctypes/test/test_numbers.py
mq_ta/bin/ctypes/test/test_prototypes.pyc
mq_ta/bin/ctypes/test/test_parameters.py
mq_ta/bin/ctypes/test/test_internals.py
mq_ta/bin/ctypes/test/test_cfuncs.pyc
mq_ta/bin/ctypes/test/test_byteswap.py
mq_ta/bin/ctypes/test/test_strings.pyc
mq_ta/bin/ctypes/test/__init__.pyc
mq_ta/bin/ctypes/test/test_libc.pyc
mq_ta/bin/ctypes/test/test_bitfields.py
mq_ta/bin/ctypes/test/test_varsize_struct.pyc
mq_ta/bin/ctypes/test/test_callbacks.py
mq_ta/bin/ctypes/test/test_random_things.py
mq_ta/bin/ctypes/__init__.pyo
mq_ta/bin/ctypes/_endian.py
mq_ta/bin/ctypes/util.py
mq_ta/bin/ctypes/_endian.pyo
mq_ta/bin/ctypes/util.pyo
mq_ta/bin/ctypes/wintypes.py
mq_ta/bin/ctypes/wintypes.pyc
mq_ta/bin/ctypes/util.pyc
mq_ta/bin/ctypes/_endian.pyc
mq_ta/bin/ctypes/__init__.pyc

For some reason I had to copy the CMQC.py from the pymqi folder back to the bin folder.

After that I was able to test with the following command from $SPLUNK_HOME/etc/apps/mq_ta/bin

/opt/splunk/bin/splunk cmd python mqinput.py

It gave me dependency errors until I got it to the above state. Now it says the following and waits for input (and the UI has stopped reporting errors and shows the input).

DEBUG ... MQINPUT: do_run() ...
0 Karma

dijikul
Communicator

Where did you get these ctypes files? I'm having a similar issue but my ctypes directory in /usr/lib/python2.7/ has a lot less files in it than your list..

0 Karma

jgbricker
Contributor

I was able to use the JMS modular input instead. However, this would still be good to learn since it is needed to set this up correctly. I'm thinking I'm not the only Splunker that would be interested in this being documented.

0 Karma

nagaraju_chitta
Path Finder

Did you get a chance to work with mq_ta app ?
Even I used JMS Modular Input now turning back to mq_ta.

0 Karma

jgbricker
Contributor

I did not use the mq_ta app because I was able to get the jms_ta working first. We are now using it in most environments.

0 Karma

nagaraju_chitta
Path Finder

ok fine ....did you implemented any threads concept in input stanza of jms_ta app? I am able to obut only 20 threads it is allowing per server..Is there anyway that we can increase more threads?

0 Karma

jgbricker
Contributor

Unfortunately I am unable to give you help there. We don't have any thread config outside default. You may want to do a separate post and see if the dev replies.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...