TA-nmon. Error ArchiveContext in splunkd.log on AIX UVF. Which 0652-141 There is no python in (PATH statement listed)

[psbromley]

New setup of Universal forward on AIX, with nmon TA-nmon app installed. All seems to be working but getting an increasing error count on the NMON home screen/dashboard and splunkd.log is showing the below....

02-04-2016 15:26:28.424 +1100 ERROR ArchiveContext - From archive='/opt/splunkforwarder/var/run/nmon/var/nmon_repository/"hostname"_160204_1434.nmon': which: 0652-141 There is no python in /opt/splunkforwarder/bin /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java5/jre/bin /usr/java5/bin /opt/ibm/director/bin.

I have checked the various scripts/files in "/opt/splunkforwarder/etc/apps/TA-nmon/bin". 2 of these files have a "which python" statement included but both send this output to /dev/null.

grep "which python" *

nmon2csv.sh:PYTHON=which python >/dev/null 2>&1
nmon_cleaner.sh:PYTHON=which python >/dev/null 2>&1

[guilmxm · Accepted solution]

Following our exchanges about this issue.

The error related to the "which python" (Python not found) has been eradicated since the version 1.6.14 of the App.

Therefore, error messages were not related to this an incorrect "LIBPATH" environment variable being set on the host.
This was generating messages related to missing module:

02-08-2016 15:20:18.916 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" Could not load program /usr/opt/freeware/bin/rpm:
02-08-2016 15:20:18.916 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" Could not load module /usr/lib/librpm.so.
02-08-2016 15:20:18.917 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh"     Dependent module /opt/splunkforwarder/lib/libz.a(libz.so.1) could not be loaded.
02-08-2016 15:20:18.917 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh"     The module has an invalid magic number.
02-08-2016 15:20:18.917 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" Could not load module rpm.
02-08-2016 15:20:18.917 +1100 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh"     Dependent module /usr/lib/librpm.so could 

This was caused by the by "LIBPATH" environment variable incorrectly set on that host.

Inserting:

unset LIBPATH

In the nmon_helper.sh before starting the topas-nmon process has prevented this message to appear (but even with the message, the TA-nmon was working correctly)

This looks like being related to a local configuration on the AIX host, and not directly with the Nmon App.

Guihem

[guilmxm]

Hi,

The error comes from the nmon_helper.sh and as you underlined the python presence verification done by the script.
On some systems and even with the stderr to stdout and finally /dev/null redirection, i have some trouble to prevent the which output to generate an error in splunkd logs.

The Home page and the nmon search associated with the error count is a report called "Number of Errors in Data Processing or Data Collect since last 24 Hours" (in savedsearches.conf)

The search is the following:

(index=nmon sourcetype=nmon_processing OR sourcetype=nmon_collect error) OR (index=_internal sourcetype=splunkd ERROR ExecProcessor nmon) NOT ("There is no python in" OR "python: not found") | stats count As trend1
| appendcols
[ search (index=nmon sourcetype=nmon_processing OR sourcetype=nmon_collect error) OR (index=_internal sourcetype=splunkd ERROR ExecProcessor nmon) NOT ("There is no python in" OR "python: not found") earliest="-1d@d" latest="@d" | stats count As trend2 ]
| transpose | fields row* | rename "row 1" As value

So the part of the search:

 (index=nmon sourcetype=nmon_processing OR sourcetype=nmon_collect error) OR (index=_internal sourcetype=splunkd ERROR ExecProcessor nmon) NOT ("There is no python in" OR "python: not found")

Should not retrieve the python error associated with the which command output, can you please check that search inline ?

I can update the application to prevent this from happening in next release, on my side even with some AIX reporting the which error output, the report does not retrieve these events, maybe you have a local copy of the report ?

[psbromley]

Hi,

Thanks for the prompt response.
Can I firstly confirm the file/script you mentioned = nmon_helper.sh
I don't see a "which python" command in this file only in the *nmon_cleaner.sh or * ????
The prefix of the lines with the error are :
2-05-2016 09:16:38.882 +1100 ERROR ArchiveContext - From archive='/opt/splunkforwarder/var/run/nmon/var/nmon_repository/[hostname_date_time.nmon]. Is this the process to cleanup/archive the processed nmon files ?

I have 12 files in this location/folder starting from installation yesterday. One every 2 hours.

I searched on the AIX server reporting with nmon (not the splunk server) but only found the savedsearches.conf file in my /$home/nmon/default/". I then realised, this was the extracted/gzip'd installation file. I have removed this entire "$home/nmon" folder structure and have restarted splunk on the Universal Forwarder/nmon host.

However, same messages in the splunkd.log on this host after restart of splunk.

I also found the same file on the splunk server in "/opt/splunk/etc/apps/nmon/default/savedsearches.conf". Content for these lines are the same.

search = (index=nmon sourcetype=nmon_processing OR sourcetype=nmon_collect error) OR (index=_internal sourcetype=splunkd ERROR ExecProcessor nmon) NOT ("There is no python in" OR "python: not found") | stats count As trend1\
| appendcols\
[ search (index=nmon sourcetype=nmon_processing OR sourcetype=nmon_collect error) OR (index=_internal sourcetype=splunkd ERROR ExecProcessor nmon) NOT ("There is no python in" OR "python: not found") earliest="-1d@d" latest="@d" | stats count As trend2 ]\
| transpose | fields row* | rename "row 1" As value

It looks the same as your example to me?

[guilmxm]

Hi,

Yes, sorry, not the "nmon_helper.sh" but the "nmon2csv.sh"

The nmon_cleaner.sh is a an input script that manages these cleaning steps (more information on the wiki: http://nmonsplunk.wikidot.com/)

The activity of this script will be visible in:

index=nmon sourcetype=nmon_clean

And Splunk schedules or output errors will also be visible in:

index=_internal sourcetype=splunkd nmon_clean

The "python" error is not a real problem, it has no effect on the App behavior.
It is only the result of the verification of python interpreter availability on the host, if python is not found, then we use Perl scripts.

Therefore, this error should not be accounted in the Home page error counter

Your nmon files should be automatically purged, can you verify the nmon_cleaner activity ?

[guilmxm]

Also, splunk reports are in your Splunk instance (the search head), not the Universal Forwarder

We can easier exchange by mail, you can contact me on the App page or my profil page