All Apps and Add-ons

NMON Perfmon App: How to troubleshoot SyntaxError: invalid syntax in splunkd.log?

fisuser1
Contributor

Hello -

Looking for some answers in regards to the NMON app for Splunk. I've deployed the app on about 15 forwarders, and all are checking in without issue. (phoning home successfully) I see the .nmon in the nmon_repository and the nmon_temp directories, however, it seems its failing when creating this into csv format. Has anyone run into this issue? I've included some output from the splunkd.log file.

Indexer: Red Hat Enterprise Linux Server release 5.10 (Tikanga)

Heterogeneous forwarders: linux-i686, aix-power, linux-x86_64

splunkd.log

09-08-2014 16:57:22.148 -0400 INFO  TailingProcessor - Archive file='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon' updated less than 10000ms ago, will not read it until it stops changing.
09-08-2014 16:57:32.150 -0400 INFO  TailingProcessor - Archive file='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon' has stopped changing, will read it now.
09-08-2014 16:57:32.154 -0400 INFO  ArchiveProcessor - handling file=/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon
09-08-2014 16:57:32.155 -0400 INFO  ArchiveProcessor - reading path=/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon (seek=0 len=25941)
09-08-2014 16:57:32.166 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':    File "/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon2csv.py", line 436
09-08-2014 16:57:32.166 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':      with open(ID_REF, "r") as ref:
09-08-2014 16:57:32.166 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':              ^
09-08-2014 16:57:32.166 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':  SyntaxError: invalid syntax
09-08-2014 16:57:32.166 -0400 ERROR ArchiveContext - archive writer failure: errno=Bad file descriptor
09-08-2014 16:57:32.257 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':    File "/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon2csv.py", line 436
09-08-2014 16:57:32.257 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':      with open(ID_REF, "r") as ref:
09-08-2014 16:57:32.257 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':              ^
09-08-2014 16:57:32.257 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':  SyntaxError: invalid syntax
09-08-2014 16:57:32.257 -0400 ERROR ArchiveContext - archive writer failure: errno=Bad file descriptor
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':    File "/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon2csv.py", line 436
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':      with open(ID_REF, "r") as ref:
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':              ^
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - From archive='/usr/local/splunk/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/servername_140908_1656.nmon':  SyntaxError: invalid syntax
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - archive writer failure: errno=Bad file descriptor

inputs.conf

$ cat inputs.conf
##################################
#                       nmon2csv stanza                 #
##################################

# Source stanza for nmon2csv.pl script
# Associated with the source stanza within props.conf
# Every nmon file present within the directory will be converted into csv files
# Splunk can manage

[monitor://$SPLUNK_HOME/etc/apps/TA-nmon/var/nmon_repository/*nmon]

disabled = false
index = nmon
sourcetype = nmon_processing
crcSalt = 

####################################################
#                       nmon csv converted files indexing                       #
####################################################

# Every file present within this directory will be indexed then deleted
# This section should not be modified under normal use

[batch://$SPLUNK_HOME/etc/apps/TA-nmon/var/csv_repository/*nmon*.csv]

disabled = false
move_policy = sinkhole
recursive = false
crcSalt = 
index = nmon
sourcetype = nmon_data
source = nmon_data

####################################################
#                       nmon config files indexing                                              #
####################################################

# Every file present within this directory will be indexed then deleted
# This section should not be modified under normal use

[batch://$SPLUNK_HOME/etc/apps/TA-nmon/var/config_repository/*nmon*.csv]

disabled = false
move_policy = sinkhole
recursive = false
crcSalt = 
index = nmon
sourcetype = nmon_config
source = nmon_config

####################################################
#                                       nmon data collect                                                       #
####################################################

# These input script sanza will run nmon and generates nmon file
# to be exploited by Splunk

# For AIX / Linux / Solaris

[script://./bin/nmon_helper.sh]
disabled = false
index = nmon
interval = 60
source = nmon_collect
sourcetype = nmon_collect

props.conf

$ cat props.conf
##################################
#                       nmon2csv stanza                 #
##################################

# Source stanza for nmon2csv.py script
# This source stanza will be called by the archive processor to convert NMON raw data into csv files
# SPlunk can manage. See inputs.conf for the associated monitor
# Light Forwarders don't have embedded Python 2x interpreter, so must have the host

[source::.../*.nmon]

invalid_cause = archive
unarchive_cmd = $SPLUNK_HOME/etc/apps/TA-nmon/bin/nmon2csv.py
sourcetype = nmon_processing
NO_BINARY_CHECK = true

###########################################
#                       nmon converted csv stanza                       #
###########################################

# This sourcetype stanza will be used to index nmon csv converted data
# Every generated csv file will contain a CSV header used by Splunk to identify fields

[nmon_data]

FIELD_DELIMITER=,
FIELD_QUOTE="
HEADER_FIELD_LINE_NUMBER=1

# your settings
INDEXED_EXTRACTIONS=csv
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=ZZZZ
TIME_FORMAT=%d-%m-%Y %H:%M:%S

# set by detected source type
KV_MODE=none
pulldown_type=true

# Overwritting default host field based on event data for nmon_data sourcetype (useful when managing Nmon central shares)
TRANSFORMS-hostfield=nmon_data_hostoverride

###########################################
#                       nmon processing stanza                          #
###########################################

[nmon_processing]

TIME_FORMAT=%Y-%m-%d %H:%M:%S

###########################################
#                       nmon config stanza                                      #
###########################################

[nmon_config]

BREAK_ONLY_BEFORE=CONFIG,
MAX_EVENTS=100000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%d-%b-%Y:%H:%M
TIME_PREFIX=CONFIG,
TRUNCATE=0

# Overwritting default host field based on event data for nmon_data sourcetype (useful when managing Nmon central shares)
TRANSFORMS-hostfield=nmon_config_hostoverride
09-08-2014 16:57:32.268 -0400 ERROR ArchiveContext - archive writer failure: errno=Bad file descriptor
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Hi,

I am the author of the Nmon Splunk App, and i will be happy to help you trouble shooting the situation.

I think your problem might come from the lack of an available Python interpreter in your forwarder hosts, this is indeed a requirement for hosts running the TA. (Splunk lightforwarder does not have any embedded Python interpreter at the contrary of full instance of heavy forwarder)

NOTE: Messages i see may also be caused by a non compatible version of Python, Python 2.7x is recommended
I think that the Python version available in your host may be an old version, coud you confirm me the Python version available ? (run: python --version)

The first thing is to check how runs the converter in one of your host, extracted from the Help page, FAQ section:

- Debugging the nmon2csv Python converter:

In case of trouble, you may be interested in debugging nmon2csv Python converter operations.

This can easily be achieved, either on nmon / TA-nmon / PA-nmon Application:

Create a temporary location for csv files, such like the normal directory structure of the App, example:

$ mkdir -p /tmp/nmon2csv_debug/etc/apps/TA-nmon 

Have an nmon file ready to test, if you don't have some to get the current copy in $SPLUNK_HOME/etc/apps/TA-nmon/var/nmon_resposity when the Application is running

Initiate conversion steps:

Adapt paths if you want to debug the nmon / TA-nmon / PA-nmon App and the type of Splunk instance (standard, light forwarder, heavy forwarder, peer node...), the following example will reproduce the conversion step for the standard Application:

$ cd /tmp/nmon2csv_debug 

$ export SPLUNK_HOME="/tmp/nmon2csv_debug" 

$ cat my_file.nmon | /opt/splunk/etc/apps/TA-nmon/bin/nmon2csv.py 

The Python converter will output its processing steps and generate various csv files in csv_repository and config_repository

Note that you can achieve the same operation in the proper normal Splunk directory, but if you do so, you need to stop Splunk before as it would immediately index and delete csv files

Also, could you inform me wat's the release version you have integrated ?
We can get in touch by mail for easier exchanges if you preferer, you can contact me in the Home page of the App in Splunk base

Guilhem

View solution in original post

guilmxm
SplunkTrust
SplunkTrust

Hi,

I am the author of the Nmon Splunk App, and i will be happy to help you trouble shooting the situation.

I think your problem might come from the lack of an available Python interpreter in your forwarder hosts, this is indeed a requirement for hosts running the TA. (Splunk lightforwarder does not have any embedded Python interpreter at the contrary of full instance of heavy forwarder)

NOTE: Messages i see may also be caused by a non compatible version of Python, Python 2.7x is recommended
I think that the Python version available in your host may be an old version, coud you confirm me the Python version available ? (run: python --version)

The first thing is to check how runs the converter in one of your host, extracted from the Help page, FAQ section:

- Debugging the nmon2csv Python converter:

In case of trouble, you may be interested in debugging nmon2csv Python converter operations.

This can easily be achieved, either on nmon / TA-nmon / PA-nmon Application:

Create a temporary location for csv files, such like the normal directory structure of the App, example:

$ mkdir -p /tmp/nmon2csv_debug/etc/apps/TA-nmon 

Have an nmon file ready to test, if you don't have some to get the current copy in $SPLUNK_HOME/etc/apps/TA-nmon/var/nmon_resposity when the Application is running

Initiate conversion steps:

Adapt paths if you want to debug the nmon / TA-nmon / PA-nmon App and the type of Splunk instance (standard, light forwarder, heavy forwarder, peer node...), the following example will reproduce the conversion step for the standard Application:

$ cd /tmp/nmon2csv_debug 

$ export SPLUNK_HOME="/tmp/nmon2csv_debug" 

$ cat my_file.nmon | /opt/splunk/etc/apps/TA-nmon/bin/nmon2csv.py 

The Python converter will output its processing steps and generate various csv files in csv_repository and config_repository

Note that you can achieve the same operation in the proper normal Splunk directory, but if you do so, you need to stop Splunk before as it would immediately index and delete csv files

Also, could you inform me wat's the release version you have integrated ?
We can get in touch by mail for easier exchanges if you preferer, you can contact me in the Home page of the App in Splunk base

Guilhem

gfuente
Motivator

You should include in the prerequisites, in the app Documentation tab in the splunk apps site, the need of a HF or a UF with a Python available.

Regards

0 Karma

mfcteda
New Member

python 2.4.3 is ok? how can i do ?the online system upgrade is difficult!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...