Would I drop or turn-off the "nmon_processing" log ?

[leo_wang]

Hi,
The "nmon2csv" script will generate some information like below and index it as sourcetype="nmon_processing" :

07-06-2017 18:35:12 Reading NMON data: 519 lines 29929 bytes
Splunk Root Directory ($SPLUNK_HOME): /opt/splunk
addon type: /opt/splunk/etc/apps/TA-nmon
addon version: 1.3.21
nmon2csv version: 1.1.34
Guest Operating System: linux
Python version: 2.7.13
NMON VERSION: 16f
.... etc

I know this information is useful when trouble shooting....
But if I am sure the TA-nmon is working well ,could I turn it off ? or just drop this information ?
Because nmon2csv script runs every minutes, and if I want to monitor 100+ hosts.... It will costs me a lot of splunk license for such log I don't need. ( And I also believe it will impact the index performance.. )

[guilmxm]

Hello,

The last version of the TA-nmon with the release 1.3.25 implements by default a "silent" option for the processing logs.
This option reduces the volume of data to be generated by removing all the per section line information.

Does it answers to your need ? Could you confirm ?

Thank you.

Guilhem

[guilmxm]

Hi,

This is a good one, thanks for asking 😉

Right, well those information are indeed useful to observe and trouble shoot any potential issue with the processing implemented in the TA.

Those information are also extracted and exploited in some administrative views I do provide like the the TCO dashboard or the Add-on reporting dashboard.

So, I ran into some analysis, as far as I can see this (the nmon_processing sourcetype) relies on 5 to 10% of the global volume of data to be generated by the Nmon app.

There are many Nmon app deployments at 1K servers and even much more, at scale this becomes indeed not that negligible.
Note that you cannot assume that this is "bad" for your indexing performance, this is indexing and this what Splunk is done for, if you have trouble indexing then you have trouble with your infrastructure, deployment and / or design.

To answer:

1. The more "clever" approach is I think an update from me, which would if not reduce at least provide an options people could use to lower the verbosity of the Nmon processing

I have logged the following enhancement request:

https://github.com/guilhemmarchand/TA-nmon/issues/37

1. Other solution at the Splunk level, redirect those events to null Queue, I have tested and validated the following configuration:

props.conf

# nmon_processing null redirection, apply the following transform
[nmon_processing]
TRANSFORMS-nullQ=eliminate-nmon-processing

transforms.conf

# Redirect to null Queue the nmon_processing events
[eliminate-nmon-processing]
REGEX=(?m)\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\sReading\sNMON\sdata
DEST_KEY=queue
FORMAT=nullQueue

Those configuration would have to be applied to all "full Splunk instances" which includes indexers, search heads and heavy forwarder. (this is not mandatory required for Universal Forwarder as it produces uncooked data)

So you would for instance includes those custom configuration with the TA-nmon and the PA-nmon_light, or anywhere you like.

This obliviously result in completely ignore and prevent from indexing the nmon_processing data. (On the network side the data is still generated and forwarded, but not indexed)

The first option will be implemented in next release of the TA-nmon.

Cheers

Guilhem