NMON Performance Monitor for Unix and Linux Systems: Why am I unable to set a custom interval for the app in nmon.conf?

[wyfwa4]

I have installed the Splunk NMON app and have started to receive data successfully from several clients. So the installation of both the app and add-on's appear to be working as required. I decided to modify the data collection frequency, so copied the nmon.conf file to the /local directory under the add-on, modified the interval/snapshot settings and pushed this change out to clients. I found that the values in the nmon.conf file were ignored. To investigate further, I modified the nmon_helper.sh script directly and found that it would use the default values in this script. So it appears to be completely ignoring any values in the default or local nmon.conf. I also checked the execute permission on the nmon.conf file to make sure it was able to run this file (as this seems to be how the values are loaded), but this did not seem to affect anything.

I am following the instructions in nmonsplunk.wikidot.com/documentation:userguide:configure:collection - is there anything I am missing to make it take the values from the nmon.conf file?

[guilmxm · Accepted solution]

Hi !

Well, it should not ignore the local/nmon.conf file.
As you saw in the nmon_helper.sh script, default values are loaded but should be overcharged by the local nmon.conf file.

Have you tried killing the running nmon process on one client ? I say that because the nmon process will not reflect changes until the next occurrence, as it starts a new process every 2 hours, you won't see any change before next occurrence.

Don't hesitate to contact me by mail, you will find my contact link on the app home page.

Cheers,

Guilhem

[wyfwa4]

Thanks for the quick response Guilhem.
I did kill the nmon process between tests - and by listing the process using PS, I could see the values that were being passed when the new nmon process started. When I added custom values to the helper script, I validated these were being used by the new nmon process - so I confirmed a new process was started by the helper script.
The reason I modified the nmon_helper.sh script directly, was to validate if the default nmon.conf values were being read in. It seems that the default config file is also ignored as well as my version in the local directory, as my understanding is that the default version would be read first and overwrite the default values in the helper script. Then the local version is read and overwrites all previous values. If the values that are being used are from the original script (I use odd values so I know which are used) - this suggests all the conf files are ignored.

Is there anyway I can easily check what is happening? does the helper script write a log file out? or write values to splunk for diagnostics?

[guilmxm]

Hi,

Can you tell me more about your nmon.conf configuration ?

Have you set the mode to "custom":

mode="custom"

# Refresh interval in seconds, Nmon will this value to refresh data each X seconds
# UNUSED IF NOT SET TO custom MODE
custom_interval="60"

# Number of Data refresh occurrences, Nmon will refresh data X times
# UNUSED IF NOT SET TO custom MODE
custom_snapshot="120"

To answer your question, technically the mon_helper.sh script will log to splunkd.log for exceptions are encountered (index=_internal sourcetype=splunkd nmon_helper.sh) and in the nmon_collect sourcetype (eventtype=nmon:collect)

The script starts by defining default values, then load the default/nmon.conf and if it exists it will load the local/nmon.conf.

When it is loaded, variables are overcharged.

Finally, you can test the very last version of the TA-nmon which is under finalization, get it from the git testing branch repository:

https://github.com/guilhemmarchand/nmon-for-splunk/tree/testing

Final questions, do you use Splunk deployment servers to push the TA-nmon to your clients ? What kind of clients are running ? (OS, OS version)

I will double check on my side that the nmon.conf works as expected.

I have fixed various things in the testing release, but still the current stable version should work as expected regarding the nmon.conf management.

Cheers,

Guilhem

[guilmxm]

I have double checked the master stable version, and i can confirm that the nmon.conf feature works as expected.

Please ensure:

• You created the local/nmon.conf on your Splunk deployment server

  /opt/splunk/etc/deployment-apps/TA-nmon/local/nmon.conf

The easiest is to just copy the default/nmon.conf to local/

• You have set the mode to "custom":

  mode="custom"

• You have set custom_interval and custom_snapshot to whatever you like. (take care of syntax and value protection)

• You hare reloaded the deployment server

  /opt/splunk/bin/splunk reload deploy-server

• The new TA-nmon has been pushed to clients, you must find the local/nmon.conf file:

  /opt/splunkforwarder/etc/apps/TA-nmon/local/nmon.conf

• Restart the client is not required for this customization as the script restarts every minute

• You kill manually the nmon process and check the new spawned process

Unless we have an unexpected issue (like something regarding your OS or Splunk configuration), the feature works fine.

Please let me know

[wyfwa4]

Just to clarify some of your questions
1) The OS version is Red Hat Enterprise Linux Server release 6.7
2) I am using deployment services (also running on rhel Linux OS)
3) the forwarders running this app are running under root account.
4) The custom values in the local nmon.conf appears to match those listed above.

However I have now fixed the issue and this seems to be caused for a formatting issue.

if I run the conf file interactively and then list my variables (as setting the variables is all the script seems to do). I get the following values

custom_interval=$'120\r'
custom_snapshot=$'120\r'
endtime_margin=$'120\r'
mode=$'custom\r'

I have also found some errors on the splunk internal logs /etc/apps/TA-nmon/bin/nmon_helper.sh: line 966: [: 120\r: integer expression expected
/etc/apps/TA-nmon/bin/nmon_helper.sh: line 875: [: 120\r: integer expression expected
/etc/apps/TA-nmon/bin/nmon_helper.sh: line 811: [: 1\r: integer expression expected

I had edited the file and the pasted the whole file over to the target server. While if I just edit locally using Vi, on the Splunk server, I do not get these errors. So I just copied over a new untouched version, manually edited locally and restarted. The errors did not appear and the new custom settings worked ok.

So the root cause seems to be that the script is sensitive for new line/carriage returns in the conf file.

[guilmxm]

Right, thank for you the analysis.

In deed, it seems the origin nmon.conf you have created had the file encoding modified for some reason (did you edited on a Windows workstation ? or modified the file encoding in anyway like editing over scp with Winscp with a bad file encoding ?)

As the file get sourced (like .profile or bash_profile file to load variables), *nix OS won't enjoy non unix encoding.

However, i took note of this, and will try to check if there is improvement i can provide for next version.

Last thing, would you please have the possibility to issue the file command "file " on the bad nmon.conf file ?

Thanks !

[wyfwa4]

Sorry for the delay. Yes I did all the editing on a Windows host originally and then copied using winscp. I recreated the issue and ran the file command, the output is as follows:

ASCII English text, with CRLF line terminators

[guilmxm]

Thank you!