We are facing issues with Solaris 10 Sparc OS servers only wherein we are getting lot of internal script errors and also while checking sourcetypes , we do not see nmon_data.
Splunk UF binary installed on Solaris 10 Sparc is splunkforwarder-7.3.0-657388c7a488-SunOS-sparc.tar.
These are the errors that we are getting in Splunkd for those host
Errors:
09-21-2019 04:54:27.560 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh: syntax error at line 188: `count=$' unexpected
09-21-2019 04:57:53.567 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/ta_custom_monitoring_solaris/bin/custom_swap_solaris.sh" ld.so.1: swap: warning: libumem.so: open failed: No such file in secure directories
09-21-2019 05:02:31.565 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nixprocess.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nixprocess.sh: syntax error at line 3: `hostname=$' unexpected
09-21-2019 05:12:07.604 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh: syntax error at line 59: `count=$' unexpected
09-21-2019 05:12:52.012 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/ta_custom_monitoring_all_unix/bin/custom_load_avg_ALLUnix.sh" ld.so.1: uptime: warning: libumem.so: open failed: No such file in secure directories
Please help on this.
try to run this troubleshoot steps : https://ta-nmon.readthedocs.io/en/latest/troubleshoot.html#
Here is a complete document that author created recently and also have the same troubleshooting steps: https://buildmedia.readthedocs.org/media/pdf/nmon-for-splunk/latest/nmon-for-splunk.pdf
If it did not work, open a case with splunk support and attach the diag file to suport case running on the UF client server and on splunk enterprise.
Per my verification, the ta-nmon addon does not support version 7.3
COMPATIBILITY
Products: Splunk Enterprise
Splunk Versions: 7.2, 7.1, 7.0, 6.6, 6.5, 6.4, 6.3, 6.2, 6.1, 6.0
further information -> https://splunkbase.splunk.com/app/3248/
Try to deploy a UF version 7.2 instead, and redeploy the addon again.
Hello,
We have installed Splunk UF version 7.2.9, still we do not get the desired sourcetypes from the host and getting same errors which we were getting initially.
I am attaching below the logs from splunkd.log having errors and the Splunk UF version info, let me know if there is anything else that needs to be checked.
11-06-2019 10:19:42.292 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.18.103.249_8089_ai-poc9-pprd.eu.corp.airliquide.com_ai-poc9-pprd_010D7307-0711-4AAC-B679-7AD93A1EDD90
11-06-2019 10:19:44.669 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh: syntax error at line 59: count=$' unexpected
count=$' unexpected
11-06-2019 10:19:50.096 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh: syntax error at line 188:
11-06-2019 10:20:06.597 +0000 INFO TcpOutputProc - Connected to idx=10.151.32.93:9997, pset=0, reuse=0. using ACK.
11-06-2019 10:21:06.400 +0000 INFO TcpOutputProc - Connected to idx=10.151.33.40:9997, pset=0, reuse=0. using ACK.
11-06-2019 10:21:46.468 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nixprocess.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nixprocess.sh: syntax error at line 3: hostname=$' unexpected
count=$' unexpected
11-06-2019 10:22:06.196 +0000 INFO TcpOutputProc - Connected to idx=10.151.33.240:9997, pset=0, reuse=0. using ACK.
11-06-2019 10:22:44.677 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/fifo_consumer.sh: syntax error at line 59:
11-06-2019 10:22:50.105 +0000 ERROR ExecProcessor - message from "/home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh" /home/splunk/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh: syntax error at line 188: `count=$' unexpected
11-06-2019 10:23:06.002 +0000 INFO TcpOutputProc - Connected to idx=10.151.32.93:9997, pset=0, reuse=0. using ACK.
11-06-2019 10:23:35.900 +0000 INFO TcpOutputProc - Connected to idx=10.151.33.40:9997, pset=0, reuse=0. using ACK.
11-06-2019 10:24:05.800 +0000 INFO TcpOutputProc - Connected to idx=10.151.32.93:9997, pset=0, reuse=0. using ACK.
bash-3.2$ /home/splunk/splunkforwarder/bin/splunk version
Splunk Universal Forwarder 7.2.9 (build 2dc56eaf3546)