Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

Issues with Splunk_TA_nix and AIX

dshakespeare_sp
Splunk Employee
Splunk Employee
‎09-05-2017 11:57 PM

Customer reports various issues with Splunk_TA_nix with regards to the vmstat.sh, bandwidth.sh, passwd.sh and df.sh functions. They are reporting errors similar to

bandwith.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" awk: The statement cannot be correctly parsed.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" BEGIN {print "Name rxPackets_PS txPackets_PS rxKB_PS txKB_PS"} ($0 ~ "Name|sar| lo") {next} {Name=$1; rxPackets_PS=$5; txPackets_PS=$7; >>> rxKB_PS=? <<<
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-cCgimMnPrsuvZ] [-f address_family] [ [-p proto] | [-@ [wparname]] ]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-D]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" netstat: illegal option -- ;
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-n] [-I interface] [interval]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" syntax error The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" The error context is
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" usage: netstat [-Aaon] [-f address_family] [-@ [wparname]]

passwd.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Could not load module libpthreads.a(shr_xpg5_64.o).
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Could not load program /opt/splunkforwarder/bin/openssl:
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Dependent module libpthreads.a(shr_xpg5_64.o) could not be loaded.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" System error: No such file or directory

vmstat.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" awk: The field -11 cannot be less than 0.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" The input line number is 5.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" The source line number is 1.

df.sh
Filesystem Type is reported as "?" with long filesystem names
eg
ilesystem Type Size Used Avail UsePct MountedOn
/dev/hd4 jfs2 5G 327.734M 4.67995G 7% /
/dev/hd2 jfs2 8G 1.74594G 6.25406G 22% /usr
/dev/hd9var jfs2 4G 482.641M 3.52867G 12% /var
/dev/hd3 jfs2 2G 84.9609M 1.91703G 5% /tmp
/dev/hd1 jfs2 50G 18.7106G 31.2894G 38% /home
/dev/hd11admin jfs2 128M 0.371094M 127.629M 1% /admin
/dev/hd10opt jfs2 2G 372.668M 1.63607G 19% /opt
/dev/livedump ? 256M 0.359375M 255.641M 1% /var/adm/ras/livedump

Tags (3)
0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

dshakespeare_sp
Splunk Employee
Splunk Employee
‎09-06-2017 12:03 AM

Unfortunately there currently are a few issues with running Splunk_TA-nix on AIX. This applies to all versions up to and including Splunk_TA_nix 5.2.4. However there are a few simple workarounds to resolve this issue.

bandwidth.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    sudo netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'

passwd.sh
Replace line 23 in passwd.sh
From:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib $SPLUNK_HOME/bin/openssl sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

To:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib "$SPLUNK_HOME/bin/splunk cmd openssl" sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

vmstat.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
        assertHaveCommand uptime
        assertHaveCommand ps
        assertHaveCommand vmstat
        assertHaveCommandGivenPath /usr/bin/sudo /usr/sbin/swap
        assertHaveCommandGivenPath /usr/bin/svmon
        CMD='eval uptime ; ps -e | wc -l ; ps -em | wc -l ; /usr/bin/sudo /usr/sbin/swap -s ; vmstat 1 1 ; vmstat -s ; svmon;dirname $0/hardware.sh;'
        PARSE_0='NR==1 {loadAvg1mi=0+$(NF-2)} NR==2 {processes=$1} NR==3 {threads=$1-processes }'
        # ps -em inclundes processes with there threads ( at least one), so processes must be excluded to count threads #
        PARSE_1='(NR==4) {swapUsed=0+$(NF-5); swapFree=0+$(NF-1)} (NR==10) {pgPageIn_PS=0+$(NF-13); pgPageOut_PS=0+$(NF-12)}'
        PARSE_2='/^memory / {memTotalMB=$2 / 256 ; memFreeMB=$4 / 256}'
        PARSE_3='/paging space page outs$/ {pgPageOut=$1 ; pgSwapOut="?" }'
        # no pgSwapOut parameter and can't be monitored in AIX (by Jacky Ho, Systex)
        PARSE_4='/cpu context switches$/ {cSwitches=$1} /device interrupts$/ {interrupts=$1 ; forks="?" }'
        PARSE_5='/^CPU_COUNT/ {cpuCount=$2}'
        MASSAGE="$PARSE_0 $PARSE_1 $PARSE_2 $PARSE_3 $PARSE_4 $PARSE_5 $DERIVE"

df.sh
Replace line 42 in df.sh
From:

CMD='eval /usr/sysv/bin/df -n ; /usr/bin/df -kP'

To:

CMD='eval /usr/sysv/bin/df -n |sed "s/:/ :/" ; /usr/bin/df -kP'

View solution in original post

Reply
Solved! Jump to solution
Solution

dshakespeare_sp
Splunk Employee
Splunk Employee
‎09-06-2017 12:03 AM

Unfortunately there currently are a few issues with running Splunk_TA-nix on AIX. This applies to all versions up to and including Splunk_TA_nix 5.2.4. However there are a few simple workarounds to resolve this issue.

bandwidth.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    sudo netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'

passwd.sh
Replace line 23 in passwd.sh
From:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib $SPLUNK_HOME/bin/openssl sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

To:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib "$SPLUNK_HOME/bin/splunk cmd openssl" sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

vmstat.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
        assertHaveCommand uptime
        assertHaveCommand ps
        assertHaveCommand vmstat
        assertHaveCommandGivenPath /usr/bin/sudo /usr/sbin/swap
        assertHaveCommandGivenPath /usr/bin/svmon
        CMD='eval uptime ; ps -e | wc -l ; ps -em | wc -l ; /usr/bin/sudo /usr/sbin/swap -s ; vmstat 1 1 ; vmstat -s ; svmon;dirname $0/hardware.sh;'
        PARSE_0='NR==1 {loadAvg1mi=0+$(NF-2)} NR==2 {processes=$1} NR==3 {threads=$1-processes }'
        # ps -em inclundes processes with there threads ( at least one), so processes must be excluded to count threads #
        PARSE_1='(NR==4) {swapUsed=0+$(NF-5); swapFree=0+$(NF-1)} (NR==10) {pgPageIn_PS=0+$(NF-13); pgPageOut_PS=0+$(NF-12)}'
        PARSE_2='/^memory / {memTotalMB=$2 / 256 ; memFreeMB=$4 / 256}'
        PARSE_3='/paging space page outs$/ {pgPageOut=$1 ; pgSwapOut="?" }'
        # no pgSwapOut parameter and can't be monitored in AIX (by Jacky Ho, Systex)
        PARSE_4='/cpu context switches$/ {cSwitches=$1} /device interrupts$/ {interrupts=$1 ; forks="?" }'
        PARSE_5='/^CPU_COUNT/ {cpuCount=$2}'
        MASSAGE="$PARSE_0 $PARSE_1 $PARSE_2 $PARSE_3 $PARSE_4 $PARSE_5 $DERIVE"

df.sh
Replace line 42 in df.sh
From:

CMD='eval /usr/sysv/bin/df -n ; /usr/bin/df -kP'

To:

CMD='eval /usr/sysv/bin/df -n |sed "s/:/ :/" ; /usr/bin/df -kP'
Reply
Solved! Jump to solution

praphulla1
Path Finder
‎11-28-2017 11:10 AM

As per your comments i have made 2 changes to bandwidth.sh.

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'
elif

separated netstat 2 separate lines and added " around ?.

we could see ? as data in Splunk for rxKB_PS, txKB_PS. Do you know how can we populate the data for this ?

0 Karma
Reply
Solved! Jump to solution

dshakespeare_sp
Splunk Employee
Splunk Employee
‎09-06-2017 12:23 AM

Note there are two further issues with regards the Splunk_TA_nix on AIX that may be worth mentioning

cpu.sh
This command uses output from the operating system command sar -P ALL 11
On Linux systems the sar command is available to "non-root" users. Unfortunately on AIX root privilege is required to run this command

vmstat.sh (again)
This command uses outputs from the following operating system commands
uptime, swap vmstat & svmon. Unfortunately on AIX root privilege is also required to run the swap command

In order to workaround theses issues you will need to consider a solution based on one of the following

  1. Run splunk as "root" on AIX Universal Forwarders.
  2. Grant Splunk "sudo" access to the swap and sar commands, without password (and call the swap and sar commands from sudo in vmstat.sh & cpu.sh.
  3. Use Role Based Access Control (RBAC) (see https://www.ibm.com/developerworks/aix/library/au-aix_rbac/index.html)
0 Karma
Reply
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...