Getting Data In

Splunk Universal Forwarder On AIX Fails To Start After Upgrade To Spunk 6.4.x

dshakespeare_sp
Splunk Employee
Splunk Employee

After upgrading Splunk Universal Forwarder to version 6.4.0 or above, Splunk will no longer start and the following error is reported.

Could not load program splunkd:
Symbol resolution failed for splunkd because:
Symbol SSL_is_server (number 648) is not exported from dependent
module /apps/splunkforwarder/lib/libssl.so.
Examine .loader section symbols with the 'dump -Tv' command.

How can I resolve this error?

dshakespeare_sp
Splunk Employee
Splunk Employee

This issue is caused by the fact some library files have failed to update. The standard AIX tar command will report certain files in $SPLUNK_HOME/lib as being in use when they are cached in library memory.

tar xvf ./splunkforwarder-6.4.1-debde650d26e-AIX-powerpc.tar
...
tar: 0511-188 Cannot create splunkforwarder/lib/libarchive.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libbz2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libcrypto.so.1.0.0: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libexslt.a, 452512 bytes, 884 media blocks.
tar: 0511-188 Cannot create splunkforwarder/lib/libpcre.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libsqlite3.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libssl.so.1.0.0: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxml2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxslt.a: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libz.a, 1353663 bytes, 2644 media blocks.

To resolve this issue;

EITHER
Run the upgrade again using GNU tar. We always used to recommend this method for Splunk Enterprise installs on AIX (see http://docs.splunk.com/Documentation/Splunk/6.2.11/Installation/InstallonAIX)

GNU tar is typically installed as part of the AIX Toolbox for Linux Applications package included in the base AIX install. It is located in /opt/freeware/bin/tar

OR
1. Run the AIX slibclean command (see man slibclean) which will attempt to remove any currently unused modules in kernel and library memory
2. Re run the upgrade procedure.

jrodman
Splunk Employee
Splunk Employee

post cleanup, you might want to try 'splunk validate files' to be sure the files on disk now match the files in the provided manifest.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...