Getting Data In

Splunk Universal Forwarder On AIX Fails To Start After Upgrade To Spunk 6.4.x

dshakespeare_sp
Splunk Employee
Splunk Employee

After upgrading Splunk Universal Forwarder to version 6.4.0 or above, Splunk will no longer start and the following error is reported.

Could not load program splunkd:
Symbol resolution failed for splunkd because:
Symbol SSL_is_server (number 648) is not exported from dependent
module /apps/splunkforwarder/lib/libssl.so.
Examine .loader section symbols with the 'dump -Tv' command.

How can I resolve this error?

dshakespeare_sp
Splunk Employee
Splunk Employee

This issue is caused by the fact some library files have failed to update. The standard AIX tar command will report certain files in $SPLUNK_HOME/lib as being in use when they are cached in library memory.

tar xvf ./splunkforwarder-6.4.1-debde650d26e-AIX-powerpc.tar
...
tar: 0511-188 Cannot create splunkforwarder/lib/libarchive.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libbz2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libcrypto.so.1.0.0: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libexslt.a, 452512 bytes, 884 media blocks.
tar: 0511-188 Cannot create splunkforwarder/lib/libpcre.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libsqlite3.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libssl.so.1.0.0: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxml2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxslt.a: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libz.a, 1353663 bytes, 2644 media blocks.

To resolve this issue;

EITHER
Run the upgrade again using GNU tar. We always used to recommend this method for Splunk Enterprise installs on AIX (see http://docs.splunk.com/Documentation/Splunk/6.2.11/Installation/InstallonAIX)

GNU tar is typically installed as part of the AIX Toolbox for Linux Applications package included in the base AIX install. It is located in /opt/freeware/bin/tar

OR
1. Run the AIX slibclean command (see man slibclean) which will attempt to remove any currently unused modules in kernel and library memory
2. Re run the upgrade procedure.

jrodman
Splunk Employee
Splunk Employee

post cleanup, you might want to try 'splunk validate files' to be sure the files on disk now match the files in the provided manifest.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...