Splunk Enterprise

Splunkforwarder installation fails on plain RHEL7.3/SELinux system

airlockOperatio
Explorer

Splunkforwarder rpm installation fails on default RHEL/CentOS 7.3 installation:

Can be reproduced with:

1) Default installation of CentOS 7.3 (e.g. with CentOS-7-x86_64-Minimal-1611.iso)

2) Install Splunk Universal Forwarder

rpm -i /tmp/splunkforwarder-6.6.0-1c4f3bbe1aea-linux-2.6-x86_64.rpm

useradd: cannot create directory /opt/splunkforwarder
complete

3) Home directory /opt/splunkforwarder of "splunk" user is created but .bashrc for example is missing:

ls -ld /opt/splunkforwarder/.bashrc

ls: cannot access /opt/splunkforwarder/.bashrc: No such file or directory

5) SELinux denies (RHEL7 prevents home directories in /opt when installed with rpm scripts):

grep -ri denied /var/log/audit/audit.log

type=AVC msg=audit(1494600643.668:245): avc: denied { write } for pid=10460 comm="useradd" name="opt" dev="dm-0" ino=33646738 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:usr_t:s0 tclass=dir 6) Reason: Missing permission for useradd_t to write into usr_t (/opt Directory).

Is splunkforwarder supportet with a default installation of RHEL7? if yes, what is the official installation procedure for this rpm? Hope your solution is not "setenforce 0".

Tags (1)

ephemeric
Contributor

One should not need to to:

sudo setfacl -m u:splunk:rwx /opt

 Not required at all. Please never do `chmod 0777 /dir` either.

This is not a DAC write perms issue in `/opt`. This is a SELinux issue. When the homedir creation "fails", files from `/etc/skel` are not copied into `/opt/splunk` so your `.bashrc` et al. are missing.

0 Karma

jethompson_splu
Splunk Employee
Splunk Employee

So there is a "Workaround" that can be used for this type of issue.

What is happening is with SELinux enabled there are context on specific Filesystem Directories which can prevent a Regular System user from making modifications. This is done "Out-of-the-Box" for any RHEL/CentOS Installation. You can take 2 Approaches for Systems Running SELinux.

1) The most common mitigation steps are to Disable SELinux. This is not always a "Good Idea" and may violate Company Security Policies. You can verify that SELinux Mode by running the following Command:

sestatus

[jthompson@jthompsonlin02 ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

So to Disable SELinux you would need to modify the following File: /etc/sysconfig/selinux. You would need to change the SELINUX=enforcing Line to match the setting that you are needing. In this case changing it from enforcing to disabled:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

If you complete the SELinux File changes then you will need to reboot the server to ensure that those changes take effect. Alternatively for a "temporary" workaround you could place SELinux into Permissive Mode which should allow the installation to complete as expected. The following provides a means of setting Permissive via the Command Line.

2) OR, You can disable SELinux from the command line using setenforce and the following is showing how to change from Enforcing to Permissive:

[jthompson@jthompsonlin02 ~]$ sudo setenforce 0
[sudo] password for jthompson:
[jthompson@jthompsonlin02 ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

As you can see from the above printout that the Mode changed from Enforcing to Permissive when using: setenforce 0 -- to change this back you would use setenforce 1.

There is another Workaround that should resolve this issue as well if you are needing to have SELinux Enabled and do not have the ability to build out the required SELinux Context for the installation and running of Splunk. This Alternative method would be to use the setfacl (Set File ACL) Command and provide the Splunk User permissions on the /opt Directory. To make this type of change for the "Workaround" you would do the following:

[jthompson@jthompsonlin02 ~]$ sudo setfacl -m u:splunk:rwx /opt

[jthompson@jthompsonlin02 ~]$ getfacl /opt
getfacl: Removing leading '/' from absolute path names
# file: opt
# owner: root
# group: root
user::rwx
user:splunk:rwx
group::r-x
mask::rwx
other::r-x

Once you have completed the installation process you would want to remove the ACL from /opt using the following steps:

[jthompson@jthompsonlin02 ~]$ sudo setfacl -x u:splunk /opt

[jthompson@jthompsonlin02 ~]$ getfacl /opt
getfacl: Removing leading '/' from absolute path names
# file: opt
# owner: root
# group: root
user::rwx
group::r-x
mask::r-x
other::r-x

Any one of these "Alternative" Methods should allow the installation of Splunk to complete with SELinux enabled.

nwieseler
Path Finder

Thank for the info on this. Question: Once the Splunk Universal Forwarder is installed (using any of the above workarounds), can SELinux be set to back to enforce 1 (enabled) or will the Splunk Universal Forwarder have issues running in the default /opt/splunkforwarder directory? In other words, are the workarounds only needed for installation and not regular operation?

murhammr
Path Finder

typically in *nix environments the product install dir and the home dir are not the same. i am using successfully using RHEL7+selinux with the universal forwarder and not having these issues. my splunk user home is /users/splunk and not /opt/splunkforwarder. my server admin team has given the splunk user permissions to write to /opt/splunkforwarder. further, using rpm is typically a root function, so this has to be resolved with your server admin team. whether they manage installing the package for you possibly via configuration management tool, or give the splunk user "sudo rpm" or similar.

airlockOperatio
Explorer

I know there are several workarounds for this problem (different home directory, creating home dir manually before installation, disabling SELinux, modifying/extending SELinux policy, ...). Nevertheless, if Splunk supports RHEL7, I think it should be possible to install this RPM (with default install options) on a default RHEL7 system without errors which is not the case at the moment.

jethompson_splu
Splunk Employee
Splunk Employee

airlockOperationsApp

Splunk 7.x & 6.x are supported for use with RHEL 7. We have many customers that are using RHEL 7 with SELinux enabled and are not having any issues with the Installation or running of Splunk Universal Forwarder.

Please take a moment to work with your Server Admin Team to correct the Permissions issue that you have experienced. The following is a Clear Sign of Permissions Violations in the Linux FileSystem:

2) Install Splunk Universal Forwarder

rpm -i /tmp/splunkforwarder-6.6.0-1c4f3bbe1aea-linux-2.6-x86_64.rpm

useradd: cannot create directory /opt/splunkforwarder
complete

Please understand that depending on how you are attempting to install the Splunk Universal Forwarder will depend on the Permissions changes that might be needed. If you are attempting to install Splunk as a Non-Root User then you will need to provide that User with either Full Access to /opt to be able to Read/Write/Traverse through that directory. If the User Account that you are attempting to install Splunk as does not have permissions on the /opt Directory (which standard users should not be able to do as that is a "Root" FileSystem Directory and owned by Root).

The following is from: ls -al /

drwxrwxr-x+ 3 root root 19 Nov 5 2016 opt

As you can see the /opt Directory is owned by Root and as such NO Regular User will be able to modify that directory. You have a couple of options here to allow the needed Permissions to complete the Installation of Splunk Universal Forwarder as a "Regular" User:

1) Use the Set File ACL Command set to provide the required permissions to your User Account being used to install Splunk. This is something that your Server Administration Team would need to complete.
2) Change the permissions of /opt to be owned by the user that you are attempting to install Splunk Universal Forwarder as.
3) Change the Permissions of /opt from 755 to 777 to allow ANY User access to Read/Write/Traverse the /opt directory.

Please understand that the issue you are experiencing is Unique to your Environment as we have many Customers that are using RHEL 7 with SELinux "Out of the Box". I in fact have a Test System that is running CentOS 7 with SELinux Enabled and I am not having any issues with that Universal Forwarder. Also as advised by Koshyk if you are going to be using SELinux you or your Server Administration Team will need to setup the required SELinux Context for your Environment to allow your installation of the Splunk Universal Forwarder to function properly.

There is going to be work that you need to do to allow this installation to complete with your current environment. If you need SELinux enabled but need to be able to install or run Splunk you may be able to use one of the "Workarounds" that I provided previously.

To answer your question about re-enabling SELinux, Sure you can, but once again YOU or YOUR TEAM will be responsible for setting up the required SELinux Context to allow Splunk to function in your Environment. The GitHub Link provided by Koshyk will get you started down the right path for using SELinux and Splunk.

koshyk
Super Champion

Welcome to the nightmare of SElinux.
1. Ensure your customer/manager knows SElinux out of box is not easy with Splunk. Ensure sufficient resourcing and maintenance overhead is allocated.
2. Please have a look into: https://github.com/doksu/selinux_policy_for_splunk and provide relevant permissions

airlockOperatio
Explorer

Thanks. https://github.com/doksu/selinux_policy_for_splunk does not help for this problem. This is a policy file for the splunkforwarder process, rather than for the installation of the rpm package.

Is RHEL 7 with default targeted SELinux policy supported by splunkforwarder? If yes, i think somesone should fix the rpm.

0 Karma

ddrillic
Ultra Champion

Looks like permission issues...

0 Karma

ephemeric
Contributor

I had the same problem.

The problem is a homedir in a non-standard location: https://www.unixmen.com/selinux-and-non-default-home-directory-locations/

$> ls -dlZ /home/robertg
drwx------. robertg robertg unconfined_u:object_r:user_home_dir_t:s0 /home/robertg

$> ls -dlZ /opt/splunk
drwxr-xr-x. splunk splunk unconfined_u:object_r:usr_t:s0   /opt/splunk

The above link explains in detail but the actual fix is:

#> semanage fcontext -a -e /home/robertg /opt/splunk
#This updates the labels of /opt/splunk, with the –R option doing it recursively for existing subdirectories.
#> restorecon –R /opt/splunk
#> ls -ldZ /opt/splunk
drwxr-xr-x. splunk splunk unconfined_u:object_r:user_home_dir_t:s0 /opt/splunk

 

If installing from RPM, see what creates the homedir:

$> rpm -q --scripts splunk

 

You can temporarily disable SELinux (not recommended):

#> setenforce 0
commands...
#> setenforce 1

 

Check any denied AVCs in `/var/log/audit/audit.log`.

Generally speaking: an app in its own dir like in /opt/appname is okay with SELinux as long as it writes there alone. Listening on ports below 1024 etc. requires additional SELinux work.

The current security in deploying to /opt/splunk using an interactive login account is wrong IMHO.

The account should be a system account (UID <1000) with no login like a classic UNIX daemon account. See `/etc/passwd` for examples.

I have tested the above and it works. Use Systemd to manage the service as `splunk:splunk` user,  group respectively.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...