We finally got stream working - but more of a work around. The problem is in part due to starting the UF using systemd, which allocates CPU slices for different processes. When using systemd to start the UF, stream fails. Disabling start on boot, and manually starting the UF from ./slunk start, stream works. The second part is that when the UF starts, ownership of all the UF files is chowned splunk:splunk. This seems logical to ensure the UF runs as splunk (or splunkfwd). However, when stream is initially installed, the set_permissions.sh changes ownership of ../Splunk_TA_stream/Linux_x86_64/streamfwd-rhel6 to root. Starting the UF undoes this, changing ownership back to splunk. We made streamfwwd-rhel6 immutable - which did prevent the ownership change back to splunk, but stream still failed when starting with systemd. Ultimately, we had to disable systemd, make streamfwd-rhel6 immutable (after running set_permissions.sh), then start the UF manually via /splunk start. Splunk needs to fix this so stream works as expected without having to disable boot-start and set the immutable flag.
... View more