Getting Data In

How to get Linux OS logs off a Splunk server, where Splunk is started as a non root account, to index in an indexer cluster?

brent_weaver
Builder

I have a Splunk indexer cluster that is using a service account (non-root) to start Splunk. How do I get the OS logs, like /var/log/messages, /var/log/secure etc... into the cluster indexes? I know that I could stream this to a syslog server and grab it there, but is there an easier way?

Any thoughts are welcome!

0 Karma

hartfoml
Motivator

@koshyk It would be great is you could find an answer here and try it out. If you do find an answer please select the answer you like.

0 Karma

koshyk
Super Champion

We faced the same issue. Assuming "splunk" have read access to the OS logs, what we have done is using Splunk_TA_nix. Put into the "local" of this app, with what files you want to collect by adding the paragraph and putting disable = false (Most of things are already part of TA_nix)

For different layers, enable Splunk_TA_nix in below fashion.
- For Splunk Forwarders push using deployment-server. It goes into $SPLUNK_HOME/etc/apps of forwarders.
- Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for deployment-server,
- Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for cluster master,
- For clustered Search Heads, package into $SPLUNK_HOME/shcluster/etc/apps and push to Search Members. In search members, it will be merged into "default", but works.
- For clustered Indexers, copy Splunk_TA_nix using cluster master via master-apps. This goes into "slave-apps" of Indexer slaves and works perfectly.

If you enable Splunk_TA_nix, then you can start colllecting every information about your Splunk Infrastructure/OS

hartfoml
Motivator

@starcher thanks for the link to github. I went to this talk and I agree with Matt. In slide #13 he basicly put in what I had said above. Matt said

Create a “log reading” group and add the spunk user to it, or simply change group ownership to splunk

groupadd syslog
chown -R :syslog /var/log
chmod -R g+s /var/log
usermod -a -G syslog splunk

0 Karma

brent_weaver
Builder

I want to collect OS logs from only the spunk servers themselves, not the forwarders. The forwarders is easy as the univfwd runs as admin on all platforms, its the spunk servers I am concerned about.
Changing the log dir permissions won't work (I do not believe) because when logrotate runs it will create the files with orig permissions.

I think my best bet is going to be to stream the logs to a remote syslog server!?!?

0 Karma

starcher
SplunkTrust
SplunkTrust

My colleague Matt Uebel gave a talk at .conf that covers this topic. His materials are in his git repo at https://github.com/MattUebel/splunk_UF_hardening

0 Karma

hartfoml
Motivator

three ways that I know of.

1) chmod -r 777 the log directory
2) add the splunk user to the wheel or root group
3) chown -R root:SplunkGroup /var/log/

Hope this helps? Don't know there may be a more restrictive way to do this?

somesoni2
SplunkTrust
SplunkTrust

The OS logs that you want to collect is from splunk cluster server only OR all other linux servers in your company?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...