Security

Really quick and basic example?

mattlemay
New Member

Hi. I've installed splunk on two machines; one is the server I want to coalesce data from various sources, and the other is one of the sources. The machine I want to monitor is a typical linux webserver running apache. I've installed the universal forwarder on my webserver. I configured it to talk to the main splunk machine, and I've told it to monitor /var/log. The results in the web interface are, frankly, astoundingly disappointing. I can see the webserver under hosts, but the sources are:

/var/log/rpmpkgs.2

/var/log/dmesg

/var/log/yum.log

/var/log/prelink/prelink.log

This seems like a pretty basic task. This is a project I'm doing on the side. I can't be a full time splunk administrator. I mainly want to monitor my web & system logs for a handful of different webservers. Eventually, if I'm feeling spunky, I may even add some windows webservers and database servers to the mix. Right now, though, I'd really like to be able to index something useful.

Is there a quick tutorial that might shed some light on this relatively trivial task?

Thanks.

Tags (3)
0 Karma

Drainy
Champion

mattlemay, your question wasn't originally clear that it wasn't picking up all the logs. It sounds as if you are unhappy that it picked up those logs you have listed.

Firstly have you got permissions to access those files if Splunk is not running as root? It shouldn't really be able to access the ones you've listed but always good to check just in-case.

Secondly could you post your inputs.conf from the forwarder please?

Thirdly if you are running syslog-ng or rsyslog you could always bypass the forwarder altogether and add a statement like;

*.* @IP

to your syslog.conf (or rsyslog.conf or someother similar purmutation) in the /etc, /etc/syslog (or where-ever it may be!)

where the IP is your splunk server and configure your splunk instance to receive on port 514 (assuming it uses this as the default, which most do)
The forwarder is great for forwarding non-syslog files, load balancing across multiple forwarders and for integration with a deployment server for centralised management. If you have a one to one relationship however you might find it easier to use the syslog method. From memory I believe the weblogs will utilize this on the system but you may need to do a quick test first.

You're right that this is trivial and Splunk normally handles this folder fine - infact its usually the first place alot of people head when demo-ing to customers to show its ease of use and ability to recursively index a folder 🙂

We just need to figure out what is stopping it on your setup

mattlemay
New Member

I am trying hard not to be snarky about this. But I didn't configure it to ignore /var/log/syslog and /var/log/messages in favour of /var/log/dmesg and /var/log/rpmpkgs.2. Yes, I ultimately want it to index /var/log/apache2. The documentation strongly insinuates that when you tell it to monitor directories, it does so recursively, which is a reasonable thing for it to do. In fact, part of the documentation explicitly says that I don't want to configure it to monitor both /var/log and /var/log/apache2.

0 Karma

MarioM
Motivator

there is tutorial here:

Splunk Tutorial!

mattlemay
New Member

Thanks for the link, but it seems to link to documentation I've already seen. I feel like I've done what the documentation told me to do and it didn't work. Now I don't know if it's a problem on the forwarder or the indexer. Is it a data source problem or an index problem?

0 Karma

sdwilkerson
Contributor

Mattlemay,
You configured Splunk to monitor (read: Index) all of /var/log and now you are disappointed that it did what you asked it to?

Generally you want to explicitly state what you want Splunk to do. For instance, if you want to monitor all of /var/log/apache then configure it to do that and it will.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...