Monitoring Splunk

Setting up splunk monitors

fsrodriguez
New Member

At my job whenever they set up a Splunk forwarder they add only one monitor. "/var/logs". Does anybody do it this way?

Shouldn't we adding monitors with stanzas on the /SPLUNK_HOME/etc/system/local/intputs.conf ?

We have forwarders installed on 29 servers. Our licence usage is currently at 8GB. Does this sound like its too much for the amount of servers?

Thanks in Advance

0 Karma

nickhills
Ultra Champion

if this is as you say, the chances are high that your ingesting duplicate data (such as when your log files role) and you likely have very few sourcetypes.

Whilst obviously this will (and does) work, its not a very sensible way to use Splunk.

In terms of estimating the licence usage, its difficult to say without knowing what sort of logs your collecting but 8GB/day for 30 servers seems like a lot unless they are quite busy.

If my comment helps, please give it a thumbs up!
0 Karma

fsrodriguez
New Member

Yeah that's what I was thinking. All of the instances have the Splunk Add-on for Unix and Linux. Some alerts are just set up to check if a service is running. I don't think we even need to add that monitor directory in order for those alerts to work.

0 Karma

nickhills
Ultra Champion

The TA for nix comes configured to collect a number of common logs from *nix systems, so its possible that's how your environment has been configured. If so you probably have sourcetype=messages or sourcetype=dmesg.

If this is the case then it may not be as you fear.
If however, all of your data is in one sourcetype I shall weep for you. 🙂

If my comment helps, please give it a thumbs up!
0 Karma

fsrodriguez
New Member

what do you mean by all of my data is once sourcetype?... and I have a feeling it is lol..

So lets say I am trying to monitor only the tomcat service and create a query with ps source. I should go into /etc/system/local/inputs.conf and add:

[monitor:/opt/tomcat/logs/catalina.out]

then this should work and it shouldn't return any results when the service is down correct?

host="server1" source=ps tomcat| stats latest(_time) as latest by host
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...