We started using Splunk deployment server after some Windows servers already had the universal forwarder installed. However, it seems some servers were installed without the Splunk UniversalForwarder being configured correctly. Currently the $decideOnStartup "host" is reporting performance metrics, but I don't see any applog, seclog, or syslog data. I know that doesn't mean the server isn't reporting any data for the logs, I just don't have any data since the Splunk history was truncated to 35 days ago.
Is there an easy way to determine what server has the messed up configuration?
At the moment, I think I will be comparing the "All Forwarers" report from the Deployment Monitor Application and
Run this search with a timeframe of 24-48hrs, and look for missing sourcetypes and or servers:
index=main sourcetype="*security" OR sourcetype="*application" OR sourcetype="*system" | stats count by host,sourcetype
If you are missing servers or source types, then check those servers to verify the forwarder configuration, and that they are configured for auditing. Windows auditing is minimal by default and must be configured by the administrator.
Run this search with a timeframe of 24-48hrs, and look for missing sourcetypes and or servers:
index=main sourcetype="*security" OR sourcetype="*application" OR sourcetype="*system" | stats count by host,sourcetype
If you are missing servers or source types, then check those servers to verify the forwarder configuration, and that they are configured for auditing. Windows auditing is minimal by default and must be configured by the administrator.
Sorry applog, syslog, seclog were short for: WinEventLog:Application, WinEventLog:Security, WinEventLog:System
No its a linux Deployment Server.
Splunk version 5.0.1
The applog, syslog, and seclog sound like unix log files. If you are configuring these boxes from a Windows box then you may have format and or permission problems.
Are you using a Windows deployment server for Linux hosts?
What version of Splunk are you running?