I'm testing Splunk with the following configuration: Splunk 4.3 indexer and Splunk Universal Forwarder 4.3 on a separate machine. Both machines are running Ubuntu 10.04, and I installed both by running the script (not using .deb) so I wasn't prompted to do any setup of inputs.conf at install time.
I have been trying to add Apache log files to inputs.conf and see them in the Manager>Data Inputs>Files and Directories section of the indexer to no avail. Here is the etc/system/local/inputs.conf on the forwarder:
[default]
host = virtapache-2
[monitor:///var/log/apache2/*.log$]
index = default sourcetype =
access_common disabled = false
./splunk list forward-server:
Active forwards:
192.168.x.x:9997 Configured but inactive forwards:
None
And ./splunk list monitor:
Monitored Directores...
....
/var/log/apache2/*.log$
/var/log/apache2/access.log
/var/log/apache2/error.log
/var/log/apache2/mod_jk.log
/var/log/apache2/other_vhosts_access.log
So for all intents and purposes it looks like the forwarder is configured correctly and should be sending those logs to the indexer. The indexer was set up to receive as per instructions. Netstat on the indexer:
tcp 0 0 192.168.x.x:9997
192.168.x.x:40189 ESTABLISHED 18711/splunkd
tcp 0 0 0.0.0.0:5506
0.0.0.0:* LISTEN 18711/splunkd
tcp 0 0 0.0.0.0:9997
0.0.0.0:* LISTEN 18711/splunkd
tcp 0 0 0.0.0.0:8089
0.0.0.0:* LISTEN 18711/splunkd
Netstat on the forwarder:
tcp 0 0 0.0.0.0:8089
0.0.0.0:* LISTEN 13611/splunkd
tcp 0 0 192.168.x.x:39265
192.168.x.x:9997 ESTABLISHED 13611/splunkd
tcp 0 0 192.168.x.x:40189
192.168.x.x:9997 ESTABLISHED 13611/splunkd
Lastly, the only thing I see that could be fishy in splunkd.log is the following on the forwarder:
01-18-2012 10:28:07.892 -0500 WARN
TcpOutputFd - Connect to
192.168.x.x:9997 failed. Connection refused
01-18-2012 10:28:07.892 -0500 ERROR
TcpOutputFd - Connection to
host=192.168.x.x:9997 failed
01-18-2012 10:28:07.892 -0500 INFO
TcpOutputProc - Detected connection to
192.168.x.x:9997 closed
01-18-2012 10:56:30.611 -0500 INFO
BatchReader - Could not send data to
output queue (parsingQueue),
retrying...
01-18-2012 10:56:30.941 -0500 INFO
TcpOutputProc - Connected to
idx=192.168.x.x:9997
I am quite confused as to what I am missing. It seems like a fairly simple matter to set up an input, set up outputs.conf, restart the service, et voila! However, I don't see these Apache logs anywhere in my Data Inputs using the web GUI on the indexer. Is there anything obvious from the configs above that could be causing this problem?
... View more