How to troubleshoot why a Windows universal forwarder can forward permon data to Splunk Light 6.3, but not Windows event logs?

[acrismatic]

Hi to all,

I'm a newbie with Splunk this week, and trying to configure a forwarder in W2008 in order to forward event logs to Splunk Light 6.3 configured as an indexer in Centos.

I've installed the universal forwarder and set the server in outputs.conf:

tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.1.175:9997

[tcpout-server://192.168.1.175:9997]

That's my inputs.conf in Splunk Server :

[splunktcp://192.168.1.102:9997]
connection_host = ip

# Windows platform specific input processor.

[WinEventLog://Application]
disabled = 0
#current_only = 1
index = wineventlog

[WinEventLog://Security]
disabled = 0
#current_only = 1
index = wineventlog

[WinEventLog://System]
disabled = 0
#current_only = 1
index = wineventlog

Well, the case is I can't seem to make my w2008 send event logs to the indexer, but perfmon events appear on it. I've set this info during universal forwarder installation, setting the ip and port for indexer, and setting Windows events logs and perfmon.

In Splunk Web, I'm trying to configure data input > forwarders > Windows events logs, but it says there's no forwarder available.

On searches, I can see perfmon events, but not so much...

Some tries I've done:

• I've use netstat to test if the connection is ok, and it is in ESTABLISHED state (so i'm receiving perfmon successfully).
• Check indexes. It's strange for me the result. I can see how wineventlog index is growing constantly, but no event appears in the search page. All perfmon events are sent to the main index.

• Review splunkd.log in w2008 forwarder and in Centos Splunk Server. I've found some errors about connections, but no idea how to solve it. This error appears in the forwarder:

  10-16-2015 11:28:37.342 +0200 INFO TcpOutputProc - Connection to 192.168.1.175:9997 closed. Connection closed by server.
  10-16-2015 11:28:38.853 +0200 WARN TcpOutputFd - Connect to 192.168.1.175:9997 failed. No connection could be made because the target machine actively refused it.
  10-16-2015 11:28:38.853 +0200 ERROR TcpOutputFd - Connection to host=192.168.1.175:9997 failed
  10-16-2015 11:28:38.853 +0200 WARN TcpOutputProc - Applying quarantine to ip=192.168.1.175 port=9997 _numberOfFailures=2
  10-16-2015 11:28:49.855 +0200 INFO TcpOutputProc - Removing quarantine from idx=192.168.1.175:9997
  10-16-2015 11:28:50.357 +0200 INFO TcpOutputProc - Connected to idx=192.168.1.175:9997
  And this appears in the indexer:

  10-16-2015 11:28:21.791 +0200 INFO TcpInputProc - Waiting for connection from src=192.168.1.102:57931 to close before shutting down TcpInputProcessor.
  10-16-2015 11:28:23.286 +0200 ERROR TcpInputProc - Error encountered for connection from src=192.168.1.102:57931. Local side shutting down
  10-16-2015 11:28:35.883 +0200 INFO TcpInputConfig - performing DNS lookup on 192.168.1.102
  I'm a bit confused with these errors. How can I receive perfmon events if indexer is refusing connections from forwarder?

Two servers are in the same subnet, with booth firewalls deactivated. Now I'm at a point where I don't know what else to check. Could someone give me some advice to look for?

I've tried to give all information possible. Don't hesitate to ask for more information, remember I'm newbie with Splunk and I'm sure I'm loosing configs and things to do...

Best Regards,

[jterry]

what does your search look like?
Are you specifying the wineventlog index? (index=wineventlog *)