Why is the Universal Forwarder on Windows 7 communicating with deployment server but not sending any Windows Logs?

[packet_hunter]

I have a stand-alone instance of Splunk running on Linux. I have a Universal Forwarder installed on Windows 7 with the intent to collect the Windows event logs. The stand-alone instance was enabled to become a deployment server when I configured the UF (Universal Forwarder) and pointed output back to the Stand-alone. I have good communication and deployed the Splunk Add-on for Microsoft Windows to the UF successfully. However the Windows events are not rolling in. Any ideas where to start troubleshooting this?

Thank you

[packet_hunter · Accepted solution]

The UF on my test win7 box was setup correctly. The standalone was setup correctly as well. So then, I used wireshark on the test box and on the standalone to monitor the communication on both network interfaces. I looked at the UF logs too. I could see that communication back from the UF to 9997 on the standalone was failing. I went back to my standalone which resides on a linux centos vm (in vmware workstation) on a windows 7 physical box. I had opened all the splunk ports on the CentOS firewall, and configured the VMware network editor to NAT correctly. The problem was the windows OS firewall. Something caused it to change/discard port 9997. I went back into the windows fw > advanced settings > inbound rules, and added port 9997 again. Then it worked. It was a windows firewall issue.

IF anyone would like full details on how to setup a similar testing lab, then please let me know and I will provide full details regarding CentOS fw, vmware nat, windows OS firewall configurations.

Thank you for all your help.

[maciep]

Are you getting the Splunk logs from that forwarder? They would be in the _internal index. If so, you can check those logs for any messages.

If not, then look on the windows box manually at the splunk logs to see if there are any errors, e.g. c:\program files\splunkuniversalforwarder\var\log\splunk\splunkd.log

Also, did you install the add-on on your standalone box as well? Not just in deployment-apps, but under apps? Just curious if you created the indexes the add-on is trying to write to as well...

[packet_hunter]

Thank you for the reply and helpful suggestions. I wiresharked the UF box to Splunk stand-alone and kept seeing the 9997 port communication failing. There were some mysterious fw changes I had to fix. I got it to work now.
Thank you all for the helpful information, I learned a lot.

[maciep]

glad to help and glad you got it working. But could you also add an answer to this question to describe what you did to fix the issue. Another user someday in the future might have a similar problem and this thread could be really helpful for them. As it stands now, the fixes are still mysterious 🙂

And then you can accept your answer as the answer which will mark this question as answered

[ddrillic]

Maybe the following can help - I can't find my data!

[packet_hunter]

thank you, but nothing is populating in either index, wineventlog or windows...
strange

[koshyk]

any errors you seeing in UF logs? normally the ports for forwarding are 9997 on the receiving side.
Please check with diagram: https://answers.splunk.com/answers/58888/what-are-the-ports-that-i-need-to-open.html to see if the ports are all open.

[packet_hunter]

Thank you, I have reviewed and my settings seem to be good. I am starting to think my windows box has some weird permission setting that is preventing the logs from being sent.