Universal forwarder issue in AWS

[partix2]

Hi , i have created 2 instances of windows in AWS and using one of the instance using universal forwarder to forward the logs on another windows instance of splunk enterprise as my indexer. But the logs are not getting forwarded and i can see the service of forwarder running on my Universal forwarder instance.Also i have enabled the receiving port 9997 on my indexer. What can be probable reason for the same?

[nickhills · Accepted solution]

In reverse probable order:

1.) Did you configure an AWS security group to allow your UF to send outbound traffic on port 9997
2.) Did you configure an AWS security group to allow your Indexer to receive inbound traffic on 9997
3.) Have you configured Windows Firewall to allow the same?
4.) Did you configure the forwarder to forward events to the indexer on 9997? - Did you use the ui, or did you set an ouputs.conf config? - Can you post the config?
5.) Does netstat show the UF trying to open port 9997 to send data on the UF?
6.) Does netstat show the indexer listening on port 9997?

[partix2]

Hi , my comments for your concerns are listed as below:
1. Did you configure an AWS security group to allow your UF to send outbound traffic on port 9997 -- YES
2. Did you configure an AWS security group to allow your Indexer to receive inbound traffic on 9997 -- YES
3. Have you configured Windows Firewall to allow the same? -- YES
4. Did you configure the forwarder to forward events to the indexer on 9997? – YES
5. Did you use the ui, or did you set an ouputs.conf config? – I used UI to configure forwarding to the indexer.
6. Can you post the config? – The outputs.conf from indexer instance in the folder “C:\Program Files\SplunkUniversalForwarder\etc\system\local” is as below:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 172.31.88.99:9997
[tcpout-server://172.31.88.99:9997]
7. Does netstat show the UF trying to open port 9997 to send data on the UF?- Netstat does not give any hint of UF trying to open port 9997
8. Does netstat show the indexer listening on port 9997? – Indexer is not listening on port 9997

Can you please help me how to proceed with this issue ..

[nickhills]

So 7 & 8 appear to be the most concerning then.

netstat -nab should show you the ports that splunk has opened.
On a UF, I would expect to see (unless you have disabled) it listening on 8089.
If it was trying to forward events to an indexer you should see the indexer IP and a listing for 9997

On the indexer you would see it listening on 8000, 8089, and 9997 (among others)
If you still don't see any ports open, are you sure that the services are running properly?

[partix2]

Hi , my comments are as below :

On indexer , I can see the established connection between indexer and forwarder on port 9997.

On forwarder I can see “TCP 172.31.37.196:49166 172.31.88.99:9997 FIN_WAIT_1” , its not showing as established or listening on port 9997, also logs are not forwarded to indexer. I also restarted the service on forwarder , but same result. What can be the probable reason for the same?
172.31.37.196- forwarder IP
172.31.88.99 - Indexer IP

[nickhills]

Have you configured inputs.conf?

Try searching for something like:
index=_internal |stats count by host
If you see two hosts returned by that search, then Splunk is working properly but it sounds like you just need to configure the universal forwarder to collect the logs.

[partix2]

Thanks for your valuable suggesstion
I tried searching with the command :
"index=_internal |stats count by host"

This was successful as I was getting logs from that forwarder but when I am simply searching with only the hostname of the Forwarder it shows no results.

May I know the reason for that?

[partix2]

Hi,

I have one more problem, I am only able to see the logs from my folder on universal forwarder:

C:\Program Files\SplunkUniversalForwarder\var\log\splunk

Apart from it am not able to see any folder logs

Can you please suggest something on this?

[eduardKiyko]

It could be a lot of reasons. Did you configure outputs.conf? Did you configure network setting? Are instances able to ping each other?