Getting Data In

Why is one of my universal forwarders trying to contact the deployment server on port 9997?

splunk_cv
Explorer

Hi,

I have a configuration where many Universal Forwarders are managed by a Deployment Server.
Today I installed a new UF on a Windows machine, and I have several problems:

  • in the internal log I see WARN TcpOutputFd - Connect to :9997 failed. No connection could be made because the target machine actively refused it and ERROR TcpOutputFd - Connection to host=:9997 failed -> Why is the UF trying to contact the DS on 9997?

  • in the internal log, I can't see the phonehome logs, but looking at the ASA logs, I can see the communication between UF and DS on port 8089 every minute and so I suppose this is the phonehome

  • Main problem: I have configured the inputs.conf to monitor a path, but I can't see the logs, maybe because the errors above.

    [monitor://C:\path\ue_*.log]
    index=ftp
    sourcetype=ftp
    In the outputs.conf I have listed, as usual, my 6 indexers on port 9997.

0 Karma
1 Solution

lguinn2
Legend

In reply to your bullet points

1 - The TcpOutputFd component is trying to send data to an indexer - it is NOT trying to contact the deployment server. It looks like there is probably an error in outputs.conf. And there is probably more than one outputs.conf...

2 - If the forwarder is not actually forwarding, you won't see any "phone home" messages in the _internal index. You can, however, go to the splunkd.log on the forwarder and check for phone home messages. If deploymentclient.conf is set up properly, you will see them there or error messages.

3 - if the forwarder is not forwarding, of course your inputs will not be sent to the indexers and the data will not be there.

There were a lot of garbled lines in your btool output. But the last 4 lines are telling:

UF\etc\system\local\outputs.conf                        [tcpout-server://10.73.21.105:9997]
UF\etc\system\local\outputs.conf                       server = 10.73.21.105:9997
UF\etc\apps\OA-output_forwarder\default\outputs.conf   [tcpout:indexmi_9997]
UF\etc\apps\OA-output_forwarde\\default\outputs.conf   server = 10.73.21.103:9997,10.73.21.104:9997,10.73.21.157:9997,
10.73.21.158:9997,10.73.21.159:9997,10.73.21.160:9997

You have a setting in etc/system/local as well as in your app. Because the stanzas are different, they do not conflict - but how does Splunk know which stanza to use? And of course, anything in etc/system/local cannot be removed with the Deployment Server, it will have to be removed manually.

If you look at splunkd.log on the forwarder, you will probably see exactly where and how it was attempting to forward. During your uninstall process, perhaps you removed the bad configuration and then the forwarder unblocked and sent its data.

btool is your friend!

View solution in original post

0 Karma

lguinn2
Legend

In reply to your bullet points

1 - The TcpOutputFd component is trying to send data to an indexer - it is NOT trying to contact the deployment server. It looks like there is probably an error in outputs.conf. And there is probably more than one outputs.conf...

2 - If the forwarder is not actually forwarding, you won't see any "phone home" messages in the _internal index. You can, however, go to the splunkd.log on the forwarder and check for phone home messages. If deploymentclient.conf is set up properly, you will see them there or error messages.

3 - if the forwarder is not forwarding, of course your inputs will not be sent to the indexers and the data will not be there.

There were a lot of garbled lines in your btool output. But the last 4 lines are telling:

UF\etc\system\local\outputs.conf                        [tcpout-server://10.73.21.105:9997]
UF\etc\system\local\outputs.conf                       server = 10.73.21.105:9997
UF\etc\apps\OA-output_forwarder\default\outputs.conf   [tcpout:indexmi_9997]
UF\etc\apps\OA-output_forwarde\\default\outputs.conf   server = 10.73.21.103:9997,10.73.21.104:9997,10.73.21.157:9997,
10.73.21.158:9997,10.73.21.159:9997,10.73.21.160:9997

You have a setting in etc/system/local as well as in your app. Because the stanzas are different, they do not conflict - but how does Splunk know which stanza to use? And of course, anything in etc/system/local cannot be removed with the Deployment Server, it will have to be removed manually.

If you look at splunkd.log on the forwarder, you will probably see exactly where and how it was attempting to forward. During your uninstall process, perhaps you removed the bad configuration and then the forwarder unblocked and sent its data.

btool is your friend!

0 Karma

splunk_cv
Explorer

Thanks for the answer. I found the problem on my own some days ago, analyzing the btool's output: the FW installation was made from a third person, and he indicates the DS also as an IX during the installation process, so of course the FW tryed to send data the DS instead of the IXs.

0 Karma

splunk_cv
Explorer

Today i tried to unistall the Universal Forwarder on the remote host, and i noticed that after it was uninstalled (or during the uninstallation operations, i can't say it exactly) I receveid all the internal log that i haven't received previously. Someone can explain this strange behavior?

0 Karma

diogofgm
SplunkTrust
SplunkTrust

did you delivered the inputs.conf to the new UF via deployment server?

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

schose
Builder

Could you provide output of the command "spunk btool outputs list --debug" from one of the forwarders?

splunk_cv
Explorer

Yes i delivered it via DS, i created an app with inputs.conf in it.
@schose this is a Windows UF, could you provide me instruction for the command under Windows systems?

0 Karma

schose
Builder

in command prompt type

"C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" btool outputs list --debug 

splunk_cv
Explorer

@schose here is the ouptut:

Note: The OA-output_forwarder is an app deployed on every forwarder with the outputs.conf file configured to send data to our 6 indexers.

UF\etc\system\default\outputs.conf type = udp
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf [tcpout]
UF\etc\system\default\outputs.conf ackTimeoutOnShutdown = 30

UF\etc\system\default\outputs.conf blockOnCloning = true

UF\etc\system\default\outputs.conf compressed = false
UF\etc\system\default\outputs.conf connectionTimeout = 20

UF\etc\system\default\outputs.conf forceTimebasedAutoLB = false
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.0.whitelist = .*
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.1.blacklist = _.*
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.filter.disable = false
UF\etc\system\default\outputs.conf indexAndForward = false
UF\etc\system\default\outputs.conf maxConnectionsPerIndexer = 2
UF\etc\system\default\outputs.conf maxFailuresPerInterval = 2
UF\etc\system\default\outputs.conf readTimeout = 300
UF\etc\system\default\outputs.conf secsInFailureInterval = 1
UF\etc\system\default\outputs.conf sslQuietShutdown = false
UF\etc\system\default\outputs.conf useACK = false
UF\etc\system\local\outputs.conf [tcpout-server://10.73.21.105:9997]
UF\etc\system\local\outputs.conf server = 10.73.21.105:9997
UF\etc\apps\OA-output_forwarder\default\outputs.conf [tcpout:indexmi_9997]
UF\etc\apps\OA-output_forwarder\default\outputs.conf server = 10.73.21.103:9997,10.73.21.104:9997,10.73.21.157:9997,
10.73.21.158:9997,10.73.21.159:9997,10.73.21.160:9997

(I reformatted the btool output for readability -@lguinn)

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...