How does a Splunk universal forwarder talk to an indexer?

[muszyngr]

Total newb here, so please be gentle. So we are on the Windows platform and have Splunk Universal Forwarder 8.0.2 installed on many Windows 10 workstations as well on a bunch of Windows Server 2012 R2 etc. I am aware of the C:\Program Files\SplunkUniversalForwarder\etc\system\local directory construct and how to modify files in here and not in the default location

My question is if in the deploymentclient.conf file all we have is the:

[target-broker:deploymentServer]
targetUri = OurDeploymentServer:8089 

Defined, how are our logs getting to our cluster of Indexers. By the way everything is working fine as I joined this team after they built and configured our Splunk environment already, I am just trying to catch up and was upgrading the UniversalForwarders to the latest version, hence my question of how does the data get to the Indexers when all the Forwarders know about is the Deployment server.

[FrankVl]

Any configuration pushed from the deployment server will be in specific apps under C:\Program Files\SplunkUniversalForwarder\etc\apps\ on your windows universal forwarders. So have a look there, what apps your team has configured to push to the UF and what is inside those apps.

You might also want to ask around in the team you joined how they set it up, hopefully they also have some documentation on the setup. After all: we can only explain how things generically work and how things are set up according to best practice. But only your teammates can tell you how it is really done in your environment 🙂

[woodcock]

That configuration is for Command-and-Control of the UF. It causes it to periodically phonehone to the Deployment Server to ask what apps it should have. You should have a YourCompany_all_outputs app that has an outputs.conf file that points to your indexers:
https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf

[muszyngr]

so I found our outputs.conf file on our Deployment servers! Does that even make sense, I thought it needs to live on the receivers aka our Indexers? The file does point to our Indexers though in this stanza

[tcpout:IndexClusterGroup]
server = Server1IP:9997, Server2IP:9997, etc.
autoLB = true

[gcusello]

Hi @muszyngr,
the deploymentclient.conf file is used to address the Deployment Server to deploy apps to Universal Forwarders.
To send data to Indexers, you have to use outputs.conf ( https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/Outputsconf ).
Anyway, I hint to see at https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/Getstartedwithgettingdatain and https://www.splunk.com/en_us/training/videos/getting-data-in-to-splunk-enterprise-windows.html to better understand how to configure your Universal Forwarder to send data to Indexers.

Anyway the best approach is that Deployment Server deploys special apps called Technical Add-Ons (TAs) that contain the inputs.conf files where are the specifications to ingest data.

Also deploymentclients.conf and outputs.conf can be deployed used a TA (it's a best practice!).

Ciao.
Giuseppe

[muszyngr]

the link to the video you provided

"Anyway, I hint to see at […] https://www.splunk.com/en_us/training/videos/getting-data-in-to-splunk-enterprise-windows.html to better understand how to configure your Universal Forwarder to send data to Indexers."

was about Monitoring not Forwarding, just letting others know

[gcusello]

Hi @muszyngr,
As I said the file the address (on UFs) the Indexers to send data is outputs.conf.
It can stay on $SPLUNK_HOME/etc/system/local or in an app.
Usually (best practice) is located in a dedicated TA that I usually call TA_Forwarder.
I usually put in this TA only two files: outputs.conf and deploymentclient.conf.
Now there the problem of the egg and the chiken: hot to have deploymentclients.conf on UFs to connect to the Deployment Server?
I usually copy the TA_Forwarder on the UF (in $SPLUNK_HOME/etc/apps) and restart Splunk (on UF).
But you have also to create a ServerClass on Deployment server to deploy apps to the UF and in these apps there must be also the TA_Forwarder.
In this way you can manage also outputs.conf and deploymentclient.conf by DS.

In outputs.con there must be the addressing of the Indexers or (when there's an Indexer Cluster) of the master Node activating the Indexer Discovery ( https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/indexerdiscovery ) .

Ciao.
Giuseppe

[muszyngr]

thank you for your prompt reply, I will check out the links you provided

that's just it though we don't even have an outputs.conf under the local directory and the one in the default directory is the vanilla original one, what stanza in the outputs.conf would I find the Index cluster under?

When installing a Universal Forwarder from scratch as a clean install the wizard allows you to configure a Deployment server or an Index server it says one or the other is required but not both, hence my confusion, if we point the Forwarder at a Deployment server how does it know where to send the logs

[nathanluke86]

Do you have an App deployed that provides the outputs for all TA's.

Example: Outputs.conf

[indexAndForward]
index = false

[tcpout]
defaultGroup = primary_if
forwardedindex.filter.disable = true
indexAndForward = false

forceTimebasedAutoLB = true

forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_if]
server = IP:Port