Help configuring a domain controller on a universal forwarder to send data to indexer

[rahulkumarfgf]

Hello Guys,
I am very new to Splunk and am trying to configure UF to send data to an indexer on port 9997. I have enabled the receiver in indexer instance. I have added [tcp://....DC IP Address:9997] and index = indexname in the inputs.conf file for UF found in $SPLUNK_HOME$/etc/system/local. I restarted splunkd services but am not getting any data coming to the specified indexer. The firewalls are OFF on the server. Indexer and UF are installed on the same server and this server is part of the domain controller. I apologize if I am not able to provide all the details as I do not have much understanding on it. Please let me know if you require any more information.

Also, when I try to stop splukd services, I get Error:1035 but the service stops and I can start it again.

Any help is much appreciated. Thank You!

[nickhills · Accepted solution]

Well, you are limited to a few options.

Text based log files you can access remotely via a UNC share/mapped drive
Metrics and instrumentation you can pull from remote WMI
Windows Event Forwarding (WEF) - configuring WEF from your DC to your Splunk Host.
See: https://hackernoon.com/the-windows-event-forwarding-survival-guide-2010db7a68c4 for more info on WEF.

If this is just POC, I would remove the UF you have deployed, and just use the copy of Splunk Core to collect all of the above (assuming it meets your needs).

A better solution is to install a UF on the DC, but I am aware this can sometimes be challenging conversation.

[rahulkumarfgf]

Yeah, I advised to install UF on the DC which would make it easier to collect logs. But that's not something we can do right now. We also need to get data from Dell and HP switches, etc. Any idea how to collect those? Also, does the port 9997 needs to be open at the DC to try and collect the logs or server port is fine? I am sorry if I sound stupid but it's all very new to me. Thanks!

[nickhills]

Add another question for the dell/hp switches and tag me in a comment with @nickhillscpl
There are a few options, but I am not an expert on those devices.

[rahulkumarfgf]

Sure, Will do! Thank You!

[rahulkumarfgf]

Hi @nickhillscpl ! Is it possible that some firewall enabled at domain controller can prevent the data from coming to splunk using UF? Will it be possible to set up firewall rule to enable tcp port 9997 to listen to traffic data, and then can it deliver data to the indexer?

[rahulkumarfgf]

Hi @nickhillscpl ! Is it possible for the data from DC to not come in if there is firewall rule enabled stopping any listening on port 9997? Just wanted to know if I can add some firewalls rule at domain controllers to allow listening at tcp port 9997. Do you think that might help?

[nickhills]

In your setup above, Splunk will not talk to the DC on 9997, and the DC will not talk to Splunk on 9997.

9997 is (by default) a Splunk -> Splunk port. The DC will only ever have Splunk trafic on 9997 if you install a UF on it, and then it will be outbound from the DC to the Splunk Indexers

[rahulkumarfgf]

in that case, is there any other port on which I can configure splunk to listen without having to install it on the domain controller? e.g. if there's any other open port on my host, and if I configure that port to listen to tcp, will that do any good?

[nickhills]

If you are not installing the UF you dont need to allow any ports.
You wont be reciving data over TCP.

Splunk will connect to the DC over WMI/RPC for instrumentation / WEF
Splunk will connect to the DC over SMB for file sharing

Your DC will have these ports open already (or it would not work as a DC)

[rahulkumarfgf]

Oh okay. Thanks! So, when I go to set up remote event logs and enter the domain ip address as host, I get the error "Unable to get wmi classes from host. The host might be unreachable or misconfigured." My host machine is a part of the DC and I am an admin user on my server. Do I need to do any other settings to resolve this?

[nickhills]

Actually WMI might be firewalled (I'm not really a windows guy) 🙂
Take a look at this: https://docs.microsoft.com/en-us/windows/win32/wmisdk/connecting-to-wmi-remotely-starting-with-vista

[rahulkumarfgf]

Thank you so much @nickhillscpl ! Will try and see what can be done. Thank You so much!

[nickhills]

No - 9997 will not be used on the DC at all.

All of the above will occur over the standard SMB/WMI ports your DC would already likely have open.

[rahulkumarfgf]

ok. Thank you! So as of now, editing the input.conf file is not going to solve the problem, I guess. I was using UF so that I don't have to use WMI as that's not configured at DC. Will try and see if I can do it though. Thanks!

[nickhills]

You still need to configure the inputs, you just do it within Splunk Core (just not the UF) - And you can use the UI to get to grips with the process:
http://yourSplunk:8000/en-GB/manager/search/datainputstats

[nickhills]

You seem to have missed a few steps, but at the very least you need an outputs.conf as well as inputs.conf on the forwarder.

Take a read of:
https://docs.splunk.com/Documentation/Forwarder/8.0.1/Forwarder/HowtoforwarddatatoSplunkEnterprise
and
https://docs.splunk.com/Documentation/Forwarder/8.0.1/Forwarder/Configureforwardingwithoutputs.conf

[rahulkumarfgf]

Hi! I have the outputs.conf and it's showing the default group, tcpout-server and tcpout:default-group configurations. My indexer and UF instance are on the same machine and I am not using the UF as a deployment server. I want to use it it to forward data from the domain controllers.

[nickhills]

I'm not sure I understand all of your comment.

Do you mean you have installed Splunk core on a server, AND installed Universal forwarder on the same host?

[rahulkumarfgf]

Sorry about that. I'm still new at this. Yes. I have windows server 2019 and I have installed splunk core and UF on the same system.

[nickhills]

Ok..
So there are some totaly valid reasons for doing that, but it does make things complicated - especially if this is a POC deployment.

Splunk Core (server) does 'work' on windows, but if this is a longer term deployment, you may want to consider Linux:
https://answers.splunk.com/answers/516059/what-are-the-pain-points-with-deploying-your-splun.html

In any case, I am assuming that this is not installed on the Domain Controller, and that you plan to collect the logs from the domain controller remotely. Is that correct?

Do you know what logs/data you need to collect from the DC?