Windows 2008R2 Universal Forwarder "Could not send data". How to configure outputs.conf?

[macser]

Hi,

I'm new to Splunk and I'm trying to set up a Universal Forwarder to forward some data to our Splunk server.
The guys who set up our present Splunk svr can't really help me as they've so far only used Splunk on *ix environment and this is the first UF on a Windows server.

I'm trying to get the UF to forward events that are periodically saved in data-files in a specific directory to our Splunk server.

Source data directory is : C:/datafile/

Destination server is : splunksvr.intranet.local

I installed the forwarder using the splunkforwarder-5.0.8-201809-x64-release.msi.
I can see a lot of things in the logs, but I don't know how to make much sense of some of it.

I found this in splunkd.log

09-15-2014 11:48:11.371 +0200 ERROR TcpOutputProc - **LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.** 
09-15-2014 11:48:16.441 +0200 INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...

One problem is, there are several "outputs.conf", so how do I know which one it's looking for?

I configured "${installdir}/etc/system/local/outputs.conf"

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunksvr.intranet.local:9997

[tcpout-server://splunksvr.intranet.local:9997]

There are others in "${installdir}/etc/system/default" and I don't know where else right now.

I also found an app.conf in "${installdir}//etc/apps/MSICreated/local/"

I've read the doc&help, tried the kb and also here but I couldn't find a solution.

I'd appreciate any help.

Serge

[neelamssantosh]

Hi ,
1. Splunk has precedence option:
system/Local --> App/local --> App/default --> system/default.
so place ur configurations in etc/system/Local and splunk will consider it on priority.
**Never edit any DEFAULT.CONF files*.
if you don't have local.conf ,then create it..

2 . Use btool commant to know from which locationS outputs.conf are considered
$SPLUNK_HOME/bin > ./splunk cmd btool props list --debug
./splunk cmd btool outputs list --debug
http://docs.splunk.com/Documentation/Splunk/6.1.3/Troubleshooting/Usebtooltotroubleshootconfiguratio...

3 . outputs.conf:
[tcpout:xxxx]
autoLB = true
indexAndForward = 1
server = 192.168.xx.1:9997,192.168.xx.2:9997,192.168.xx.3:9997

ALL THE BEST..

[wrangler2x]

If you are still seeing this ERROR about the connection being forcibly closed by the remote host, read my comment here: http://answers.splunk.com/answers/138307/indexer-send-tcp-rst-to-forwarders-trying-sending-data.html

[triest]

First, can the forwarders resolve splunksvr.intranet.local? You might try using an ip address instead of a hostname just to rule that out.

I have a virtual development environment and my outputs.conf looks like:

[tcpout]
indexAndForward = false
autoLB = true
defaultGroup = my_indexers

[tcpout:my_indexers]
server =10.0.0.1:9997, 10.0.0.2:9997

That's basically may way of saying, I can't swear what you have is wrong, but here's another suggestion.

Honestly, Splunk on Windows isn't that different from Splunk on Unix, so make sure your outputs.conf matches theirs (assuming you want it to do the same thing and they aren't using syslog) and you should be good to go. Just as a point on reference, we push the same enable_outputs app from our deployment server to both Windows and Unix systems

In terms of knowing which config files Splunk is looking at, the config files get merged:

1. $SPLUNK_HOME/etc/system/default
2. $SPLUNK_HOME/etc/apps (in alphabetical order)
3. $SPLUNK_HOME/etc/system/local

If you want to see what a merged configuration looks like you can use btool (assuming C:\Program Files\SplunkUniversalForwarder is the installation directory:

"C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" btool outputs list

If you have outputs.conf in $SPLUNK_HOME/etc/apps make sure that it follows the app folder structure (so outputs.conf should be in $SPLUNK_HOME/etc/apps/app-name/local or $SPLUNK_HOME/etc/apps/app-name/default) and you need to have a metadata directory with a default.meta file. While its a bit more work to do it as an app, it lets you easily update things from a deployment server in the future.

[triest]

Good to hear you got it working and thanks for sharing what was wrong; hopefully that can help some one else avoid this frustration in the future.

Best of luck!

[macser]

First, thank you for your answer & suggestions.

While reading the answers if found out that I'd misspelled the outputs.conf file. I fixed this and have now a new error message.

From "splunkd.log"

09-16-2014 09:40:17.829 +0200 INFO  TcpOutputProc - Connection to 192.168.18.66:9997 closed. Read error. An existing connection was forcibly closed by the remote host.
09-16-2014 09:40:47.719 +0200 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
09-16-2014 09:40:48.249 +0200 INFO  TcpOutputProc - Connected to idx=192.168.18.66:9997

[macser]

First, thank you for your answer & suggestions.

I put splunksvr.intranet.local to use human-readable server names. I checked and the server can be resolved at DNS level.
However, while reading the answers if found out that I'd misspelled the outputs.conf file (as output.conf). I fixed this and have now a new error message.

I can't use other outputs.conf source as this is the first UF we're using for this Splunk server, other systems use syslog. Another team is using a Splunk server and they have forwarders but they couldn't help me find what's wrong, their setup is too different.