Windows: Sending Splunk logs to third party server

[spectrum2035]

I need to send Windows Event logs to the third party syslog solutions. Logs from Windows Universal Forwarder is sent to HFWD and from there it is routed both Splunk IDX and Syslog Aggregator. For some reasons its not hitting the syslog server. I have checked btool for input, output, props and transforms and couldn't find anything there.

Config on the HFWD to accept logs from the Windows server and to send it to syslog

=========================================================================

props.conf

[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog

transforms.conf

[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog

outputs.conf
#Sent to Indexer

[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp

inputs.conf
[splunktcp://10.20.10.10:9997]
#machine not part of the domain so need to use the IP address
#_SYSLOG_ROUTING = to_syslog

=========================================================================
Config on Windows UF

[tcpout]
defaultGroup = send_to_syslog
maxQueueSize = 7MB
autoLBFrequency=15


[tcpout:send_to_syslog]
server = 10.175.108.40:9997
#sendCookedData = false
=========================================================================

One of the base app to send logs from Heavy FWD to INDX

[tcpout]
default_group = indexer_fwd
axQueueSize = 7MB
autoLBFrequency=15

[tcpout:indexer_fwd]
server = IDX1.abcd.com:9997, IDX2.abcd.com:9997, IDX3.abcd.com:9997

[koshyk · Accepted solution]

Please try following

ALL below settings have to be done on the Heavy Forwarder

In props.conf

#props.conf
[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog

in transforms.conf

#transforms.conf
[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog

In outputs.conf (alongside your indexer outputs, you need to add syslog stanza separately)

# outputs.conf
[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp

If this is not working,
1. you need to get tcpdump of HeavyForwarder to see if there is some network connection issue. You need to see destination traffic going to 10.172.148.186 on port 1514
2. Check for firewall issues
3. Check tcpdump at the destination server level to ensure the message is captured at wire
4. Try setting up this connection to another server which you own with no firewall etc.
5. Try removing the indexer setting just to see if there is any conflict of stanza. You can double check using btool

[spectrum2035]

The issue was with the firewall, Network team didnt open the firewall rule (though they said its been done).

[codebuilder]

You have a typo in your tcpout stanza in your app (based on the values you provided):

 [tcpout]
 default_group = indexer_fwd
 axQueueSize = 7MB
 autoLBFrequency=15

Should be:

 [tcpout]
 default_group = indexer_fwd
 maxQueueSize = 7MB
 autoLBFrequency=15

That is likely causing errors/issues. Also, make sure that you have an index created to receive your data. This should match your DEST_KEY.

[evania]

Hi @codebuilder ,

Did you have a chance to check out answers? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

[spectrum2035]

It didnt work.. as commented by koshyk we are still waiting for tcpdump data from the linux team.

[spectrum2035]

Thanks codebuilder for your response. I did check "maxQueueSize" settings, it was the typo while i put here. Logs are receiving at the indexer without any issues.

We have issues only at the syslog aggregator (10.172.148.186 at port 1514)

[codebuilder]

Do you have the pass4SymmKey set correctly on your HF?

[spectrum2035]

To add, we are not using SSL as we are currently testing phase. HF are working for other data sources and is able to send data to the Indexers. So pass4SymmKey wont be an issue here.
The issue is with the HF to syslog.

[spectrum2035]

Just to give one more update: When i have checked the internal index with port 1514 i am getting following output..

06-06-2019 16:46:36.584 +0100 INFO Metrics - group=syslog_connections, ingest_pipe=1, to_syslog:10.172.148.186:1514:10.172.148.186:1514, sourcePort=8089, destIp=10.172.148.186, destPort=1514, _tcp_Bps=2744.97, _tcp_KBps=2.68, _tcp_avg_thruput=2.74, _tcp_Kprocessed=46355, _tcp_eps=12.68

[spectrum2035]

Forgot to add, i have also added to_syslog to the tcpout default_group

[tcpout]
default_group = indexer_fwd, to_syslog as some else mentioned that it has resolved their issue. But for me No Luck 😞