Getting Data In

Graylog sending to SPLUNK over 9997

pfabrizi
Path Finder

I have Graylog forwarding to a UF over port 9997 and I see events streaming in but not being picked up by SPLUNK. I have a inputs.conf set to [splunktcp:9997] . I tried to setup a syslog.conf to tream to a file but realized that is port 514 and 9997. Can any one provide some debugging hints?
How can I see if the events are getting picked up by the UF and just not forwarding?
I looked in the Metrics.log but see nothing, is that the correct place to look?

Thanks!

Tags (1)
0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

You are trying to send to a port that expects the (proprietary) Splunk-2-Splunk protocol; the message indicates that: "Possible invalid source sending data to splunktcp port". It will not understand the wire format Graylog is using.
You may be more successful by creating a network input for a different port and use that as your Graylog destination.

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

Usually the port 9997 is used for the splunk protocol splunktcp. (only used by splunk forwarders)
If your "graylog" software is sending logs, it is probably not using this protocol.

looking at the internet, it seems that some people created code to have graylog send data over TCP to splunk :
https://github.com/graylog-labs/graylog-plugin-splunk

If it is sending data as syslog , please setup splunk to listen to UDP or TCP on a different port, and try send the data to it to see. (you may have to create a sourcetype to get proper event parsing)

If the graylog is able to send data to a splunk HEC "http event collector" API, try to setup such an input on splunk, grab the token, and use it to configure the graylog sender.

pfabrizi
Path Finder

Hi SSIEVENT,
I was just thinking that and in the process to tell the Graylog folks to send under another port. I am using a UF, so I have no UI, but i should be able to setup a inputs.conf with a tcp listener.

Thank You!

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

The link I sent also includes instructions on how to do it using inputs.conf (or the CLI).

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

You are trying to send to a port that expects the (proprietary) Splunk-2-Splunk protocol; the message indicates that: "Possible invalid source sending data to splunktcp port". It will not understand the wire format Graylog is using.
You may be more successful by creating a network input for a different port and use that as your Graylog destination.

0 Karma

pfabrizi
Path Finder

OK, I changed the port to 9996. I see it listening on that port. I no longer get the error messages but I am not seeing any data flow to indexer.
am I missing anything? This is running on a Linux UF.

This is my inputs.conf
[tcp://9996]
index=wineventlog

I see these messages in the metrics.log but don't know what they mean.

06-20-2018 10:35:13.269 -0400 INFO Metrics - group=tcpin_connections, 10.xx.xx.4:51718:9996, connectionType=raw, sourcePort=51718, sourceHost=server.doamin.net, sourceIp=10.xx.xx.4, destPort=9996, kb=0.00, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.01, _tcp_Kprocessed=2.14, _tcp_eps=0.00, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.00, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00

0 Karma

yannK
Splunk Employee
Splunk Employee

So we see some connections coming in. (do the sum of kb over some time to see the volume).

now you need to get a proper parsing.
- define a sourcetype on your input on the UF
- on the indexers, define the sourcetype in props.conf with the proper rules to : break the events, find the timestamp, consider multiline events, define the timezone etc..

If you are not sure, try with sourcetype=syslog (on the UF) and see what it does.

0 Karma

pfabrizi
Path Finder

Thank You!

0 Karma

pfabrizi
Path Finder

Sorry, I wasn't seeing this yesterday but I am today.

06-19-2018 07:16:00.830 -0400 ERROR TcpInputProc - Message rejected. Received unexpected message of size=842019128 bytes from src=10.00.0.7:52640 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

I am going to go back to our Graylog folks and see if they can decrease the payload, is this correct?

Thanks!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...