Forwarder SSL and NO-SSL Forwarding

[fabrizioalleva]

Hi all,
I've an enviroment like this:

1 Search Head Cluester ( 3 servers ) ;
1 Indexers Cluster ( 4 server );
1 Deployment Server;
1 Cluster Master;
1 Heavy Forwarder;
and N universal forwarder.

On each server, of the indexer cluster, I've opened the 9997 port to receive and ingest data.

I configured the Heavy forwarder to send data to the IDX cluster through an'APP with an outputs.conf like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
indexerDiscovery = idxc1
useACK = true
autoLBVolume = 65536

[indexer_discovery:idxc1]
master_uri = https://[cm_ip]:8090
pass4SymmKey = **********

This output is not SSL and all the data which use this output are ingested from IDX cluster.

Now I've to send from the same Heavy Forwarder other data but in SSL. So after creating the certificates for Indexers and for the forwarders, after copying the RootCA on each server, I've created an app on the forwarder with an inputs.conf like this:

[tcp-ssl://6514]
disabled = false
sourcetype = syslog
index = ads

[SSL]
serverCert = $SPLUNK_HOME/etc/certs/hforwarder.pem
sslPassword = ***********
requireClientCert = false

and outputs.conf like this :

[tcpout]
defaultGroup=splunkindexer-ssl

[tcpout:splunkindexer-ssl]
autoLBFrequency = 30
compressed  = false
server  = idx1:9996,idx2:9996,idx3:9996,idx4:9996
clientCert  = $SPLUNK_HOME/etc/certs/splunk.pem
useSSL=true
sslPassword  = ***********

On the each indexers node I've create an app to open the 9996 port on SSL, with an inputs.conf like this:

[tcp-ssl://9996]
disabled = false

[SSL]
serverCert = $SPLUNK_HOME/etc/certs/splunk.pem
sslPassword = **********
requireClientCert = false

After deploying this configuration the first flow through the 9997 doesn't receive any data and not all data are forwarded through 9996.
With tcpdump I see the right flow from the source to Heavy Forwarder (6514) , I see some data receveived from the indexer cluster on 9996 but no data where indexed on ads index and on other index.

So now I ask myself what should be wrong and if a forwarder can support two outputs.conf so configured.

Any help is appreciated and welcome
Fabrizio

[harsmarvania57]

Why are you sending part of data over SSL and other data on non-SSL from same Heavy Forwarders to same set of Indexers ? If there is requirement to send some data over SSL then I'll send all data over SSL.

[fabrizioalleva]

Ok,
Probably the easiest way is to configure the 9997 port on indexer in SSL, and all the comunication will go over SSL
But If I want to configure the other port ?
Does the Heavy Forwarder support a configuration like the one I've posted in my question ?

Fabrizio

[harsmarvania57]

You can configure to send different data on different tcpout stanza but in that case you need to use _TCP_ROUTING parameter in inputs.conf

[fabrizioalleva]

Hi,
in the last days I've succeded in establishing the right comunication between HF and IDX. So now when i bring up the port 9996 all the other forwarding still work fine.
Only the one which runs on 6514 (hf) ---> 9996 (idx_cl ) give me those messagges in metrics.log:

01-27-2020 14:41:26.012 +0100 INFO Metrics - group=tcpin_connections, a.b.c.d:40386:6514, connectionType=rawSSL, sourcePort=40386, sourceHost=a.b.c.d, sourceIp=a.b.c.d, destPort=6514, kb=0, _tcp_Bps=0, _tcp_KBps=0, _tcp_avg_thruput=0, _tcp_Kprocessed=0, _tcp_eps=0, _process_time_ms=0, evt_misc_kBps=0, evt_raw_kBps=0, evt_fields_kBps=0, evt_fn_kBps=0, evt_fv_kBps=0, evt_fn_str_kBps=0, evt_fn_meta_dyn_kBps=0, evt_fn_meta_predef_kBps=0, evt_fn_meta_str_kBps=0, evt_fv_num_kBps=0, evt_fv_str_kBps=0, evt_fv_predef_kBps=0, evt_fv_offlen_kBps=0, evt_fv_fp_kBps=0

Any suggestion?
Thanks

[harsmarvania57]

Hi this is not error message, it is INFO message that source a.b.c.d connected to HF on port 6514 but no events sent (Based on logs you provided)

[fabrizioalleva]

I've tried even this way, but after the deployment, was alway the same.
In all the NO-SSL app's input.conf I put

_TCP_ROUTING = default-autolb-group

and in the SSL one :

_TCP_ROUTING=splunkindexer-ssl

but it is always the same.