How to Forward indexers data to an external system?

[gabribot]

Hi, I have a Splunk Enterprise installation composed of 3 clustered indexers.

I need to forward all the events received on the 9997 port to an external system.

Data must be indexed locally but also sent to this external system.

I can't do this operation directly from universal forwarders because of network restrictions.

Is there a way to achieve this goal on indexers side?

[gabribot]

Thanks for your response.

We tried to distribute this configuration on our indexes, but it didn't work.

We saw coming data on the external system, but Splunk became not searchable and replication factor was not met.

Do you see something wrong with this one?

[indexAndForward]
index=true
selectiveIndexing=false

[tcpout]
defaultGroup=external_system
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

[external_system]
indexAndForward = true

[tcpout:external_system]
disabled=false
sendCookedData=false
server=<external_system>:<external_port>

[richgalloway]

The indexAndForward setting must be in the [tcpout] stanza.

Please elaborate on "it didn't work".

[gabribot]

We didn't put indexAndForward under the [tcpout] because the documentation says:

* This setting is only available for heavy forwarders.

But we also tried with this configuration and it didn't work the same:

[tcpout]
indexAndForward = true
defaultGroup=external_system
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

[tcpout:external_system]
disabled=false
sendCookedData=false
server=<external_host>:<external_port>

We applied this config by bundle push on the indexers.

The main issue is that the restart never ends, as you can see from the attached picture. At least one indexer remains in a "pending" state.

After apply this config, search factor and replication factor cannot be met and ALL the indexes were not fully searchable.

Despite of the invalide state of the cluster, we saw coming data on the external system.

[isoutamo]

Hi

You could forward also to Splunk as S2S traffic. 

This should be enough for that on your indexers outputs.conf

[tcpout]
indexAndForward=true

[tcpout:<Your server name or something]
server=<target server ip>:<used port like 9997 for s2s>
# other parameter what you want to use like blacklist

Then you should remember that it that connection didn't work then your indexing in local node will be stopped after remote queue is full!

r. Ismo 

[gabribot]

Hi, thanks for your reply.

We tried this approach but we had the problem described in the previous answer.

Maybe they are related to the remote queue size as you said.

Is there a way to control the remote queue size or length in tcpout mode?

[isoutamo]

You could set queue sizes on remote side only.

I think your real issue is that default group definition under tcpout section? I think that this should'n be there.

[gabribot]

We followed current documentation:

[tcpout]

defaultGroup = <comma-separated list>
* A comma-separated list of one or more target group names, specified later
  in [tcpout:<target_group>] stanzas.
* The forwarder sends all data to the specified groups.
* If you don't want to forward data automatically, don't configure this setting.
* Can be overridden by the '_TCP_ROUTING' setting in the inputs.conf file, 
  which in turn can be overridden by a props.conf or transforms.conf modifier.
* Starting with version 4.2, this setting is no longer required.

Data forwarding is working, but the state of the cluster is invalid.

We also noted these crash logs and we think they are related to this problem:

Received fatal signal 6 (Aborted) on PID 2521742.
Cause:
   Signal sent by PID 2521742 running under UID 1001.
Crashing thread: TcpOutEloop
...........
Backtrace (PIC build):
  [0x00007F02A9F91A7C] pthread_kill + 300 (libc.so.6 + 0x6EA7C)
  [0x00007F02A9F3D476] raise + 22 (libc.so.6 + 0x1A476)
  [0x00007F02A9F237F3] abort + 211 (libc.so.6 + 0x7F3)
  [0x0000556B5A5B0FA9] ? (splunkd + 0x1A52FA9)
  [0x0000556B5BA12B6E] _ZN11TimeoutHeap18runExpiredTimeoutsER13MonotonicTime + 670 (splunkd + 0x2EB4B6E)
  [0x0000556B5B939260] _ZN9EventLoop18runExpiredTimeoutsER13MonotonicTime + 32 (splunkd + 0x2DDB260)
  [0x0000556B5B93A690] _ZN9EventLoop3runEv + 208 (splunkd + 0x2DDC690)
  [0x0000556B5A97185E] _ZN11Distributed11EloopRunner4mainEv + 206 (splunkd + 0x1E1385E)
  [0x0000556B5BA0957D] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2EAB57D)
  [0x0000556B5BA0A413] _ZN6Thread8callMainEPv + 147 (splunkd + 0x2EAC413)
  [0x00007F02A9F8FB43] ? (libc.so.6 + 0x6CB43)
  [0x00007F02AA021A00] ? (libc.so.6 + 0xFEA00)
Linux / splunk-indexer01 / 5.15.0-76-generic / #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 / x86_64
assertion_failure="!_current_timeout_was_readded" assertion_function="void TimeoutHeap::assert_didnt_get_readded() const" assertion_file="/builds/splcore/main/src/util/TimeoutHeap.h:527"
/etc/debian_version: bookworm/sid
Last errno: 0
Threads running: 85
Runtime: 61.996351s
argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd]
Regex JIT enabled
.......

[isoutamo]

You should look what you need to put on UF and HF vs. what is needed on indexer in outputs.conf. Those are different thing as normally indexers just write events to disks.

On props.conf is this one for indexing and clone events to another destination. As you could see there is no default group definition.

# Clone events to groups indexer1 and indexer2. Also, index all this data
# locally as well.

[tcpout]
indexAndForward=true

[tcpout:indexer1]
server=Y.Y.Y.Y:9997

[tcpout:indexer2]
server=X.X.X.X:6666

I suppose than when you set default group here it just changing this behaviour somehow and then it cannot store events inside this cluster.

Seems to be some kind of timeout which happened before that crash.

Have you see any events on target system? Based on port I assume that target is also splunk?

If so, you should remove "sendCookedData = false" to send S2S data to remote.

My guess is that this should work

[tcpout]
indexAndForward=true

[tcpout:external_system]
disabled=false
forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection)
server=<external_system>:9997

[gabribot]

We tried to remove the default group as you suggested but it gave us the same error.

We don't have to send data to another Splunk, on the other side there will be Fluentd that will capture the data.

At the moment we are trying to send data to a socket opened with netcat on another device in the same subnet.

We see coming data on netcat, but Splunk crashes on the indexers.

This is the btool output related to output.conf:

/opt/splunk/etc/system/local/outputs.conf   [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf enableOldS2SProtocol = false
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)
/opt/splunk/etc/system/local/outputs.conf   forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/local/outputs.conf   indexAndForward = true
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf useClientSSLCompression = true
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf   [tcpout:external_system]
/opt/splunk/etc/system/local/outputs.conf   disabled = false
/opt/splunk/etc/system/local/outputs.conf   sendCookedData = false
/opt/splunk/etc/system/local/outputs.conf   server = <external_server>:<external_port>

[isoutamo]

As your receiver is fluentd, I assume that you have syslog source listener on it? You probably have something similar than 

<source>
  @type syslog
  port 8080
  bind 0.0.0.0
  tag cf.app
  message_length_limit 99990
  frame_type octet_count
  <transport tcp>
  </transport>
  <parse>
    message_format rfc5424
  </parse>
</source>

In splunk side you must format sending events to be a valid syslog message (RFC5424). Otherwise fluentd didn't accept those and quite soon splunk's queues are full and so on...

Unfortunately I haven't currently any syslog server to test this. But I suppose that it's goes something like this

https://docs.splunk.com/Documentation/Splunk/9.1.1/Admin/Outputsconf#Syslog_output----

[syslog]
defaultGroup = syslog:syslog_out

[syslog:syslog_out]
server = <Your fluentd server>:<receiving port>
type = tcp
timestampformat = %b %e %H:%M:%S
maxEventSize = <XXXX if greater than 1024>

Probably you are also needing a props.conf & transforms.conf to route events also to this syslog output instead of that pure tcpout (or maybe you don't need tcpout-stanza?)? I hope that those instructions are enough clear on docs. There is also some old posts, but unfortunately those seems to be for HF configuration not for indexer.

Please inform us what is actually configuration which is working after you have get it. 

[richgalloway]

Splunk can index data locally and forward it to another system.  The assumption, however, is that the other system is Splunk.

Splunk indexers can forward to non-Splunk systems, but only as raw TCP or syslog. 

See https://docs.splunk.com/Documentation/Splunk/9.1.0/Admin/Outputsconf#TCP_Output_stanzas and https://docs.splunk.com/Documentation/Splunk/9.1.0/Forwarding/Forwarddatatothird-partysystemsd for more information.