Deployment Architecture

How can I forward the internal Splunk logs of a Splunk deployment to another Splunk

Ledio_Ago
Splunk Employee
Splunk Employee

Let's say I have a distributed Splunk environment, n indexers, one search head and a forwarder load balancing input data to these indexers. I like to pull out all of the internal Splunk logs from this deployment and have them forwarded to another Splunk for monitoring purposes. What's the best way of doing that?

2 Solutions

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...