I have a situation where I want to send just the content of one local log file on one indexer ("test_indexer") to another indexer ("production_indexer"). Apart from that, the sending indexer in this scenario ("test_indexer") should continue to function as usual (indexing everything else locally). My plan was to just add an additional tcpout stanza in outputs.conf (in my case [tcpout:production_indexer] in /opt/splunk/etc/system/local/outputs.conf) and declare the _TCP_ROUTING parameter for the specific stanza in inputs.conf.  Problem: The sending indexer ("test_indexer") stops indexing any incoming and local data completely after I add the following configurations: /opt/splunk/etc/system/local/inputs.conf   [monitor:///path/to/my/file.log] index = my_index sourcetype = my_sourcetype _TCP_ROUTING = production_indexer   /opt/splunk/etc/system/local/outputs.conf   [tcpout:production_indexer] clientCert = $SPLUNK_HOME/etc/auth/server.pem server = xyz:9998 sslPassword = $abc== sslVerifyServerCert = false useSSL = true   To me, this behavior is wrong. I am just adding an additional, non-default tcpout stanza (on top of the default one defined in /opt/splunk/etc/system/default/outputs.conf) that is used only by one specific input stanza. According to my understanding, this change should neither impact any other inputs not the default tcpout definition.  Debugging output before adding the above configuration:    $ splunk btool --debug outputs list /opt/splunk/etc/system/default/outputs.conf [syslog] /opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024 /opt/splunk/etc/system/default/outputs.conf priority = <13> /opt/splunk/etc/system/default/outputs.conf type = udp /opt/splunk/etc/system/default/outputs.conf [tcpout] /opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30 /opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30 /opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0 /opt/splunk/etc/system/default/outputs.conf blockOnCloning = true /opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100 /opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz /opt/splunk/etc/system/default/outputs.conf compressed = false /opt/splunk/etc/system/default/outputs.conf connectionTTL = 0 /opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20 /opt/splunk/etc/system/default/outputs.conf disabled = false /opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5 /opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1 /opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1 /opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false /opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .* /opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.* /opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup) /opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false /opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30 /opt/splunk/etc/system/default/outputs.conf indexAndForward = false /opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2 /opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2 /opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto /opt/splunk/etc/system/default/outputs.conf readTimeout = 300 /opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1 /opt/splunk/etc/system/default/outputs.conf sendCookedData = true /opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false /opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2 /opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0 /opt/splunk/etc/system/default/outputs.conf useACK = false /opt/splunk/etc/system/default/outputs.conf writeTimeout = 300   Debugging output after adding the above configuration:    $ splunk btool --debug outputs list /opt/splunk/etc/system/default/outputs.conf [syslog] /opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024 /opt/splunk/etc/system/default/outputs.conf priority = <13> /opt/splunk/etc/system/default/outputs.conf type = udp /opt/splunk/etc/system/default/outputs.conf [tcpout] /opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30 /opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30 /opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0 /opt/splunk/etc/system/default/outputs.conf blockOnCloning = true /opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100 /opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz /opt/splunk/etc/system/default/outputs.conf compressed = false /opt/splunk/etc/system/default/outputs.conf connectionTTL = 0 /opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20 /opt/splunk/etc/system/default/outputs.conf disabled = false /opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5 /opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1 /opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1 /opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false /opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .* /opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.* /opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup) /opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false /opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30 /opt/splunk/etc/system/default/outputs.conf indexAndForward = false /opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2 /opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2 /opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto /opt/splunk/etc/system/default/outputs.conf readTimeout = 300 /opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1 /opt/splunk/etc/system/default/outputs.conf sendCookedData = true /opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false /opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2 /opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0 /opt/splunk/etc/system/default/outputs.conf useACK = false /opt/splunk/etc/system/default/outputs.conf writeTimeout = 300 /opt/splunk/etc/system/local/outputs.conf [tcpout:production_indexer] /opt/splunk/etc/system/local/outputs.conf clientCert = $SPLUNK_HOME/etc/auth/server.pem /opt/splunk/etc/system/local/outputs.conf server = xyz:9998 /opt/splunk/etc/system/local/outputs.conf sslPassword = $abc== /opt/splunk/etc/system/local/outputs.conf sslVerifyServerCert = false /opt/splunk/etc/system/local/outputs.conf useSSL = true   Note: Setting  indexAndForward to true is not an option as I really only want to forward the contents of the one specific local log file to the other indexer.  ... View more

I'm trying to figure out the sizing of a Splunk environment that will only be used for a very short time but by a substantial amount of users (20-40 workshop participants). All these users will be running searches simultaneously against the same index. The idea of the workshop is to build a dashboard to visualize some previously indexed data. In a way it's very similar to the official Splunk4Rookies workshop, just with different data. My concern is that the user experience will be terrible due to too many searches being attempted at same time. This raises the following questions: - how many concurrent searches need to be possible to support 30 users simultaneously building dashboards? - how far can I increase the max_searches_per_cpu parameter in limits.conf? what are the downsides? - assuming all data resides in one index (and all searches being run on this index) is this enough or should one index be replicated by implementing indexer clustering? how many searchable copies would be necessary? I'm hoping to be able to use an all-in-one Splunk instance (so no indexer clustering) but I have no means to realistically test the search performance/experience with 20-40 simultaneous users before the actual workshop. Has anyone have any experience with such a setup or does anyone know how Splunk does this for their Splunk4Rookies workshops? Thanks ... View more

Going through the documentation of Connected Experiences (https://docs.splunk.com/Documentation/Gateway/1.8.0/Installation/ConfigureCloudGatewayConf) I noticed the attribute spacebridge_server in cloudgateway.conf. Obviously, the default value for this is currently prod.spacebridge.spl.mobi where the Splunk-hosted Cloud Gateway service (also seems to be known as spacebridge) resides. This raises the questions for me if an on-prem version of the spacebridge will become available soon? This "feature" would be extremly important (if not to say necessary) for many of my enterprise clients to have as that would mean they wouldn't have to "open" their productive environments to the outside world (through the Cloud Gateway app on a Search Head that currently requires an open port to prod.spacebridge.spl.mobi). ... View more

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how? And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly? Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert? ... View more

I'm trying to get Splunk TV on Apple TV approved by Security in my company. The connection from the Cloud Gateway app on the SHC to the Cloud Bridge is sufficently documented here: https://docs.splunk.com/Documentation/Gateway/1.8.0/Installation/Installation, but I'm struggling to find the equivalent for the connection between the Cloud Bridge and Apple TV with the Splunk TV app on it. What I'm trying to find out is what connections on which port(s) I need to open in the Firewall for Apple TV and the Splunk TV app to work. Does anyone have more details here? Perhaps from your own attempts of getting Splunk TV on Apple TV approved? ... View more

In Indexer Clustering, is there a way to push the configuration bundle from a Cluster Master to the Indexers and explicitly don't restart the Indexers? I'm thinking there might be an updated configuration bundle (which requires a restart of the Indexers) that I want to distribute to the Indexers but only execute the restart of the Indexers at a later moment (i.e. a scheduled time-window when the load is low on the Indexers). For Search Head Clustering, this is possible and documented (https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/PropagateSHCconfigurationchanges#Control_the_restart_process), but for Indexer Clustering I didn't find any mention in the documentation. ... View more