Cannot remove forward-server

[andreasknutsson]

I have inherited an old on-prem Splunk 7.0.2 installation that I'm now trying to reconfigure to forward data to our SplunkCloud instance.
I have installed the SplunkCloud app on the search-head that is acting as deploy-server. It is now forwarding its internal logs to the cloud as expected. Now I want to remove the old forward-servers.

When I execute: ./splunk list forward-server
I get this:

Active forwards:
10.yy.167.67:9997 (ssl)
inputs1.q.splunkcloud.com:9997
(ssl) Configured but inactive
forwards:
10.yy.167.68:9997
inputs10.q.splunkcloud.com:9997
inputs11.q.splunkcloud.com:9997
inputs12.q.splunkcloud.com:9997
inputs13.q.splunkcloud.com:9997
inputs14.q.splunkcloud.com:9997
inputs15.q.splunkcloud.com:9997
inputs2.q.splunkcloud.com:9997
inputs3.q.splunkcloud.com:9997
inputs4.q.splunkcloud.com:9997
inputs5.q.splunkcloud.com:9997
inputs6.q.splunkcloud.com:9997
inputs7.q.splunkcloud.com:9997
inputs8.q.splunkcloud.com:9997
inputs9.q.splunkcloud.com:9997

This looks correct.
Then I execute: ./splunk remove forward-server 10.yy.167.68:9997 and get this message:

In handler 'tcpout-server': Type = outputs, Context = (user: nobody, app:, root: /opt/splunk/etc), Acting as = nobody: Invalid configuration context: Cannot read configuration if user context is set but app context is not

I get the same kind of message if I try to do this in the web gui.

I have tried to find the configuration file that contains this forward-server config but I cannot locate it.
It should be /splunk/etc/system/local/outputs.conf but there is no such file. I have tried grabbing for the IP address in the entire /etc directory structure but got no matches!

Does anyone have any input on how to proceed?

Regards,
Andreas

[harsmarvania57]

Hi,

Try to find out tcpout stanza from outputs.conf using btool and find that server.

$PLUNK_HOME/bin/splunk cmd btool outputs list --debug

[andreasknutsson]

Good idea!
I executed that command and found one suspect:

[tcpout:clustered_indexers]

[indexer_discovery:cluster_master]

Could this be the issue? Is it safe to remove? We are going to retire the entire on-prem indexer setup, we do already have all data in SplunkCloud.

[harsmarvania57]

Server 10.yy.167.67 under any tcpout stanza with parameter server ?

It looks like you are using Indexer discovery and due to that 10.yy.167.67 indexer is coming from Cluster Master, in that case that server will be removed from forward-server list when you'll decommission that indexer from Indexer Cluster.