This is interesting.  https://docs.splunk.com/Documentation/Splunk/9.0.2/DistSearch/Forwardsearchheaddata Discusses how to forward internal logs from search head cluster members. It appears to recommend using the deployer to propagate the outputs.conf file. The outputs.conf files sits in $SPLUNK_HOME/etc/system/local folder which the deployer can not propagate to. Am I reading this wrong?  Forward data from search head cluster members You perform the same configuration steps to forward data from search head cluster members to their set of search peers. However, you must ensure that all members use the same outputs.conf file. To do so, do not edit the file on the individual search heads. Instead, use the deployer to propagate the file across the cluster.  ... View more

Thanks. This is what I expected. No problem, it is a static change and I can make changes on each server. ... View more

I do have the full path on my Splunk server to run the tower-cli script. I can run the actual tower-cli command that is in the alerts scripts from any path on the CLI.... "/opt/rh/python27/root/usr/bin/tower-cli user list >> "/home/splunk/bin/scripts/nac_interfaces.log" Just can't execute it from the alerts scripts in Splunk Not sure what path it can't find. The tower-cli is attempting to login to another server to run the playbook. Here is the content of the tower-cli file: [root@splunk bin]# cat tower-cli #!/opt/rh/python27/root/usr/bin/python2 # -*- coding: utf-8 -*- import re import sys from tower_cli.cli.run import cli if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(cli()) Here is the tower-cli config. foobar would be the host that has the playbook to run [root@splunk bin]# tower-cli config # User options (set with `tower-cli config`; stored in ~/.tower_cli.cfg). host: foobar.com username: splunk.service password: xxxxxxxxx verify_ssl: False # Defaults. use_token: False verbose: False certificate: format: human color: True description_on: False Could it be a permissions error? When I attempt to run the script at the CLI as a non-root user I get this error: "/opt/rh/python27/root/usr/bin/python2: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory" ... View more

I am not sure why you want to remove the data from your index. Why not just filter out through your searches? If you still want to "remove" the old data you can either use the "delete" operator by entering something like this... index=blah source="prog.log.run" | delete. This will basically just hide the data from searching and viewing. If your index only contains this source and nothing else you would not need to specify the source in the search command. If you want to completely remove/clean all data from the index you can do this at the CLI with the clean command splunk stop splunk clean eventdata -index < *index_name >* Be careful as this will remove all event data from the (index_name) and is NOT recoverable. If you neglect to provide the -index name ALL indexes are cleaned! ... View more

In Linux it is easy enough to give users access to their own application folders/files, however, When a knowledge object is created in the GUI the ownership of that file is Splunk or Root and that user cannot modify it at the CLI. Am I missing something fundamental here? ... View more