As of version 6.0, it is now possible for admin users or users with the get_diag capability to remotely pull diags from REST endpoint "/streams/diag". Just use curl -k -u admin:changeme https://server_name:port/services/streams/diag >> diag_server.tar.gz" or you can just pass a remote uri to the already existing diag command: ./splunk diag -uri remote-server-name:remote-splunk-port ... View more

As of version 6.0, it is now possible for admin users or users with the get_diag capability to remotely pull diags from REST endpoint "/streams/diag". Just use curl -k -u admin:changeme https://server_name:port/services/streams/diag >> diag_server.tar.gz" or you can just pass a remote uri to the already existing diag command: ./splunk diag -uri remote-server-name:remote-splunk-port ... View more

This snapshot indicates that some bucket(or buckets) from the _audit index is missing the required number of copies to meet the replication factor, search factor configurations at this point. ie some bucket from _audit might be missing a replicated copy. Are there any messages on the master instance (when you login as the admin)? If there are, can you paste them here? Alternatively, in order to find out what the exact problem is, I'd suggest looking at $SPLUNK_HOME/var/log/splunk/splunkd.log and looking for any errors for the _audit buckets. Typically, this situation should get remedied automatically after some time. ... View more

In case the restart doesn't work, just trying making a small change to the $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf file (like adding a comment), and try to push the bundle again ... View more