1) Try to do the following: Increase distsearch.conf timeouts on the SH as: [distributedSearch] statusTimeout = 120 connectionTimeout = 120 authTokenConnectionTimeout = 120 authTokenSendTimeout = 120 authTokenReceiveTimeout = 120 [replicationSettings] connectionTimeout = 120 sendRcvTimeout = 120 On the indexers at distsearch.conf [replicationSettings] connectionTimeout = 120 sendRcvTimeout = 120 and in server.conf [httpServer] busyKeepAliveIdleTimeout = 120 It seems it has a little bit improvement after the change. But the error message is still shown from time to time. 2) Checking the pstack output, find the I/O thread is busy with SSL handshakes. SSL operation slow down the process causing timeout. 3) SSL operation (especially compression) are CPU intensive routes. It needs to be invoked within the main IO thread (management port) that IO operation will be slowed down. 4) It's a known issue limitation of OpenSSL design - compression is done during the write operation blocking IO 5) Disable the SSL client compression in the search head server.conf [sslConfig] useClientSSLCompression = false 6) The system is running back to normal after disabling the ssl client compression in the search head ... View more

We finally find that there is no supported way to change the password via a deployment server. However, we can reset the admin password by using the following command: splunk edit user -auth admin: --newpassword and we would change the password across all the UFs is to set up a script with this command the have the script deployed and ran on all the deployment clients Try to create a dump app and deploy it to the UF - create the inputs.conf e.g. [script://$SPLUNK_HOME/etc/apps//bin/change_pass.sh] disabled = false interval = 100000 (any big value) - create the change_pass.sh script and issue the splunk change admin password command. As it expects to run it once, we could have some checking to stop it from changing password again if it’s run already. ... View more

Highly suspect it's the Anti-virus and firewall issue in the window server but cannot find any related log in the AV or firewall blocking it. Upgrade to the v667 and have the issue fixed as the WinHostInfo process can be restarted in that version if it's down for some reasons. It's kind of workaround but it helps to resolve the issue. ... View more

Encounter a known issue on kvstore in v7.1.x. Splunk Search head cluster initial kv sync continues to fail. Try to apply the following workaround and it helps. Add the following attribute for all SH members and restart all SH members server.conf [kvstore] allowUnsafeRenamesDuringInitialSync = true NOTE: allowUnsafeRenamesDuringInitialSync attribute is only available in mongodb 3.4.10 and is not available prior to Splunk version 7.1.4. ... View more

1) Check there is any anti-virus running in the Windows 2) Check there is any firewall blocking the process 3) Upgrade to the v659 or v667. It has a fix to startup the process again. ... View more

Find a way to fix it. Update the idpCert.pem file after the ADFS certificate is updated Generation of the federationmedata.xml from ADFS Open the xml file by the xml viewer Find the X509Data -> X509Certificate Backup the idpCert.pem and create a new one -----BEGIN CERTIFICATE----- < content of the X509 Certificate> -----END CERTIFICATE----- ... View more

There are TWO ways to do it. 1) Do the backup of the ITSI. Inside the backup, it has the json of the glass table 2) You could try the following http://xxxx.splunk.com:8000/en-US/custom/SA-ITOA/itoa_interface/nobody/glass_table/ ... View more

After the dirty shutdown, the bucket got corrupted and Splunk marked it for further investigation. ls -laR emptybucket-hot_v1_xxx Check that it has the journal.gz and necessary files... Then do the following 1) Stop Splunk 2) make backup of that bucket 3) rename the bucket back to hot_v1_xxx 4) repair using fsck (and adding --include-hots) (save log output) 5) Start Splunk ... View more

I finally find the answer. When using the SDK to get the dashboard data, GET data/ui/views/{name} method detail We can get the eai:data (xml definition for the dashboard) and the SPL will be inside the xml. Have to go deeper level of xml. ... View more

Is there any error message in the splunkd.log ? Do you still see the Jasonlinbreaker error right now ? Have you tried to disable the SSL and see it recover ? ... View more

Is it possible they are referring different date ? Have you tried to add the date together with the time ? ... View more

you want to run the alert again and again until it got the result more than 0. Isn't it ? Or alert return result 0 and just rerun the alert once only ? ... View more

Please check the cipherSuite parameter and see they are matched in Indexer and forwarder ... View more

You could use the search id to find the corresponding search folder in the $SPLUNK_HOME/var/run/splunk/dispatch. Inside the search folder, you could find the search content for that particular search. ... View more

Please try to check there is any error in splunkd.log and metrics.log ($SPLUNK_HOME/var/log/splunk folder). It should give you some hints if there is anything wrong in Splunk. ... View more

the notification of orphaned searches are new searches or the previous 11 ? ... View more