Splunk search process is consuming high memory causing splunkd to crash. How to collect jemalloc heap data for Splunk Technical Support to troubleshoot high memory growth issues contributed by the specific search process?    ... View more

Splunk web is returning HTTP 500 internal server error after entering login credential as splunkd fails to start up. We have checked the server.conf:[httpServer] :disableDefaultPort is set to false. ... View more

After performing "./splunk clean all" on one of the search head, we are having issue to add the search head back to the search head cluster. Permission error message is reported when adding the search head back to the cluster: ./splunk add shcluster-member -current_member_uri URI: You (user=admin) do not have permission to perform this operation (requires capability: edit_search_head_clustering). ... View more

Overview dashboard shows "No results found" for CPU, Memory, Disk, Network when Splunk app for Infrastructure is installed on Splunk version 7.1.x. ... View more

Rsyslogd server is setup to send syslog messages to Splunk HTTP Event Collector (HEC) using omhttp module. During high load, HEC sending the following response to rsyslogd server: http status code: 503 reply: {"text":"Server is busy", "code":9, "invalid-event-number":0}. How to address this issue? ... View more

KV Store is not starting up after upgrading Splunk to version 7.2.5.1. ./splunk show kvstore-status shows status as failed. When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error: ./splunk migrate migrate-kvstore [App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds. WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds. App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function. mongod.log reported the following error: F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail ... View more

Splunk is not starting up when performing an upgrade to Splunk 7.1.6 on the indexer. Error message when starting up Splunk: mongod exited abnormally(exit code 100, status: exited with code 100) - look at mogod.log to investigate.  ERROR: Failed to fix non-voting members Priority value during mogod version migration from 3.0.x to 3.4.x. Reason: ERROR while running mogod-fix-voting-priority migration.  ... View more

Windows Event Log files (.evtx) monitoring stop working after a while and the Splunk universal forwarder has to be restarted to start data collection again. Here is the [monitor] stanza configured to monitor the Windows Event Log files (.evtx): [monitor://C:\Windows\System32\winevt\Logs\VisualSVNServerActivity.evtx] disabled = 0 index = WinEvent [monitor://C:\Windows\System32\winevt\Logs\VisualSVNServerManagement.evtx] disabled = 0 index = WinEvent ... View more

Indexer in the cluster was abruptly shutdown and subsequently fail to join back to the cluster. Please help to provide the steps to clean up the standalone buckets to allow the indexer to join back to the cluster. warning message in splunkd.log: xx-xx-xxxx xx:xx:xx.xxx -0500 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=xxx.xxx.xxx:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line=“Internal Server Error” socket_error=“No error” remote_error=Cannot add peer=xxx.xxx.xxx.xxx mgmtport=8089 (reason: bucket already added as clustered, peer attempted to add again as standalone. guid=C199873F-6E72-43D8-B54F-554750ACE904 bid= mi_batch~314~C199873F-6E72-43D8-B54F-554750ACE904). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=403F2E7869E35F5BB8C945D993035AA2 add_type=Initial-Add base_generation_id=0 batch_serialno=7 batch_size=18 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=403F2E7869E35F5BB8C945D993035AA2 mgmt_port=8089 name=C199873F-6E72-43D8-B54F-554750ACE904 register_forwarder_address= register_replication_address= register_search_address= replication_port=8003 replication_use_ssl=0 replications= server_name=xxx.xxx.xxx site=default splunk_version=7.2.0 splunkd_build_number=8c86330ac18 status=Up } ]. ... View more

Would Splunk deployment servers have compatibility issue with deployment clients that are on earlier version? ... View more