you have to run the REST API with the passwords.conf on the same Splunk instance which you're trying to decrypt the pass4SymmKey since the splunk.secret key file used for encryption is different on each Splunk instance. It's autogenerated during splunk installation. You can test the procedure on a dev instance first with same procedure before doing so in production. ... View more

so the old “Load Existing Settings” UI is gone, but the data is still available via search: | kvstorelookup collection_name=showcase_linear_regression_history *replace collection_name with the correct one for your assistant nothing new will get written to this (it’s there so that old data is not lost), but you can retrieve your settings and migrate them to an Experiment The values of collection_name are one of: showcase_linear_regression_history, showcase_classification_history, showcase_categorical_outlier_detection_history, showcase_clustering_history, or showcase_forecast_history ... View more

If you run the following two REST endpoint searches: | rest splunk_server=local /servicesNS/nobody/SplunkEnterpriseSecuritySuite/alerts/alert_actions | search (is_custom=1 OR name="email" OR name="script") AND disabled!=1 | table title and |rest splunk_server=local /servicesNS/nobody/SplunkEnterpriseSecuritySuite/data/ui/alerts |table title Do you see more than 30 results? SOLNESS-18523 is due to the fact we are truncating the results at 30 hence not all the action are visible. If you have all the entries stop showing assuming it was working previously, i would suggest first try clearing browser and splunkd cache: https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/CustomizationOptions#Clear_client_and_server_assets_caches_after_customization ... View more

are you running Splunk version 7.2.2 or above? It's only introduced from 7.2.2. ... View more

when you delete lookup through REST API it actually gets synced through SHC which means you only need to delete the lookup from one SH and the same lookup file will be removed from all the SHs in the cluster per my lab test. ... View more

From Splunk version 7.2.2 and above, you may run below command to decrypt the encrypted password to find the original clear text password on the same splunk instance: *./splunk show-decrypted --value '< pass4SymmKey >' * https://docs.splunk.com/Documentation/Splunk/7.2.2/Security/ConfigureS2Sonnewcipher Pre 7.2.2: You may obtain a clear text password for pass4SymmKey through below steps: 1) Create passwords.conf in $SPLUNK_HOME/etc/apps/search/local folder 2) Copy encrypted pass4SymmKey under each stanza in server.conf from CM into passwords.conf: Example passwords.conf $SPLUNK_HOME/etc/apps/search/local/passwords.conf [credential:general] password = $1$q5jsBxheBw== [credential:clustering] password = $1$q5jsBxheBw== [credential:license] password = $1$q5jsBxheBw== [credential:shclustering] password = $1$q5jsBxheBw== 3) Run http(s)://server:mgmt_port/en-US/debug/refresh to read the new configuration 4) Run http(s)://server:mgmt_port/services/storage/passwords and look for clear_password for each stanza You can also use following format of REST API either through Splunk web or Splunk search: http(s)://<server>:<mgmt_port>/servicesNS/-/-/storage/passwords OR | rest /servicesNS/-/-/storage/passwords | rest /services/storage/passwords ... View more

This error message itself indicates splunk is having issue to decrypt encrypted parameters in system configuration files. This can be from any system/local conf files. What you want to do is to find all the encrypted password (begin with "$" in your conf files. (e.g. pass4Symmkey, sslPassword, etc), re-enter them in clear text format and restart splunk. From Splunk version 7.2.2 and above, you may run below command to decrypt the encrypted password to find the original clear text password: *./splunk show-decrypted --value < pass4SymmKey > * https://docs.splunk.com/Documentation/Splunk/7.2.2/Security/ConfigureS2Sonnewcipher If you're getting "Decryption operation failed: AES-GCM Decryption failed!" from running above decryption attempt on the same splunk instance, that means your encrypted key might be corrupt or incorrect and you should re-enter the correct clear text one. ... View more

LineBreaking is done as part of parsing pipeline: http://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/Howindexingworks#Event_processing_and_the_data_pipeline More details about event processing pipelines: https://wiki.splunk.com/Community:HowIndexingWorks Please note sometime parsing can be done by the Heavy Forwarder: http://docs.splunk.com/Documentation/Splunk/7.2.1/Deploy/Componentsofadistributedenvironment#How_components_support_the_data_pipeline If by any chance you have HF doing the parsing, perhaps you should apply relevant props.conf on the Heavy Forwarder. You may check splunkd.log on relevant Splunk components searching for "truncating" and look for if the particular log is getting truncated due to any setting. ... View more

detailed steps also posted in https://answers.splunk.com/answers/559610/wrong-retention-caused-buckets-to-freeze-in-cluste.html?childToView=700643#answer-700643 ... View more

Detailed steps with correction on some syntax: WARNING - make sure you have a full backup before going forward. Step 1 - place CM in maintenance-mode ./splunk enable maintenance-mode Step 2 - On each peer: stop splunkd go to each index directory (db and colddb): 2.1 grep -Rle '1$' --include bucket_info.csv . | xargs sed -i 's/1$/0/g' 2.2 mv .bucketManifest /tmp/.bucketManifest //backup bucket manifest file for each index repeat step 2 till all indexes done 4. start splunkd Step 3 - restart CM <----------------IMPORTANT: this will allow the latest cluster frozen flag to be updated on CM. Step 4 - confirm all peers "UP" then disable maintenance-mode ./splunk disable maintenance-mode Step 5 - confirm buckets being replicated through GUI ============================================== P.S. Additional script to do the job automatically through all buckets: #! /bin/ksh for CSV in $(grep -Rle '1$' --include bucket_info.csv . ) do sed 's/1$/0/g' $CSV > /tmp/HOLD.out cat /tmp/HOLD.out > $CSV DIR_NAME=$(dirname $(dirname $CSV)) if [ -f ${DIR_NAME}/.bucketManifest ] then Index_Name=$(print $DIR_NAME | awk -F "/" '{print $2}') print " ${DIR_NAME}/.bucketManifest /tmp/${Index_Name}.bucketManifest" mv ${DIR_NAME}/.bucketManifest /tmp/${Index_Name}.bucketManifest fi done ... View more

There's feature enhancement request raised under ADDON-14309 ... View more

looks like it's missing /splunk/var/lib/splunk/kvstore/mongo/local.* file which stores the cluster information. You may want to check if you have /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 file existing from your kvstore backup. Also if by any chance you did run "splunk clean kvstore --cluster" command after restoring the kvstore backup, it would wipe out all the local.* files which contains the cluster information. ... View more

@coltwanger is actually right on this. The maxHotSpanSecs and maxHotIdleSecs won't kick in until the idle time gap between the last most recent event arriving at the hot bucket and the newly arriving event exceeding the value defined. Hence if events are constantly coming into the hot bucket, it won't roll even maxHotSpanSecs/maxHotIdleSecs set to 1hr/1day etc. maxHotSpanSecs = * Upper bound of timespan of hot/warm buckets in seconds. * NOTE: If you set this too small, you can get an explosion of hot/warm buckets in the filesystem. * NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all events to the single hot bucket and maxHotSpanSeconds will not be enforced. * If you set this parameter to less than 3600, it will be automatically reset to 3600. * This is an advanced parameter that should be set with care and understanding of the characteristics of your data. * Highest legal value is 4294967295 * Defaults to 7776000 seconds (90 days). * Note that this limit will be applied per ingestion pipeline. For more information about multiple ingestion pipelines see parallelIngestionPipelines in server.conf.spec file. * With N parallel ingestion pipelines, each ingestion pipeline will write to and manage its own set of hot buckets, without taking into account the state of hot buckets managed by other ingestion pipelines. Each ingestion pipeline will independently apply this setting only to its own set of hot buckets. * NOTE: the bucket timespan snapping behavior is removed from this setting. See the 6.5 spec file for details of this behavior. maxHotIdleSecs = * Provides a ceiling for buckets to stay in hot status without receiving any data. * If a hot bucket receives no data for more than maxHotIdleSecs seconds, Splunk rolls it to warm. * This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll. * A value of 0 turns off the idle check (equivalent to infinite idle time). * Highest legal value is 4294967295 * Defaults to 0. Feedback from engineering: "maxHotSpanSecs would be of lesser priority from the default settings within indexes.conf. We'd like to make sure first that we don't make too many small buckets so we'd prefer to "expand" a hot bucket time range rather than be strict with it. " "There is no rule about rolling to warm when exceeding the maxHotSpanSecs. We roll a bucket to warm because it's been idle too long, we've reached maxHotBuckets (-1 if no quarantine bucket) or a bucket has reached a certain size in disk." "If we cannot find a bucket with the correct time range for an event, we find the next best bucket and expand that range. To check that we still honor the maxHotSpanSecs without the influence of the above behavior" ... View more