Thanks for clarifying. I have added the new Ubuntu servers to the cluster and will initiate the data rebalance today.  Doing some reading on removing a peer and it looks like it's as simple as running the offline command on the peer, correct? Assuming the cluster is healthy, just run this wait for it to finish, and the server can be shut down.    splunk offline --enforce-counts   ... View more

@richgalloway wrote: I think you have right idea on all counts.  Migrating the CM is similar to migrating a SH.  Do migrate the CM before the indexers. Working on this project. I have the new CM stood up on Ubuntu 22 and it has replaced the Centos 7 CM which is now offline. The Indexers are still on Centos 7. I see in the docs that the CM and indexers need to be the same OS. Is this true? The cluster seems to be working fine so far and I'm working on the new Ubuntu indexers that will be added to the cluster. Still safe to proceed or will I run into issues adding the Ubuntu indexers to the cluster? Found under "Operating system requirements" "All indexer cluster nodes (manager node, peer nodes, and search heads) must run on the same operating system and version." System requirements and other deployment considerations for indexer clusters - Splunk Documentation ... View more

I've been working on an issue where I need to be able to filter out a registry entry as well, but this solution doesn't get me there.   I need to filter out the languagelist entry, just can't get the regex to work. Anyone else have success filtering reg entries? I need to filter out this entry. HKU\.default\software\classes\local settings\muicache\2c4\52c64b7e\languagelist ... View more

This is great! Good steps to follow, thank you! ... View more

Thanks for responding. I'll proceed and see how it goes! ... View more

Found another app that needs kvstore, but this one is a vendor TA. kvstore was not referenced in any documentation and I only found out after I stopped getting data. Fixed now, just keep an eye out for missing data after disabling kvstore on a HF. ... View more

Maybe not a common TA or app, but Splunk App for Stream uses kvstore. Found this out recently doing some troubleshooting. So on stream servers make sure in server.conf to set [kvstore] disabled = false ... View more

I posted my script in this thread and it's the sample script, just edited for my environment. It's been working fine for a long time until this issue arose. ... View more

I'm passing a bucket path and getting this error now. Replaced some path values to hide internal names and bucket name. bash-4.2$ /opt/splunk/bin/splunk cmd "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py" "/opt/splunk/var/lib/splunk/indexname/db/db_string" Traceback (most recent call last):    File "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py", line 51, in <module>       handleOldBucket(bucket, files) NameError: name 'handleOldBucket' is not defined ... View more

It looks like this line     if len(sys.argv) != 2: is where the script runs into trouble. I've also tried    if len(sys.argv) < 2: but same results. Not sure what values I need to pass or where to pass them to get the script to proceed. Maybe something in indexes.conf on the coldToFrozenScript line ?    coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py" ... View more

Thanks for replying. I had been running splunk 8.2 with it working fine and then upgraded to 9.0.3 on CentOS 7. I'm pretty sure 9.0.3 had been working fine as well, but I may be mistaken on the timing. I did have to do a server name change on the indexers, but the frozen server stayed the same. My indexers write to a frozen server over NFS and nfs exports uses ip address, not name. My coldtofrozen.py script hasn't changed, but when I run it this is what I get. [bin]# ./splunk cmd "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py" usage: python coldToFrozenExample.py <bucket_dir_to_archive> [bin]# Full script, edited to remove comments and internal server names. #!/opt/splunk/bin python import sys, os, gzip, shutil, subprocess, random, socket hname = socket.gethostname() ARCHIVE_DIR = os.path.join('/mnt/nfs/frozensrv', hname) def handleNewBucket(base, files):    print('Archiving bucket: ' + base)    for f in files:       full = os.path.join(base, f)       if os.path.isfile(full):          os.remove(full) if __name__ == "__main__":    if len(sys.argv) != 2:       sys.exit('usage: python coldToFrozenExample.py <bucket_dir_to_archive>')    if not os.path.isdir(ARCHIVE_DIR):          try:             os.mkdir(ARCHIVE_DIR)          except OSError:    # Ignore already exists errors, another concurrent invokation may have already created this dir    sys.stderr.write("mkdir warning: Directory '" + ARCHIVE_DIR + "' already exists\n")    bucket = sys.argv[1]    if not os.path.isdir(bucket):       sys.exit('Given bucket is not a valid directory: ' + bucket)    rawdatadir = os.path.join(bucket, 'rawdata')    if not os.path.isdir(rawdatadir):          sys.exit('No rawdata directory, given bucket is likely invalid: ' + bucket)    files = os.listdir(bucket)    journal = os.path.join(rawdatadir, 'journal.gz')    if os.path.isfile(journal):          handleNewBucket(bucket, files)    else:          handleOldBucket(bucket, files)    if bucket.endswith('/'):          bucket = bucket[:-1]    indexname = os.path.basename(os.path.dirname(os.path.dirname(bucket)))    destdir = os.path.join(ARCHIVE_DIR, indexname, os.path.basename(bucket))    while os.path.isdir(destdir):          print('Warning: This bucket already exists in the archive directory')          print('Adding a random extension to this directory...')          destdir += '.' + str(random.randrange(10))    shutil.copytree(bucket, destdir) ... View more

I'll give this a try, thank you! ... View more

Downgraded to 1.4.15 and got it working again. Also tried downgrading to 1.4.16, but it did not work. ... View more

Yes, also added the intermediate cert and tried adding the ca cert, but the ca cert was already in the store. Rebooted the search head and tried creating a new data input, but I get the same error. ... View more

Having same issue. Splunk 9.0.3 OpenJDK 1.8.0 Akamai app 1.4.17 Tried adding the host url cert to cacerts, restarted splunk, but no change, still fails with error Encountered the following error while trying to update: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Not sure what to try next. Might try downgrading to 1.4.15 as time allows. ... View more

I'm getting these same errors when running the upgrade readiness app. Do these need to be fixed prior to upgrading to 9.0? I'm on 8.2 today. ... View more