Hi, we want to implement a strict 120 day time retention policy for some indexes. So this config should be fine. For my understanding it´s mandatory to set als MaxHotspanSecs for 24h (of course the frozen path) MaxHotSpanSecs=86400 (1day) FrozenTimePeriodinSec= 10368000 (120 days) Thank you ... View more

Hi there, A question regarding the retention policy approach in a clustered multi site-cluster two sites with each 3 indexers (replication factor 2+1). We are planning a retention policy over 120 days and I feel the indexer's attitude towards cold to frozen is still somewhat unclear. Is that true that the cluster master handles the backup handling (coldToFrozen) and thus not every indexer pushes the cold buckets too frozen, otherwise we would have a huge storage space requirement. https://answers.splunk.com/answers/241066/how-is-bucket-deletion-due-to-retention-managed-in.html Many thanks! ... View more

Hi there, i expand the coldToFrozenScript.py to a compressing & encryption function (tar and python gpg) . It runs well when i start this manualy with python3 . I cant run this in python 2.7 because i got several errors. So when spunk starts the script, i got errors.. Have you any idea which i can solve this issue? I know, splunk use python 2.7 but iḿ not python experienced very well. Thx ! BucketMover - coldToFrozenScript cmd='"/usr/bin/python3.6" "/home/phil/Nextcloud/Work/coldToFrozen_testing_python3_manual.py" /opt/splunk/var/lib/splunk/test/colddb/db_1552398887_1552398884_47' exited with non-zero status='exited with code 1' 03-12-2019 15:40:37.126 +0100 ERROR BucketMover - coldToFrozenScript cmd='"/usr/bin/python3.6" "/home/phil/Nextcloud/Work/coldToFrozen_testing_python3_manual.py" /opt/splunk/var/lib/splunk/test/colddb/db_1552398895_1552398893_48' exited with non-zero status='exited with code 1' 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript Error in sitecustomize; set PYTHONVERBOSE for traceback: 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript AttributeError: module 'sys' has no attribute 'setdefaultencoding' 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript Traceback (most recent call last): 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript File "/home/phil/Nextcloud/Work/coldToFrozen_testing_python3_manual.py", line 5, in 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript import fs 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript File "/usr/local/lib/python3.6/dist-packages/fs/init.py", line 4, in 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript import("pkg_resources").declare_namespace(name) 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript File "/opt/splunk/lib/python2.7/site-packages/pkg_resources.py", line 33 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript def _bypass_ensure_directory(name, mode=0777): 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript ^ 03-12-2019 15:40:37.170 +0100 ERROR BucketMover - coldToFrozenScript SyntaxError: invalid token 03-12-2019 15:40:37.175 +0100 ERROR BucketMover - coldToFrozenScript Error in sitecustomize; set PYTHONVERBOSE for traceback: ... View more

Hello Community, I'm relatively new to Splunk and now I'm struggling with a bigger challenge for me. The following scenario: Configuration of a retention policy over 120 days. All directories that arrive in Frozen must be compressed and encrypted. Concerning encryption and compression I already have a little shellscript doing this job manually. Of course, this should be done automatically. prof of concept - script: #! / Bin / bash #Compresses the most recent directory and encrypts the transfer.tar.gz with gpg (testing-encryption-key). newest = $ (ls -td / opt / splunk / var / lib / splunk / test / frozendb / * / | head -1) echo $ newest tar -czvf transfer.tar.gz $ newest   gpg -e -r testing-encryption-key -o / tmp / backup / $ (date +% Y% m% d-% H% M% S) .tar.gz.gpg transfer.tar.gz rm transfer.tar.gz sleep 2 ---------------------------- The script works successfully if you do it manually. Now there are two variants to bring data from cold to frozen. Either in the indexes.conf with: coldToFrozenDir or coldToFrozenScript When I use "ColdToFrozenDir", I need a triger around the script (compression and encryption). My idea would be a cronjob every 10 minutes, but here natülrich could forget a directory if there are overlaps. Is there any better idea here? I think the better way would be to work with "coldToFrozenScript" and combine the copy process with my prof of concept script. Do you have a good approach or a path for variant one? The whole of course should run stable 🙂 Thank you in advance ... View more