We need to move our archives to different storage and I'm looking for a way to blast this out to our 48 indexers all at once rather than needing to walk through it indexer by indexer. I want to make sure we don't lose any in-flight writes while I swap the NFS mounts for the archive directory on each indexer and I can't see a way to do that outside of stopping Splunk on the node. It doesn't look like maintenance mode will do this. Best I have is for each node: 1. Drop cluster to maintenance mode 2. splunk stop on node 3. mount old archive to temp location 4. mount new archive to permanent location 5. splunk start Then at the end I can kick off a script to move all the old data off to the new location. Is there a magically way to tell Splunk to just stop archiving for a few minutes so I can make the mount swap? ... View more

We're struggling to do OS patching of our indexer cluster in a reasonable timeframe. It currently takes about 24 hours with the vast majority of that time just waiting on bucket fixups tasks to complete between reboots. Wondering how others are doing it without impacting any searching or filling up index process queues. Our current process: Blast out an apt update && apt upgrade -y && apt autoremove -y to all indexes. Takes about 10 - 15 minutes to complete Blast out a puppet no-noop to all indexers - takes about 5 minutes to complete The for each indexer: splunk offline - takes 5 - 10 minutes reboot - takes < 30 seconds Wait for bucket fix ups to complete - around 30 minutes We've had issues using the rolling restart - it sometimes gets stuck in the middle and you have to bounce the cluster master and it doesn't resume where it left off. It also by default defers saved searches, which effectively disables alerting in our environment for a few hours (we are enabling running saved searches during rolling restarts to address this). Does this just work out of the box for others or are there secret "gold" settings that you've had to tweak? Some information about our environment: 2 sites 24 indexers per site Splunk 7.3.3 buckets only replicate between sites, no intra-site replication factor. Wondering if this is contributing to our problems... we've been looking into increasing storage to account for this. 5ms between sites, multi-10gbit links ... View more

I'm running this query to get average event counts per day by index. When I run this, each site does an aggregate 2.5GB/s of IO each for 15 - 30 minutes to satisfy the query. Our understanding is that this is just metadata, so this is fairly surprising. Does anyone know why this is and have a better way to write this query? | tstats count where (index=* OR index=_*) AND earliest=-w@w+1d latest=-w@w+6d by index | eval eventCountPerDay=count/5 | table index, eventCountPerDay ... View more

I'm trying to find a way to programmatically get the average size of data flowing into each index on a daily basis so we can set indexes.conf max data retention based on how many days we want to hold for each index. There seem to be a variety of ways to get the size of indexes per day, none of which seem to match. There are a lot of posts on this over the last decade, but reading all of them has just left me more confused. Ideally I can figure out the average amount of MB required on disk (across all indexers in a site) per index. Then I can define how many days I want to retain on a per index basis and set homePath.maxDataSizeMB to avg_per_day_size_mb * number_of_days_to_retain. Ignore cold/frozen for the moment and assume that all hot/warm is going to the same path (e.g. no smartstore). How can I do this? The basic ways I've seen: 1. Use dbinspect. I can filter by state = "hot" or "warm" as well as filter by bucketId to throw away replicated copies of the bucket (either due to replication factor or site replication factor). Given that dbinspect shows you a snapshot in time of buckets, I don't think it accurately shows me the amount of new data coming into the index per day. 2. Use license usage log. When I average this metric out to a per day per index value, it feels like it is fairly close to the amount of data I'm storing across splunk (looking at du values on linux hosts). However, it doesn't count all indexes, notable events, internal logs, etc, which I need to account for. 3. Estimate raw data size for messages (e.g. len(_raw)) over a short period of time and then use those values with tstats over a longer period of time to estimate size. This feels like it will give the closest value, however it feels like a really dirty way to get the data. 4. Use metrics log to look at throughput values. Reading documentation, this just seems to be plain wrong as it stores only samples and doesn't show size of the message after it comes out of the various parsing queues. What are folks doing to reliably size their indexes.conf based on # of days they want to retain per index? ... View more