This was a combination of two bugs that were fixed in later versions of splunk (7.0.8+, 7.1.6+, 7.2.4+)
For a workaround, its safe to
delete older generation files, keeping the last 10 or so per site
don't delete the gen0 file
for example, if i have:
search_sitedefault_gen1000.csv.gz as the latest file, i can delete search_sitedefault_gen(1-990).csv.gz safely
but remember this is per site, so if i have the latest:
search_site0_gen1000.csv.gz (delete gen1-990 for site0, dont delete gen0)
search_site1_gen3500.csv.gz (delete gen1-3490 for site1, dont delete gen0)
... View more