Search head out of disk space because ${SPLUNK_HOME}/var/lib/splunk has HUGE db directories?

[woodcock]

I have 2 search heads that are very similar but one has some extra apps installed (such as SoS). The one with more apps is continuously out of disk space and I just found out why. On the search head that is fine, /opt/splunk/var/lib/splunk has 531M used but on the loaded one, it has 35G!!! What is taking up all the space? Many directory pairs like this and .dat. Inside each directory are 3 directories: "colddb", "db", and "thaweddb". The "db" directories are where all the space is consumed. What is creating these and how can I rein it in?

[woodcock]

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

[woodcock]

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

[Ayn · Accepted solution]

These are Splunk's indexes. The directory names in var/lib/splunk usually correspond to the index names, so if you want to see what all that data is, just search the corresponding index name in Splunk on your search head.

[nawazns5038]

what does the datamode_summary contain.
how can we move data from one path to another in an indexer cluster.

[woodcock]

I do not understand what this is telling me:
splunk_server VALUE_audit VALUE_internal summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes
splunk-search04 20752 / 500000MB (4%) 5947 / 500000MB (1%) 1316 / 500000MB (0%) 1326 / 500000MB (0%) 129 / 500000MB (0%) 19 / 500000MB (0%) 7551 / 500000MB (2%) 377 / 500000MB (0%)

[sowings]

And you might consider the Fire Brigade app, appropriate to your Splunk version. In particular the "Indexer Host Overview" page could help explain what's going on with that search head.

[sowings]

The above search would show summary index usage local to the search head. If you're using report acceleration, you might try | rest /services/admin/summarization splunk_server=local, and pay attention to summary.size. Some apps (like bluecoat or Palo Alto) may call "tscollect" directly to create tsidx name spaces. These are a bit harder to track down (as in, I don't yet have a search for identifying that space). There may also be summary space in use by accelerated data models, but that space would be on the indexers and not on the search head.

[woodcock]

We are not running any summary indices but we do have some apps that may have setup some, which was my theory (and why I mentioned SoS app). Is there a way to map these files to the app that created them?

[dwaddle]

Given it's a search head, my bet is on summary indexes and/or tsidx files for apps like bluecoat or palo alto..

[sowings]

To add to Ayn's comment, you can run this search on your search head:


| rest /services/data/indexes splunk_server=local
| search totalEventCount!=0
| eval cell=tostring(currentDBSizeMB) + " / " + tostring(maxTotalDataSizeMB) + "MB (" + tostring(round(currentDBSizeMB * 100 / maxTotalDataSizeMB)) + "%)"
| chart first(cell) over splunk_server by title