srtemp is the workspace used for search 'post-process' actions, which are used in Splunk dashboards to provide additional search pipeline processing of searches.
This is useful for dashboards when you want one base search to gather information, and then portions of that information presented in multiple ways, and permits creation of dashboards with complex displays efficiently.
Unfortunately post process actions aren't full searches, and run inside main splunkd, rather than as safe separate search processes, so don't get all the protections built for normal searches. One of the protections they lack is the cleanup logic for stale data in var/run/splunk/dispatch, so if a main splunkd crashes, or power is lost, or similar, then the in-flight data lives forever.
If this location is growing rapidly, while splunk is continuing to work and not crashing, this more likely represents a case where you are post-processing a very large base search in a very popular dashboard (or many dashboards), in which case you may have to hunt down the expensive dashboard and redesign the base search to emit a smaller result set.
Obviously, of course, Splunk should be changed to:
clean up srtemp contents over time
Apply storage quotas to post process actions in addition to normal searches.
Less obviously, Splunk needs to be changed so that post process actions are fully converted into normal searches. This is somewhat delayed by the need to make full searches start up as quickly as post process actions, which is why they are special in the first place.
,Apparently srtemp is the temp dir for post-process actions, which are used in dashboards when rendering the search results with some additional search logic (usually filtering or charting a small number of items, less than 10 thousand).
Since these post processes run inside main splunkd, and since the search machinery will delete the temp files on completion of the search (or really on each chunk of the search but hopfully you have only one chunk for a postprocess), the implication is that the splunkd main process crashed at some point, or possibly multiple points.
If main splunkd crashes, there's not any current machinery to prune old data out of srtemp. So you could wipe old directories in there live, or you could stop splunk and wipe the entire contents.
There should definitely be automatic cleanup code added to handle cases like splunkd crashing, operating system crash, or power loss. I would argue of course that post process actions should be transformed into proper full searches with all the safety of normal searches applying.
... View more
