I found a solution to my specific breed of the problem. After toying around with the Splunk_SA_CIM and SplunkAppForWebAnalytics, which both define the Web datamodel, my log quickly filled with these messages:
08-14-2016 09:40:58.919 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:40:58.919 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv
08-14-2016 09:41:28.922 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:41:28.923 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv
08-14-2016 09:41:58.918 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:41:58.918 +0200 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv
The reason for this error message in my special case was, by often restarting the splunk service, it was unable to finish correctly writing out the dispatch data and just left corrupt (empty) files there instead.
ll scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/
total 36K
prw------- 1 root root 0 Aug 14 05:40 alive.token|
-rw------- 1 root root 0 Aug 14 05:40 args.txt
-rw------- 1 root root 0 Aug 14 05:53 audited
-rw------- 1 root root 28K Aug 14 05:40 info.csv
-rw------- 1 root root 0 Aug 14 05:40 metadata.csv
-rw------- 1 root root 0 Aug 14 05:40 peers.csv
-rw------- 1 root root 0 Aug 14 05:40 pipeline_sets
-rw------- 1 root root 0 Aug 14 05:40 request.csv
-rw------- 1 root root 0 Aug 14 05:40 search.log
-rw------- 1 root root 6.8K Aug 14 05:40 status.csv
After removing the corrupt directories, the error messages went away with them.
... View more