Hi all,
Need help over here, Splunk cant start after a crash yesterday. Now, when I try to run "Splunk start", it will not launch, but the splunkd service will run. Also causing the server to be very slow. Kindly refer to below logs. Last thing that I know was that my scheduled report did not complete, and when I tried to export from dashboard, the problem occurred.
Many thanks in advance!
07-02-2019 15:43:50.807 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:43:50.822 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:08.092 +0800 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:44:12.599 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:44:12.605 +0800 WARN PeriodicReapingTimeout - Spent 21812ms reaping search artifacts in C:\Program Files\Splunk\var\run\splunk\dispatch
07-02-2019 15:44:20.685 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:44:25.382 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:25.387 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:44:53.644 +0800 WARN PeriodicReapingTimeout - Spent 14250ms reaping search processes
07-02-2019 15:44:54.536 +0800 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:44:55.564 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:44:55.580 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:56.544 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:45:01.529 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:45:03.537 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_scheduler__nobody__sos__RMD5fe2b0603bfc33e11_at_1562053181_6_1562053502.10\metadata.csv
07-02-2019 15:45:05.113 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:45:20.788 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:45:25.188 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:45:25.202 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:45:52.227 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:45:52.286 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:45:54.281 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:46:05.040 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:46:36.338 +0800 WARN PeriodicReapingTimeout - Spent 21281ms Reaping srtemp of old files
07-02-2019 15:46:36.341 +0800 INFO PipelineComponent - MetricsManager:probeandreport() took longer than seems reasonable (10515 milliseconds) in callbackRunnerThread. Might indicate hardware or splunk limitations.
07-02-2019 15:46:36.343 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:46:36.370 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:46:36.380 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:46:36.789 +0800 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:46:50.677 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:46:50.694 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:46:50.700 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:11.208 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:47:16.305 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:16.305 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:20.450 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:20.450 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:20.658 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:47:20.675 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:47:20.685 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:24.885 +0800 INFO PipelineComponent - Performing early shutdown tasks
07-02-2019 15:47:24.885 +0800 INFO IndexProcessor - handleSignal : Disabling streaming searches.
07-02-2019 15:47:24.885 +0800 INFO IndexProcessor - request state change from=RUN to=SHUTDOWN_SIGNALED
07-02-2019 15:47:24.895 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:24.895 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:29.038 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:29.038 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:33.181 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:33.181 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:37.320 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:37.320 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:40.420 +0800 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:47:41.464 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:41.464 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:45.704 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:45.704 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:49.843 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:51.371 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:54.361 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:47:54.386 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:47:54.393 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:55.517 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:55.517 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:56.896 +0800 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:47:59.661 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:10.547 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:14.696 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:14.696 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:18.835 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:18.835 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:20.615 +0800 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:48:22.981 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:22.981 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
... View more