Splunk down | Unable to open database file

[junshenchoo]

Hi all,

Need help over here, Splunk cant start after a crash yesterday. Now, when I try to run "Splunk start", it will not launch, but the splunkd service will run. Also causing the server to be very slow. Kindly refer to below logs.
Last thing that I know was that my scheduled report did not complete, and when I tried to export from dashboard, the problem occurred.

Many thanks in advance!

07-02-2019 15:43:50.807 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:43:50.822 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:08.092 +0800 WARN  HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:44:12.599 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:44:12.605 +0800 WARN  PeriodicReapingTimeout - Spent 21812ms reaping search artifacts in C:\Program Files\Splunk\var\run\splunk\dispatch
07-02-2019 15:44:20.685 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:44:25.382 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:25.387 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:44:53.644 +0800 WARN  PeriodicReapingTimeout - Spent 14250ms reaping search processes 
07-02-2019 15:44:54.536 +0800 WARN  HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:44:55.564 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:44:55.580 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:44:56.544 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:45:01.529 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:45:03.537 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_scheduler__nobody__sos__RMD5fe2b0603bfc33e11_at_1562053181_6_1562053502.10\metadata.csv
07-02-2019 15:45:05.113 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:45:20.788 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:45:25.188 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:45:25.202 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:45:52.227 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:45:52.286 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:45:54.281 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:46:05.040 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:46:36.338 +0800 WARN  PeriodicReapingTimeout - Spent 21281ms Reaping srtemp of old files
07-02-2019 15:46:36.341 +0800 INFO  PipelineComponent - MetricsManager:probeandreport() took longer than seems reasonable (10515 milliseconds) in callbackRunnerThread. Might indicate hardware or splunk limitations.
07-02-2019 15:46:36.343 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:46:36.370 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:46:36.380 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:46:36.789 +0800 WARN  HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:46:50.677 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:46:50.694 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:46:50.700 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:11.208 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__sos__RMD59d4672721e98f163_at_1561973400_76273\metadata.csv
07-02-2019 15:47:16.305 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:16.305 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:20.450 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:20.450 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:20.658 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:47:20.675 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:47:20.685 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:24.885 +0800 INFO  PipelineComponent - Performing early shutdown tasks
07-02-2019 15:47:24.885 +0800 INFO  IndexProcessor - handleSignal : Disabling streaming searches.
07-02-2019 15:47:24.885 +0800 INFO  IndexProcessor - request state change from=RUN to=SHUTDOWN_SIGNALED
07-02-2019 15:47:24.895 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:24.895 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:29.038 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:29.038 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:33.181 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:33.181 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:37.320 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:37.320 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:40.420 +0800 WARN  HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:47:41.464 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:41.464 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:45.704 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:45.704 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:49.843 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:51.371 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:54.361 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:47:54.386 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__nobody__SplunkAppForFortinet__RMD5293e1d270510edf8_at_1561973400_76274\metadata.csv
07-02-2019 15:47:54.393 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_tmp_1561973408.1\metadata.csv
07-02-2019 15:47:55.517 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:47:55.517 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:47:56.896 +0800 WARN  HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs/1.0/ta_mscs_azure_audit_inputs: Winsock error 10053
07-02-2019 15:47:59.661 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:10.547 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:14.696 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:14.696 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:18.835 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:18.835 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing
07-02-2019 15:48:20.615 +0800 WARN  DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv
07-02-2019 15:48:22.981 +0800 ERROR SQLitePersistentStorageImpl - Error executing: select primarykey, value from keyvaluepairs_t where secondary1 = ?1 Msg=unable to open database file file=C:\Program Files\Splunk\var\lib\splunk\persistentstorage\fschangemanager_state
07-02-2019 15:48:22.981 +0800 ERROR FSChangeMonitor - Exception thrown in update(2) - continuing

[Richfez]

While there's a chance that you might get some sort of answer here, this seems like it might be FAR better taken up with Splunk Support. If you have a support contract - use it.

If you do not, this sounds like the level of problem that we volunteers in the community won't be able to resolve for you.

But maybe you can resolve this yourself. Look at the very first line of the error you posted, and start applying some troubleshooting 101 steps.

It says could not read metadata file C:\Program Files\Splunk\var\run\splunk\dispatch\rt_scheduler__admin_VEEtQ2JfRGVmZW5zZQ__RMD5da33e6a6a5c2d83a_at_1561973400_76272\metadata.csv

So, can YOU read that file? Does it exist? What contents does it have? Does it appear formatted well, or maybe it stops in the middle of an item? Are permissions appropriate?

That's just some starting points.

To reiterate:

If you are using a version of Splunk with support, I'd be calling them up for support.

If instead you are using Splunk Free, then I assume it's not actually that important of stuff. Either restore from your last known good backup, or if you don't make backups then it's doubly unimportant ("community supported software" and also not backed up), so just uninstall then reinstall Splunk. That should resolve this.

Note that you SHOULD be able to copy off your configs ($SPLUNKHOME/etc) and use those to rebuild stuff afterwards. And maybe you want to look into copying index files out of the way too, it's possible you can drop those back in without issue after wiping out Splunk and reinstalling. What I'm trying to say here is take your time doing this, do a little research, and maybe you can recover nearly everything you had before.

And to THAT end, if you try restoring indexes and things and run into trouble, feel free to make a new question!

[junshenchoo]

Hi rich7177,

Appreciate your response, I have done some troubleshooting on the issue. I have tried to open up this file, I can do so, however it contains only a few empty spaces (1KB file size). Security permission looks fine, it was working fine for the past 3-4 months. Unfortunately, we do not have Splunk Support, as we are using a NFR version. We are resellers and we do collect logs on behalf of our customers to show them Splunk's functionalities.

Thanks!

[Richfez]

YW, and sorry I can't help. Still, this question's out here and maybe someone else can chime in.

Also, you do fit into one of the gaps I think Splunk has, in that partners and resellers have no good mechanism to report bugs or get support. I can see that (in their mind) it's opening a can of worms, but maybe if the worms are really there, it's the right thing to do anyway.

Anyway!

Since there's not much to lose, you could possibly delete all those corrupted or broken directories/files and see if it works afterwards?

I do not think a straight "reinstall" of splunk over top of the existing install would work. But I still think move your apps and indexes out, get a fresh copy of Splunk put down and ... maybe you can just put your apps and data right back and it'll be fine.

Do let us know if you do something like that and it works!

Happy (ish) Splunking!
-Rich