Splunk DB Connect 1: Why am I getting "ERROR HttpListener...Connection closed by peer" when trying to reindex data?

[despachoSTD]

I have my Splunk indexing data from a database using DB Connect 1. As data in the DB is fairly static, I have two DBInputs configured as dump, and to be executed every two days. This way, when I know there is a change in the data, or I want to improve the indexing process I empty the index, enable the dbinputs, wait until the indexing has finished, and disable them, and they remain disabled until the next indexing is needed. This has worked like charm so far.

However, this morning I needed to reindex my data, I emptied the index and enabled the dbinputs as usual. However, I got no data in. In the dbx.log everything seems to be working well:

2016-04-07 14:43:24.049 dbx2080:INFO:DumpDatabaseMonitor - Database monitor=[dbmon-dump://DBConnect1/DBInput1] finished with status=true resultCount=38373 in duration=553 ms
2016-04-07 14:43:24.049 dbx2080:INFO:ExecutionContext - Execution finished in duration=553 ms

However, in the splunkd.log I get the following error:

04-07-2016 13:28:03.854 +0200 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /en-US/manager/dbx/dbx/dbmon?app_only=True&msgid=6970000.493133157297: Connection closed by peer
04-07-2016 13:28:03.854 +0200 ERROR HttpListener - Handler for /en-US/manager/dbx/dbx/dbmon?app_only=True&msgid=6970000.493133157297 sent a 0 byte response after earlier claiming a Content-Length of 14521!

None of the DBInputs work now, although I can execute DBQuery and DBInfo normally.

I have updated to the latest versions of Splunk Enterprise and DB Connect 1, just to see if it helps. I have deleted the indexes, and created them again, and the same with the DBInputs, I have double checked the permissions on the Splunk files and folders (Windows environment), I have disabled the Windows Firewall, and everything I could imagine that could help, but nothing has changed.

What is the error on the HttpListener and how can it be overcome? What am I doing wrong?

Thanks in advance!

[despachoSTD · Accepted solution]

I answer my own question. After crashing my head against the keyboard for almost one month, I have been able to solve the problem.

It turns out that the dbx app stores the results of the queries in the $SPLUNK_HOME$/var/spool/dbmon dir, in files with ".dbmonevt" extension. And, after that, Splunk indexes them with one of the default data inputs (not surprisingly, the one that looks for $SPLUNK_HOME$/var/spool/dbmon/*.dbmonevt") and "acquires" the query results. In my case, I had stopped all the data inputs, but the ones for my apps, and that was causing Splunk not reading the results of the queries. I suppose that any kind of problems to write the results files in that dir would cause a similar behaviour. Turning on the data input again inmediatly solved the problem.

Besides, the data input is configured to delete the files once indexed. As it was not indexing them, I had around 20 Gb of files in the mentioned directory which could be deleted.

Hope it helps!!!

[fuld]

Thanks for posting the answer!