My indexes aren't being populated - I can;t see why

[phil__tanner]

We have an on-prem Splunk instance (was 7.0.3, have now upgraded to 8.0.4 but are still seeing the same behaviour).

When I try to index files, or DBX connections, the file indexer correctly reports the number of matching files in the directory, but the Index shows 0 events.

I have also tried indexing databases using DBConnect, which again, shows results during initial testing and configuration, but after setup, the index remains with 0 events in it. The $SPLUNK_HOME/var/log/splunk/splunk_app_db_connect_server.log file shows this:

2020-04-24 04:15:27.525 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Job '<JOBNAME>' starting
2020-04-24 04:15:27.525 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Batch size: 1,000
2020-04-24 04:15:27.525 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Error threshold: N/A
2020-04-24 04:15:27.525 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Jmx monitoring: false
2020-04-24 04:15:27.626 +0000  [QuartzScheduler_Worker-25] INFO  c.s.d.s.dbinput.recordreader.DbInputRecordReader - action=db_input_record_reader_is_opened task=<JOBNAME> query=SELECT * FROM "<DATABASE>"."dbo"."<TABLE>"
2020-04-24 04:15:27.726 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Job '<JOBNAME>' started
2020-04-24 04:15:27.776 +0000  [QuartzScheduler_Worker-25] INFO  c.s.dbx.server.dbinput.recordwriter.HecEventWriter - action=write_records batch_size=50
2020-04-24 04:15:27.776 +0000  [QuartzScheduler_Worker-25] INFO  c.s.d.s.dbinput.recordwriter.HttpEventCollector - action=writing_events_via_http_event_collector
2020-04-24 04:15:27.776 +0000  [QuartzScheduler_Worker-25] INFO  c.s.d.s.dbinput.recordwriter.HttpEventCollector - action=writing_events_via_http_event_collector record_count=50
2020-04-24 04:15:27.778 +0000  [QuartzScheduler_Worker-25] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
      ******<snip>*******
2020-04-24 04:15:27.778 +0000  [QuartzScheduler_Worker-25] ERROR org.easybatch.core.job.BatchJob - Unable to write records
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
      ******<snip>*******
2020-04-24 04:15:27.778 +0000  [QuartzScheduler_Worker-25] INFO  org.easybatch.core.job.BatchJob - Job '<JOBNAME>' finished with status: FAILED

I've turned off SSL checkboxes, so assume it's a mismatch on port expectations based on some other googling, and can confirm that a SPL of:

| dbxquery query="SELECT TOP 10 * FROM \"<DATABASE>\".\"dbo\".\"<TABLE>\"" connection="<CONNECTION>"

returns results, just like the DB Connect configuration does.

I'm really struggling to discover any reason why my indexes aren't being populated, and would really appreciate any help.

P

[phil__tanner · Accepted solution]

So - there were two associated problems here.

The first that the MSSQL database drivers that I downloaded from the linked pages in the DB Connect were 0.0.02 versions above the "tested with" table in DBConnect app. Downgrading the drivers to the "tested with" made the data populate into my temp index without any issues.

But populating into new indexes still didn't work, and that was because the index needed to be added to the db_connect option under the HTTP event collectors section.

[0YAoNnmRmKDg]

Hi Phil,

Is there anything in the internal index working?

index=_internal | timechart count by host have any results for example?

also to check HEC is working you can follow some steps here to test with curl

https://docs.splunk.com/Documentation/Splunk/8.0.3/Data/HTTPEventCollectortokenmanagement

specifically

https://docs.splunk.com/Documentation/Splunk/8.0.3/Data/HTTPEventCollectortokenmanagement#Send_an_ev...

curl -k "https://mysplunkserver.example.com:8088/services/collector" \
    -H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \
    -d '{"event": "Hello, world!", "sourcetype": "manual"}'

[phil__tanner]

Hi.

Yup, internal index being populated (146,398 events in last 60 mins).

And yes, other events into other indexes are all being populated still just fine - I have anything (including HTTP events) still populating into their indexes just fine. So long as those indexes were created before November.

If I try creating any new data inputs, with a new index, the initial searches (like DBX, or file monitor counts) work fine, but the indexes sit there with 0 events, 1Mb size, and return no results.

So it's something to do with new indexes I think - everything else is ticking along fine....