I have DBconnect 3.1.3 running on a 7.0.1 instance with 3 DB Inputs, 2 of them work perfectly but the third one puts out the error below and then fails. Two of the connections are almost identical except they go to different DBs, but one works and one doesn't. I have tried restarting and reconfiguring the input.
2018-04-12 18:48:43.627 +0000 [QuartzScheduler_Worker-29] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch
java.io.IOException: HTTP Error 400: Bad Request
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:79)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Found the problem, the HEC fails to write when a single event has 800000+ characters. Had 1 log message that was ~800x the size of a normal log message and that was causing the failure, when that message is skipped (by changing the rising column) everything goes back to working as intended.
I have this issue with my DB Connection on the customer environment. I tried the following:
- Add hardcoded metadata
- Downgrade "db connect" to earlier version
- Install splunk on other server
- The total size of all columns in the table is not bigger than 800000+ characters
Have anyone had this issue and solved it in another way?
Many cases turned out to be caused by lack of meta data. For example, in DB Connect 3 the data fetched from DB are forwarded via HEC configured in local.
It appears to get 404 error when the payload lacks meta data, like source, sourcetype or host.
You can get the payload and confirm it by enabling DEBUG for the inputs or connections.
Then you can find the payload being forwarded via HEC - check the fields for default metadata such as host, source and sourcetype, below is just for example purpose.
If any of default fields are not populated you need to add them through the configuration for metadata.
Hi @andreacorvini ,
I am facing same issue. Can you please send me the link for dbconnect[previous version] as well.
Thanks
I shared the app with you.
Can you send me the app as well? Thanks.
@andreacorvini
Dear Matt,
I have a copy of the splunk-db-connect_311. If you provide me an email address i'll gladly send it to you!
BR,
Tamas
Thanks for the quick reply. My Email is -> zauberpferd12@gmail.com
Thanks again.
Mat
I'm going to install this version and report the result here.
Thank you very much!
Hi @andreacorvini .
Really Worked!
Thank you!
I would suggest to use HEC /collector REST API http://docs.splunk.com/Documentation/SplunkCloud/7.0.3/Data/Sendmetricstoametricsindex to get server verbose response body and learn more about failure reason. For example :
curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
HEC authorization token can be found in Settings->Data Inputs->HTTP Event Collector->db-connect-http-input (or create your own token. Also uncheck 'Enable indexer acknowledgement' if the api returns code=10 in the response)
HTTP can be found in Settings->Data Inputs->Global Settings
Had this exact same symptom but a very different cause in DB Connect 3.1.2
After debugging it was found that the host field on the input was empty (which is allowed as it defaults to the connection hostname), however since the connection was set to use the JDBC URL the hostname was blank there as well.
This appears to be only possible if while creating the connection the edit JDBC URL button is pressed and no hostname is left in the box.
I have case open to ensure the application gets updated to prevent this scenario but it was very difficult to find the issue, the root cause was DB Connect attempting to send a hostname of "null" via HTTP Event Collector which also has minimal logging available
after giving value to host field in the inputs its started writing/indexing records.
Thank you.
Hi.
Tell me where can I download the version DB Connect 3.1.2?
I can not download it on the site (((
Could you send it to my e-mail dyuran@list.ru?
I sent you an email with a link to download it.
Ciao
Andrea
May I have that version as well? Please send it here: marxsabandana@gmail.com
Thank you, this version works for me.
Hello @andreacorvini,
I've the same problem with 3.1.3.
Can you send the link to lukahoff@gmx.de please?
Thanks so much.
LH
ok, shared.
Thanks!
LH