Ingestion stuck due to overload

[meshorer]

Hi, 

I have an app that ingest offenses from a SIEM system (qradar).  One time there were a few thousands offenses to ingest at the same time, and it caused to an error in the app ingestion. But none of the offenses were ingested for a few hours. Is there a way to alert when there is an ingestion error for an app, and maybe a way to fix it?

[meshorer]

thank you, is there a remediation for that issue? I mean ok I monitored and an alert was fired, now what?

[phanTom]

@meshorer 

You will need to monitor the ingestd.log on the platform to check for any ingestion failures. It's best to get this into Splunk and it depends on the version you have as to how it gets there. 

In the latest version there is a UF on the box that you can configure in "Forwarder Settings" and this can send all of the SOAR Logs into the splunk_app_soar index:

index=splunk_app_soar source=*ingestd.log

 You should be able to make some detections there. 

In the older versions most data is sent via HEC but DOESN'T include these logs so you will need to put a UF on the server yourself and then load in the splunk_app_for_soar to it and that should grab the Daemon logs and send to splunk in the same way as above.

-- Did this fix the issue? If so please mark as a solution. Happy SOARing! --

[marnall]

If you know which error to look for, or can make a good guess that it includes the word "ingestion", then you could search in the internal logs:

index=_internal log_level=error ingestion

You could also make a "maintenance alert" which looks for a drop in logs for an index, source, sourcetype, or some other field. If you expect logs at a certain time but there are zero, then it could be because of a log ingestion error.