I will add that as with all DevOps / SRE efforts it's extremely important to thoughtfully set and enforce standards. In Splunk's case, for both ingestion patterns, sourcetype names, and log structures. It is much easier to manage rapidly changing microservices and teams when you have a strict pattern like standardized log4j template -> HEC, or a shared library function for javascript for posting to HEC, each with sourcetype=stackName:microServiceName. ... View more

Dark purple and dark green look like likely suspects, take note of those sourcetypes. Can you confirm the same spike in indexing load from those sourcetypes on other hosts during the time window when they had issues? This is normally going to happen because of a single forwarder sending a very high amount of bandwidth. It can be addressed in a couple of different ways- 1. Increase the number of threads on the forwarder, since each thread can send to a distinct indexer. 2. If the data is coming from a centralized data source (like syslog etc) spread the load out between hosts. For some perspective, 6MB/s over a 24h period would result in over 500GB/day, which is well outside the recommended 200-250GB/day per indexer. No wonder the poor servers are struggling! ... View more

Okay, so that means that the forwarder(s) in question are successfully sending to multiple indexers, that's great! Now we need to find out what datasource is causing that indexing bandwidth. Go to the monitoring console and open the "Indexing Performance: Instance" dashboard. Scroll down to the "Estimated Indexing Rate Per Sourcetype" panel and see if there are any outliers. EDIT: That feast/famine cycle (Where an instance has an enormous indexing rate with full queues then drops to nearly none) is just the data load balancing to another server and the queues emptying to disk after backing up. Very normal in this circumstance. ... View more

Usually there's 3 things that block up queues; Input volume Performance Configuration In this case, it's pretty clear the indexer in question is getting 2x the instantaneous indexing rate of the other indexers. My question is; is this server usually that much higher than the others? ... View more

I'm assuming that there are events that truly are only one line mixed with these exception dumps? I'd recommend looking into "MUST_BREAK_AFTER" and "MUST_NOT_BREAK_AFTER" in props.conf So you'd end up with something like; MUST_BREAK_AFTER= \] MUST_NOT_BREAK_AFTER=\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\,\d{3}\s.+?LogData\[ The idea here is that Splunk will continue to linebreak each line unless it detects that "LogData[" string, at which point it will stop linebreaking until it encounters the close bracket. Hopefully this will make one event out of many. ... View more

Use the default field date_second. https://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Knowledge/Usedefaultfields#date_second ... View more