During some searches the number of events that are supposed to be returned does not match the number of events that are actually displayed. In one instance the Events counter showed 13 events, but the timeline showed "No events found" and none were displayed. In other instances fewer events are displayed than the counter states that there should be. In the search log there are errors for Timeliner like: "08-30-2017 12:58:47.035 ERROR Timeliner - Ignored 2 events because they were after the commit time (0).". If you add up the number of ignored events you get a number equaling the number of events that are missing from the timeline. There are also log entries like: "08-30-2017 12:58:38.909 WARN SearchResultCollator - Collector X produced chunk with startTime 1503348584.000000 when our cursor time was already 0.000000, time ordering has failed!" that may or may not be related. Running the search again usually fixes the issue, but I'd like to resolve the underlying issue or be able to explain the cause to users that report the issue. Has anyone seen this? Can you provide details as to why events are ignored? ... View more

I am looking for the best way to provide redundancy to my Splunk solution and plan for the future at the same time. Current ingestion is roughly 20 GB/day and am planning on that doubling within the next few years. The current solution is a single server acting as a search head and indexer. My concern is that a single system does not provide redundancy. I would like to go ahead and build redundancy into my setup, but I'd like an informed opinion on which option to take. The two options that I'm currently aware of is a Splunk Cluster setup and a duplicate Indexer setup. The Splunk Cluster has the benefit of providing redundancy while not requiring me to purchase additional licensing. On the other hand I will have to procure additional servers to build the cluster. Taking a duplicate Indexer setup I can have my forwarders send a copy of everything to two(or more) Indexers. This will require doubling my Splunk license and doubling the network traffic required to deliver the data to the indexers. I know that the recommended number of indexers for 20 GB/day is one so a cluster may be a little bit excessive. I appreciate any advice you can provide. ... View more