For me, it is due to the cliff function that BDB goes off of when the data set goes out of RAM -- I've been placing a higher value on consistent latency than raw throughput. The BDB issues hark back to the original Dynamo paper where the Amazon folks had a bunch of trouble tuning BDB-JE for consistent performance. When I measured the performance of the Voldemort BDB storage engine a year or 2 ago, I found the same issue here.
... View more