At 500GB per day, that works out to a little under 25k events per second.
So why does it take 7 (quite expensive) servers with all of these resources to do what could be done on a $2k PC?
I'll give you a hint: You're doing it wrong 😉
Here's what I used (an older pc):
http://www.newegg.com/Product/Product.aspx?Item=N82E16856101117
It has 32GB ram and Ubuntu 16 with a Samsung EVO NVME being used as a cache drive using lvmcache.
I'm able to ingest at around 40k eps and searches take a few seconds.
Side note:
If you do insist on spending a whole bunch of money for something you don't need, be sure your disks are set up correctly both in raid (10 is best, but 6 is fine). Be sure you have the correct strip size in both the raid bios/config and when you partition the disks in Linux. As well as the mount options you use in /etc/fstab - and be VERY sure you partition on the right sector boundaries (4k by default) - not doing so will drastically reduce performance.
Be sure your OS disks are not the same as your data disks. You will thank me later 😉
I spent a few hours and went from around 300MB/s Read, 76MB/s Write to 1.3GB/s read and 800MB/s write.
Happy to share my notes if you want to PM me.
... View more