A FusionIO ioDrive card with splunk?

[gfriedmann]

Has anyone used an ioDrive for their splunk? I'm really curious if anyone can speak to the crazy high IOPS & throughput significantly helping the splunk search. I've seen a benchmark where the crazy fastness only really was apparent when there were multiple threads using the storage heavily. Maybe the equivalent of 4-6 active searches.

My platform might be considered "low log volume & high search concurrency".

I know the usual advice is to scale out to additional indexers/search servers instead of beefing up individual servers. I'm generally cool with that (i'm planning on adding multiple servers). But this is a specific storage IO question for the crowd.

Do you have any numbers or story to tell?

[silvermail]

Prices has came down somewhat, and I am really tempted to get the OCZ RevoDrive x2 card - which is essentially 4 * 60 GB SF-1200 drives running under RAID 0. Would love to do some benchmarking once I have the card.

[araitz · Accepted solution]

Yes, we have had customers use FusionIO drives with Splunk.

Since indexing is largely sequential writing, you won't see a tremendous benefit. The same can be said generically for dense reporting searches.

However, for needle-in-the-haystack types of searches (this token appears only a handful of times in millions of results), and to a lesser extent semi-sparse searches (this token appears thousands of times in millions of results), you should experience excellent performance due to the random seek performance of such drives.

In general, because they are so small and so expensive, there is not widespread adoption of FusionIO or similar technology with Splunk, but if money is no object then they help as described above.

Update: Splunk 4.3 has a great use case for SSD drives: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Bloomfilters