Hey folks,
Long time Splunk fan here. Initially when we started using Splunk, our queries were simple, and so searches ran fast.
So I loved Splunk.
However, over time as useage grew, queries got more complex and our datasets grew large.
Unfortunately, neither Splunk or any other log analysis products deal with this well ( and Splunk probably deals with it best amongst all others for that matter ).
Our "needle in a haystack" class of queries work okay, e.g. user=badman or ip=evil_ipp.
We have a fair number of those, and while they have gotten slower with how massive our data has become, they are still tolerable.
To some extent, we've built our own apps on top of Splunk that maintain important aggregates, though that still doesn't work for ad hoc queries, since the queries are such that they don't hit indexes.
Specifically, we have queries of these types that just run too slow, of the order of many minutes, or 10's of minutes:
Queries which generate lots of results., e.g. error
Queries which have wildcard prefixes, e.g. *Error AND *.php
Queries which use splunk for complex analysis, and do data manipulation ( e.g. using rex and regex replacement )
These are queries which don't benefit from indexing, and effectively "break" Splunk despite having many nodes setup.
It feels like I am better off sshing into one machine and just grepping logs, and in some cases this isn't even possible due to security restrictions. So much as I loved Splunk as an introductory user, I've become very frustrated with search speed as data volumes and how queries have shifted from the simple "searching for a word" to the more complex examples outlined above.
1. Is search speed a major pain point for users of advanced queries and/or large datasets ?
[ Hopefully, if you can chime in and be vocal, the good folks at Splunk will hear our feedback 🙂 ]
Personally, it is incredibly frustrating to watch queries run for so long, to the point I have to go surf reddit while the advanced searches run.
2. What are folks doing to make searches that can't use indexes run fast in these scenarios ?
Modifying applications has helped in some cases, but due to not having control of some libraries, it isn't feasible at other times. Other times, admittedly some of my regexes aren't as optimized as they could be, but when I'm debugging a fire, its impractical for me to find out what is or isn't in my log so I have to play it safe with a regex.
... View more