Today one of my users did a search and it took 99% of the 32GB of RAM on the Splunk Production host and made it unusable for others, this must be a bug, no users could log into splunk and all alerts ceased to go out. How can one poor search kills the box. Any help would be appreciated.
This user only has 2 indexes as his default.
The user ran this search.....over a 1 day period
source="/proj/logs/prd/*systemout.log" UserName can't be null
He should have run this search........
index=claims source="/proj/logs/prd/*systemout.log" "UserName can't be null"
How can one poor search kills the box.
There is unfortunately no protections against expensive searches.
But it's strange that a single quote causes such an issue.
Please fill a support case to report it.
We found the issue, the default indexes included the OS index, as soon as we removed it, the search took 8 seconds and competed just fine