Security: Is a fake alert app useless?

[einervonvielen2]

Hi everyone,

preparing for my master´s thesis my supervisor at the uni suggested to create an app that produces fake alerts with suspicious log files in splunk to maintain admins´s attention on security issues. L like at the airport security where regularly fake guns and knifes are displayed on the scanner to catch the guard´s attention.

However, after some research I get the feeling most admins have an opposite issue, having to many false alerts. As I have no experience with Splunk in a security context, I am looking for some opinions on that. Can someone give me some insights?

[marycordova]

Echoing @richgalloway alert fatigue is a real thing.

Also, there are plenty of security related events happening all the time anyway, why not write an app that detects some of those? There's a lot of good stuff happening in the "risk based" surfacing of mundane events that you could look into also. Several good talks about it from Splunk .conf 2018 (or 17 maybe?).

Alternatively, you could write an app to periodically test existing use cases, and now that I'm coming up with this idea it think it's great. Here goes:

1. Install the Splunk Security Essentials app
2. Take a look at the uses cases
3. Pick some of the most relevant to today's environment
4. Write an app that would on a schedule generate log events to trigger the use cases
5. The premise being you want to test that your indexing, parsing, scheduling, and alerting layers are all working properly, as @richgalloway said, this is especially important for very rare alerts and something I've practiced at various locations

Now that I think about it...there are entire companies built on this premise already, so much for the genius idea 😛

Maybe work on something in Splunk with the Machine Learning Toolkit...that's pretty hot these days.

[einervonvielen2]

Hi @marycordova,

thanks for your hints. I will try it. So far I installed, the Security Essentials App but I feel overwhelmed with all the different documentation and I dont know where to start. Do you have some hint for something like a step by step beginners guide?

[marycordova]

Maybe take a look at cyber events in the news in the last 6 months, see if there are any use cases in Security Essentials, then see if you can write a few "fake" events to trigger those rules to make sure they are operating properly, then write up a detailed document for people to use on what the requirements are to make sure those use cases are going to work for their environment.

[LukeMurphey]

Some thoughts below:

"suggested to create an app that produces fake alerts with suspicious log files in splunk to maintain admins´s attention on security issues"
I think this could have value depending on the circumstances. I would think it would be better for testing the automation though. I happen to know that some customers do generate fake alerts in order to make sure that their tools properly detect the issue (basically using fake alerts as a sort of unit test to the detection logic).

This could also be used for training too. Splunk does something like this with its BOTS program ("Boss of the SOC").

"after some research I get the feeling most admins have an opposite issue, having too many false alerts."
This is very much true. Admins tend to get a massive volume of alerts and all of them could be security issues. The difficulty is that security alerts are oftentimes a little fuzzy and you cannot have 100% confidence in all of the alerts. This is why you have humans analyze them.

I used to be a Security Analyst at a Security Operations Center and well meaning managers would often pop in and start asking about specific alerts on the screen. The conversation would go like this:

• Manager: points at alert regarding an authentication failure, "how do you know with certainty that isn't a valid attack?"
• Analyst: "I don't; but I don't see any other activity indicating malicious behavior"
• Manager: "so you cannot be certain that it isn't an attack, maybe you should escalate it"
• Analyst: "I could but I would also have to escalate 1,000 other similar alerts and then none of the alerts would get fixed because it would have overwhelmed the IT team and they would just ignore all of them"

The fact is that much of security monitoring is a judgment call. If one is not careful, then you could easily generate a contrived situation that the admins miss but could not realistically be escalated anyways (i.e. if they escalated the fake alert, they would also have to escalate thousands of other alerts that are similar but aren't real issues).

[honey4sec]

Hi.
Short intro, im a Senior Security Analyst who also happens to be a Splunk Certified power user.
I have been working in the security industry for 8y in a company with more then 90k employees.

What do you want to write about in your thesis?
What is your field of study
I can tell you from knowledge that throwing false grenades is generally a bad idea.
It creates distrust and disruptions in regards to real incidents.
Your free to dm me if you want to talk.

[einervonvielen2]

I am not sure what I am gonna write about. As I said this idea came from my supervisor and what you said confirmed me in my thoughts. However, maybe the idea @richgalloway mentioned above to create something like a tester for incidents (it could be challenging to differentiate this from something like Eventgen). I am studying business informatics. The thesis should have an security context.

[richgalloway]

Most shops experience a lot of false positives. The term "alert fatigue" is also very real. That said, there is something to be gained by occasionally injecting an event to trigger an alert that is normally never seen. This helps to ensure the alert logic is still valid and workflow for handling the alert is sound.