Security Alert Overload: Causes, Costs, & Solutions
In 2023, the Los Angeles Police Department responded to a series of triggered alarms at a GardaWorld cash storage warehouse in a suburban neighborhood in the San Fernando Valley. All thirteen were deemed to be false positives.
A year later, four more alarms rang at the same facility: one just before midnight on March 30th and the other three on Easter Day. Three of the four were determined to be false alarms and the one that was considered valid only resulted in a notified supervisor and written report. The response times between all four occurrences ranged from several minutes to multiple hours.
Due to the frequency of police dispatches to this cash storage facility that had previously cried wolf, the LAPD had become desensitized to the potential severity of such alarms, a phenomenon known as alert fatigue.
However, between 11:30 p.m. on March 30th and 3:51 p.m. the following day, thieves had breached the building and the safe within it, resulting in a heist totaling $30 million.
Alert Fatigue in Cybersecurity
Alert fatigue, also referred to as alert burnout, exists both in the real world and virtual realm. In regards to cybersecurity, alert fatigue is caused by a combination of alert overdose and poor prioritization which disregards business context, environmental architecture, and sufficient integration with incident response protocols.
When defensive teams become desensitized to alerts due to an overwhelming number of them, especially in cases where most consist of low-priority or false positive issues, actual attacks being conducted by malicious adversaries can be missed. Response times to valid threats can also be greatly increased when a large quantity of alerts must be parsed through or due to a queue of less severe events being dealt with.
According to a report published by Coro in 2023, in a survey of 500 cybersecurity experts, an alarming 73% admit they have missed, ignored, or failed to respond to high-priority security alerts. The percentage of participants that report to have muted a security alert entirely amounts to 26%.
While these figures may be shocking, research conducted by Forrester in 2020 found that security teams deal with an average of 11,000 security alerts per day. The same study also found that out of those 11,000 alerts, 28% are never addressed, for a total of 3,080 security alerts that are left unattended on a daily basis.
With the weight of this workload, it comes as no surprise that 84% of cybersecurity professionals claim to have experienced burnout in 2024 according to a study performed by Hack the Box. The study also found that cybersecurity employee burnout can have a substantial negative financial impact on an organization. On average, due to lost productivity attributed to stress and fatigue, medium to large organizations within the United States lose over $626 million annually. Out of the 3,208 surveyed cybersecurity professionals, 89% attributed being overworked as one of the key causes of their burnout state.
Causes of Alert Fatigue
Although a resilient security posture requires the implementation of multiple tools, the sheer amount of cybersecurity noise they produce can quickly lead to alert fatigue. In the same Coro report cited earlier, on average, security subject matter experts reported that they are managing over ten cybersecurity tools at a time. Those surveyed also reported that five hours a day are spent on tool management. Additionally, 32% of the survey’s participants stated that they manage between 501 to 1,000 endpoint devices with each having an average of 4 security agents installed.
Commonly used tools such as Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), Security Information and Event Management (SIEM) solutions, endpoint security systems, firewalls, anti-virus software, Advanced Persistent Threat (APT) detection software, etc. can all generate security alerts.
Each and everyone of these tools, whether due to their configuration or general poor design, can prioritize less severe issues over threats that require immediate attention. These faults can contribute to inefficiencies in incident response processes. Additionally, systems that lack quality threat classification mechanisms can flag normal activity events as abnormal, creating a torrent of trivial notifications and adding unnecessary entries to the log files. Furthermore, certain threats are more pertinent to an organization depending on their industry or sector. If there is a lack of customization options, this too could render incident response processes less efficient.
Even if the tools used do possess adequate filtering capabilities, multiple tools may output the same alerts, leading to a bloat in the number of issues to analyze due to redundancy. This overlap can be even more detrimental to a cybersecurity program if the alerts are not detailed enough and require manual comparison in order to match them.
Combating Alert Fatigue
In order to counter against alert fatigue and avoid the ill effects that come with it, a number of actionable steps can be taken.
1. Prioritize threats:
To begin, time should be taken in order to ensure any tool that generates security alerts is configured to better assess risk and prioritize alerts based on context, business impact, and severity that is relevant to your environment. By customizing the tools in the technology stack to your specific organization, you ensure attention is diverted to the issues that matter the most. Proper calibration practices also include reducing the number of false positives or false negatives received via custom filters. This can all be achieved through the alteration of settings such as those associated with discovery, anomaly detection, and alert thresholds. This shift from the default settings will reduce the frequency of irrelevant security notifications. A kit of customized tooling also provides the benefit of added depth to your security program by addressing the risks you are most likely to encounter.
IONIX provides a number of prioritization features aimed at reducing alert fatigue. First, our discovery evidence functionality shows our customers – with full transparency – why we attributed a given asset to them (and consequent security findings on those assets). Second, we prioritize findings by running exploit validation tests. So assets with confirmed exploitability will be critical, and warrant alerts, but others will not – greatly helping reduce noise. One final alerting feature of the IONIX platform is our ‘Action Items’. These remediation instructions aggregate multiple findings into a single alert, greatly reducing noise.
2. Centralize alerts:
Integrate an alert management platform into your technology stack. These solutions consolidate the alerts produced across multiple independently functioning tools and present them in a single interface. With these tools, instead of making configuration changes to each tool individually, thresholds and settings are automatically cast across the board. This reduction in manual effort will enable you and your teams to spend your time with incident response and remediation instead of finding the valid threats to begin with.
3. Leverage artificial intelligence:
By taking advantage of the processing and assessment capabilities of AI, suspicious activity can be more accurately identified and brought to your attention. AI tooling can use real-time context in order to prioritize any activity indicative of malicious intent to ensure your team is in the best position to thwart an attack or respond to one as quickly as possible. Research published by IBM in their 2024 Cost of a Data Breach Report, found that organizations that extensively use AI for security and automation were able to identify and contain data breaches about 100 days faster than organizations that do not utilize the technology at all. This reduction in response time was found to reduce the cost of a data breach by 45.6%.
4. Integrate threat intelligence:
Use a threat intelligence platform in order to aggregate, normalize, and manage threat information from various sources, allowing for easier access and analysis. These platforms can be integrated with other forms of security tooling to aggregate data. With this data, you can cross-check security alerts to match them to known vulnerabilities and identify if they are related to modern adversarial campaigns. By doing so, prioritization can be bolstered based on real attack scenarios, resulting in an even more hardened attack surface.
5. Conduct regular reviews:
All of these practices and implementations should be well documented and considered in your incident response processes. Every tool that is used should be accounted for and added to an asset ledger. Also, the documentation should reflect everytime a configuration is changed or a new piece of technology is added to the security program. Assessments should be performed at regular intervals to ensure that configurations are up to date with the latest threat environment and organizational changes. If available, the CIS Benchmark for the technology should be adhered to.
Conclusion
Alert fatigue presents a significant challenge that can persist if not properly addressed in a timely manner. If ignored, it can lead to successful attacks with consequences such as data breaches, financial loss, regulatory non-compliance, and reputational damage. To combat alert fatigue effectively, organizations must establish a system that includes alert prioritization, leveraging security alert solutions, and reviewing them all frequently. By implementing these strategies, you can minimize noise in cybersecurity and ensure your cybersecurity team is as responsive and efficient as they can be.
