Image courtesy of Shutterstock
One of the biggest challenges for IT departments is the ability to detect and respond to critical events such as cyber attacks in real time. This problem will only compound as the amount of machine data organizations produce is expected to grow 15 times by 2020. There is simply no way traditional monitoring systems will be able to effectively manage this volume of data.
Managing machine logs – the output of every application, website, server and supporting IT infrastructure component in the enterprise – is the starting point for data analysis. Many IT departments hope they will be able to improve system or application availability, prevent downtime, detect fraud and identify important changes in customer and application behavior by studying machine logs. However, traditional operational analytics and log management tools fail to help users proactively discover events they don’t anticipate.
The explosion of machine data has made it absolutely impossible for humans to write every rule to detect relevant events. Most of these events are unknown, new (or rather “anomalous”), or indescribable, and as a result, they go undetected. IT organizations need a mechanism to automatically “tell” users what is happening inside of their data without the administrator’s prerequisite knowledge of the event.
Enter Anomaly Detection. Anomaly Detection enables enterprises to automatically detect events in streams of machine data, generate previously undiscoverable insights within a company’s entire IT and security infrastructure and allow remediation before an issue impacts key business services. This type of machine intelligence capability works to continuously handle the aggregate data output of the enterprise in real time, mitigating the risks of both benign and malicious events as they occur.
An Anomaly Detection service enables IT and security teams to do three things:
- Uncover the unknown unknowns that were not previously detected or identified
- Alert relevant users about the anomalous event, who can then classify and document it based on relevance and severity
- Share that knowledge with others so that the organization can better understand how to handle it.
An Anomaly Detection service deciphers how a series of events and their patterns vary from that which is considered normal and enables IT to quickly determine the significance to the business. It doesn’t replace the role of the experienced IT professional, but rather enhances the efficiency and efficacy of his/her performance.
Effective anomaly detection engines are “fuzzy” in nature. They don’t need to know anything about the nature of the data, or the expected types or numbers of events. It also doesn’t know the difference between harmful and benign events. This may sound confusing or overly broad, but it puts separating wrong from right into human hands as opposed to leaving it to pre-determined rules that may respond improperly or not at all to an anomalous event. The important part is that real-time action can occur to deal with these events. While it’s prudent to take certain types of anomalies offline for resolution sometime after they were witnessed, a breach or other such severe event must be dealt with while the event is, quite literally, in progress.
It’s important to note that Anomaly Detection is not just about automation. It is best enhanced with the domain knowledge of an experienced IT or security professional who can spot the difference between an attack or failure and a benign blip. This is the case especially if the anomaly detection engine can capture and encode an analyst’s feedback and use that feedback to better detect future events and equip other users with the captured knowledge. The next generation of IT professionals will need to be flexible and adaptive in order to fight cyberattack; and Anomaly Detection will be one of the tools they’ll need in order to do that
About the author:
Bruno Kurtic is founding vice president of product and strategy at Sumo Logic.