Prevention should be the holy grail of any cybersecurity strategy, but we know that 100% prevention is not realistic. So, what exactly happens when a serious threat is detected? Using a recent incident as an example, it is enlightening to follow the chain of events that starts when human SOC analysts decide that CSIRT action is required.
Effectively dealing with critical security incidents that potentially have a major impact on the business of our customers is the essential function of the ON2IT 24/7 Security Operations Center (SOC). Time is of the essence, and it is vitally important to know what to do and how to act.
ON2IT’s Cybersecurity Incident Response Team (CSIRT) is prepared to deal with all types of cybersecurity incidents thanks to adequate and efficient procedures, checklists, expertise and training.
The CSIRT consists of screened cybersecurity specialists and is activated in case of a major incident that requires specific and immediate strategic attention.
Identify: Are we in trouble?
The ability to monitor, detect and identify threats in our customers’ IT environments is a key capability of the ON2IT SOC. State-of-the art cybersecurity technology typically generates tens of thousands of alerts and events per day. Most of them are informational and all obvious attacks have been stopped, for instance at the endpoint or by a firewall.
The EventFlow™ engine that’s part of our AUXO™ platform uses advanced technology and a wide array of Threat Intelligence feeds to evaluate all these alerts and events. They are automatically classified by our proprietary Indicator-of-Good technology, reducing the workload of our SOC analysts dramatically. This is a different approach than the one used by most MDR-vendors, who are essentially looking for a needle-in-a-haystack by evaluating endless stacks of Indicators-of-Compromise.
Out of every 100.000 events, EventFlow™ can classify 99.999 automatically as
non-suspicious. During every SOC-shift, EventFlow™ singles out a collection of (sometimes related) events for evaluation by a human SOC analyst (the remaining 1 event out of 100.000).
This is what happened on a recent Tuesday evening at 20:08 CET. In isolation, the events singled out by EventFlow™ that day do not seem critical. However, the SOC analyst on duty notices a suspicious pattern in the events that EventFlow™ has grouped together. The traffic seems to originate from two legitimate logged in users, but the particular requests to a particular server are highly unusual.
Time to Escalate
After a quick investigation, the SOC analyst decides to consult with the SOC Shift Lead, who only needs a couple of seconds to agree that an escalation to the CSIRT is required. At this point, the severity of the incident is yet unknown, but the chance that this is a high priority incident cannot be ruled out. And because the clock is ticking, the CSIRT procedures are started. Better safe than sorry.
The first call is to the CSIRT Lead Coordinator. Other specialists familiar with the technology and configuration of the customers equipment are added to the CSIRT.
Containment and Communication
The CSIRT has switched to high-gear. Analysis and investigation of logs and traffic patterns reveals that the unusual requests originate from a single source that at first sight seems to be a legitimate part of the customer’s infrastructure. The requests, however, are suspicious, involving scripts and actions that are rarely used by typical users. Although the potentially dangerous requests are effectively blocked by fine-grained firewall policies, at this point there is no absolute certainty that this is a single isolated incident. Information from the customer is required for further analysis.
Primary damage control is initiated according to established protocols. In this case, the suspect user accounts and connections are blocked and firewall policies are further fine-tuned to detect and block specific traffic patterns in the environment that was potentially accessed by the suspicious users. At the same time, evidence is secured and thorough analysis takes place to identify actual impact and a structural mitigation strategy.
The CSIRT team contacts the relevant stakeholders at the customer following established communication protocols. The incident related communication details are captured in our AUXO™ platform. This advanced case management platform provides 24/7 insights during and after the incident.
Alignment with the customer
Communication with the security team at the customer makes it clear that the credentials used to set up the suspicious communication are associated with a
third party that no longer works for the customer. The threat seems to be isolated to these particular credentials, that have not been revoked (which they should have been). The third party cannot be reached at this time. Plausible scenarios are that the credentials were acquired via a data breach in the third party’s environment, or via a disgruntled employee. The customer’s security team will continue the investigation on their end and contact the third party first thing in the morning. In the meantime, suspicious login attempts on other accounts are reviewed to rule out that other users are compromised as well.
The communication protocol til then involves frequent updates that include current status of the incident, current risk and exposure, mitigation steps that have been taken and next steps.
Eradicate and Recover
The CSIRT team, in close communication with the customer, commences clean-up and possibly removal of attack traces. Additional investigation is performed to determine if other environments are potentially affected as well and require protective measures.
They also perform a final sanity check to validate mitigation. Systems are subject to intensified monitoring to detect suspicious behavior.
Post-incident evaluation and improvement
A postmortem analysis is initiated in order to identify improvements and inform security improvements advisories (SIAs) that aim to avoid the reoccurrence of a similar incident. An evaluation meeting is scheduled for the next day to discuss postmortem analysis, lessons learned and to monitor follow-up on SIAs. The SIAs are registered in the AUXO™ portal so that their status is real-time available.
Cases are created for recommendations that require follow-up actions. The ON2IT SOC engineers go way beyond their traditional monitoring and response responsibilities. They play an important role in proactively implementing changes, together with the customer, that mitigate known vulnerabilities and further strengthen security posture. In this situation, a new case is created to roll out an additional layer of authentication for customer-managed devices so that login attempts from unknown devices are no longer possible at all. Another action is to update the metadata and relevance score for some of the defined protect surfaces. This provides the SOC with useful context so they can fine-tune their triage on events.
What can you learn from this?
Many service providers of so-called managed detection and response (MDR) or remote SOC services limit their activities to alerting their customers when suspicious activity is detected. These alerts might also be false positives, or minor incidents that require no immediate action when the technical context is known. The burden is on the customer to figure this out.
From that point on, it is the customer’s responsibility to assemble the team of specialists required to analyze the threat in their infrastructure and proceed with mitigating actions. When this happens during non-office hours, especially when the operational management of the cybersecurity equipment is outsourced, it can take hours to sometimes days to effectively respond to the threat.
SOC, operational and CSIRT capabilities are required as soon as a high priority breach is detected. In practice this means 24/7 since incidents can occur outside of regular working hours. During the initial investigation, communication and the ability to implement mitigating measures within minutes are of critical importance. Large organizations traditionally have the scale and ability to run internal SOC, operations and CSIRT functions.
But organizations that outsource the SOC function, should be extremely cautious in making sure that CSIRT and required operational management skills are available at short notice and can work together effectively.
At ON2IT, we firmly believe in the advantages of being responsible for the design, implementation and continuous verification of the cybersecurity technology in our customers’ infrastructure. This deep knowledge enables our SOC engineers and CSIRT to act decisively when every minute counts to decrease the impact of a breach.
Disclaimer: Some details in this article are fictitious or slightly modified in order to preserve anonymity of those involved.
