Hyperscale ML threat intelligence for early detection & disruption

Microsoft

Dec 23, 2024

In today's rapidly evolving cybersecurity landscape, the ability to swiftly identify and mitigate threats is more critical than ever. Attackers are increasingly well-resourced, enabling them to keep adding new components to their toolkits that keep their infrastructure fresh and hard to detect. Traditional labeling methods used to identify and block malicious infrastructure are struggling to keep up. At Microsoft, we recognize the pressing need for innovative solutions that not only keep pace with these threats but stay ahead of them.

This past Ignite, we announced Threat Intelligence Tracking via Dynamic Networks (TITAN)—a groundbreaking approach that uses the power of machine learning to transform threat intelligence and attack disruption by automatically neutralizing malicious activity at scale. By leveraging real-time ML-driven analytics, TITAN uncovers previously hidden threat actor infrastructure, enabling the disruption capabilities built into our unified security operations platform to detect and stop attacks significantly earlier in the attack chain (Figure 1).

The power of machine-scale threat intelligence

TITAN represents a new wave of innovation built on Microsoft threat intelligence capabilities, introducing a real-time, adaptive threat intelligence (TI) graph that integrates first and third-party telemetry from the unified security operations platform, Microsoft Defender for Threat Intelligence, Microsoft Defender for Experts, and customer feedback. This graph employs guilt-by-association techniques to propagate known TI labels to unknown neighboring entities (e.g., IP, file, email) at machine scale. By analyzing relationships between entities, TITAN can identify attacker infrastructure before they are leveraged in attacks, providing an invaluable window of opportunity to prevent harm.

Figure 1. Architectural overview of TITAN, comprising four key steps: (1) constructing a graph using telemetry from 1^st and 3^rd party detectors in the Unified Security Operations Platform, (2) integrating known threat intelligence from across Microsoft, (3) applying reputation propagation algorithms to classify previously unknown entities as either benign or malicious, and (4) updating the reputation score for each entity in the graph.

By leveraging guilt-by-association methods, TITAN can swiftly identify hidden threat actor infrastructure through cross-organizational associations with known malicious entities within the TI graph. Specifically, we employ a semi-supervised label propagation technique that iteratively assigns reputation scores to nodes based on their neighbors’ scores, refining the graph’s score distribution until convergence. These high-confidence entity reputation scores empower the unified security operations platform to implement proactive containment and remediation actions via attack disruption.

A key advantage of our constantly evolving threat intelligence is that we can provide clear and explainable reputation scores for each entity by examining the neighboring entities that contribute to the overall score.

Preventing attacks before they happen

Consider a scenario where TITAN detects unusual activity from a seemingly benign IP address that has connections to known malicious domains. Traditional systems might not flag this IP until after malicious activity is confirmed. However, TITAN's guilt-by-association techniques elevate the reputation score of the IP address, immediately triggering detection and disruption rules that block the threat before any damage occurs.

With an impressive average macro-F1 score of 0.89 and a precision-recall AUC of 0.94, TITAN identifies millions of high-risk entities each week, enabling a 6x increase in non-file threat intelligence. Since its deployment, TITAN has reduced the time to disrupt by a factor of 1.9x while maintaining 99% precision, as confirmed by customer feedback and thorough manual evaluation by security experts—ultimately saving customers from costly security breaches.

Dynamic threat intelligence graph construction

At the heart of TITAN is a dynamic, time-evolving threat intelligence graph that captures complex relationships between millions of interlinked entities, alerts, and incidents. By combining telemetry across both 1^st and 3^rd party sources in the unified security operations platform, TITAN is uniquely positioned for comprehensive view of the threat landscape, essential for early detection and disruption. Key features include:

Real-time updates – In cybersecurity, speed is critical. TITAN operates in real-time, with graph creation and reputation propagation algorithms running every hour. This frequency ensures that security teams receive fresh and active threat intelligence, enabling swift and effective responses to emerging threats. The ability to act quickly can mean the difference between thwarting an attack and being breached.
Infusing security domain knowledge via edge weights – Edges in the TI graph carry weights that signify the strength or relevance of the relationships between entities. We introduce edge weight decay functions that automatically reduce edge weights based on the time elapsed since the edge was formed. This ensures that newer and more relevant relationships have a greater impact on reputation assessments, aligning the dynamic graph with the real-time nature of security incidents.
Pruning outdated nodes and edges – To maintain the relevance and efficiency of the TI graph, we implement pruning mechanisms that remove nodes and edges when their weights fall below certain thresholds. This approach keeps the graph focused on the most current and meaningful connections, ensuring optimal performance.

Evolving cybersecurity defense with TI

TITAN represents a monumental step forward in the mission to protect organizations from cyber threats. By infusing the power of AI with advanced threat intelligence, we are equipping security teams with the tools they need to stay ahead of the attackers. This is only possible with a unified platform that consolidates threat intelligence across 1^st and 3^rd party workloads and products, organizations benefit not only from streamlining their security operations but also gain deeper insights into potential threats and vulnerabilities.

TITAN is just one of the many examples of how powerful bringing together the full capabilities of an industry-leading cloud-native security information and event management (SIEM), comprehensive extended detection and response (XDR), and generative AI built specifically for cybersecurity. Integrating all of this data, advanced analysis, threat intel and automation enables an entirely new era of defense for security teams and we’re so energized by the potential. TITAN is just the start – look forward to new capabilities announced in the coming months.