Enhancing Security Monitoring: Integrating GitLab Cloud Edition with Microsoft Sentinel
Maximize your security operations by combining GitLab Cloud Edition with Microsoft Sentinel. This blog covers how to fill the void of a missing native connector for GitLab in Sentinel. Utilize GitLab's API endpoints, Azure Monitor Data Collection Rules, and Data Collection Endpoints, as well as Azure Logic Apps and Key Vault, to simplify log collection and improve immediate threat identification. Our detailed guide will help you integrate smoothly and strengthen your security defences.Integrating Fluent Bit with Microsoft Sentinel
This guide will walk you through the steps required to integrate Fluent Bit with Microsoft Sentinel. Beware that in this article, we assume you already have a Sentinel workspace, a Data Collection Endpoint and a Data Collection Rule, an Entra ID application and finally a Fluent Bit installation. As mentioned above, log ingestion API supports ingestion both in custom tables as built-in tables, like CommonSecurityLog, Syslog, WindowsEvent and more. In case you need to check which tables are supported please the following article: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables Prerequisites: Before beginning the integration process, ensure you have the following: An active Azure subscription with Microsoft Sentinel enabled; Microsoft Entra ID Application taking note of the ClientID, TenantID and Client Secret – create one check this article: https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app?tabs=certificate A Data Collection Endpoint (DCE) – to create a data collection endpoint, please check this article: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-endpoint-overview?tabs=portal A Data Collection Rule (DCR) – fields from the Data Collection Rule need to match exactly to what exists in table columns and also the fields from the log source. To create a DCR please check this article: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-create-edit?tabs=cli Depending on the source, it might require a custom table to be created or an existing table from log analytics workspace; Fluent Bit installed on your server or container – In case you haven’t yet installed Fluent Bit, in the following article you'll find the instructions per type of operating system: https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit High level architecture: Step 1: Setting up Fluent Big configuration file Before we step-in into the configuration, Fluent Bit has innumerous output plugins and one of those is through Log Analytics Ingestion API both to supported Sentinel tables but also for custom tables. You can check more information about it here in Fluent Bit documentation: https://docs.fluentbit.io/manual/pipeline/outputs/azure_logs_ingestion Moving forwarder, in order to configure Fluent Bit to send logs into Sentinel log analytics workspace, please take note of the specific input plugin you are using or intend to use to receive logs and how can you use it to output the logs to Sentinel workspace. For example most of the Fluent Bit plugins allow to set a “tag” key which can be used within the output plugin so that there’s a match in which logs are intended to send. On the other hand, in a scenario where multiple input plugins are used and all are required send logs to Sentinel, then a match of type wildcard "*" could be used as well. Another example, in a scenario where there are multiple input plugins of type “HTTP” and you want to just send a specific one into Sentinel, then the “match” field must be set according to the position of the required input plugin, for example “match http.2”, if the input plugin would the 3 rd in the list of HTTP inputs. If nothing is specified in the "match" field, then it will assume "http.0" by default. For better understanding, here’s an example of how a Fluent Bit config file could look: First, the configuration file is located under the path ”/etc/fluent-bit/fluent-bit.conf” The first part is the definition of all “input plugins”, then follows the “filter plugins” which you can use for example to rename fields from the source to match for what exists within the data collection rule schema and Sentinel table columns and finally there’s the output plugins. Below is a screenshot of a sample config file: INPUT plugins section: In this example we’re going to use the “dummy input” to send sample messages to Sentinel. However, in your scenario you could leverage other’s input plugins within the same config file. After everything is configured in the input section, make sure to complete the “FILTER” section if needed, and then move forward to the output plugin section, screenshot below. OUTPUT plugins section: In this section, we have output plugins to write on a local file based on two tags “dummy.log” and “logger”, an output plugin that prints the outputs in json format and the output plugin responsible for sending data to Microsoft Sentinel. As you can see, this one is matching the “tag” for “dummy.log” where we’ve setup the message “{“Message”:”this is a sample message for testing fluent bit integration to Sentinel”, “Activity”:”fluent bit dummy input plugn”, “DeviceVendor”:”Ubuntu”}. Make sure you insert the correct parameters in the output plugin, in this scenario the "azure_logs_ingestion" plugin. Step 2: Fire Up Fluent Bit When the file is ready to be tested please execute the following: sudo /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf Fluent bit will start initialization all the plugins it has under the config file. Then you’re access token should be retrieved if everything is well setup under the output plugin (app registration details, data collection endpoint URL, data collection rule id, sentinel table and important to make sure the name of the output plugin is actually “azure_logs_ingestion”). In a couple of minutes you should see this data under your Microsoft Sentinel table, either an existing table or a custom table created for the specific log source purpose. Summary Integrating Fluent Bit with Microsoft Sentinel provides a powerful solution for log collection and analysis. By following this guide, hope you can set up a seamless integration that enhances your organization's ability to monitor and respond to security threats, just carefully ensure that all fields processed in Fluent Bit are mapped exactly to the fields in Data Collection Rule and Sentinel table within Log Analytics Workspace. Special thanks to “Bindiya Priyadarshini” that collaborated with me on this blog post. Cheers!Announcing Public Preview: New STIX Objects in Microsoft Sentinel
Security teams often struggle to understand the full context of an attack. In many cases, they rely solely on Indicators of Compromise (IoCs) without the broader insights provided by threat intelligence developed on Threat Actors, Attack Patterns, Identities - and the Relationships between each. This lack of context available to enrich their workflows limits their ability to connect the dots, prioritize threats effectively, and respond comprehensively to evolving attacks. To help customers build out a thorough, real-time understanding of threats, we are excited to announce the public preview of new Threat Intelligence (TI) object support in Microsoft Sentinel and in the Unified SOC Platform. In addition to Indicators of Compromise (IoCs), Microsoft Sentinel now supports Threat Actors, Attack Patterns, Identities, and Relationships. This enhancement empowers organizations to take their threat intelligence management to the next level. In this blog, we’ll highlight key scenarios for which your team would use STIX objects, as well as demos showing how to create objects and new relationships and how to use them to hunt threats across your organization Key Scenarios STIX objects are a critical tool for incident responders attempting to understand an attack and threat intelligence analysts seeking more information on critical threats. It is designed to improve interoperability and sharing of threat intelligence across different systems and organizations. Below, we’ve highlighted four ways Unified SOC Platform customers can begin using STIX objects to protect their organization. Ingesting Objects: You can now ingest these objects from various commercial feeds through several methods including STIX TAXII servers, API, files, or manual input. Curating Threat Intelligence: Curate and manage any of the supported Threat Intelligence objects. Creating Relationships: Establish connections between objects to enhance threat detection and response. For example: Connecting Threat Actor to Attack Pattern: The threat actor "APT29" uses the attack pattern "Phishing via Email" to gain initial access. Linking Indicator to Threat Actor: An indicator (malicious domain) is attributed to the threat actor "APT29". Associating Identity (Victim) with Attack Pattern: The organization "Example Corp" is targeted by the attack pattern "Phishing via Email". Hunt and Investigate Threats More Effectively: Match curated TI data against your logs in the unified SOC platform powered by Microsoft Sentinel. Use these insights to detect, investigate, and hunt threats more efficiently, keeping your organization secure. Get Started Today with the new Hunting Model The ability to ingest and manage these new Threat Intelligence objects is now available in public preview. To enable this data in your workspaces for hunting and detection, submit your request here and we will provide further details. Demo and screen shots Demo 1: Hunt and detect threats using STIX objects Scenario: Linking an IOC to a Threat Actor: An indicator (malicious domain) is attributed to the threat actor " Sangria tempest " via the new TI relationship builder. Please note that the Sangria tempest actor object and the IOC are already present in this demo. These objects can be added automatically or created manually. To create new relationship, sign into your Sentinel instance and go to Add new à TI relationship. In the New TI relationship builder, you can select existing TI objects and define how it's related to one or more other TI objects. After defining a TI object’s relationship, click on “Common” to provide metadata for this relationship, such as Description, Tags, and Confidence score: Another type of meta data a customer can add to a relationship is the Traffic Light Protocol (TLP). The TLP is a set of designations used to ensure that sensitive information is shared with the appropriate audience. It uses four colors to indicate different levels of sensitivity and the corresponding sharing permissions: TLP:RED: Information is highly sensitive and should not be shared outside of the specific group or meeting where it was originally disclosed. TLP:AMBER: Information can be shared with members of the organization, but not publicly. It is intended to be used within the organization to protect sensitive information. TLP:GREEN: Information can be shared with peers and partner organizations within the community, but not publicly. It is intended for a wider audience within the community. TLP:WHITE: Information can be shared freely and publicly without any restrictions. Once the relationship is created, your newly created relationship can be viewed from the “Relationships” tab. Now, retrieve information about relationships and indicators associated with the threat actor 'Sangria Tempest'. For Microsoft Sentinel customers leveraging the Azure portal experience, you can access this in Log Analytics. For customers who have migrated to the unified SecOps platform in the Defender portal, you can go find this under “Advanced Hunting”. The following KQL query provides you with all TI objects related to “Sangria Tempest.” You can use this query for any threat actor name. let THREAT_ACTOR_NAME = 'Sangria Tempest'; let ThreatIntelObjectsPlus = (ThreatIntelObjects | union (ThreatIntelIndicators | extend StixType = 'indicator') | extend tlId = tostring(Data.id) | extend StixTypes = StixType | extend Pattern = case(StixType == "indicator", Data.pattern, StixType == "attack-pattern", Data.name, "Unkown") | extend feedSource = base64_decode_tostring(tostring(split(Id, '---')[0])) | summarize arg_max(TimeGenerated, *) by Id | where IsDeleted == false); let ThreatActorsWithThatName = (ThreatIntelObjects | where StixType == 'threat-actor' | where Data.name == THREAT_ACTOR_NAME | extend tlId = tostring(Data.id) | extend ActorName = tostring(Data.name) | summarize arg_max(TimeGenerated, *) by Id | where IsDeleted == false); let AllRelationships = (ThreatIntelObjects | where StixType == 'relationship' | extend tlSourceRef = tostring(Data.source_ref) | extend tlTargetRef = tostring(Data.target_ref) | extend tlId = tostring(Data.id) | summarize arg_max(TimeGenerated, *) by Id | where IsDeleted == false); let SourceRelationships = (ThreatActorsWithThatName | join AllRelationships on $left.tlId == $right.tlSourceRef | join ThreatIntelObjectsPlus on $left.tlTargetRef == $right.tlId); let TargetRelationships = (ThreatActorsWithThatName | join AllRelationships on $left.tlId == $right.tlTargetRef | join ThreatIntelObjectsPlus on $left.tlSourceRef == $right.tlId); SourceRelationships | union TargetRelationships | project ActorName, StixTypes, ObservableValue, Pattern, Tags, feedSource You now have all the information your organization has available about Sangria Tempest, correlated to maximize your understanding of the threat actor and its associations to threat infrastructure and activity. Demo 2: Curate and attribute objects We have created new UX to streamline TI object creation, which includes the capability to attribute to other objects, so while you are creating a new IoC, you can also attribute that indicator to a Threat Actor, all from one place. To create a new TI object and attribute it to one or multiple threat actors, follow the steps below: Go to Add new a TI Object. In the Context menu, select any object type. Enter all the required information in the fields on the right-hand side for your selected indicator type. While creating a new TI object, you can do TI object curation. This includes defining the relationship. You can also quickly duplicate TI objects, making it easier for those who create multiple TI objects daily. Please note that we also introduced an “Add and duplicate” button to allow customers to create multiple TI objects with the same metadata to streamline a manual bulk process. Demo 3: New supported IoC types The attack pattern builder now supports the creation of four new indicator types. These enable customers to build more specific attack patterns that boost understanding of and organizational knowledge around threats. These new indicators include: X509 certificate X509 certificates are used to authenticate the identity of devices and servers, ensuring secure communication over the internet. They are crucial in preventing man-in-the-middle attacks and verifying the legitimacy of websites and services. For instance, if a certificate is suddenly replaced or a new, unknown certificate appears, it could indicate a compromised server or a malicious actor attempting to intercept communications. JA3 JA3 fingerprints are unique identifiers generated from the TLS/SSL handshake process. They help in identifying specific applications and tools used in network traffic, making it easier to detect malicious activities For example, if a network traffic analysis reveals a JA3 fingerprint matching that of the Cobalt Strike tool, it could indicate an ongoing cyber attack. JA3S JA3S fingerprints extend the capabilities of JA3 by also including server-specific characteristics in the fingerprinting process. This provides a more comprehensive view of the network traffic and helps in identifying both client and server-side threats For instance, if a server starts communicating with an unknown external IP address using a specific JA3S fingerprint, it could be a sign of a compromised server or data exfiltration attempt. User agent User Agents provide information about the client software making requests to a server, such as the browser or operating system. They are useful in identifying and profiling devices and applications accessing a network For example, if a User Agent string associated with a known malicious browser extension appears in network logs, it could indicate a compromised device. Conclusion: The ability to ingest, curate, and establish relationships between various threat intelligence objects such as Threat Actors, Attack Patterns, and Identities provides a powerful framework for incident responders and threat intelligence analysts. The use of STIX objects not only improves interoperability and sharing of threat intelligence but also empowers organizations to hunt and investigate threats more efficiently. As customers adopt these new capabilities, they will find themselves better equipped to understand the full context of an attack and build robust defenses against future threats. With the public preview of Threat Intelligence (TI) object support, organizations are encouraged to explore these new tools and integrate them into their security operations, taking the first step towards a more informed and proactive approach to cybersecurity.Introducing Threat Intelligence Ingestion Rules
Microsoft Sentinel just rolled out a powerful new public preview feature: Ingestion Rules. This feature lets you fine-tune your threat intelligence (TI) feeds before they are ingested to Microsoft Sentinel. You can now set custom conditions and actions on Indicators of Compromise (IoCs), Threat Actors, Attack Patterns, Identities, and their Relationships. Use cases include: Filter Out False Positives: Suppress IoCs from feeds known to generate frequent false positives, ensuring only relevant intel reaches your analysts. Extending IoC validity periods for feeds that need longer lifespans. Tagging TI objects to match your organization's terminology and workflows Get Started Today with Ingestion Rules To create new “Ingestion rule”, navigate to “Intel Management” and Click on “Ingestion rules” With the new Ingestion rules feature, you have the power to modify or remove indicators even before they are integrated into Sentinel. These rules allow you to act on indicators currently in the ingestion pipeline. > Click on “Ingestion rules” Note: It can take up to 15 minutes for the rule to take effect Use Case #1: Delete IOC’s with less confidence score while ingesting When ingesting IOC's from TAXII/Upload API/File Upload, indicators are imported continuously. With pre-ingestion rules, you can filter out indicators that do not meet a certain confidence threshold. Specifically, you can set a rule to drop all indicators in the pipeline with a confidence score of 0, ensuring that only reliable data makes it through. Use Case #2: Extending IOC’s The following rule can be created to automatically extend the expiration date for all indicators in the pipeline where the confidence score is greater than 75. This ensures that these high-value indicators remain active and usable for a longer duration, enhancing the overall effectiveness of threat detection and response. Use Case #3: Bulk Tagging Bulk tagging is an efficient way to manage and categorize large volumes of indicators based on their confidence scores. With pre-ingestion rules, you can set up a rule to tag all indicators in the pipeline where the confidence score is greater than 75. This automated tagging process helps in organizing indicators, making it easier to search, filter, and analyze them based on their tags. It streamlines the workflow and improves the overall management of indicators within Sentinel. Managing Ingestion rules In addition to the specific use cases mentioned, managing ingestion rules gives you control over the entire ingestion process. 1. Reorder Rules You can reorder rules to prioritize certain actions over others, ensuring that the most critical rules are applied first. This flexibility allows for a tailored approach to data ingestion, optimizing the system's performance and accuracy. 2. Create From Creating new ingestion rules from existing ones can save you a significant amount of time and offer the flexibility to incorporate additional logic or remove unnecessary elements. Effectively duplicating these rules ensures you can quickly adapt to new requirements, streamline operations, and maintain a high level of efficiency in managing your data ingestion process. 3. Delete Ingestion Rules Over time, certain rules may become obsolete or redundant as your organizational needs and security strategies evolve. It's important to note that each workspace is limited to a maximum of 25 ingestion rules. Having a clean and relevant set of rules ensures that your data ingestion process remains streamlined and efficient, minimizing unnecessary processing and potential conflicts. Deleting outdated or unnecessary rules allows for a more focused approach to threat detection and response. It reduces clutter, which can significantly enhance the performance. By regularly reviewing and purging obsolete rules, you maintain a high level of operational efficiency and ensure that only the most critical and up-to-date rules are in place. Conclusion By leveraging these pre-ingestion rules effectively, you can enhance the quality and reliability of the IOC’s ingested into Sentinel, leading to more accurate threat detection and an improved security posture for your organization.Improve SecOps collaboration with case management
Are you using a 3rd party case management system for the SecOps work you do in Microsoft Sentinel or Defender XDR? Do you struggle to find a solution that encompasses the specific needs of your security team? We are excited to announce a new case management solution, now in public preview. This is our first step towards providing a native, security-focused case management system that spans all SecOps workloads in the Defender portal, removing customer reliance on 3rd party SIEM/XDR and ticketing systems. This will be available for all Microsoft Sentinel customers that have onboarded to the unified SecOps platform.Go agentless with Microsoft Sentinel for SAP
What a title during Agentic AI times 😂 Dear community, Bringing SAP workloads under the protection of your SIEM solution is a primary concern for many customers out there. The window for defenders is small “Critical SAP vulnerabilities being weaponized in less than 72 hours of a patch release and new unprotected SAP applications provisioned in cloud (IaaS) environments being discovered and compromised in less than three hours.” (SAP SE + Onapsis, Apr 6 2024) Having a turn-key solution as much as possible leads to better adoption of SAP security. Agent-based solutions running in Docker containers, Kubernetes, or other self-hosted environemnts are not for everyone. Microsoft Sentinel for SAP’s latest new capability re-uses the SAP Cloud Connector to profit from already existing setups, established integration processes, and well-understood SAP components. Meet agentless ❌🤖 The new integration path leverages SAP Integration Suite to connect Microsoft Sentinel with your SAP systems. The Cloud integration capability of SAP Integration Suite speaks all relevant protocols, has connectivity into all the places where your SAP systems might live, is strategic for most SAP customers, and is fully SAP RISE compatible by design. Are you deployed on SAP Business Technology Platform yet? Simply upload our Sentinel for SAP integration package (see bottom box in below image) to your SAP Cloud Integration instance, configure it for your environment, and off you go. Best of all: The already existing SAP security content (detections, workbooks, and playbooks) in Microsoft Sentinel continues to function the same way as it does for the Docker-based collector agent variant. The integration marks your steppingstone to bring your SAP threat signals into the Unified Security Operations Platform – a combination of Defender XDR and Sentinel – that looks beyond SAP at your whole IT estate. Microsoft Sentinel solution for SAP applications is certified for SAP S/4HANA Cloud, Private Edition RISE with SAP, and SAP S/4HANA on-premises. So, you are all good to go😎 You are already dockerized or agentless? Then proceed to this post to learn more about what to do once the SAP logs arrived in Sentinel. Final Words During the private preview we saw drastically reduced deployment times for SAP customers being less familiar with Docker, Kubernetes and Linux administration. Cherry on the cake: the network challenges don’t have to be tackled again. The colleagues running your SAP Cloud Connector went through that process a long time ago. SAP Basis rocks 🤘 Get started from here. Find more details on our blog on the SAP Community and latest Microsoft Learn article. Cheers MartinIntroducing SOC Optimization Recommendations Based on Similar Organizations
One of the key challenges that security teams in modern SOCs regularly face is determining which new data sources to onboard and which detections to activate. This ongoing process takes time and requires constant evaluation of the organization’s assets and the value that the data brings to the SOC. "…determining which logs to ingest for better threat coverage is time-consuming and requires significant effort. I need to spend a long time identifying the appropriate logs..." Elie El Karkafi, Senior Solutions Architect, ampiO Solutions Today, we’re excited to announce the public preview of recommendations based on similar organizations - a first-of-its-kind capability for SOC optimizations. Recommendations based on similar organizations use peer-based insights to guide and accelerate your decision-making process. We believe that applying insights learned from the actions of organizations with similar profiles can provide great value. Recommendations based on similar organizations use advanced machine learning to suggest which data to ingest, based on organizations with similar ingestion patterns. The recommendations also highlight the security value you can gain by adding the data. They list out-of-the-box rules that are provided by Microsoft research, which you can activate to enhance your coverage. Use the new recommendations to swiftly pinpoint the next recommended data source for ingestion and determine the appropriate detections to apply. This can significantly reduce the time and costs typically associated with research or consulting external experts to gain the insights you need. Recommendations based on similar organizations are now available in the SOC optimization page, in both the Azure portal and the unified security operations platform: - unified security operations platform Use cases Let’s take a tour of the unified security operations platform, stepping into the shoes of a small tech company that benefited from recommendations based on similar organizations during its private preview phase. In the following image, the new recommendation identifies that the AADNonInteractiveUserSignInLogs table is used by organizations similar to theirs: Selecting View details button on the recommendation card allowed them to explore how other organizations use the recommended table. This includes insights into the percentage of organizations using the table for detection and investigation purposes. By selecting See details hyperlink, the SOC engineer was able to explore how coverage could be improved with respect to the MITRE ATT&CK framework, using Microsoft’s out-of-the box rules: By selecting Go to Content hub, the SOC engineer was able to view all the essential data connectors needed to start ingesting the recommended tables. This page also includes a detailed list of out of the box, recommended analytics rules, which can provide immediate value and enhanced protection for your environment: Finally, by following the recommendation, which uses the security practices of similar organizations as a benchmark, the tech company quickly ingested the AADNonInteractiveUserSignInLogs table and activated several recommended analytics rules. Overall, this resulted in improved security coverage, corresponding to the company's specific characteristics and needs. Feedback from private preview: “I think this is a great addition. Like being able to identify tables not being used, it is useful to understand what tables other organizations are utilizing which could reveal things that so far haven't been considered or missed...” Chris Hoard, infinigate.cloud "In my view, those free recommendations are always welcomed and we can justify cost saving and empowering SOC analysts (that we know are more and more difficult to find)." Cyrus Irandoust, IBM “These recommendations will help us to take a look at the left out stuffs” Emmanuel Karunya, KPMG “Nice overview and insights! Love the interface too - nice and easy overview!” Michael Morten Sonne, Microsoft MVP Q&A: Q1: Why don’t I see these recommendations? A: Not all workspaces are eligible for recommendations based on similar organizations. Workspaces only receive these recommendations if the machine learning model identifies significant similarities between your organization and others, and discovers tables that they have but you don’t. If no such similarities are identified, no extra recommendations are provided. You’re more likely to see these recommendations if your SOC is still in its onboarding process, rather than a more mature SOC. Q2: What makes an organization similar to mine? A: Similarity is determined based on ingestion trends, as well as your organization's industry and vertical, when available in our databases. Q3: Is any of my PII being used to make recommendations to other customers? A: No. The recommendations are generated using machine learning models that rely solely on Organizational Identifiable Information (OII) and system metadata. Customer log content is never accessed or analyzed, and no customer data, content, or End User Identifiable Information (EUII) is exposed during the analysis process. Microsoft prioritizes customer privacy and ensures that all processes comply with the highest standards of data protection. Looking forward Microsoft continues to use artificial intelligence and machine learning to help our customers defend against evolving threats and provide enhanced protection against cyberattacks. This ongoing innovation is a key part of SOC optimization’s commitment to help you maximize your value from your SIEM & XDR. Learn More: SOC optimization documentation: SOC optimization overview ; Recommendation's logic Short overview and demo: SOC optimization Ninja show In depth webinar: Manage your data, costs and protections with SOC optimization SOC optimization API: Introducing SOC Optimization API | Microsoft Community Hub
