Cribl o Logstash vs AMA CEF: What’s the Best Choice for Ingesting Firewall Logs?
Hi everyone, what are the advantages of using Cribl or Logstash over a CEF log collector via AMA for ingesting firewall logs such as Palo Alto for example into Microsoft Sentinel? In a typical scenario, how would you configure the ingestion to optimize performance, scalability, and cost? What do you think? Let’s discuss and share experiences!Microsoft Cost Management updates—February 2025 (summary)
Here's a quick run-down of the Cost Management updates for February 2025: Cost details datasets now include AccountId and InvoiceSectionId columns to support more cost allocation scenarios. Note: These columns are already available in FOCUS exports. Copilot is now one click away from the Cost Management overview with new sample prompts that can help you get started with Copilot for Azure. Learn about the FinOps Open Cost and Usage Specification with the Learning FOCUS blog series. New ways to save money with Microsoft Cloud: Generally available: Changes to instance size flexibility ratios for Azure Reserved Virtual Machine Instances for M-series. Generally available: Azure NetApp Files now supports minimum volume size of 50 GiB. Public preview: Reduce costs with Hibernation in Azure DevTest Labs. Public preview: Troubleshoot disk performance using Microsoft Copilot in Azure. Public preview: Azure Monitor integrates performance diagnostics for enhanced VM troubleshooting. Public preview: Introducing the new AKS Monitoring Experience—Unified Insights at your fingertips. Documentation updates for Cost Management API modernization, programmatically creating MCA subscriptions, and more. This is just a quick summary. For the full details, please see Microsoft Cost Management updates—February 2025.
Understanding Cloud Cost Fluctuations with Power BI
Staying on top of your cloud costs requires regular reviews. There are many ways to slice and dice your cloud costs; one approach I find helpful is comparing daily and monthly cost deltas. Below is a visual from my Power BI report showing how my previous month’s costs compare to the month prior. The visual is filtered to only show delta increases/decreases over $1K. I can quickly see we spent $5K more on Azure SQL Database in the selected month compared to the previous month. I call this my 'large cost swings' graph. I understand that everything is not linear, nor do things translate nicely from one day or month to the next. However, the data has a story to tell. What I ask my team to focus on is the story the data is telling. In this case, we made some modifications to ADF and SQL, leading to a $4K net reduction in costs. Some stories explain the outcome of one or more actions. Then there are those stories which can help shape your future consumption and spending.Navigating Azure Retail Pricing Data in Power BI: My Journey
Recently, I embarked on integrating Azure Retail Pricing data into my Power BI Cost Management dashboard. Initially, this seemed daunting, but with the right orientation and assistance from Copilot, I successfully navigated through it. Did you know? Microsoft offers a Retail Pricing API that allows you to import data into Power BI in your preferred currency. The first hurdle I encountered was the API’s paginated results. To overcome this, I created a function in Power BI that iterates through the paginated results using a base URL. To my surprise, both the Retail Price table and the Azure Cost Management table had only one common column: meterID. This led to a many-to-many relationship, which is less than ideal as it introduces data ambiguity. Since there were multiple matching meterIDs with different retail prices, I addressed this by creating Measures. Additionally, I created another measure to calculate the retail cost, as the Retail Price table did not contain any consumption data. Happy to share more details if anyone's interested. #Azure #PowerBi #AzureCostManagement #AzureRetailPricingMoving to a Microsoft Customer Agreement (MCA) from an Enterprise Agreement (EA)
Microsoft has introduced a new contractual model for Azure enterprise customers, called the Microsoft Customer Agreement. Guidance on changes to the billing hierarchy can be found here: Set up billing for Microsoft Customer Agreement - Azure Microsoft Senior Cloud Solution Architect Dina Fatkulbayanova has written a great article on LinkedIn sharing her knowledge and recommendations of the transition process and billing hierarchy changes: Moving from EA to MCA: what you should know Check it out and please share with your networks on LinkedIn!Microsoft Cost Management updates—November 2024 (summary)
Here's a quick run-down of the Cost Management updates for November 2024: Export to Microsoft Fabric preview signup. New Azure OpenAI view in Cost analysis. Estimate costs for Azure OpenAI in Copilot for Azure. Build Cost analysis views in Copilot for Azure. New ways to save money with Microsoft cloud: Storage account default egress limit increase to 200 gbps. Azure backup reduced protected instance fees hana backup. Linux VM promotional offer. Autoscale in vCore based Azure Cosmos DB for MangoDB (Preview). Extended security updates enabled by Azure Arc. New capabilities to aid Migration and Hybrid Cloud Management (Preview). Documentation updates Azure OpenAI, Red Hat Linux, and Nutanix reservations. This is just a quick summary. For the full details, please see Microsoft Cost Management updates—November 2024.Azure VMWare (AVS) Cost Optimization Using Azure Migrate Tool
What is AVS? Azure VMware Solution provides private clouds that contain VMware vSphere clusters built from dedicated bare-metal Azure infrastructure. Azure VMware Solution is available in Azure Commercial and Azure Government. The minimum initial deployment is three hosts, with the option to add more hosts, up to a maximum of 16 hosts per cluster. All provisioned private clouds have VMware vCenter Server, VMware vSAN, VMware vSphere, and VMware NSX. As a result, you can migrate workloads from your on-premises environments, deploy new virtual machines (VMs), and consume Azure services from your private clouds. Learn More: https://learn.microsoft.com/en-us/azure/azure-vmware/introduction What is Azure Migrate Tool? Azure Migrate is a comprehensive service designed to help you plan and execute your migration to Azure. It provides a unified platform to discover, assess, and migrate your on-premises resources, including servers, databases, web apps, and virtual desktops, to Azure. The tool offers features like dependency analysis, cost estimation, and readiness assessments to ensure a smooth and efficient migration process. Learn More: https://learn.microsoft.com/en-us/azure/migrate/migrate-services-overview How Azure Migrate can be used to Discover and Assess AVS? Azure Migrate enables the discovery and assessment of Azure VMware Solution (AVS) environments by collecting inventory and performance data from on-premises VMware environments, either through direct integration with vCenter (via Appliance) or by importing data from tools like RVTools. Using Azure Migrate, organizations can analyze the compatibility of their VMware workloads for migration to AVS, assess costs, and evaluate performance requirements. The process involves creating an Azure Migrate project, discovering VMware VMs, and generating assessments that provide insights into resource utilization, right-sizing recommendations, and estimated costs in AVS. This streamlined approach helps plan and execute migrations effectively while ensuring workloads are optimized for the target AVS environment. Note: We will be narrating the RVtools Import method in this article. What Is RVTools? RVTools is a lightweight, free utility designed for VMware administrators to collect, analyze, and export detailed inventory and performance data from VMware vSphere environments. Developed by Rob de Veij, RVTools connects to vCenter or ESXi hosts using VMware's vSphere Management SDK to retrieve comprehensive information about the virtual infrastructure. Key Features of RVTools: Inventory Management: Provides detailed information about virtual machines (VMs), hosts, clusters, datastores, networks, and snapshots. Includes details like VM names, operating systems, IP addresses, resource allocations (CPU, memory, storage), and more. Performance Insights: Offers visibility into resource utilization, including CPU and memory usage, disk space, and VM states (e.g., powered on/off). Snapshot Analysis: Identifies unused or orphaned snapshots, helping to optimize storage and reduce overhead. Export to Excel: Allows users to export all collected data into an Excel spreadsheet (.xlsx) for analysis, reporting, and integration with tools like Azure Migrate. Health Checks: Identifies configuration issues, such as disconnected hosts, orphaned VMs, or outdated VMware Tools versions. User-Friendly Interface: Displays information in tabular form across multiple tabs, making it easy to navigate and analyze specific components of the VMware environment. Hand-on LAB Disclaimer: The data used for this LAB has no relationship with real world scenarios. This sample data is self-created by the author and purely for understanding the concept. To discover and assess your Azure VMware Solution (AVS) environment using an RVTools extract report in the Azure Migrate tool, follow these steps: Prerequisites RVTools Setup: Download and install RVTools from the Official Website Ensure connectivity to your vCenter server. Extract the data by running RVTools and saving the output as an Excel (.xlsx) file Permissions: You need at least the Contributor role on the Azure Migrate project. Ensure that you have appropriate permissions in your vCenter environment to collect inventory and performance data. File Requirements: The RVTools file must be saved in .xlsx format without renaming or modifying the tabs or column headers. Note: Sample Sheet: Please check the attachment included with this article. Note that this is not the complete format; some tabs and columns have been removed for simplicity. During the actual discovery and assessment process, please do not modify the tabs or columns. Procedure Step 1: Export Data from RVTools Follow the steps provided in official website to get RVTools Extract Sample Sheet: Please check the attachment included with this article. Note that this is not the complete format; some tabs and columns have been removed for simplicity. During the actual discovery and assessment process, please do not modify the tabs or columns. Step 2: Discover Log in to the Azure portal. Navigate to Azure Migrate and select your project or create new project. Under Migration goals, select Servers, databases and web apps. On Azure Migrate | Servers, databases and web apps page, under Assessment tools, select Discover and then select Using import. In Discover page, in File type, select VMware inventory (RVTools XLSX). In the Step 1: Import the file section, select the RVTools XLSX file and then select Import. Wait for some time to Import Once import completed check for Error Messages if any and rectify those and re upload, otherwise wait 10-15 minutes to reflect imported VMs in the discovery. Post discovery Reference Link: https://learn.microsoft.com/en-us/azure/migrate/vmware/tutorial-import-vmware-using-rvtools-xlsx?context=%2Fazure%2Fmigrate%2Fcontext%2Fvmware-context Step 3: Assess After the upload is complete, navigate to the Servers tab. Click on Assess -->Azure VMware Solution to assess the discovered machines. Edit assessment settings based on your requirements and Save Target region: Select the Azure region for the migration. Node Type: Specify the Azure VMware Solution series (e.g., AV36, AV36P). Pricing model: Select pay-as-you-go or reserved instance pricing. Discount: Specify any available discounts. Note: We will be explaining all the parameters in optimize session. As of now just review and leave parameters as it is. In Assess Servers, select Next. In Select servers to assess > Assessment name > specify a name for the assessment. In Select or create a group > select Create New and specify a group name. Select the appliance and select the servers you want to add to the group. Then select Next. In Review + create assessment, review the assessment details, and select Create Assessment to create the group and run the assessment. Step 4: Review the Assessment View an assessment In Windows, Linux and SQL Server > Azure Migrate: Discovery and assessment, select the number next to Azure VMware Solution. In Assessments, select an assessment to open it. As an example (estimations and costs, for example, only): Review the assessment summary. You can select Sizing assumptions to understand the assumptions that went in node sizing and resource utilization calculations. You can also edit the assessment properties or recalculate the assessment. Step 5: Optimize We have received a report without any optimization in our previous steps. Now we can follow below steps to optimize the cost and node count even further High level steps: Find limiting factor Find which component in settings are mapped for optimization depending on limiting factor Try to adjust the mapped component according to Scenario and Comfort Find Limiting factor: First understand which component (CPU, memory and storage) is deciding your ESXI Node count. This will be highlighted in the report The limiting factor shown in assessments could be CPU or memory or storage resources based on the utilization on nodes. It is the resource, which is limiting or determining the number of hosts/nodes required to accommodate the resources. For example, in an assessment if it was found that after migrating 8 VMware VMs to Azure VMware Solution, 50% of CPU resources will be utilized, 14% of memory is utilized and 18% of storage will be utilized on the 3 Av36 nodes and thus CPU is the limiting factor. Find which option in the setting can be used to optimize: This is depending on the limiting factor. For eg: If Limiting factor is CPU, which means you have high CPU requirement and CPU oversubscription can be used to optimize ESXI Node. Likewise, if storage is the limiting factor editing FTT, RAID or introducing External storage like ANF will help you to reduce Node count. Even reducing one node count will create a huge impact in dollar value. Let's understand how over commitment or over subscription works with simple example. Let's suppose I have two VMs with below specification Name CPU Memory Storage VM1 9 vCPU 200 GB 500 GB VM2 4 vCPU 200 GB 500 GB Total 13 vCPU 400 GB 1000 GB We have EXSI Node which has below capacity: vCPU 10 Memory 500 GB storage 1024 GB Now without optimization I need two ESXI node to accommodate 13 vCPU of total requirement. But let's suppose VM1 and VM2 doesn't consume entire capacity all the time. The total capacity usage at a time will not go beyond 10. then I can accommodate both VM in same ESXI node, Hence I can reduce my node count and cost. Which means it is possible to share resources among both VMs. Without optimization With optimization Parameters effecting Sizing and Pricing CPU Oversubscription Specifies the ratio of number of virtual cores tied to one physical core in the Azure VMware Solution node. The default value in the calculations is 4 vCPU:1 physical core in Azure VMware Solution. API users can set this value as an integer. Note that vCPU Oversubscription > 4:1 may impact workloads depending on their CPU usage. Memory overcommit factor Specifies the ratio of memory overcommit on the cluster. A value of 1 represents 100% memory use, 0.5, for example is 50%, and 2 would be using 200% of available memory. You can only add values from 0.5 to 10 up to one decimal place. Deduplication and compression factor Specifies the anticipated deduplication and compression factor for your workloads. Actual value can be obtained from on-premises vSAN or storage configurations. These vary by workload. A value of 3 would mean 3x so for 300GB disk only 100GB storage would be used. A value of 1 would mean no deduplication or compression. You can only add values from 1 to 10 up to one decimal place. FTT : How many device failure can be tolerated for a VM RAID : RAID stands for Redundant Arrays of Independent Disks Explains how data should be stored for redundancy Mirroring : Data will be duplicated as it is to another disk E.g.: To protect a 100 GB VM object by using RAID-1 (Mirroring) with an FTT of 1, you consume 200 GB. Erasure Coding : Erasure coding divides data into chunks and calculates parity information (redundant data) across multiple storage devices. This allows data reconstruction even if some chunks are lost, similar to RAID, but typically more space-efficient E.g.: to protect a 100 GB VM object by using RAID-5 (Erasure Coding) with an FTT of 1, you consume 133.33 GB. Comfort Factor: Azure Migrate considers a buffer (comfort factor) during assessment. This buffer is applied on top of server utilization data for VMs (CPU, memory and disk). The comfort factor accounts for issues such as seasonal usage, short performance history, and likely increases in future usage. For example, a 10-core VM with 20% utilization normally results in a 2-core VM. However, with a comfort factor of 2.0x, the result is a 4-core VM instead. AVS SKU Sizes Optimization Result In this example we got to know that CPU is my limiting factor hence I have adjusted CPU over subscription value from 4:1 to 8:1 Reduced node count from 6 (3 AV36P+3 AV64) to 5 AV36P Reduced Cost by 31% Note: Over-provisioning or over-committing can put your VMs at risk. However, in Azure Cloud, you can create alarms to warn you of unexpected demand increases and add new ESXi nodes on demand. This is the beauty of the cloud: if your resources are under-provisioned, you can scale up or down at any time. Running your resources in an optimized environment not only saves your budget but also allows you to allocate funds for more innovative ideas.
