Optimizing your Hyper-V hosts
The good news is that Hyper-V is designed to provide optimal performance and functionality based on the dynamics of what’s running on the host and the needs of the VMs regarding memory and CPU resources. In most cases, running with the defaults will work great. For those cases where some tuning to optimize for host CPU or live migration is desired, this blog will cover some opportunities to adjust your Hyper-V systems. Overview of CPU Scheduling Before we get into the ways to configure and manage Hyper-V there are a few key concepts that are important to understand. For more details, reference: Manage Hyper-V hypervisor scheduler types | Microsoft Learn Physical CPUs and Logical Processors Modern systems allow for multiple physical CPUs, with each CPU having multiple Logical Processors (LPs). LPs are what the host and VMs virtual processors map to for thread execution. Here is an outline of how CPUs, Cores, and Logical Processors relate: A host has X number of sockets in which a CPU plugs into Each CPU has X number of cores Each Core has one or more Logical Processors (LP) o Simultaneous Multithreading (SMT) allows for 2 LPs per Core, if enabled. If not enabled, there is 1 LP per core. Diagram 1: Showing the host having a single Socket and the CPU with 4 cores. With SMT enabled, this will show 2 logical processors per core. Note, the diagram above also shows that each core has 2 LPs, and the virtual processors are mapped to LPs of the Core. A core will not execute threads on its LPs from VPs of different VMs. This is why it’s recommended to configure VMs with an even number of VPs. The image below shows how this configuration looks in Task Manager. It’s an easy way to confirm that SMT is configured, there are twice as many logical processors as there are cores: This is the powershell to get the same information: PS C:\Users\Administrator> (Get-WmiObject -Class Win32_Processor) | fl NumberOfEnabledCore,NumberOfLogicalProcessors NumberOfEnabledCore : 4 NumberOfLogicalProcessors : 8 Virtual Processors and Scheduling Part of the configuration of a VM is the number of Virtual Processors (VPs) it has. As the name implies, the operating system in the VM sees these as CPUs to execute threads on. The scheduler then manages execution on an LP. The sum of the VPs configured for the running VMs can be greater than the available LPs. To manage this, there is a “scheduler” in the hypervisor or host that manages which VP can execute on which LP. Diagram 2: Showing a configuration where there are more Virtual Processors than Logical Processors. The scheduler will assign which VPs each set of LPs in a core will execute at any moment in time. Because there are more VP’s than LPs in the above example, when the other VPs have threads waiting to execute it will stop the execution of LPs of some cores switch them to execution for waiting VPs, so another moment in time for the same system might look like this: Diagram 3: Showing the same configuration as diagram 2, but in a different moment in time where there is a difference in the Virtual Processor to Logical Processor assignment For the purposes of this blog, most examples will be specific to the “Core Scheduler” with SMT enabled on the system, unless otherwise noted. The Core Scheduler is the default and recommended option for Windows Server and Azure Local. For more information on the different schedulers refer to: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types#hypervisor-scheduler-types Optimize Host and VM CPU by Dedicating CPUs to the Host The host OS is running processes critical to the system, like managing live migrations for VMs, hosting services like the Failover Cluster Service, and managing virtual machines’ resources. If the host is showing high CPU utilization, it may be that the host is impacted by the CPU demand of the VMs. Dedicating CPUs for the host’s use can alleviate the impact of contention between its processes and the resources that the VMs it’s hosting require. This separation can be beneficial to the VMs also, as it keeps host processes from causing contention with the VMs CPU scheduling. Using MinRoot MinRoot is a feature that limits CPUs that are used for the host. This article describes, in detail, how it works and how to configure MinRoot. Essentially, a BCDedit command is used to specify the number of Virtual Processors (VPs) to assign to the Host. It assigns a Logical Processor (LPs) directly to each host VP for the host’s use. The remaining LPs of the system will be available to VMs without the host VPs being scheduled on them. The LPs assigned to the host may share time with VPs from VMs. The MinRoot setting will become effective once the host is restarted. Diagram 4: Showing a configuration where MinRoot is used to dedicate 2 logical processors to the host. The other 6 logical processors are shared between the VMs Choosing the number of LPs dedicated to the host is similar to the process of identifying the number of VPs to assign to a VM. It depends on what’s running inside of it (its workloads). Common variables include the CPU load from networking (RDMA enabled networks can help reduce this) and other software that has CPU demands. The CPU utilization from the OS may increase with the increase of the number or type of VMs running on it. Loads are specific to each deployment and often change over time, so occasional monitoring of host CPU utilization, particularly at times of peak loads, is recommended. This article has some helpful information on how to use Performance Monitor, which is a tool that is built into Windows Server and Azure Local and Windows Client, which is commonly used to analyze CPU utilization. While the article references Microsoft SQL Server, the premise is the same for general CPU utilization monitoring. Optimizing Live Migration Live migration (LM) is used to load balance VMs between hosts and to facilitate host maintenance with limited impact on the workloads running inside of the VMs. For example, Failover clusters use LM to automatically move VMs for activities like updating hosts using Cluster Aware Updating (CAU). More information: Live Migration Overview | Microsoft Learn Limiting the interruption of network communications and CPU while a VM it is moving from one host to another is often important for workloads that have low “jitter” requirements. Hyper-V’s Live Migration has been optimized over the years for speed and reliability. The initial phase of a live migration copies the VM state from the source host to the destination host using multiple passes, since the VMs are in use and the memory and processor state are continuously changing. Once the delta between the source and destination state is minimal, the source VM is paused, and the final memory and state copy is completed. It is at that point that the destination VM is resumed and becomes active. This final phase is often completed in under a second, but the time can depend on the amount of CPU/Memory churn caused by what’s running in the VM, and network bandwidth and latency that is used to transfer the state. Here are some settings that can optimize live migration. It’s also good to remember that if the hosts are part of a failover cluster, all nodes of the cluster should be configured identically. Number of Simultaneous Live Migrations The number of LM sessions that a host can run at the same time, whether source or destination, is controlled by setting the -MaximumVirtualMachineMigrations parameter. Live migration uses host resources like CPU and networking. If the goal is to reduce the time for individual VMs to be live migrated, then a lower number is better. If the goal is to take a set of VMs and live migrate them all in the shortest amount of time, then a higher number might be better. However, since VMs are constantly changing memory state because they are running and active, simultaneously live migrating more VMs might actually take more overall time to accomplish. The recommendation is to keep this number low and analyze the effect on the LM time and the system resources when trying higher settings, to find the sweet spot for your system. To get the value: Get-VMHost -MaximumVirtualMachineMigrations To set the value to 2: Set-VMHost -MaximumVirtualMachineMigrations 2 Reminder: ensure this setting is the same across all nodes of a failover cluster Host Networking Considerations Live migrations use a network connection between the source and destination nodes. Optimizing the network configuration can allow the initial and final memory and state transfers to complete faster. RDMA and SMB RDMA is a network offload that reduces host CPU overhead for the network traffic, while increasing the throughput. It requires NICs on the hosts and switches that the traffic flows to support RDMA and to be properly configured. When enabled, the same network traffic can have as much as 20% higher throughput (more data per second) and 20% less CPU usage. Therefore, the live migration will be completed faster with less system CPU impact. RDMA does require the servers to have NICs that have the capability, and the proper configuration for the NICs and switches that the network traffic flows through between the source/destination servers. RDMA configuration is specific to the NICs and switches in the environment, so details are not in-scope for this blog. If the source/destination systems for a live migration are configured to allow RDMA networking, live migration needs to be set to use SMB as the transport protocol. Set live migration to use SMB: Live migration can transfer data between nodes using 3 different mechanisms. The default is “TCP/IP” which uses a Hyper-V managed connection between the source and destination systems. “Compression” is useful to reduce the amount of data that is transmitted over the networking by compressing it. Compression will use more host CPU to do the compression and decompression on each node. This is often the best option for scenarios like doing live migration over a WAN where bandwidth is limited or charged. “SMB” allows the data to be transmitted using SMB optimizations like SMB Direct, which is what allows the LM traffic to go over RDMA enabled connections. If the nodes have more than one network path with the same characteristics (throughput, RDMA/Non-RDMA), SMB will manage using all of them using SMB Mulitchannel. To change the setting, use Set-VMHost from a PowerShell Console, opened as Administrator. This is the command to use to change to SMB. Ensure this is set consistently all the nodes of a failover cluster. Set-VMHost –VirtualMachineMigrationPerformanceOption SMB Set SMB bandwidth limit for Live Migration Live migration will use as much of the network bandwidth as possible, which can starve other components that may need to use the same connection, like failover cluster traffic. When the system is set to use SMB for live migration, it is recommended to set a limit for live migration traffic. The amount of bandwidth to limit live migration depends on the available bandwidth between the source and destination systems. For example, if there are 2 connections between the host and destination which are RDMA capable and have a throughput of 10Gb (Gigabit), the aggregate would be 20Gb. A general recommendation would be to limit live migration traffic to ~50%, which would be 10Gb. While a good starting point, if live migrations are too slow the limit could be increased. Likewise, if other components of the system, like Failover Cluster and Cluster Shared Volumes, are having issues during live migration, reducing the limit for Live Migration may be needed. To set the bandwidth limit for live migration traffic over SMB (example value of 10Gb): Install the SMB bandwidth limit feature Add-WindowsFeature -Name FS-SMBBW Then set the limit for LiveMigration Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 10GB Resources: Virtual Machine Resource Controls | Microsoft Learn Live Migration Overview | Microsoft Learn Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure - Microsoft Community Hub https://learn.microsoft.com/en-us/troubleshoot/windows-server/virtualization/troubleshoot-live-migration-guidance Set up hosts for live migration without Failover Clustering | Microsoft Learn Monitor CPU Usage - SQL Server | Microsoft Learn
