Blog Post

ITOps Talk Blog
3 MIN READ

Troubleshooting Azure Arc for Servers - Status Offline

SoniaCuff's avatar
SoniaCuff
Icon for Microsoft rankMicrosoft
Apr 28, 2020

While trying out the capabilities of Azure Arc for Servers, I came across a couple of times when the on-premises virtual machine showed as ‘Offline’ in the Azure Portal. To help troubleshoot this, there’s a command line tool that can help reconnect the machine to Azure, called azcmagent.exe. 

 

1. Check the machine state

My experience as a systems administrator have ingrained in me the approach to check the simple things first. In this case, is the virtual machine actually running?

 

If you have a way to verify connecting to or getting a response from the server another way, do that first. Many things can cause a virtual machine to stop responding, including issues with the host. So we want to first check that the server is running and otherwise healthy.

 

2. Check the Azure Arc for Servers state
Azure Arc for Servers uses an agent known as the Connected Machine Agent. If installed successfully, you can find it listed in Add or remove programs:

It runs two services: the Azure Hybrid Instance Metadata Service and the Guest Configuration Service.
From a command prompt, we can use the azcmagent command to check the status of the agent:

 

 

 

azcmagent show

 

 

 

 

With the error below, the root cause was that one of the services was not running.


So, maybe the program was accidentally uninstalled, is being blocked by aggressive third party security software, or maybe the server has other underlying issues preventing the service from running.


Not pictured, it’s also possible to get an Agent Status: Disconnected response. This indicates that things are functioning from an agent perspective, but it’s not communicating with Azure. Our challenge is to re-establish that connection or figure out what’s preventing it.

 

3. Double check the prerequisites
Azure Arc for Servers has some pre-requisites, including supported operating systems, supported Azure regions and Azure subscription and service limits. If you’ve previously onboarded and connected a machine successfully, make sure none of those requirements listed here, have changed.


There are no special network ports required, as Azure Arc for Servers uses TCP Port 443. If your firewall is very restrictive, you’ll want to check you’re still allowed access to the required Azure URLs (preferably) or IP ranges and service tags. For details, see Networking Configuration.

 

4. Disconnect and Connect
Another use of the azcmagent tool is to remove a server and then to connect it again, even if the Azure Arc Connected Machine resource has been deleted from Azure. This does not remove or reinstall the agent and its services on the virtual machine. Both parameters do require you to provide valid credentials, in the way the disconnect method does.

 

Note: If you leave out the service principal ID and secret components, Windows will give you the interactive browser login experience to provide authentication credentials. Learn more about the authentication options.

 

This disconnect and connect method is also recommended for machines that have not sent a heartbeat signal to Azure in more than 45 days, after which they will automatically be deregistered.

 

 

azcmagent disconnect --service-principal-id <serviceprincipalAppID> --service-principal-secret <serviceprincipalPassword> --tenant-id <tenantID>
azcmagent connect --service-principal-id <serviceprincipalAppID> --service-principal-secret <serviceprincipalPassword> --tenant-id <tenantID> --subscription-id <subscriptionID> --resource-group <ResourceGroupName> --location <resourceLocation>

 

 

 

 

 

 

Learn more:
For more information, visit Managing and maintaining the Connected Machine agent.

 

-SCuffy



 

 

 

Updated Jul 29, 2024
Version 2.0
azure
Sonia Cuff
  • SoniaCuff's avatar
    SoniaCuff
    Icon for Microsoft rankMicrosoft
    Jul 29, 2024

    Carmine7  Cgoff1195  Thanks for your comments. This article has now had the -reconnect option removed since it has been deprecated, and the product group have confirmed the official guidance of using disconnect and connect instead.

  • Karl-WE's avatar
    Karl-WE
    MVP
    Jul 09, 2024

    agree with Cgoff1195  SoniaCuff this artcile could require and updated version, if you have time

    Check Status 

    Identify disconnected servers >Azure Portal > Azure Update Manager > sort by column "Status"
    click on the disconnected server in the first column (alternatively Navigate to the Arc Resource in Azure Portal (top search bar enter DNS name))

    Check connection status, should look like this. if it is giving the error about attestation service HIDS, and (re)connected does not work consider the next steps below.

     

    cd C:\Program Files\AzureConnectedMachineAgent
.\azcmagent.exe check

. This might take a few minutes."
Use Case  |Endpoint                                              |Reachable  |Private  |TLS      |Proxy
core      |https://agentserviceapi.guestconfiguration.azure.com  |true       |false    |TLS 1.2  |not used
core      |https://gbl.his.arc.azure.com                         |true       |false    |TLS 1.3  |not used
core      |https://login.microsoftonline.com                     |true       |false    |TLS 1.3  |not used
core      |https://management.azure.com                          |true       |false    |TLS 1.3  |not used
core      |https://pas.windows.net                               |true       |false    |TLS 1.3  |not used
core      |https://westeurope-gas.guestconfiguration.azure.com   |true       |false    |TLS 1.2  |not used
core      |https://weu.his.arc.azure.com                         |true       |false    |TLS 1.3  |not used

     



    Reconnect:

    • Identify disconnected servers >Azure Portal > Azure Update Manager > sort by column "Status"
    • click on the disconnected server in the first column (alternatively Navigate to the Arc Resource in Azure Portal (top search bar enter DNS name))
    • (Open Server Manager) > Remote PowerShell > Connect to affected server using the copy icon and paste into the PowerShell

       


      Copy the data from the Azure Object 

     

    cd C:\Program Files\AzureConnectedMachineAgent
.\azcmagent.exe connect -l location -g resourcegroupname -s subscriptionID

     

     

     

    If this does not work, this is the fastest way to fix it.

    1. Connect to the machine using Windows Admin Center > Apps > uninstall Connected Machine agent

    2. in Azure Portal note the resource group and tags

    3. delete the Arc object

    4. onboard the machine again using Windows Admin Center

     

  • Nickolas_McKibben's avatar
    Nickolas_McKibben
    Copper Contributor
    Jun 13, 2024

    We are finding that if a on-prem Windows 2012 server that we have set up in Azure for ESU, is showing offline in Azure, all we have to do is to RDP to the server and then wait several minutes and the server will show "online" in Azure and when you run the azcmagent show command on the server, it shows connected.

     

    We are not really sure why just RDPing to the server seems to fix the offline status in Azure, but it works. Can anyone explain this.

  • SoniaCuff's avatar
    SoniaCuff
    Icon for Microsoft rankMicrosoft
    Jan 15, 2024

    Nice write up, LMHutch  and thanks for the cross-link!
    When I started in tech, it used to frustrate me that sometimes "uninstall and resinstall" is the answer, especially when it was a full clean uninstall with manual removal of components. But the fact is that sometimes something can go wrong during an installation that isn't captured by a log or would take too long (or be impossible) to find as the root cause. The goal is to get up and running as quickly as possible and sometimes trying the process again from scratch is the most efficient way to do that!