Typical Storage access issues troubleshooting

We get a big number of cases with Storage Account connection failing and sometimes we see that our customers are not aware of the troubleshooting steps they can take to accelerate the resolution of this issue.

 

As such, we've compiled some scenarios and the usual troubleshooting steps we ask you to take.

Always remember that if you have done changes to your infrastructure, consider rolling them back to ensure that this is not the root cause. Even a small change that apparently has no effect, may cause downtime on your application.

Common messages

The errors that are shown in the portal when the Storage Account connectivity is down are very similar, and they may not indicate correctly the cause.

 

 

The errors don't really indicate what the root cause is, but it's very common to be a broken connection with the Storage.

What to verify?

There are 4 major components to verify in these cases:

• Logic App environment variables and network settings
• Storage Account networking settings
• Network settings
• DNS settings

Logic App environment variables and Network

From an App Settings point of view, there is not much to verify, but these are important steps, that sometimes are overlook.

 

At this time, all or nearly all Logic Apps have been migrated to dotnet Functions_Worker_Runtime (under Environmental Variables tab), but this is good to confirm. It's also good to confirm if your Platform setting is set to 64 bits (under Configuration tab/ General Settings). We've seen that some deployments are using old templates and setting this as 32 bits, which doesn't make full use of the available resources.

 

Check if Logic App has the following environment variables with value:

• WEBSITE_CONTENTOVERVNET - set to 1 OR WEBSITE_VNET_ROUTE_ALL - set to 1.
• OR vnetRouteAllEnabled set to 1.

Configure virtual network integration with application and configuration routing. - Azure App Service | Microsoft Learn

 

These settings can also be replaced with the UI setting in the Virtual Network tab, when you select "Content Storage" in the Configuration routing.

For better understanding, vnetContentShareEnabled takes precedence.

 

In other words, if it is set (true/false), WEBSITE_CONTENTOVERVNET is ignored. Only if vnetContentShareEnabled is null, WEBSITE_CONTENTOVERVNET is taken into account.

 

Also keep this in mind:

Storage considerations for Azure Functions | Microsoft Learn

• WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and AzureWebJobsStorage have the connection string as in the Storage Account
• Website_contentazurefileconnectionstring | App settings reference for Azure Functions | Microsoft Learn
• Azurewebjobsstorage | App settings reference for Azure Functions | Microsoft Learn

 

• WEBSITE_CONTENTSHARE has the Fileshare name
• Website_contentshare | App settings reference for Azure Functions | Microsoft Learn

These are the first points to validate.

Storage Account settings

If all these are matching/properly configured and still the Logic App is in error, we move to the next step, that is to validate the Storage Account network settings.

 

When the Storage Account does not have Vnet integration enabled, there should be no issues, because the connection is made through the public endpoints.

Still, even with this, you must ensure that at least the "Allow storage account key access" is enabled.

 

This is because at this time, the Logic App is dependent on the Access key to connect to the Storage Account. Although you can set the AzureWebJobsStorage to run with Managed Identity, you can't fully disable storage account key access for Standard logic apps that use the Workflow Service Plan hosting option.

 

However, with ASE v3 hosting option, you can disable storage account key access after you finish the steps to set up managed identity authentication.

Create example Standard workflow in Azure portal - Azure Logic Apps | Microsoft Learn

 

If this setting is enabled, you must check if Storage Account is behind Firewall. The Access may be Enabled for select networks or fully disabled. Both options require Service Endpoints or Private Endpoints configured.

Deploying Standard Logic App to Storage Account behind Firewall using Service or Private Endpoints | Microsoft Community Hub

 

So check the Networking tab under the Storage Account and confirm the following:

 

In case you select the "selected networks" option, confirm that the VNET is the same as the Logic App is extended to. Your Logic App and Storage may be hosted in different Vnets, but you must ensure that there is full connectivity between them. They must be peered and with HTTPS and SMB traffic allowed (more explained in the Network section).

You can select "Disabled" network access as well.

 

You should also confirm that the Fileshare is created. Usually this is created automatically with the creation of the Logic App, but if you use Terraform or ARM, it may not create the file share and you must do it manually.

 

Confirm if all 4 Private Endpoints are created and approved (File, Table, Queue and Blob). All these resources are used for different components of the Logic App. This is not fully documented, as it is internal engine documentation and not publicly available.

 

For Azure Functions, the runtime base, this is partially documented, as you can read in the article:

Storage considerations for Azure Functions | Microsoft Learn

 

If a Private Endpoint is missing, create it and link it to the Vnet as Shared Resource. Not having all Private Endpoints created may end in runtime errors, connections errors or trigger failures. For example, if a workflow is not generating the URL even if it saves correctly, it may be the Table and Queue Private Endpoints missing, as we've seen many times with customers.

 

You can read a bit more about the integration of the Logic App and a firewall secured Storage Account and the needed configuration in these articles:

Secure traffic between Standard workflows and virtual networks - Azure Logic Apps | Microsoft Learn

Deploy Standard logic apps to private storage accounts - Azure Logic Apps | Microsoft Learn

 

You can use the Kudu console (Advanced tools tab) to further troubleshoot the connection with the Storage Account by using some network troubleshooting commands. If the Kudu console is not available, we recommend using a VM in the same Vnet as the Logic App, to mimic the scenario.

 

Nslookup [hostname or IP] [DNS HOST IP]

TCPPing [hostname or IP]:[PORT]

Test-Netconnection [hostname] -port [PORT]

 

If you have Custom DNS, the command NSLookup will not return the results from your DNS unless you specify the IP address as a parameter. Instead, you can use the nameresolver command for this, which will use the Vnet DNS settings to check for the endpoint name resolution.

 

nameresolver [endpoint hostname or IP address]

Networking Related Commands for Azure App Services | Microsoft Community Hub

Vnet configuration

Having configured the Private Endpoint for the Logic App will not affect traffic to the Storage. This is because the PE is only for Inbound traffic. The Storage Communication will the considered as outbound traffic, as it's the Logic App that actively communicates with the Storage.

 

Secure traffic between Standard workflows and virtual networks - Azure Logic Apps | Microsoft Learn

 

So consider that the link between these resources must not be interrupted. This forces you to understand that the Logic App uses both HTTPS and SMB protocols to communicate with the Storage Account, meaning that traffic under the ports 443 and 445 needs to be fully allowed in your Vnet.

If you have a Network Security Group associated with the Logic App subnet, you need to confirm that the rules are allowing this traffic. You may need to explicitly create rules to allow this.

 

 

In case you have forced routing to your Network Virtual Appliance (i.e. Firewall), you must also ensure that this resource is not filtering the traffic or blocking it. Having TLS inspection enabled in your Firewall must also be disabled, for the Logic App traffic.

 

In short, this is because the Firewall will replace the certificate in the message, thus making the Logic App not recognizing the returned certificate, invalidating the message.

 

You can read more about TLS inspection in this URL:

Azure Firewall Premium features | Microsoft Learn

DNS

If you are using Azure DNS, this section should not apply, because all records are automatically created once you create the resources, but if you're using a Custom DNS, when you create the Azure resource (ex: Storage Private Endpoint), the IP address won't be registered in your DNS, so you must do it manually.

 

You must ensure that all A Records are created and maintained, also keeping in mind that they need to point to the correct IP and name. If there are mismatches, you may see the communications severed between the Logic App and other resources, such as the Storage Account.

 

So double-check all DNS records, and confirm that all is in proper state and place.

 

And to make it even easier, with the help of my colleague Mohammed_Barqawi , this information is now translated into a easy to understand flowchart.

If you continue to have issues after all these steps are verified, I suggest you open a case with us, so that we can validate what else may be happening, because either a step may have been missed, or some other issue may be occurring.