Securely Integrating Azure API Management with Azure OpenAI via Application Gateway

Sabyasachi-Samaddar

Microsoft

Feb 25, 2025

Introduction

As organizations increasingly integrate AI into their applications, securing access to Azure OpenAI services becomes a critical priority. By default, Azure OpenAI can be exposed over the public internet, posing potential security risks. To mitigate these risks, enterprises often restrict OpenAI access using Private Endpoints, ensuring that traffic remains within their Azure Virtual Network (VNET) and preventing direct internet exposure.

However, restricting OpenAI to a private endpoint introduces challenges when external applications, such as those hosted in AWS or on-premises environments, need to securely interact with OpenAI APIs. This is where Azure API Management (APIM) plays a crucial role. By deploying APIM within an internal VNET, it acts as a secure proxy between external applications and the OpenAI service, allowing controlled access while keeping OpenAI private.

To further enhance security and accessibility, Azure Application Gateway (App Gateway) can be placed in front of APIM. This setup enables secure, policy-driven access by managing traffic flow, applying Web Application Firewall (WAF) rules, and enforcing SSL termination if needed.

What This Blog Covers

This blog provides a technical deep dive into setting up a fully secure architecture that integrates Azure OpenAI with APIM, Private Endpoints, and Application Gateway. Specifically, we will walk through:

Configuring Azure OpenAI with a Private Endpoint to restrict public access and ensure communication remains within a secure network.
Deploying APIM in an Internal VNET, allowing it to securely communicate with OpenAI while being inaccessible from the public internet.
Setting up Application Gateway to expose APIM securely, allowing controlled external access with enhanced security.
Configuring VNET, Subnets, and Network Security Groups (NSGs) to enforce network segmentation, traffic control, and security best practices.

By the end of this guide, you will have a production-ready, enterprise-grade setup that ensures:

End-to-end private connectivity for Azure OpenAI through APIM.
Secure external access via Application Gateway while keeping OpenAI hidden from the internet.
Granular network control using VNET, Subnets, and NSGs.

This architecture provides a scalable and secure solution for enterprises needing to expose OpenAI securely without compromising privacy, performance, or compliance.

Prerequisites

Before diving into the integration of Azure API Management (APIM) with Azure OpenAI in a secure, private setup, ensure you have the following in place:

1. Azure Subscription & Required Permissions

An active Azure Subscription with the ability to create resources.
Contributor or Owner access to deploy Virtual Networks (VNETs), Subnets, Network Security Groups (NSGs), Private Endpoints, APIM, and Application Gateway.

2. Networking Setup Knowledge

Familiarity with Azure Virtual Network (VNET) concepts, Subnets, and NSGs is helpful, as we will be designing a controlled network environment.

3. Required Azure Services

The following services are needed for this integration:

Azure Virtual Network (VNET) – To establish a private, secure network.
Subnets & NSGs – For network segmentation and traffic control.
Azure OpenAI Service – Deployed in a region that supports private endpoints.
Azure API Management (APIM) – Deployed in an Internal VNET mode to act as a secure API proxy.
Azure Private Endpoint – To restrict Azure OpenAI access to a private network.
Azure Application Gateway – To expose APIM securely with load balancing and optional Web Application Firewall (WAF).

4. Networking and DNS Requirements

Private DNS Zone: Required to resolve private endpoints within the VNET.
Custom DNS Configuration: If using a custom DNS server, ensure proper forwarding rules are in place.
Firewall/NSG Rules: Ensure necessary inbound and outbound rules allow communication between services.

5. Azure CLI or PowerShell (Optional, but Recommended)

Azure CLI (az commands) or Azure PowerShell for efficient resource deployment.

Once you have these prerequisites in place, we can proceed with designing the secure architecture for integrating Azure OpenAI with APIM using Private Endpoints and Application Gateway.

Architecture Overview

The architecture ensures secure and private connectivity between external users and Azure OpenAI while preventing direct public access to OpenAI’s APIs. It uses Azure API Management (APIM) in an Internal VNET, an Azure Private Endpoint for OpenAI, and an Application Gateway for controlled public exposure.

Key Components & Flow

User Requests
- External users access the API via a public endpoint exposed by Azure Application Gateway.
- The request passes through App Gateway before reaching APIM, ensuring security and traffic control.
Azure API Management (APIM) – Internal VNET Mode
- APIM is deployed in Internal VNET mode, meaning it does not have a public endpoint.
- APIM serves as a proxy between external applications and Azure OpenAI, ensuring request validation, rate limiting, and security enforcement.
- The Management Plane of APIM still requires a public IP for admin operations, but the Data Plane (API traffic) remains fully private.
Azure Private Endpoint for OpenAI
- APIM cannot access Azure OpenAI publicly since OpenAI is secured with a Private Endpoint.
- A Private Endpoint allows APIM to securely connect to Azure OpenAI within the same VNET, preventing internet exposure.
- This ensures that only APIM within the internal network can send requests to OpenAI.
Managed Identity Authentication
- APIM uses a Managed Identity to authenticate securely with Azure OpenAI.
- This eliminates the need for hardcoded API keys and improves security by using Azure Role-Based Access Control (RBAC).
Application Gateway for External Access
- Since APIM is not publicly accessible, an Azure Application Gateway (App Gateway) is placed in front of it.
- App Gateway acts as a reverse proxy that securely exposes APIM to the public while enforcing:
  - SSL termination for secure HTTPS connections.
  - Web Application Firewall (WAF) for protection against threats.
  - Load balancing if multiple APIM instances exist.
Network Segmentation & Security
- VNET & Subnets: APIM, OpenAI Private Endpoint, and App Gateway are deployed in separate subnets within an Azure Virtual Network (VNET).
- NSGs (Network Security Groups): Strict inbound and outbound rules ensure that only allowed traffic flows between components.
- Private DNS: Required to resolve Private Endpoint addresses inside the VNET.

Security Enhancements

No direct internet access to Azure OpenAI, ensuring full privacy.
Controlled API exposure via App Gateway, securing public requests.
Managed Identity for authentication, eliminating hardcoded credentials.
Private Endpoint enforcement, blocking unwanted access from external sources.

This architecture ensures that Azure OpenAI remains secure, APIM acts as a controlled gateway, and external users can access APIs safely through App Gateway.

Azure CLI Script for VNet, Subnets, and NSG Configuration

# Variables
RESOURCE_GROUP="apim-openai-rg"
LOCATION="eastus"
VNET_NAME="apim-vnet"
VNET_ADDRESS_PREFIX="10.0.0.0/16"

# Subnets
APP_GATEWAY_SUBNET="app-gateway-subnet"
APP_GATEWAY_SUBNET_PREFIX="10.0.1.0/24"

APIM_SUBNET="apim-subnet"
APIM_SUBNET_PREFIX="10.0.2.0/24"

OPENAI_PE_SUBNET="openai-pe-subnet"
OPENAI_PE_SUBNET_PREFIX="10.0.3.0/24"

# NSGs
APP_GATEWAY_NSG="app-gateway-nsg"
APIM_NSG="apim-nsg"
OPENAI_PE_NSG="openai-pe-nsg"

# Step 1: Create Resource Group
az group create --name $RESOURCE_GROUP --location $LOCATION

# Step 2: Create Virtual Network
az network vnet create \
    --resource-group $RESOURCE_GROUP \
    --name $VNET_NAME \
    --address-prefix $VNET_ADDRESS_PREFIX \
    --subnet-name $APP_GATEWAY_SUBNET \
    --subnet-prefix $APP_GATEWAY_SUBNET_PREFIX

# Step 3: Create Additional Subnets (APIM & OpenAI Private Endpoint)
az network vnet subnet create \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $APIM_SUBNET \
    --address-prefix $APIM_SUBNET_PREFIX

az network vnet subnet create \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $OPENAI_PE_SUBNET \
    --address-prefix $OPENAI_PE_SUBNET_PREFIX

# Step 4: Create NSGs
az network nsg create --resource-group $RESOURCE_GROUP --name $APP_GATEWAY_NSG
az network nsg create --resource-group $RESOURCE_GROUP --name $APIM_NSG
az network nsg create --resource-group $RESOURCE_GROUP --name $OPENAI_PE_NSG

# Step 5: Add NSG Rules for APIM (Allow 3443 for APIM Internal VNet)
az network nsg rule create \
    --resource-group $RESOURCE_GROUP \
    --nsg-name $APIM_NSG \
    --name AllowAPIMInbound3443 \
    --priority 120 \
    --direction Inbound \
    --access Allow \
    --protocol Tcp \
    --source-address-prefixes ApiManagement \
    --destination-address-prefixes VirtualNetwork \
    --destination-port-ranges 3443

# Step 6: Associate NSGs with Subnets
az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $APP_GATEWAY_SUBNET \
    --network-security-group $APP_GATEWAY_NSG

az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $APIM_SUBNET \
    --network-security-group $APIM_NSG

az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $OPENAI_PE_SUBNET \
    --network-security-group $OPENAI_PE_NSG

# Step 7: Configure Service Endpoints for APIM Subnet
az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $APIM_SUBNET \
    --service-endpoints Microsoft.EventHub Microsoft.KeyVault Microsoft.ServiceBus Microsoft.Sql Microsoft.Storage Microsoft.AzureActiveDirectory Microsoft.CognitiveServices Microsoft.Web

Creating an Azure Open AI with private endpoint

# Create an Azure OpenAI Resource
az cognitiveservices account create \
    --name $AOAI_NAME \
    --resource-group $RESOURCE_GROUP \
    --kind OpenAI \
    --sku S0 \
    --location $LOCATION \
    --yes \
    --custom-domain $AOAI_NAME

#Create a Private Endpoint
az network private-endpoint create \
    --name $PRIVATE_ENDPOINT_NAME \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --subnet $SUBNET_NAME \
    --private-connection-resource-id $(az cognitiveservices account show --name $AOAI_NAME --resource-group $RESOURCE_GROUP --query id -o tsv) \
    --group-id account \
    --connection-name "${PRIVATE_ENDPOINT_NAME}-connection"

# Create a Private DNS Zone
az network private-dns zone create \
    --resource-group $RESOURCE_GROUP \
    --name $PRIVATE_DNS_ZONE_NAME

# Link Private DNS Zone to VNet
az network private-dns link vnet create \
    --resource-group $RESOURCE_GROUP \
    --zone-name $PRIVATE_DNS_ZONE_NAME \
    --name "myDNSLink" \
    --virtual-network $VNET_NAME \
    --registration-enabled false

# Retrieve the Private IP Address from the Private Endpoint
PRIVATE_IP=$(az network private-endpoint show \
    --name $PRIVATE_ENDPOINT_NAME \
    --resource-group $RESOURCE_GROUP \
    --query "customDnsConfigs[0].ipAddresses[0]" -o tsv)

# Create a DNS Record for Azure OpenAI
az network private-dns record-set a add-record \
    --resource-group $RESOURCE_GROUP \
    --zone-name $PRIVATE_DNS_ZONE_NAME \
    --record-set-name $AOAI_NAME \
    --ipv4-address $PRIVATE_IP

# Disable Public Network Access
az cognitiveservices account update \
    --name $AOAI_NAME \
    --resource-group $RESOURCE_GROUP \
    --public-network-access Disabled

Conclusion

Securing Azure OpenAI with private endpoints, APIM, and Application Gateway ensures a robust, enterprise-grade architecture that balances security, accessibility, and performance. By leveraging private endpoints, Azure OpenAI remains shielded from public exposure, while APIM acts as a controlled gateway for managing external API access. The addition of Application Gateway provides an extra security layer with SSL termination, WAF protection, and traffic management.

With this setup, organizations can:
✔ Ensure end-to-end private connectivity for Azure OpenAI.
✔ Enable secure external access via APIM and Application Gateway.
✔ Enforce strict network segmentation with VNETs, Subnets, NSGs, and Private DNS.
✔ Strengthen security with Managed Identity authentication and controlled API exposure.

By following this guide, you now have a scalable, production-ready solution to securely integrate Azure OpenAI with external applications, whether they reside in AWS, on-premises, or other cloud environments. Implement these best practices to maintain compliance, minimize security risks, and enhance the reliability of your AI-powered applications.

Updated Feb 24, 2025

Version 1.0

artificial intelligence

Sabyasachi-Samaddar

Microsoft

Joined August 12, 2021

View Profile

Azure Architecture Blog

Follow this blog board to get notified when there's new activity