Using NVIDIA Triton Inference Server on Azure Container Apps
TOC Introduction to Triton System Architecture Architecture Focus of This Tutorial Setup Azure Resources File and Directory Structure ARM Template ARM Template From Azure Portal Testing Azure Container Apps Conclusion References 1. Introduction to Triton Triton Inference Server is an open-source, high-performance inferencing platform developed by NVIDIA to simplify and optimize AI model deployment. Designed for both cloud and edge environments, Triton enables developers to serve models from multiple deep learning frameworks, including TensorFlow, PyTorch, ONNX Runtime, TensorRT, and OpenVINO, using a single standardized interface. Its goal is to streamline AI inferencing while maximizing hardware utilization and scalability. A key feature of Triton is its support for multiple model execution modes, including dynamic batching, concurrent model execution, and multi-GPU inferencing. These capabilities allow organizations to efficiently serve AI models at scale, reducing latency and optimizing throughput. Triton also offers built-in support for HTTP/REST and gRPC endpoints, making it easy to integrate with various applications and workflows. Additionally, it provides model monitoring, logging, and GPU-accelerated inference optimization, enhancing performance across different hardware architectures. Triton is widely used in AI-powered applications such as autonomous vehicles, healthcare imaging, natural language processing, and recommendation systems. It integrates seamlessly with NVIDIA AI tools, including TensorRT for high-performance inference and DeepStream for video analytics. By providing a flexible and scalable deployment solution, Triton enables businesses and researchers to bring AI models into production with ease, ensuring efficient and reliable inferencing in real-world applications. 2. System Architecture Architecture Development Environment OS: Ubuntu Version: Ubuntu 18.04 Bionic Beaver Docker version: 26.1.3 Azure Resources Storage Account: SKU - General Purpose V2 Container Apps Environments: SKU - Consumption Container Apps: N/A Focus of This Tutorial This tutorial walks you through the following stages: Setting up Azure resources Publishing the project to Azure Testing the application Each of the mentioned aspects has numerous corresponding tools and solutions. The relevant information for this session is listed in the table below. Local OS Windows Linux Mac V How to setup Azure resources and deploy Portal (i.e., REST api) ARM Bicep Terraform V 3. Setup Azure Resources File and Directory Structure Please open a terminal and enter the following commands: git clone https://github.com/theringe/azure-appservice-ai.git cd azure-appservice-ai After completing the execution, you should see the following directory structure: File and Path Purpose triton/tools/arm-template.json The ARM template to setup all the Azure resources related to this tutorial, including a Container Apps Environments, a Container Apps, and a Storage Account with the sample dataset. ARM Template We need to create the following resources or services: Manual Creation Required Resource/Service Container Apps Environments Yes Resource Container Apps Yes Resource Storage Account Yes Resource Blob Yes Service Deployment Script Yes Resource Let’s take a look at the triton/tools/arm-template.json file. Refer to the configuration section for all the resources. Since most of the configuration values don’t require changes, I’ve placed them in the variables section of the ARM template rather than the parameters section. This helps keep the configuration simpler. However, I’d still like to briefly explain some of the more critical settings. As you can see, I’ve adopted a camelCase naming convention, which combines the [Resource Type] with [Setting Name and Hierarchy]. This makes it easier to understand where each setting will be used. The configurations in the diagram are sorted by resource name, but the following list is categorized by functionality for better clarity. Configuration Name Value Purpose storageAccountContainerName data-and-model [Purpose 1: Blob Container for Model Storage] Use this fixed name for the Blob Container. scriptPropertiesRetentionInterval P1D [Purpose 2: Script for Uploading Models to Blob Storage] No adjustments are needed. This script is designed to launch a one-time instance immediately after the Blob Container is created. It downloads sample model files and uploads them to the Blob Container. The Deployment Script resource will automatically be deleted after one day. caeNamePropertiesPublicNetworkAccess Enabled [Purpose 3: For Testing] ACA requires your local machine to perform tests; therefore, external access must be enabled. appPropertiesConfigurationIngressExternal true [Purpose 3: For Testing] Same as above. appPropertiesConfigurationIngressAllowInsecure true [Purpose 3: For Testing] Same as above. appPropertiesConfigurationIngressTargetPort 8000 [Purpose 3: For Testing] The Triton service container uses port 8000. appPropertiesTemplateContainers0Image nvcr.io/nvidia/tritonserver:22.04-py3 [Purpose 3: For Testing] The Triton service container utilizes this online resource. ARM Template From Azure Portal In addition to using az cli to invoke ARM Templates, if the JSON file is hosted on a public network URL, you can also load its configuration directly into the Azure Portal by following the method described in the article [Deploy to Azure button - Azure Resource Manager]. This is my example. Click Me After filling in all the required information, click Create. And we could have a test once the creation process is complete. 4. Testing Azure Container App In our local environment, use the following command to start a one-time Docker container. We will use NVIDIA's official test image and send a sample image from within it to the Triton service that was just deployed to Container Apps. # Replace XXX.YYY.ZZZ.azurecontainerapps.io with the actual FQDN of your app. There is no need to add https:// docker run --rm nvcr.io/nvidia/tritonserver:22.04-py3-sdk /workspace/install/bin/image_client -u XXX.YYY.ZZZ.azurecontainerapps.io -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg After sending the request, you should see the prediction results, indicating that the deployed Triton server service is functioning correctly. 5. Conclusion Beyond basic model hosting, Triton Inference Server's greatest strength lies in its ability to efficiently serve AI models at scale. It supports multiple deep learning frameworks, allowing seamless deployment of diverse models within a single infrastructure. With features like dynamic batching, multi-GPU execution, and optimized inference pipelines, Triton ensures high performance while reducing latency. While it may not replace custom-built inference solutions for highly specialized workloads, it excels as a standardized and scalable platform for deploying AI across cloud and edge environments. Its flexibility makes it ideal for applications such as real-time recommendation systems, autonomous systems, and large-scale AI-powered analytics. 6. References Quickstart — NVIDIA Triton Inference Server Deploying an ONNX Model — NVIDIA Triton Inference Server Model Repository — NVIDIA Triton Inference Server Triton Tutorials — NVIDIA Triton Inference ServerAccelerating Azure Container Apps with the Azure CLI and Compose Files
I’d like to introduce the containerapp-compose Azure CLI extension. This extension adds a command group to the containerapps extension, allowing you to start with a Compose file and a resource group in Azure and end up with a deployed Azure Container Apps environment and running services.Giving Granular Controls to Azure EasyAuth
When working with web applications, implementing authentication and authorization is crucial for security. Traditionally, there have been various ways from simple username/password inputs to OAuth-based approaches. However, implementing authentication and authorization logics can be quite complex and challenging. Fortunately, Azure offers a feature called EasyAuth, which simplifies this process. EasyAuth is built in Azure PaaS services like App Service, Functions, Container Apps, and Static Web Apps. The best part of using EasyAuth is that you don't have to modify existing code for the EasyAuth integration. Having said that, since EasyAuth follows one of the cloud architecture patterns – Sidecar pattern, it secures the entire web application. However, fine-tuned control – such as protecting only specific areas or excluding certain parts – can be tricky to achieve. To address this limitation, the developer community has explored various solutions: EasyAuth for App Service by Maxime Rouiller EasyAuth for Azure Container Apps by John Reilly EasyAuth for Azure Static Web Apps by Anthony Chu These approaches are still valid today but require updates to align with the latest .NET features. Throughout this post, I'm going to explore how to transform the authentication token generated by EasyAuth in Azure App Service and Azure Container Apps into a format usable by Blazor and other ASP.NET Core web applications. Please refer to my previous post about EasyAuth on Azure Static Web Apps. Download NuGet Package A few NuGet packages are available for ASP.NET Core web applications to integrate the Client Principal token generated by EasyAuth. Currently, the packages support Microsoft Entra ID and GitHub OAuth. NOTE: These packages are not the Microsoft official ones. Therefore, use these packages with your own care. Package Version Downloads Aliencube.Azure.Extensions.EasyAuth Aliencube.Azure.Extensions.EasyAuth.EntraID Aliencube.Azure.Extensions.EasyAuth.GitHub You can find the source code of these NuGet packages on this GitHub repository. Getting Started with Azure Developer CLI The NuGet package repository has sample apps for you to deploy ASP.NET Core web apps to both Azure App Service and Azure Container Apps. Clone the repository. Make sure you have the latest version of Azure CLI, Azure Bicep CLI and Azure Developer CLI. Login to Azure by running the azd auth login command. Deploy the sample apps by running azd up command. You might be asked to provide both Client ID and Client Secret values of a GitHub OAuth app. If you don't have it, leave them blank. Apply NuGet Package to Web App The example uses a Blazor web app, but the same approach applies to any ASP.NET Core web application. Microsoft Entra ID Add the NuGet package to your Blazor application. dotnet add package Aliencube.Azure.Extensions.EasyAuth.EntraID NOTE: At the time of writing, it's still in preview. Therefore, use the --prerelease tag at the end of the command like dotnet add package Aliencube.Azure.Extensions.EasyAuth.EntraID --prerelease . Open the Program.cs file and find the code line, var app = builder.Build(); . Then, add dependencies for authentication and authorisation. // 👇👇👇 Add dependencies for authentication/authorisation builder.Services.AddAuthentication(EasyAuthAuthenticationScheme.Name) .AddAzureEasyAuthHandler<EntraIDEasyAuthAuthenticationHandler>(); builder.Services.AddAuthorization(); // 👆👆👆 Add dependencies for authentication/authorisation var app = builder.Build(); In the same Program.cs file and find the code line, app.Run(); . Then activate authentication and authorisation. // 👇👇👇 Activate authentication/authorisation app.UseAuthentication(); app.UseAuthorization(); // 👆👆👆 Activate authentication/authorisation app.Run(); Apply the Authorize attribute to a specific page component. PAGE "/random-page-url" @* 👇👇👇 Apply authorisation *@ @using Aliencube.Azure.Extensions.EasyAuth @using Microsoft.AspNetCore.Authorization @attribute [Authorize(AuthenticationSchemes = EasyAuthAuthenticationScheme.Name)] @* 👆👆👆 Apply authorisation *@ Deploy the app to either Azure App Service or Azure Container Apps. After deployment, configure the authentication settings like: Open your web browser and navigate the page component you applied authorisation, and see the 401 (Unauthorized) error. Sign into the web app through /.auth/login/aad and visite the page again, and verify you're able to see the content on that page. GitHub OAuth It's mostly the same as using Microsoft Entra ID, except using a different NuGet package. Install the GitHub-specific NuGet package. dotnet add package Aliencube.Azure.Extensions.EasyAuth.GitHub NOTE: At the time of writing, it's still in preview. Therefore, use the --prerelease tag at the end of the command like dotnet add package Aliencube.Azure.Extensions.EasyAuth.GitHub --prerelease . Open the Program.cs file and find the code line, var app = builder.Build(); . Then, add dependencies for authentication and authorisation. Make sure this time you use GitHubEasyAuthAuthenticationHandler instead of EntraIDEasyAuthAuthenticationHandler . // 👇👇👇 Add dependencies for authentication/authorisation builder.Services.AddAuthentication(EasyAuthAuthenticationScheme.Name) .AddAzureEasyAuthHandler<GitHubEasyAuthAuthenticationHandler>(); builder.Services.AddAuthorization(); // 👆👆👆 Add dependencies for authentication/authorisation var app = builder.Build(); Follow the same steps as above. How Does It Work? When authenticated through Azure EasyAuth, a request header X-MS-CLIENT-PRINCIPAL is created. It contains a base-64 encoded JSON object, which needs to be decoded and transformed into a ClaimsPrincipal instance for role-based access control. Here's the JSON object. For brevity, most claims are omitted except two. Here's the JSON object generated from Microsoft Entra ID authentication. { "auth_typ": "aad", "name_typ": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress", "role_typ": "http://schemas.microsoft.com/ws/2008/06/identity/claims/role", "claims": [ { "typ": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress", "val": "john.doe@contoso.com" }, { "typ": "roles", "val": "User" } ] } And, here's the JSON object generated from GitHub authentication. { "auth_typ": "github", "name_typ": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name", "role_typ": "http://schemas.microsoft.com/ws/2008/06/identity/claims/role", "claims": [ { "typ": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name", "val": "John Doe" }, { "typ": "urn:github:type", "val": "User" } ] } This JSON object should be converted into the ClaimsPrincipal instance for granular control on each page. Let's understand the JSON object. auth_typ : It indicates the authentication method currently used. For example, Microsoft Entra ID says aad and GitHub says github . name_typ : After the authentication, the claim with this key is bound to HttpContext.User.Identity.Name . role_typ : They claim key for authorisation by roles. claims : A collection of claims generated by either Microsoft Entra ID or GitHub. But the challenge on the X-MS-CLIENT-PRINCIPAL token is that it has no role claims matching to the role_typ key. The Entra ID one contains the roles claim and GitHub one contains the urn:github:type claim, which is not following the OpenID Connect standard, while ClaimsPrincipal object on ASP.NET Core web apps do follow the standard. The NuGet packages above convert the role values. By adding and activating the conversion handler, users are now able to login to the web app with EasyAuth and access to certain pages with proper permissions. Here's the example that uses the AuthenticationSchemes property to let users access to pages as long as they sign in with EasyAuth. PAGE "/random-page-url" @attribute [Authorize(AuthenticationSchemes = EasyAuthAuthenticationScheme.Name)] This is another example that uses the Roles property to let users having the User permission to access to the given page. PAGE "/random-page-url" @attribute [Authorize(Roles = "User")] NOTE: Make sure that X-MS-CLIENT-PRINCIPAL token from Microsoft Entra ID doesn't contain the roles claims out-of-the-box because it really depends on each tenant's policy. Roadmap Azure EasyAuth supports multiple identity providers, but the NuGet packages currently support Microsoft Entra ID and GitHub OAuth. ✅ Microsoft Entra ID ✅ GitHub ⏹️ OpenID Connect ⏹️ Google ⏹️ Facebook ⏹️ X ⏹️ Apple Contributions are always welcome! So far, we've walked through how to transform Azure EasyAuth authentication token to the ClaimsPrincipal object to be used within ASP.NET Core web apps. With these packages, you can control individual pages based on the users' role and increase apps' security with less efforts. Learn More For further details on Azure EasyAuth, check out: Azure App Service EasyAuth Azure Functions EasyAuth Azure Container Apps EasyAuth Azure Static Web Apps EasyAuth Azure EasyAuth Extensions (Code) Azure EasyAuth Samples (Code)Custom scaling on Azure Container Apps based on Redis Streams
ACA's autoscaling feature internally leverages KEDA and gives you the ability to configure the number of replicas to deploy based on rules (event triggers). Apart from HTTP and TCP rule based scaling, container apps also support custom rules giving it the flexibility and opening up a lot more configuration options since all of KEDA's event-based scalers are supported.Enhancing Security for Azure Container Apps with Aqua Security
Azure Container Apps (ACA) is a developer-first serverless platform that allows you to run scalable containerized workloads at any scale. Being serverless provides inherent security benefits by reducing the attack surface, but it also presents some unique challenges for any security solution. Hence, we’re happy to announce that our partner, Aqua has just certified Azure Container Apps for their suite of security solutions. Azure Container Apps: Built-In Security Features Due to its purpose-built nature ACA offers several built-in security features that help protect your containerized applications: Isolation: ACA runs your workload without the need for root access to the underlying host. Additionally, it’s trivial and requires minimal overhead to isolate different teams in their own environments without the need to painfully cordon off each team via Kubernetes namespaces. Network Security: ACA supports virtual network integration, allowing you to control inbound and outbound traffic to your applications on a both a per app basis as well as for an entire environment all at once. Additionally, we provide protection against common layer-7 vulnerabilities such as redirection attacks. Managed Identity: ACA integrates with Azure Active Directory, enabling secure access to other Azure services without managing credentials. While these features provide a solid foundation, securing containerized workloads requires a comprehensive approach that addresses the entire lifecycle of your applications. This is where Aqua’s suite of tools excels. Elevating ACA's Security Posture using Aqua Aqua Security is a certified security solution for ACA, offering a full-lifecycle approach to securing your containerized applications. Here’s how Aqua enhances ACA's security capabilities: Supply Chain Security: Aqua scans container images for tampering and potential supply chain attacks, ensuring that only verified and secure images are deployed. Comprehensive Image Scanning: Aqua scans container images in Azure Container Registry (ACR) and CI/CD pipelines for vulnerabilities, misconfigurations, malware, and embedded secrets, enabling developers to address issues early. Image Assurance Policies: Aqua enforces policies to ensure that only compliant images are deployed, minimizing risks and ensuring adherence to security and compliance standards. Agentless Discovery and Scanning: Aqua automatically discovers and scans all running services and assets, providing broad visibility into your ACA workloads. Runtime Protection with MicroEnforcer: Aqua's MicroEnforcer provides non-invasive runtime security, detecting and preventing threats such as cryptocurrency mining, reverse shell execution, and unauthorized access. By leveraging Aqua's security solutions, organizations can confidently meet the most stringent security requirements for their ACA workloads. For more information on how to use Aqua's tooling with ACA, visit the Aqua blog: Securing Azure Container Apps
Leveraging Azure Container Apps Labels for Environment-based Routing and Feature Testing
Azure Container Apps offers a powerful feature through labels and traffic splitting that can help developers easily manage multiple versions of an app, route traffic based on different environments, and enable controlled feature testing without disrupting live users. In this blog, we'll walk through a practical scenario where we deploy an experimental feature in a staging revision, test it with internal developers, and then switch the feature to production once it’s validated. We'll use Azure Container Apps labels and traffic splitting to achieve this seamless deployment process.
