Calculating Chargebacks for Business Units/Projects Utilizing a Shared Azure OpenAI Instance
Azure OpenAI Service is at the forefront of technological innovation, offering REST API access to OpenAI's suite of revolutionary language models, including GPT-4, GPT-35-Turbo, and the Embeddings model series. Enhancing Throughput for Scale As enterprises seek to deploy OpenAI's powerful language models across various business units, they often require granular control over configuration and performance metrics. To address this need, Azure OpenAI Service is introducing dedicated throughput, a feature that provides a dedicated connection to OpenAI models with guaranteed performance levels. Throughput is quantified in terms of tokens per second (tokens/sec), allowing organizations to precisely measure and optimize the performance for both prompts and completions. The model of provisioned throughput provides enhanced management and adaptability for varying workloads, guaranteeing system readiness for spikes in demand. This capability also ensures a uniform user experience and steady performance for applications that require real-time responses. Resource Sharing and Chargeback Mechanisms Large organizations frequently provision a singular instance of Azure OpenAI Service that is shared across multiple internal departments. This shared use necessitates an efficient mechanism for allocating costs to each business unit or consumer, based on the number of tokens consumed. This article delves into how chargeback is calculated for each business unit based on their token usage. Leveraging Azure API Management Policies for Token Tracking Azure API Management Policies offer a powerful solution for monitoring and logging the token consumption for each internal application. The process can be summarized in the following steps: ** Sample Code: Refer to this GitHub repository to get a step-by-step instruction on how to build the solution outlined below : private-openai-with-apim-for-chargeback 1. Client Applications Authorizes to API Management To make sure only legitimate clients can call the Azure OpenAI APIs, each client must first authenticate against Azure Active Directory and call APIM endpoint. In this scenario, the API Management service acts on behalf of the backend API, and the calling application requests access to the API Management instance. The scope of the access token is between the calling application and the API Management gateway. In API Management, configure a policy (validate-jwt or validate-azure-ad-token) to validate the token before the gateway passes the request to the backend. 2. APIM redirects the request to OpenAI service via private endpoint. Upon successful verification of the token, Azure API Management (APIM) routes the request to Azure OpenAI service to fetch response for completions endpoint, which also includes prompt and completion token counts. 3. Capture and log API response to Event Hub Leveraging the log-to-eventhub policy to capture outgoing responses for logging or analytics purposes. To use this policy, a logger needs to be configured in the API Management: # API Management service-specific details $apimServiceName = "apim-hello-world" $resourceGroupName = "myResourceGroup" # Create logger $context = New-AzApiManagementContext -ResourceGroupName $resourceGroupName -ServiceName $apimServiceName New-AzApiManagementLogger -Context $context -LoggerId "OpenAiChargeBackLogger" -Name "ApimEventHub" -ConnectionString "Endpoint=sb://<EventHubsNamespace>.servicebus.windows.net/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<key>" -Description "Event hub logger with connection string" Within outbound policies section, pull specific data from the body of the response and send this information to the previously configured EventHub instance. This is not just a simple logging exercise; it is an entry point into a whole ecosystem of real-time analytics and monitoring capabilities: <outbound> <choose> <when condition="@(context.Response.StatusCode == 200)"> <log-to-eventhub logger-id="TokenUsageLogger">@{ var responseBody = context.Response.Body?.As<JObject>(true); return new JObject( new JProperty("Timestamp", DateTime.UtcNow.ToString()), new JProperty("ApiOperation", responseBody["object"].ToString()), new JProperty("AppKey", context.Request.Headers.GetValueOrDefault("Ocp-Apim-Subscription-Key",string.Empty)), new JProperty("PromptTokens", responseBody["usage"]["prompt_tokens"].ToString()), new JProperty("CompletionTokens", responseBody["usage"]["completion_tokens"].ToString()), new JProperty("TotalTokens", responseBody["usage"]["total_tokens"].ToString()) ).ToString(); }</log-to-eventhub> </when> </choose> <base /> </outbound> EventHub serves as a powerful fulcrum, offering seamless integration with a wide array of Azure and Microsoft services. For example, the logged data can be directly streamed to Azure Stream Analytics for real-time analytics or to Power BI for real-time dashboards With Azure Event Grid, the same data can also be used to trigger workflows or automate tasks based on specific conditions met in the incoming responses. Moreover, the architecture is extensible to non-Microsoft services as well. Event Hubs can interact smoothly with external platforms like Apache Spark, allowing you to perform data transformations or feed machine learning models. 4: Data Processing with Azure Functions An Azure Function is invoked when data is sent to the EventHub instance, allowing for bespoke data processing in line with your organization’s unique requirements. For instance, this could range from dispatching the data to Azure Monitor, streaming it to Power BI dashboards, or even sending detailed consumption reports via Azure Communication Service. [Function("TokenUsageFunction")] public async Task Run([EventHubTrigger("%EventHubName%", Connection = "EventHubConnection")] string[] openAiTokenResponse) { //Eventhub Messages arrive as an array foreach (var tokenData in openAiTokenResponse) { try { _logger.LogInformation($"Azure OpenAI Tokens Data Received: {tokenData}"); var OpenAiToken = JsonSerializer.Deserialize<OpenAiToken>(tokenData); if (OpenAiToken == null) { _logger.LogError($"Invalid OpenAi Api Token Response Received. Skipping."); continue; } _telemetryClient.TrackEvent("Azure OpenAI Tokens", OpenAiToken.ToDictionary()); } catch (Exception e) { _logger.LogError($"Error occured when processing TokenData: {tokenData}", e.Message); } } } In the example above, Azure function processes the tokens response data in Event Hub and sends them to Application Insights telemetry, and a basic Dashboard is configured in Azure, displaying the token consumption for each client application. This information can conveniently be used to compute chargeback costs. A sample query used in dashboard above that fetches tokens consumed by a specific client: customEvents | where name contains "Azure OpenAI Tokens" | extend tokenData = parse_json(customDimensions) | where tokenData.AppKey contains "your-client-key" | project Timestamp = tokenData.Timestamp, Stream = tokenData.Stream, ApiOperation = tokenData.ApiOperation, PromptTokens = tokenData.PromptTokens, CompletionTokens = tokenData.CompletionTokens, TotalTokens = tokenData.TotalTokens Azure OpenAI Landing Zone reference architecture A crucial detail to ensure the effectiveness of this approach is to secure the Azure OpenAI service by implementing Private Endpoints and using Managed Identities for App Service to authorize access to Azure AI services. This will limit access so that only the App Service can communicate with the Azure OpenAI service. Failing to do this would render the solution ineffective, as individuals could bypass the APIM/App Service and directly access the OpenAI Service if they get hold of the access key for OpenAI. Refer to Azure OpenAI Landing Zone reference architecture to build a secure and scalable AI environment. Additional Considerations If the client application is external, consider using an Application Gateway in front of the Azure APIM If "streaming" is set to true, tokens count is not returned in response. In that that case libraries like tiktoken (Python), orgpt-3-encoder(javascript) for most GPT-3 models can be used to programmatically calculate tokens count for the user prompt and completion response. A useful guideline to remember is that in typical English text, one token is approximately equal to around 4 characters. This equates to about three-quarters of a word, meaning that 100 tokens are roughly equivalent to 75 words. (P.S. Microsoft does not endorse or guarantee any third-party libraries.) A subscription key or a custom header like app-key can also be used to uniquely identify the client as appId in OAuth token is not very intuitive. Rate-limiting can be implemented for incoming requests using OAuth tokens or Subscription Keys, adding another layer of security and resource management. The solution can also be extended to redirect different clients to different Azure OpenAI instances. For example., some clients utilize an Azure OpenAI instance with default quotas, whereas premium clients get to consume Azure Open AI instance with dedicated throughput. Conclusion Azure OpenAI Service stands as an indispensable tool for organizations seeking to harness the immense power of language models. With the feature of provisioned throughput, clients can define their usage limits in throughput units and freely allocate these to the OpenAI model of their choice. However, the financial commitment can be significant and is dependent on factors like the chosen model's type, size, and utilization. An effective chargeback system offers several advantages, such as heightened accountability, transparent costing, and judicious use of resources within the organization.Checklist for Migrating Web Apps to App Service
App Service continues to invest in migration tooling to allow customers to easily migrate their web apps to App Service. The current set of tools enable discovery, assessment, and migration of web apps across various scenarios and scopes viz. standalone web app, single IIS server and even a datacenter.
How to remove secrets from Container Apps linked to ACR
Azure Container Apps allows your application to securely store sensitive configuration values. Once secrets are defined at the application level, secured values are available to revisions in your container apps. Additionally, you can reference secured values inside scale rules. This blog provides a detailed, step-by-step procedure for removing secrets associated with an Azure Container Registry (ACR). In this example, we will walk through the process of creating a Container App with an image reference from the ACR, which automatically generates a secret. We will then attempt to remove this secret and observe its behaviour throughout the process. Secrets are scoped to an application, outside of any specific revision of an application. Adding, removing, or changing secrets doesn't generate new revisions. Each application revision can reference one or more secrets. Multiple revisions can reference the same secret(s). An updated or deleted secret doesn't automatically affect existing revisions in your app. When a secret is updated or deleted, you can respond to changes in one of two ways: Deploy a new revision. Restart an existing revision. Before you delete a secret, deploy a new revision that no longer references the old secret. Then deactivate all revisions that reference the secret. Create an Azure Container Registry: az acr create \ --name "$CONTAINER_REGISTRY_NAME"\ --resource-group "$RESOURCE_GROUP"\ --location "$LOCATION"\ --sku Basic \ --admin-enabled true Explanation: This command creates an Azure Container Registry (ACR) with the specified name, resource group and location. The --sku Basic specifies the pricing tier for the registry, and --admin-enabled true enables admin access to the registry. Build and push image from a Dockerfile. Now use Azure Container Registry to build and push an image. First, create a local working directory and then create a Dockerfile named Dockerfile with the single line: FROM mcr.microsoft.com/hello-world. This is a simple example to build a Linux container image from the hello-world image hosted at Microsoft Container Registry. You can create your own standard Dockerfile and build images for other platforms. If you are working at a bash shell, create the Dockerfile with the following command: echo "FROM mcr.microsoft.com/hello-world" > Dockerfile Run the az acr build command, which builds the image and, after the image is successfully built, pushes it to your registry. The following example builds and pushes the sample/hello-world:v1 image. The . at the end of the command sets the location of the Dockerfile, in this case the current directory. az acr build --image sample/hello-world:v1 \ --registry myContainerRegistry008 \ --file Dockerfile . Create and Deploy the Container App from ACR Create and deploy your container app with the containerapp up command. This command will: Create the resource group Create the Container Apps environment Create the Log Analytics workspace Create and deploy the container app using a public container image Note that if any of these resources already exist, the command will use them instead of creating new ones. az containerapp up \ --name my-container-app \ --resource-group my-container-apps \ --location centralus \ --environment 'my-container-apps' \ --image azuredockerregistry.azurecr.io/image:latest \ --target-port 80 \ --ingress external \ --query properties.configuration.ingress.fqdn 1. Once the Container App gets created , check if there are any secrets added to your container app. 2. If you attempt to delete an existing secret and encounter an error, it's because there is a running revision that still references the secret. 3. In this case, ensure you deactivate all existing revisions that reference the secret and create a new revision that references an image from a public repository (e.g., docker.io). 4. After verifying that there are no references to secrets in existing revisions, you can use the `az containerapp registry remove` command to remove a registry associated with your container app. az containerapp registry remove -n MyContainerapp -g MyResourceGroup --server MyContainerappRegistry.azurecr.io If successful, the command will return "Registry Successfully Removed." 5. Upon checking the "Secrets" section in the portal you should see the secret removed. 6. If you continue to see secrets in the portal UI even after following the above steps, now try deleting the secrets directly using the delete option in the portal. It's crucial to manage secrets carefully to maintain the security and integrity of your Azure Container Apps and associated resources. !!HAPPY LEARNING !!
Use managed identity instead of AzureWebJobsStorage to connect a function app to a storage account
In a function app, usually we use appsetting AzureWebJobsStorage to connect to storage. This blog shows you how to configure a function app using Azure Active Directory identities instead of secrets or connection strings, where possible. Using identities helps you avoid accidentally leaking sensitive secrets and can provide better visibility into how data is accessed. This will not work if the storage account is in a sovereign cloud or has a custom DNS. IMPORTANT! When running in a Consumption or Elastic Premium plan, your app uses the WEBSITE_AZUREFILESCONNECTIONSTRING and WEBSITE_CONTENTSHARE settings when connecting to Azure Files on the storage account used by your function app. Azure Files doesn't support using managed identity when accessing the file share. That is to say, if your functio app is running on Consumption/EP, plan, you can only delete and recreate function app on app service plan to avoid using File Share. For more information, see Azure Files supported authentication scenarios Below are the steps to do configuration. 1. Enable system assigned identity in your function app and save it. 2. Give storage access to your function app. Search for Storage Blob Data Owner, select it. 3. If you configure a blob-triggered function app, repeat the step 2 to add Storage Account Contributor and Storage Queue Data Contributor roles which will be used for blob trigger. 4. Return to Access Control (IAM), click Role assignments, search for your function app name to confirm the roles are added successfully. 5. Navigate to your function app. Select Configuration and edit AzureWebJobsStorage. Change the name to AzureWebJobsStorage__accountname. Change the value to your storage account name. (The new setting uses a double underscore ( __ ), which is a special character in application settings.) 6. Delete the previous AzureWebJobsStorage. Then you will find your function app still works fine.
Introducing Code-to-Cloud with Azure Portal, get your apps in the cloud fast and easy
We are excited to announce Code-to-Cloud (preview), one of the fastest and easiest way to deploy and manage your web apps on Azure. Code-to-Cloud is a portal-based experience that takes an app-first approach to building, deploying, and running your apps. Code-to-Cloud makes it easier for developers to get started using Azure, without needing to be an expert on the hundreds of different cloud services.Azure Container Apps Announces New Pricing Plan and Enhanced Networking
In Public Preview today - You can optimize the hosting of each app in an environment by choosing between Serverless and Dedicated compute options. You can also leverage new network enhancements for more optimized and secure networking.
