Introducing GenAI Gateway Capabilities in Azure API Management
We are thrilled to announce GenAI Gateway capabilities in Azure API Management – a set of features designed specifically for GenAI use cases. Azure OpenAI service offers a diverse set of tools, providing access to advanced models like GPT3.5-Turbo to GPT-4 and GPT-4 Vision, enabling developers to build intelligent applications that can understand, interpret, and generate human-like text and images. One of the main resources you have in Azure OpenAI is tokens. Azure OpenAI assigns quota for your model deployments expressed in tokens-per-minute (TPMs) which is then distributed across your model consumers that can be represented by different applications, developer teams, departments within the company, etc. Starting with a single application integration, Azure makes it easy to connect your app to Azure OpenAI. Your intelligent application connects to Azure OpenAI directly using API Key with a TPM limit configured directly on the model deployment level. However, when you start growing your application portfolio, you are presented with multiple apps calling single or even multiple Azure OpenAI endpoints deployed as Pay-as-you-go or Provisioned Throughput Units (PTUs) instances. That comes with certain challenges: How can we track token usage across multiple applications? How can we do cross charges for multiple applications/teams that use Azure OpenAI models? How can we make sure that a single app does not consume the whole TPM quota, leaving other apps with no option to use Azure OpenAI models? How can we make sure that the API key is securely distributed across multiple applications? How can we distribute load across multiple Azure OpenAI endpoints? How can we make sure that PTUs are used first before falling back to Pay-as-you-go instances? To tackle these operational and scalability challenges, Azure API Management has built a set of GenAI Gateway capabilities: Azure OpenAI Token Limit Policy Azure OpenAI Emit Token Metric Policy Load Balancer and Circuit Breaker Import Azure OpenAI as an API Azure OpenAI Semantic Caching Policy (in public preview) Azure OpenAI Token Limit Policy Azure OpenAI Token Limit policy allows you to manage and enforce limits per API consumer based on the usage of Azure OpenAI tokens. With this policy you can set limits, expressed in tokens-per-minute (TPM). This policy provides flexibility to assign token-based limits on any counter key, such as Subscription Key, IP Address or any other arbitrary key defined through policy expression. Azure OpenAI Token Limit policy also enables pre-calculation of prompt tokens on the Azure API Management side, minimizing unnecessary request to the Azure OpenAI backend if the prompt already exceeds the limit. Learn more about this policy here. Azure OpenAI Emit Token Metric Policy Azure OpenAI enables you to configure token usage metrics to be sent to Azure Applications Insights, providing overview of the utilization of Azure OpenAI models across multiple applications or API consumers. This policy captures prompt, completions, and total token usage metrics and sends them to Application Insights namespace of your choice. Moreover, you can configure or select from pre-defined dimensions to split token usage metrics, enabling granular analysis by Subscription ID, IP Address, or any custom dimension of your choice. Learn more about this policy here. Load Balancer and Circuit Breaker Load Balancer and Circuit Breaker features allow you to spread the load across multiple Azure OpenAI endpoints. With support for round-robin, weighted (new), and priority-based (new) load balancing, you can now define your own load distribution strategy according to your specific requirements. Define priorities within the load balancer configuration to ensure optimal utilization of specific Azure OpenAI endpoints, particularly those purchased as PTUs. In the event of any disruption, a circuit breaker mechanism kicks in, seamlessly transitioning to lower-priority instances based on predefined rules. Our updated circuit breaker now features dynamic trip duration, leveraging values from the retry-after header provided by the backend. This ensures precise and timely recovery of the backends, maximizing the utilization of your priority backends to their fullest. Learn more about load balancer and circuit breaker here. Import Azure OpenAI as an API New Import Azure OpenAI as an API in Azure API management provides an easy single click experience to import your existing Azure OpenAI endpoints as APIs. We streamline the onboarding process by automatically importing the OpenAPI schema for Azure OpenAI and setting up authentication to the Azure OpenAI endpoint using managed identity, removing the need for manual configuration. Additionally, within the same user-friendly experience, you can pre-configure Azure OpenAI policies, such as token limit and emit token metric, enabling swift and convenient setup. Learn more about Import Azure OpenAI as an API here. Azure OpenAI Semantic Caching policy Azure OpenAI Semantic Caching policy empowers you to optimize token usage by leveraging semantic caching, which stores completions for prompts with similar meaning. Our semantic caching mechanism leverages Azure Redis Enterprise or any other external cache compatible with RediSearch and onboarded to Azure API Management. By leveraging the Azure OpenAI Embeddings model, this policy identifies semantically similar prompts and stores their respective completions in the cache. This approach ensures completions reuse, resulting in reduced token consumption and improved response performance. Learn more about semantic caching policy here. Get Started with GenAI Gateway Capabilities in Azure API Management We’re excited to introduce these GenAI Gateway capabilities in Azure API Management, designed to empower developers to efficiently manage and scale their applications leveraging Azure OpenAI services. Get started today and bring your intelligent application development to the next level with Azure API Management.Choosing the right Azure API Management tier for your networking scenarios
There are different options when it comes to integrating your API Management with your Azure Virtual Network (VNet) which are important to understand. These options will depend on your network perimeter access requirements and the available tiers and features in Azure API Management. This blog post aims to guide you through the different options available on both the classic tiers and v2 tiers of Azure API Management, to help you decide which choice works best for your requirements. We need to define how are we going to call the tiers : developer, basic, standard , premium. For example v1 tiers, classical tiers, etc…
Designing and running a Generative AI Platform based on Azure AI Gateway
Are you in a platform team who has been tasked with building an AI Platform to serve the needs of your internal consumers? What does that mean? It’s a daunting challenge to be set, and even harder if you’re operating in a highly regulated environment. As enterprises scale out usage of Generative AI past a few initial use-cases they will face into a new set of challenges - scaling, onboarding, security and compliance to name a few. In this article we outline a set of common requirements and provide a reference implementation for an AI Platform.
Announcing Public Preview of API Management WordPress plugin to build customized developer portals
Azure API Management WordPress plugin enables our customers to leverage the power of WordPress to build their own unique developer portal. API managers and administrators can bring up a new developer portal in a matter of few minutes and customize the theme, layout, add stylesheet or localize the portal into different languages.The Rising Significance of APIs - Azure API Management & API Center
As we venture deeper into the digital era, APIs (Application Programming Interfaces) have become the cornerstone of modern software development and digital communication. APIs continue to be pivotal, acting as the conduits through which different systems, applications, and devices interact and exchange data. This growing reliance on APIs is reflected in the investment trends, with a significant 92% of global respondents indicating that investments in APIs will either remain steady or increase over the next year (2023 State of the API Report, Postman, 2023). Recognizing the critical role that APIs play in modern software architecture, Microsoft has been consistently investing in and expanding its API suite to fulfill diverse needs. Managing APIs is not a one-size-fits-all solution with different API ecosystems requiring different API management approaches and tools. Azure API Center, a new API Azure service, was recently announced General Availability (GA). Azure API Center is engineered to function independently, yet it seamlessly integrates with Azure API Management, providing customers options to manage various aspects of their API ecosystem. In the following sections, we will dive into the specifics of Azure API Management and Azure API Center, highlighting their differences, use cases, and guiding you on when to use each product to best manage and leverage your API ecosystem. Azure API Management: Your Gateway to Digital Transformation Azure API Management (APIM) is a managed cloud service designed to streamline and secure the use of APIs. API Management acts as a secure front door to facilitate, manage, and analyze the interactions between an organization’s APIs and their users with some of their core functionalities listed below: API gateway - operational management: API Management acts as a gateway, managing API exposure, security, and analytics during runtime. Traffic routing: Acts as a facade to backend services by accepting API calls and routing them to appropriate backends. API access: Verifies API keys and other credentials such as JWT tokens and certificates presented with requests to access APIs published through an API Management instance. Operational stability: API Management allows you to enforce usage quotas and rate limits to manage the flow of requests to your APIs effectively to prevent API overuse. It also validates requests and responses in compliance with the specification - e.g., JSON and XML validation, validation of headers and query params. Request transformation: The rich policy engine of API Management allows you to modify incoming and outgoing requests to your needs with more than 60 built-in policies and the option to build your own custom policies. API logging: API Management provides the capability to emit logs, metrics, and traces, which are essential for monitoring, reporting, and troubleshooting your APIs. Self-hosted gateway: API Management also offers self-hosted gateway capabilities, a containerized version of the default managed gateway to place your gateways in the same environments where you host your APIs. Developer Portal: API Management features a developer portal, which can be generated automatically and is a fully customizable website with the documentation of your APIs. It facilitates API discovery, testing, and consumption by internal and external developers. Azure API Center: Your Inventory for API Lifecycle Management Azure API Center is the newest addition to the Azure API suite, focusing on the design-time aspects of API governance and centralizing your API inventory for tracking reasons. It acts as a repository and governance tool for all APIs within an organization, regardless of where they are in their lifecycle or where they are deployed. API inventory management: API Center allows you to register all your organization’s APIs in a centralized inventory, regardless of their type, lifecycle stage, or deployment location, for better tracking and accessibility. Tackling API sprawl: APIs' runtime might be managed in multiple different API Management services, multiple different API gateways from different vendors, or are unmanaged at all. Azure API Center allows you to develop and maintain a structured API inventory. Holistic API view: While API Management excels in runtime API mediation, its inventory management capabilities are limited to the types of APIs that are supported at runtime and to the versions that are actively managed in runtime. Azure API Center supports any kind of API types, such as AsyncAPIs, and you can easily track APIs across different deployment environments. Real-world API representation: You can add detailed information about each API, including versions, definitions, custom metadata, and associate them with deployment environments (e.g. Dev, Test, Production). API governance: Azure API Center provides tools to organize and filter APIs using metadata, set up linting and analysis to check API design consistency for better conformance on API style guidelines. Additionally, shift-left API compliance to API teams to ensure that developers can more productively and efficiently create compliant APIs. API discovery and reuse: It enables internal developers and API program managers to discover APIs through the Azure portal, an API Center portal, and developer tools, including a Visual Studio Code extension. Navigating Your API Ecosystem - Example Scenarios for Azure API Center and API Management While both services are integral to the API ecosystem, they serve distinct purposes: Azure API Management is geared towards runtime API governance and observability, focusing on the operational aspects of API management, such as securing, publishing, and analyzing APIs in use. Azure API Center, in contrast, is tailored for design-time API governance, helping organizations to maintain a structured inventory of all APIs for better discovery and governance. API Management and API Center are complementary services that, when used together, provide a comprehensive API management solution from design to deployment. API Mangement excels in the operational phase, while API Center, as your organizational API inventory, shines in the design and governance stages, ensuring that APIs are not only functional but also adhere to organizational standards and best practices. Note: The following scenarios are not mutually exclusive. They have been separated for better display and clarity. Scenario 1: Azure API Center serves as an API design governance tool for analyzing API definitions based on linting rules to check on API design consistency and quality: Scenario 2: Azure API Center serves as a centralized inventory solution for managing APIs across different API lifecycle stages (e.g. development, testing, production): Scenario 3: Azure API Center serves as a centralized inventory solution for managing APIs across different regional or organizational deployments (e.g. Asia, Europe, America): Scenario 4: Azure API Center serves as a centralized inventory solution for managing APIs across different cloud platforms (e.g. Azure, AWS, and Google Cloud): Azure API Center and API Management Workspaces Note: Azure API Management workspaces currently only apply to the Premium API Management tier (see Workspaces in Azure API Management for further details). API management workspaces are a feature within Azure API Management that allows decentralized API development teams to manage and productize their own APIs. Workspaces enable aggregation of multiple teams with proper isolation in a single APIM service. For Azure API Center, all previously mentioned scenarios still apply, regardless of whether APIM services use workspaces. Azure API Center will continue to improve the management of APIs across different API Managment services, as organizations... have APIM services across environments, for example, dev, test, and prod. have more than one production APIM service in their company. have APIm platforms from multiple vendors. Conclusion In conclusion, the launch of Azure API Center and the continued evolution of Azure API Management underscore Microsoft’s commitment to empowering organizations in their API first journey. By leveraging Azure API Management for runtime efficiency and Azure API Center for inventory and governance, organizations can navigate their API ecosystems with confidence, knowing they have the tools to foster innovation, efficiency, and growth. Share Your Thoughts! Your insights are invaluable to us. We're eager to hear what you think about Azure API Center and API Management, and to understand your needs. Is there something specific that would make you and your organization even more successful? Your feedback is the key to our continuous improvement. If you prefer a more personal touch, feel free to reach out via LinkedIn to Julia Kasper, Pierce Boggan and Mike Budzynski. Thank you for being a part of our journey!Expanding GenAI Gateway Capabilities in Azure API Management
In May 2024, we introduced GenAI Gateway capabilities – a set of features designed specifically for GenAI use cases. Today, we are happy to announce that we are adding new policies to support a wider range of large language models through Azure AI Model Inference API. These new policies work in a similar way to the previously announced capabilities, but now can be used with a wider range of LLMs. Azure AI Model Inference API enables you to consume the capabilities of models, available in Azure AI model catalog, in a uniform and consistent way. It allows you to talk with different models in Azure AI Studio without changing the underlying code. Working with large language models presents unique challenges, particularly around managing token resources. Token consumption impacts cost and performance of intelligent apps calling the same model, making it crucial to have robust mechanisms for monitoring and controlling token usage. The new policies aim to address challenges by providing detailed insights and control over token resources, ensuring efficient and cost-effective use of models deployed in Azure AI Studio. LLM Token Limit Policy LLM Token Limit policy (preview) provides the flexibility to define and enforce token limits when interacting with large language models available through the Azure AI Model Inference API. Key Features Configurable Token Limits: Set token limits for requests to control costs and manage resource usage effectively Prevents Overuse: Automatically blocks requests that exceed the token limit, ensuring fair use and eliminating the noisy neighbour problem Seamless Integration: Works seamlessly with existing applications, requiring no changes to your application configuration Learn more about this policy here. LLM Emit Token Metric Policy LLM Emit Token Metric policy (preview) provides detailed metrics on token usage, enabling better cost management and insights into model usage across your application portfolio. Key Features Real-Time Monitoring: Emit metrics in real-time to monitor token consumption. Detailed Insights: Gain insights into token usage patterns to identify and mitigate high-usage scenarios Cost Management: Split token usage by any custom dimension to attribute cost to different teams, departments, or applications Learn more about this policy here. LLM Semantic Caching Policy LLM Semantic Caching policy (preview) is designed to reduce latency and reduce token consumption by caching responses based on the semantic content of prompts. Key Features Reduced Latency: Cache responses to frequently requested queries based to decrease response times. Improved Efficiency: Optimize resource utilization by reducing redundant model inferences. Content-Based Caching: Leverages semantic similarity to determine which response to retrieve from cache Learn more about this policy here. Get Started with Azure AI Model Inference API and Azure API Management We are committed to continuously improving our platform and providing the tools you need to leverage the full potential of large language models. Stay tuned as we roll out these new policies across all regions and watch for further updates and enhancements as we continue to expand our capabilities. Get started today and bring your intelligent application development to the next level with Azure API Management.
