Continuously monitor your GenAI application with Azure AI Foundry and Azure Monitor

Now, Azure AI Foundry and Azure Monitor seamlessly integrate to enable ongoing, comprehensive monitoring of your GenAI application's performance from various perspectives, including token usage, operational metrics (e.g. latency and request count), and the quality and safety of generated outputs.

With online evaluation, now available in public preview, you can continuously assess your application's outputs, regardless of its deployment or orchestration framework, using built-in or custom evaluation metrics. This approach can help organizations identify and address security, quality, and safety issues in both pre-production and post-production phases of the enterprise GenAIOps lifecycle. Additionally, online evaluations integrate seamlessly with new tracing capabilities in Azure AI Foundry, now available in public preview, as well as Azure Monitor Application Insights. Tying it all together, Azure Monitor enables you to create custom monitoring dashboards, visualize evaluation results over time, and set up alerts for advanced monitoring and incident response.

Let’s dive into how all these monitoring capabilities fit together to help you be successful when building enterprise-ready GenAI applications.

Observability and the enterprise GenAIOps lifecycle

The generative AI operations (GenAIOps) lifecycle is a dynamic development process that spans all the way from ideation to operationalization. It involves choosing the right base model(s) for your application, testing and making changes to the flow, and deploying your application to production. Throughout this process, you can evaluate your application’s performance iteratively and continuously. This practice can help you identify and mitigate issues early and optimize performance as you go, helping ensure your application performs as expected. 

Evaluation is a critical component of the GenAIOps lifecycle

You can use the built-in evaluation capabilities in Azure AI Foundry, which now include remote evaluation and continuous online evaluation, to support end-to-end observability into your app’s performance throughout the GenAIOps lifecycle. Online evaluation can be used in many different application development scenarios, including:

• Automated testing of application variants.
• Integration into DevOps CI/CD pipelines.
• Regularly assessing an application’s responses for key quality metrics (e.g. groundedness, coherence, recall).
• Quickly responding to risky or inappropriate outputs that may arise during real-world use (e.g. containing violent, hateful, or sexual content)
• Production application monitoring and observability with Azure Monitor Application Insights.

Now, let explore how you can use tracing for your application to begin your observability journey.

Gain deeper insight into your GenAI application's processes with tracing

Tracing enables comprehensive monitoring and deeper analysis of your GenAI application's execution. This functionality allows you to trace the process from input to output, review intermediate results, and measure execution times. Additionally, detailed logs for each function call in your workflow are accessible. You can inspect parameters, metrics, and outputs of each AI model utilized, which facilitates debugging and optimization of your application while providing deeper insights into the functioning and outputs of the AI models. 

The Azure AI Foundry SDK supports tracing to various endpoints, including local viewers, Azure AI Foundry, and Azure Monitor Application Insights. Learn more about new tracing capabilities in Azure AI Foundry.

Continuously measure the quality and safety of generated outputs with online evaluation 

With online evaluation, now available in public preview, you can continuously evaluate your collected trace data for troubleshooting, monitoring, and debugging purposes. Online evaluation with Azure AI Foundry offers the following capabilities:

• Integration between Azure AI services and Azure Monitor Application Insights
• Monitor any deployed application, agnostic of deployment method or orchestration framework
• Support for trace data logged via the Azure AI Foundry SDK or a logging API of your choice
• Support for built-in and custom evaluation metrics via the Azure AI Foundry SDK
• Can be used to monitor your application during all stages of the GenAIOps lifecycle

To get started with online evaluation, please review the documentation and code samples.

Use tracing to view performance and debug your application in Azure AI Foundry portal

Dig into tracing details for your application's inputs and outputs

Monitor your app in production with Azure AI Foundry and Azure Monitor

Azure Monitor Application Insights excels in application performance monitoring (APM) for live web applications, providing many experiences to help enhance the performance, reliability, and quality of your applications. Once you’ve started collecting data for your GenAI application, you can access an out-of-the-box dashboard view to help you get started with monitoring key metrics for your application directly from your Azure AI project. Insights are surfaced to you via an Azure Monitor workbook that is linked to your Azure AI project, helping you quickly observe trends for key metrics, such as token consumption, user feedback, and evaluations. You can customize this workbook and add tiles for additional metrics or insights based on your business needs. You can also share it with your team so they can get the latest insights as well.

Build enterprise-ready GenAI apps with Azure AI Foundry

Ready to learn more? Here are other exciting announcements from Microsoft Ignite to support your GenAIOps workflows:

• New tracing and debugging capabilities to drive continuous improvement
• New ways to evaluate models and applications in pre-production
• New ways to document and share evaluation results with business stakeholders

Whether you’re joining in person or online, we can’t wait to see you at Microsoft Ignite 2024. We’ll share the latest from Azure AI and go deeper into best practices for GenAIOps with these breakout sessions:

• Multi-agentic GenAIOps from prototype to production with dev tools
• Trustworthy AI: Advanced risk evaluation and mitigation
• Azure AI and the dev toolchain you need to infuse AI in all your apps