Ignite 2024: Streamlining AI Development with an Enhanced User Interface, Accessibility, and Learning Experiences in Azure AI Foundry portal
Announcing Azure AI Foundry, a unified platform that simplifies AI development and management. The platform portal (formerly Azure AI Studio) features a revamped user interface, enhanced model catalog, new management center, improved accessibility and learning, making it easier than ever for Developers and IT Admins to design, customize, and manage AI apps and agents efficiently.Announcing management center and other tools to secure and govern Azure AI Foundry
We’re pleased to share new security and IT governance capabilities in Azure AI Foundry that can help organizations build and scale GenAI solutions that are secure by default, including a new management center, granular networking controls, and the general availability of data and service connections.Announcing Model Fine-Tuning Collaborations: Weights & Biases, Scale AI, Gretel and Statsig
As AI continues to transform industries, the ability to fine-tune models and customize them for specific use cases has become more critical than ever. Fine-tuning can enable companies to align models with their unique business goals, ensuring that AI solutions deliver results with greater precision However, organizations face several hurdles in their model customization journey: Lack of end-to-end tooling: Organizations struggle with fine-tuning foundation models due to complex processes, and the absence of tracking and evaluation tools for modifications. Data scarcity and quality: Limited access to large, high-quality datasets, along with privacy issues and high costs, complicate model training and fine-tuning. Shortage of fine-tuning expertise and pre-trained models: Many companies lack specialized knowledge and access to refined models for fine-tuning. Insufficient experimentation tools: A lack of tools for ongoing experimentation in production limits optimization of key variables like model diversity and operational efficiency. To address these challenges, Azure AI Foundry is pleased to announce new collaborations with Weights & Biases, Scale AI, Gretel and Statsig to streamline the process of model fine-tuning and experimentation through advanced tools, synthetic data and specialized expertise. Weights & Biases integration with Azure OpenAI Service: Making end-to-end fine-tuning accessible with tooling The integration of Weights & Biases with Azure OpenAI Service offers a comprehensive end-to-end solution for enterprises aiming to fine-tune foundation models such as GPT-4, GPT-4o, and GPT-4o mini. This collaboration provides a seamless connection between Azure OpenAI Service and Weights and Biases Models which offers powerful capabilities for experiment tracking, visualization, model management, and collaboration. With the integration, users can also utilize Weights and Biases Weave to evaluate, monitor, and iterate on the performance of their fine-tuned models powered AI applications in real-time. Azure's scalable infrastructure allows organizations to handle the computational demands of fine-tuning, while Weights and Biases offers robust capabilities for fine-tuning experimentation and evaluation of LLM-powered applications. Whether optimizing GPT-4o for complex reasoning tasks or using the lightweight GPT-4o mini for real-time applications, the integration simplifies the customization of models to meet enterprise-specific needs. This collaboration addresses the growing demand for tailored AI models in industries such as retail and finance, where fine-tuning can significantly improve customer service chatbots or complex financial analysis. Azure Open AI Service and Weights & Biases integration is now available in public preview. For further details on Azure OpenAI Service and Weights & Biases integration including real-world use-cases and a demo, refer to the blog here. Scale AI and Azure Collaboration: Confidently Implement Agentic GenAI Solutions in Production Scale AI collaborates with Azure AI Foundry to offer advanced fine-tuning and model customization for enterprise use cases. It enhances the performance of Azure AI Foundry models by providing high-quality data transformation, fine-tuning and customization services, end-to-end solution development and specialized Generative AI expertise. This collaboration helps improve the performance of AI-driven applications and Azure AI services such as Azure AI Agent in Azure AI Foundry, while reducing production time and driving business impact. "Scale is excited to partner with Azure to help our customers transform their proprietary data into real business value with end-to-end GenAI Solutions, including model fine-tuning and customization in Azure." Vijay Karunamurthy, Field CTO, Scale AI Checkout a demo in BRK116 session showcasing how Scale AI’s fine-tuned models can improve agents in Azure AI Foundry and Copilot Studio. In the coming months, Scale AI will offer fine-tuning services for Azure AI Agents in Azure AI Foundry. For more details, please refer to this blog and start transforming your AI initiatives by exploring Scale AI on the Azure Marketplace. Gretel and Azure OpenAI Service Collaboration: Revolutionizing data pipeline for custom AI models Azure AI Foundry is collaborating with Gretel, a pioneer in synthetic data and privacy technology, to remove data bottlenecks and bring advanced AI development capabilities to our customers. Gretel's platform enables Azure users to generate high-quality datasets for ML and AI through multiple approaches - from prompts and seed examples to differential privacy-preserved synthetic data. This technology helps organizations overcome key challenges in AI development including data availability, privacy requirements, and high development costs with support for structured, unstructured, and hybrid text data formats. Through this collaboration, customers can seamlessly generate datasets tailored to their specific use cases and industry needs using Gretel, then use them directly in Azure OpenAI Service for fine-tuning. This integration greatly reduces both costs and time compared to traditional data labeling methods, while maintaining strong privacy and compliance standards. The collaboration enables new use cases for Azure AI Foundry customers who can now easily use synthetic data generated by Gretel for training and fine-tuning models. Some of the new use cases include cost-effective improvements for Small Language Models (SLMs), improved reasoning abilities of Large Language Models (LLMs), and scalable data generation from limited real-world examples. This value is already being realized by leading enterprises. “EY is leveraging the privacy-protected synthetic data to fine-tune Azure OpenAI Service models in the financial domain," said John Thompson, Global Client Technology AI Lead at EY. "Using this technology with differential privacy guarantees, we generate highly accurate synthetic datasets—within 1% of real data accuracy—that safeguard sensitive financial information and prevent PII exposure. This approach ensures model safety through privacy attack simulations and robust data quality reporting. With this integration, we can safely fine-tune models for our specific financial use cases while upholding the highest compliance and regulatory standards.” The Gretel integration with Azure OpenAI Service is available now through Gretel SDK. Explore this blog describing a finance industry case study and checkout details in technical documentation for fine-tuning Azure OpenAI Service models with synthetic data from Gretel. Visit this page to learn more Statsig and Azure Collaboration: Enabling Experimentation in AI Applications Statsig is a platform for feature management and experimentation that helps teams manage releases, run powerful experiments, and measure the performance of their products. Statsig and Azure AI Foundry are collaborating to enable customers to easily configure and run experiments (A/B tests) in Azure AI-powered applications, using Statsig SDKs in Python, NodeJS and .NET. With these Statsig SDKs, customers can manage the configuration of their AI applications, manage the release of new configurations, run A/B tests to optimize model and application performance, and automatically collect metrics at the model and application level. Please check out this page to learn more about the collaboration and get detailed documentation here. Conclusion The new collaborations between Azure and Weights & Biases, Scale AI, Gretel and Statsig represent a significant step forward in simplifying the process of AI model customization. These collaborations aim to address the common pain points associated with fine-tuning models, including lack of end-to-end tooling, data scarcity and privacy concerns, lack of expertise and experimentation tooling. Through these collaborations, Azure AI Foundry will empower organizations to fine-tune and customize models more efficiently, ultimately enabling faster, more accurate AI deployments. Whether it’s through better model tracking, access to synthetic data, or scalable data preparation services, these collaborations will help businesses unlock the full potential of AI.
Continuously monitor your GenAI application with Azure AI Foundry and Azure Monitor
Now, Azure AI Foundry and Azure Monitor seamlessly integrate to enable ongoing, comprehensive monitoring of your GenAI application's performance from various perspectives, including token usage, operational metrics (e.g. latency and request count), and the quality and safety of generated outputs. With online evaluation, now available in public preview, you can continuously assess your application's outputs, regardless of its deployment or orchestration framework, using built-in or custom evaluation metrics. This approach can help organizations identify and address security, quality, and safety issues in both pre-production and post-production phases of the enterprise GenAIOps lifecycle. Additionally, online evaluations integrate seamlessly with new tracing capabilities in Azure AI Foundry, now available in public preview, as well as Azure Monitor Application Insights. Tying it all together, Azure Monitor enables you to create custom monitoring dashboards, visualize evaluation results over time, and set up alerts for advanced monitoring and incident response. Let’s dive into how all these monitoring capabilities fit together to help you be successful when building enterprise-ready GenAI applications. Observability and the enterprise GenAIOps lifecycle The generative AI operations (GenAIOps) lifecycle is a dynamic development process that spans all the way from ideation to operationalization. It involves choosing the right base model(s) for your application, testing and making changes to the flow, and deploying your application to production. Throughout this process, you can evaluate your application’s performance iteratively and continuously. This practice can help you identify and mitigate issues early and optimize performance as you go, helping ensure your application performs as expected. You can use the built-in evaluation capabilities in Azure AI Foundry, which now include remote evaluation and continuous online evaluation, to support end-to-end observability into your app’s performance throughout the GenAIOps lifecycle. Online evaluation can be used in many different application development scenarios, including: Automated testing of application variants. Integration into DevOps CI/CD pipelines. Regularly assessing an application’s responses for key quality metrics (e.g. groundedness, coherence, recall). Quickly responding to risky or inappropriate outputs that may arise during real-world use (e.g. containing violent, hateful, or sexual content) Production application monitoring and observability with Azure Monitor Application Insights. Now, let explore how you can use tracing for your application to begin your observability journey. Gain deeper insight into your GenAI application's processes with tracing Tracing enables comprehensive monitoring and deeper analysis of your GenAI application's execution. This functionality allows you to trace the process from input to output, review intermediate results, and measure execution times. Additionally, detailed logs for each function call in your workflow are accessible. You can inspect parameters, metrics, and outputs of each AI model utilized, which facilitates debugging and optimization of your application while providing deeper insights into the functioning and outputs of the AI models. The Azure AI Foundry SDK supports tracing to various endpoints, including local viewers, Azure AI Foundry, and Azure Monitor Application Insights. Learn more about new tracing capabilities in Azure AI Foundry. Continuously measure the quality and safety of generated outputs with online evaluation With online evaluation, now available in public preview, you can continuously evaluate your collected trace data for troubleshooting, monitoring, and debugging purposes. Online evaluation with Azure AI Foundry offers the following capabilities: Integration between Azure AI services and Azure Monitor Application Insights Monitor any deployed application, agnostic of deployment method or orchestration framework Support for trace data logged via the Azure AI Foundry SDK or a logging API of your choice Support for built-in and custom evaluation metrics via the Azure AI Foundry SDK Can be used to monitor your application during all stages of the GenAIOps lifecycle To get started with online evaluation, please review the documentation and code samples. Monitor your app in production with Azure AI Foundry and Azure Monitor Azure Monitor Application Insights excels in application performance monitoring (APM) for live web applications, providing many experiences to help enhance the performance, reliability, and quality of your applications. Once you’ve started collecting data for your GenAI application, you can access an out-of-the-box dashboard view to help you get started with monitoring key metrics for your application directly from your Azure AI project. Insights are surfaced to you via an Azure Monitor workbook that is linked to your Azure AI project, helping you quickly observe trends for key metrics, such as token consumption, user feedback, and evaluations. You can customize this workbook and add tiles for additional metrics or insights based on your business needs. You can also share it with your team so they can get the latest insights as well. Build enterprise-ready GenAI apps with Azure AI Foundry Ready to learn more? Here are other exciting announcements from Microsoft Ignite to support your GenAIOps workflows: New tracing and debugging capabilities to drive continuous improvement New ways to evaluate models and applications in pre-production New ways to document and share evaluation results with business stakeholders Whether you’re joining in person or online, we can’t wait to see you at Microsoft Ignite 2024. We’ll share the latest from Azure AI and go deeper into best practices for GenAIOps with these breakout sessions: Multi-agentic GenAIOps from prototype to production with dev tools Trustworthy AI: Advanced risk evaluation and mitigation Azure AI and the dev toolchain you need to infuse AI in all your apps
AI reports: Improve AI governance and GenAIOps with consistent documentation
AI reports are designed to help organizations improve cross-functional observability, collaboration, and governance when developing, deploying, and operating generative AI applications and fine-tuned or custom models. These reports support AI governance best practices by helping developers document the purpose of their AI model or application, its features, potential risks or harms, and applied mitigations, so that cross-functional teams can track and assess production-readiness throughout the AI development lifecycle and then monitor it in production. Starting in December, AI reports will be available in private preview in a US and EU Azure region for Azure AI Foundry customers. To request access to the private preview of AI reports, please complete the Interest Form. Furthermore, we are excited to announce new collaborations with Credo AI and Saidot to support customers’ end-to-end AI governance. By integrating the best of Azure AI with innovative and industry-leading AI governance solutions, we hope to provide our customers with choice and help empower greater cross-functional collaboration to align AI solutions with their own principles and regulatory requirements. Building on learnings at Microsoft Microsoft’s approach for governing generative AI applications builds on our Responsible AI Standard and the National Institute of Standards and Technology’s AI Risk Management Framework. This approach requires teams to map, measure, and manage risks for generative applications throughout their development cycle. A core asset of the first—and iterative—map phase is the Responsible AI Impact Assessment. These assessments help identify potential risks and their associated harms, as well as mitigations to address them. As development of an AI system progresses, additional iterations can help development teams document their progress in risk mitigation and allow experts to review the evaluations and mitigations and make further recommendations or requirements before products are launched. Post-deployment, these assessments become a source of truth for ongoing governance and audits, and help guide how to monitor the application in production. You can learn more about Microsoft’s approach to AI governance in our Responsible AI Transparency Report and find a Responsible AI Impact Assessment Guide and example template on our website. How AI reports support AI impact assessments and GenAIOps AI reports can help organizations govern their GenAI models and applications by making it easier for developers to provide the information needed for cross-functional teams to assess production-readiness throughout the GenAIOps lifecycle. Developers will be able to assemble key project details, such as the intended business use case, potential risks and harms, model card, model endpoint configuration, content safety filter settings, and evaluation results into a unified AI report from within their development environment. Teams can then publish these reports to a central dashboard in the Azure AI Foundry portal, where business leaders can track, review, update, and assess reports from across their organization. Users can also export AI reports in PDF and industry-standard SPDX 3.0 AI BOM formats, for integration into existing GRC workflows. These reports can then be used by the development team, their business leaders, and AI, data, and other risk professionals to determine if an AI model or application is fit for purpose and ready for production as part of their AI impact assessment processes. Being versioned assets, AI reports can also help organizations build a consistent bridge across experimentation, evaluation, and GenAIOps by documenting what metrics were evaluated, what will be monitored in production, and the thresholds that will be used to flag an issue for incident response. For even greater control, organizations can choose to implement a release gate or policy as part of their GenAIOps that validates whether an AI report has been reviewed and approved for production. Key benefits of these capabilities include: Observability: Provide cross-functional teams with a shared view of AI models and applications in development, in review, and in production, including how these projects perform in key quality and safety evaluations. Collaboration: Enable consistent information-sharing between GRC, development, and operational teams using a consistent and extensible AI report template, accelerating feedback loops and minimizing non-coding time for developers. Governance: Facilitate responsible AI development across the GenAIOps lifecycle, reinforcing consistent standards, practices, and accountability as projects evolve or expand over time. Build production-ready GenAI apps with Azure AI Foundry If you are interested in testing AI reports and providing feedback to the product team, please request access to the private preview by completing the Interest Form. Want to learn more about building trustworthy GenAI applications with Azure AI? Here’s more guidance and exciting announcements to support your GenAIOps and governance workflows from Microsoft Ignite: Learn about new GenAI evaluation capabilities in Azure AI Foundry Learn about new GenAI monitoring capabilities in Azure AI Foundry Learn about new IT governance capabilities in Azure AI Foundry Whether you’re joining in person or online, we can’t wait to see you at Microsoft Ignite 2024. We’ll share the latest from Azure AI and go deeper into capabilities that support trustworthy AI with these sessions: Keynote: Microsoft Ignite Keynote Breakout: Trustworthy AI: Future trends and best practices Breakout: Trustworthy AI: Advanced AI risk evaluation and mitigation Demo: Simulate, evaluate, and improve GenAI outputs with Azure AI Foundry Demo: Track and manage GenAI app risks with AI reports in Azure AI Foundry We’ll also be available for questions in the Connection Hub on Level 3, where you can find “ask the expert” stations for Azure AI and Trustworthy AI.Compare and select models with new benchmarking tools in Azure AI Foundry
Explore the latest updates to the model benchmarks experience in Azure AI Foundry portal. These updates include: direct integration with the Azure AI model catalog, new performance and cost metrics, and the ability to evaluate and compare models using your own private data.Accelerate enterprise GenAI application development with tracing in Azure AI Foundry
We are excited to announce the public preview of tracing in Azure AI Foundry, a powerful capability designed to enhance monitoring and debugging capabilities for your machine learning models and applications. Tracing allows you to gain deeper insights into the performance and behavior of your models, to help ensure they operate efficiently and effectively. Enable comprehensive monitoring and analysis of your application's execution Tracing allows you to trace application processes from input to output, review intermediate results, and measure execution times. Additionally, detailed logs for each function call in your workflow are accessible. You can inspect parameters, metrics, and outputs of each AI model used, for easier debugging and optimization of your application. The Azure AI Foundry SDK supports tracing to various endpoints including local viewers (Prompty trace viewer and Aspire dashboard), Azure AI Foundry, and Azure Monitor Application Insights. This flexibility helps you integrate tracing with any application, facilitating testing, evaluation, and deployment across different orchestrations and existing GenAI frameworks. Key Capabilities Basic debugging In situations where your application encounters an error, the trace functionality becomes extremely useful. It allows you to delve into the function causing the error, assess the frequency of exceptions, and troubleshoot using the provided exception message and stack trace. Detailed execution logs Tracing captures detailed traces of your model's execution, including data preprocessing, feature extraction, model inference, and post-processing steps. These details provide valuable insights into the inner workings of your models, helping you identify bottlenecks and optimize performance. For example, understanding the call flow of an application is crucial for complex AI systems where multiple components and services interact. By enabling tracing, developers can identify bottlenecks, understand dependencies, and optimize the flow for better performance. Performance metrics In addition to execution logs, tracing collects key performance metrics, such as latency and token utilization. These metrics allow you to monitor the efficiency of your models and make data-driven decisions to improve their performance. Building monitoring dashboards with the data collected from tracing can provide real-time visibility into the system's health. These dashboards can track key performance indicators (KPIs), provide alerts on anomalies, and help ensure that the AI services are running as expected. Error tracking Tracing helps you identify and troubleshoot errors in your models by capturing detailed error logs. Whether it's a data preprocessing issue or a model inference error, tracing provides the information you need to diagnose and fix problems quickly. This is particularly useful for capturing runtime exceptions, such as rate-limiting, which are critical for maintaining the reliability of your applications. Evaluations and user feedback You can attach evaluations metrics and user feedback to traces via online evaluation capabilities in Azure AI Foundry. Online evaluation allows you to incorporate real-world performance data and user insights into your monitoring process, to assess whether your models meet the desired quality standards. The Azure AI Foundry SDK simplifies the process of downstream evaluation, facilitating continuous improvement and validation of AI models against real-world data. Additionally, capturing user evaluations and interactions can provide insights into how users are engaging with the AI features, to inform user-centric improvements. Visualize Traces Azure AI Foundry provides robust tools for visualizing traces, both for local debugging and production-level monitoring. You can use these tools to gain a better understanding of your model's behavior and performance. The visualization capabilities include: Local debugging: Visualize traces during development to identify and resolve issues early, helping ensure that models are optimized before deployment. Visualize the data via Azure AI Foundry portal and Azure Monitor: In the post-deployment phase, developers often want to delve deeper into their applications' performance to optimize it further. For instance, you might want to monitor your GenAI application's performance, usage, and costs. In this scenario, the trace data for each request, the aggregated metrics, and user feedback become vital. Tracing seamlessly integrates with Azure Monitor, allowing you to visualize and analyze your model's performance metrics and logs using a customizable dashboard in Azure Monitor Application Insights. This integration provides a holistic view of your model's health and performance, enabling you to make informed decisions. Getting Started To start using tracing in Azure AI Foundry and Azure Monitor, follow these simple steps: Log Traces: Enable Tracing via Azure AI SDK for enabling tracing on Model inference API. Configure Logging: Set up the logging configuration to capture the desired level of detail for your model's execution. Enable Tracing in AI Studio: In your Azure AI Project, navigate to the Tracing and enable the feature for your models. Monitor and Analyze: Use Azure Monitor to visualize and analyze the collected logs and metrics, gaining insights into your model's performance. Find detailed guidance in our documentation: Overview of tracing capabilities in Azure AI Foundry Learn how to implement and use tracing with the Azure AI Foundry SDK Visualize your traces Build production-ready GenAI apps with Azure AI Foundry Want to learn about more ways to build and monitor enterprise-ready GenAI applications? Here are other exciting announcements from Microsoft Ignite to support your GenAIOps workflows: New ways to evaluate generative AI outputs for quality and safety New ways to monitor performance with Azure AI Foundry and Azure Monitor Whether you’re joining in person or online, we can’t wait to see you at Microsoft Ignite 2024. We’ll share the latest from Azure AI and go deeper into best practices for GenAIOps with these sessions: Microsoft Ignite Keynote Multi-agentic GenAIOps from prototype to production with dev tools Azure AI and the dev toolchain you need to infuse AI in all your apps
