Data Storage in Azure OpenAI Service
Data Stored at Rest by Default Azure OpenAI does store certain data at rest by default when you use specific features (continue reading) In general, the base models are stateless and do not retain your prompts or completions from standard API calls (they aren't used to train or improve the base models). However, some optional service features will persist data in your Azure OpenAI resource. For example, if you upload files for fine-tuning, use the vector store, or enable stateful features like Assistants API Threads or Stored Completions, that data will be stored at rest by the service. This means content such as training datasets, embeddings, conversation history, or output logs from those features are saved within your Azure environment. Importantly, this storage is within your own Azure tenant (in the Azure OpenAI resource you created) and remains in the same geographic region as your resource. In summary, yes – data can be stored at rest by default when using these features, and it stays isolated to your Azure resource in your tenant. If you only use basic completions without these features, then your prompts and outputs are not persisted in the resource by default (aside from transient processing). Location and Deletion of Stored Data Location: All data stored by Azure OpenAI features resides in your Azure OpenAI resource’s storage, within your Azure subscription/tenant and in the same region (geography) that your resource is deployed. Microsoft ensures this data is secured — it is automatically encrypted at rest using AES-256 encryption, and you have the option to add a customer-managed key for double encryption (except in certain preview features that may not support CMK). No other Azure OpenAI customers or OpenAI (the company) can access this data; it remains isolated to your environment. Deletion: You retain full control over any data stored by these features. The official documentation states that stored data can be deleted by the customer at any time. For instance, if you fine-tune a model, the resulting custom model and any training files you uploaded are exclusively available to you and you can delete them whenever you wish. Similarly, any stored conversation threads or batch processing data can be removed by you through the Azure portal or API. In short, data persisted for Azure OpenAI features is user-managed: it lives in your tenant and you can delete it on demand once it’s no longer needed. Comparison to Abuse Monitoring and Content Filtering It’s important to distinguish the above data storage from Azure OpenAI’s content safety system (content filtering and abuse monitoring), which operates differently: Content Filtering: Azure OpenAI automatically checks prompts and generations for policy violations. These filters run in real-time and do not store your prompts or outputs in the filter models, nor are your prompts/outputs used to improve the filters without consent. In other words, the content filtering process itself is ephemeral – it analyzes the content on the fly and doesn’t permanently retain that data. Abuse Monitoring: By default (if enabled), Azure OpenAI has an abuse detection system that might log certain data when misuse is detected. If the system’s algorithms flag potential violations, a sample of your prompts and completions may be captured for review. Any such data selected for human review is stored in a secure, isolated data store tied to your resource and region (within the Azure OpenAI service boundaries in your geography). This is used strictly for moderation purposes – e.g. a Microsoft reviewer could examine a flagged request to ensure compliance with the Azure OpenAI Code of Conduct. When Abuse Monitoring is Disabled: if you disabled content logging/abuse monitoring (via an approved Microsoft process to turn it off). According to Microsoft’s documentation, when a customer has this modified abuse monitoring in place, Microsoft does not store any prompts or completions for that subscription’s Azure OpenAI usage. The human review process is completely bypassed (because there’s no stored data to review). Only the AI-based checks might still occur, but they happen in-memory at request time and do not persist your data at rest. Essentially, with abuse monitoring turned off, no usage data is being saved for moderation purposes; the system will check content policy compliance on the fly and then immediately discard those prompts/outputs without logging them. Data Storage and Deletion in Azure OpenAI “Chat on Your Data” Azure OpenAI’s “Chat on your data” (also called Azure OpenAI on your data, part of the Assistants preview) lets you ground the model’s answers on your own documents. It stores some of your data to enable this functionality. Below, we explain where and how your data is stored, how to delete it, and important considerations (based on official Microsoft documentation). How Azure Open AI on your data stores your data Data Ingestion and Storage: When you add your own data (for example by uploading files or providing a URL) through Azure OpenAI’s “Add your data” feature, the service ingests that content into an Azure Cognitive Search index (Azure AI Search). The data is first stored in Azure Blob Storage (for processing) and then indexed for retrieval: Files Upload (Preview): Files you upload are stored in an Azure Blob Storage account and then ingested (indexed) into an Azure AI Search index. This means the text from your documents is chunked and saved in a search index so the model can retrieve it during chat. Web URLs (Preview): If you add a website URL as a data source, the page content is fetched and saved to a Blob Storage container (webpage-<index name>), then indexed into Azure Cognitive Search. Each URL you add creates a separate container in Blob storage with the page content, which is then added to the search index. Existing Azure Data Stores: You also have the option to connect an existing Azure Cognitive Search index or other vector databases (like Cosmos DB or Elasticsearch) instead of uploading new files. In those cases, the data remains in that source (for example, your existing search index or database), and Azure OpenAI will use it for retrieval rather than copying it elsewhere. Chat Sessions and Threads: Azure OpenAI’s Assistants feature (which underpins “Chat on your data”) is stateful. This means it retains conversation history and any file attachments you use during the chat. Specifically, it stores: (1) Threads, messages, and runs from your chat sessions, and (2) any files you uploaded as part of an Assistant’s setup or messages. All this data is stored in a secure, Microsoft-managed storage account, isolated for your Azure OpenAI resource. In other words, Azure manages the storage for conversation history and uploaded content, and keeps it logically separated per customer/resource. Location and Retention: The stored data (index content, files, chat threads) resides within the same Azure region/tenant as your Azure OpenAI resource. It will persist indefinitely – Azure OpenAI will not automatically purge or delete your data – until you take action to remove it. Even if you close your browser or end a session, the ingested data (search index, stored files, thread history) remains saved on the Azure side. For example, if you created a Cognitive Search index or attached a storage account for “Chat on your data,” that index and the files stay in place; the system does not delete them in the background. How to Delete Stored Data Removing data that was stored by the “Chat on your data” feature involves a manual deletion step. You have a few options depending on what data you want to delete: Delete Chat Threads (Assistants API): If you used the Assistants feature and have saved conversation threads that you want to remove (including their history and any associated uploaded files), you can call the Assistants API to delete those threads. Azure OpenAI provides a DELETE endpoint for threads. Using the thread’s ID, you can issue a delete request to wipe that thread’s messages and any data tied to it. In practice, this means using the Azure OpenAI REST API or SDK with the thread ID. For example: DELETE https://<your-resource-name>.openai.azure.com/openai/threads/{thread_id}?api-version=2024-08-01-preview . This “delete thread” operation will remove the conversation and its stored content from the Azure OpenAI Assistants storage (Simply clearing or resetting the chat in the Studio UI does not delete the underlying thread data – you must call the delete operation explicitly.) Delete Your Search Index or Data Source: If you connected an Azure Cognitive Search index or the system created one for you during data ingestion, you should delete the index (or wipe its documents) to remove your content. You can do this via the Azure portal or Azure Cognitive Search APIs: go to your Azure Cognitive Search resource, find the index that was created to store your data, and delete that index. Deleting the index ensures all chunks of your documents are removed from search. Similarly, if you had set up an external vector database (Cosmos DB, Elasticsearch, etc.) as the data source, you should delete any entries or indexes there to purge the data. Tip: The index name you created is shown in the Azure AI Studio and can be found in your search resource’s overview. Removing that index or the entire search resource will delete the ingested data. Delete Stored Files in Blob Storage: If your usage involved uploading files or crawling URLs (thereby storing files in a Blob Storage container), you’ll want to delete those blobs as well. Navigate to the Azure Blob Storage account/container that was used for “Chat on your data” and delete the uploaded files or containers containing your data. For example, if you used the “Upload files (preview)” option, the files were stored in a container in the Azure Storage account you provided– you can delete those directly from the storage account. Likewise, for any web pages saved under webpage-<index name> containers, delete those containers or blobs via the Storage account in Azure Portal or using Azure Storage Explorer. Full Resource Deletion (optional): As an alternative cleanup method, you can delete the Azure resources or resource group that contain the data. For instance, if you created a dedicated Azure Cognitive Search service or storage account just for this feature, deleting those resources (or the whole resource group they reside in) will remove all stored data and associated indices in one go. Note: Only use this approach if you’re sure those resources aren’t needed for anything else, as it is a broad action. Otherwise, stick to deleting the specific index or files as described above. Verification: Once you have deleted the above, the model will no longer have access to your data. The next time you use “Chat on your data,” it will not find any of the deleted content in the index, and thus cannot include it in answers. (Each query fetches data fresh from the connected index or vector store, so if the data is gone, nothing will be retrieved from it.) Considerations and Limitations No Automatic Deletion: Remember that Azure OpenAI will not auto-delete any data you’ve ingested. All data persists until you remove it. For example, if you remove a data source from the Studio UI or end your session, the configuration UI might forget it, but the actual index and files remain stored in your Azure resources. Always explicitly delete indexes, files, or threads to truly remove the data. Preview Feature Caveats: “Chat on your data” (Azure OpenAI on your data) is currently a preview feature. Some management capabilities are still evolving. A known limitation was that the Azure AI Studio UI did not persist the data source connection between sessions – you’d have to reattach your index each time, even though the index itself continued to exist. This is being worked on, but it underscores that the UI might not show you all lingering data. Deleting via API/portal is the reliable way to ensure data is removed. Also, preview features might not support certain options like customer-managed keys for encryption of the stored data(the data is still encrypted at rest by Microsoft, but you may not be able to bring your own key in preview). Data Location & Isolation: All data stored by this feature stays within your Azure OpenAI resource’s region/geo and is isolated to your tenant. It is not shared with other customers or OpenAI – it remains private to your resource. So, deleting it is solely your responsibility and under your control. Microsoft confirms that the Assistants data storage adheres to compliance like GDPR and CCPA, meaning you have the ability to delete personal data to meet compliance requirements Costs: There is no extra charge specifically for the Assistant “on your data” storage itself. The data being stored in a cognitive search index or blob storage will simply incur the normal Azure charges for those services (for example, Azure Cognitive Search indexing queries, or storage capacity usage). Deleting unused resources when you’re done is wise to avoid ongoing charges. If you only delete the data (index/documents) but keep the search service running, you may still incur minimal costs for the service being available – consider deleting the whole search resource if you no longer need it Residual References: After deletion, any chat sessions or assistants that were using that data source will no longer find it. If you had an Assistant configured with a now-deleted vector store or index, you might need to update or recreate the assistant if you plan to use it again, as the old data source won’t resolve. Clearing out the data ensures it’s gone from future responses. (Each new question to the model will only retrieve from whatever data sources currently exist/are connected.) In summary, the data you intentionally provide for Azure OpenAI’s features (fine-tuning files, vector data, chat histories, etc.) is stored at rest by design in your Azure OpenAI resource (within your tenant and region), and you can delete it at any time. This is separate from the content safety mechanisms. Content filtering doesn’t retain data, and abuse monitoring would ordinarily store some flagged data for review – but since you have that disabled, no prompt or completion data is being stored for abuse monitoring now. All of these details are based on Microsoft’s official documentation, ensuring your understanding is aligned with Azure OpenAI’s data privacy guarantees and settings. Azure OpenAI “Chat on your data” stores your content in Azure Search indexes and blob storage (within your own Azure environment or a managed store tied to your resource). This data remains until you take action to delete it. To remove your data, delete the chat threads (via API) and remove any associated indexes or files in Azure. There are no hidden copies once you do this – the system will not retain context from deleted data on the next chat run. Always double-check the relevant Azure resources (search and storage) to ensure all parts of your data are cleaned up. Following these steps, you can confidently use the feature while maintaining control over your data lifecycle.Prototyping Agents with visual tools
Introduction Agents are gaining wide adoption in the emerging generative AI applications for organizations, transforming the way we interact with technology. Agent development using visual tools provides a low code / no code approach in prototyping agentic behavior. They help in creating preliminary versions of agentic applications, enabling development, testing, refining the functionalities before full-scale deployment. Prototyping tools for agents typically have the below features: Visual tools that allow for rapid creation, management and interaction with agentic applications Enable users to define and modify agents and multi-agent workflows through a point-and-click, drag-and-drop interface The interface should make it easier to set parameters for agents within a user-friendly environment and modify flows Chat interface to create chat sessions and view results in a conversational and interactive interface. This will enable interactive agent development and testing Enable adding memory and tools for agents Support for popular OSS agentic frameworks like autogen, langflow, llamaindex, etc Access to built-in add-ins and connectors to build sophisticated workflows Ability to extend the add-ins and build custom connectors Enable tracing for visualization, audit and governance of agents Ability to generate deployment code or provide an API and deploy the resulting workflows By leveraging these tools, developers can quickly prototype and iterate on agent designs, ensuring that the final product is robust, efficient, and capable of delivering a seamless user experience. In this blog, we will look at some OSS options for prototyping and developing agents. AutoGen Studio AutoGen Studio is a low-code interface built to help you rapidly prototype AI agents, enhance them with tools, compose them into teams and interact with them to accomplish tasks. While it is still not meant to be a production-ready app, AutoGen Studio can help users rapidly create, manage, and interact with agents that can learn, adapt, and collaborate. Declaratively define and modify agents and multi-agent workflows through a point and click, drag and drop interface (e.g., you can select the parameters of two agents that will communicate to solve your task). Create chat sessions with the specified agents and view results (e.g., view chat history, generated files, and time taken). Explicitly add capabilities to your agents and accomplish more tasks. Publish chat sessions to a local gallery. Agent Development Canvas Provides a visual interface for creating agent teams through declarative specification (JSON) or drag-and-drop Supports configuration of all core components: teams, agents, tools, models, and termination conditions Fully compatible with Autogen AgentChat component definitions Component map Edit Components: Code based editor: Playground Provides an interactive environment for testing and running agent teams Live message streaming between agents Visual representation of message flow through a control transition graph Interactive sessions with teams using UserProxyAgent Full run control with the ability to pause or stop execution Tracing and audit Deployment: AutoGen Studio provides options through Docker and python options for depoying the agents. Semantic Workbench Semantic Workbench is another tool to prototype agents. The workbench provides a user-friendly UI for creating conversations with one or more agents, configuring settings, and exposing various behaviours. The Semantic Workbench is composed of three main components: Workbench Service (Python): The backend service that handles core functionalities. Workbench App (React/Typescript): The frontend web user interface for interacting with workbench and assistants. Assistant Services (Python, C#, etc.): any number of assistant services that implement the service protocols/APIs, developed using any framework and programming language of your choice. Designed to be agnostic of any agent framework, language, or platform, the Semantic Workbench facilitates experimentation, development, testing, and measurement of agent behaviours and workflows. Assistants integrate with the workbench via a RESTful API, allowing for flexibility and broad applicability in various development environments. Dashboard Provides a view on existing agents added to the workbench. Agent Development Canvas Canvas to add and import new assistants to the workbench. Agent landing page Option for viewing past conversations, add new conversations to test the flow and assistant configurations. Configure Agents Designing instruction prompts, guardrails, etc.,. Conversation Testing Interface to test the assistant flow. Debugging conversations Logging the conversation trace and using the trace information for debugging Ironclad- Rivet Rivet is a visual programming environment for building AI agents with LLMs. Iterate on your prompt graphs in Rivet, then run them directly in your application. With Rivet, teams can effectively design, debug, and collaborate on complex LLM prompt graphs, and deploy them in their own environment. Agent Development Canvas Sample Flow Flow output Plugins Prompt Designer Testing Letta ADE: The Letta ADE is a graphical user interface for creating, deploying, interacting and observing with agents. Letta enables developers to build and deploy stateful AI agents - agents that maintain memory and context across long-running conversations. The Agent Development Environment (ADE) provides a visual interface for building and monitoring agents, with real-time visibility into agent memory and behavior. Letta’s context management system intelligently manages memory. Post version 0.5.0, the UI interface is not available in local and we are dependent on a web based interface hosted in letta servers – though the backend can still be local. Letta enables developers to build and deploy stateful AI agents - agents that maintain memory and context across long-running conversations. Letta Desktop Letta agents live inside a Letta Server, which persists them to a database. You can interact with the Letta agents inside your Letta Server with the ADE (a visual interface) and connect your agents to external application via the REST API and Python & TypeScript SDKs. Letta Desktop bundles together the Letta Server and the Agent Development Environment (ADE) into a single application Adding LLM backends The Letta server can be connected to various LLM API backends Flowise: Flowise is an open source low-code tool for developers to build customized LLM orchestration flows & AI agents. Authoring Canvas offers advanced interface with options to visually add langchain and llamaindex objects for chatflow. Some of the key features include Authoring canvas for chat flows and agents Chains: Manage the flow of interaction with users, providing a framework to design and implement flows tailored to specific tasks or objectives. Language Models: Responsible for language generation and understanding, optimized for various needs. Prompts: Keywords or patterns that trigger specific actions or responses based on user inputs. Output Parsers: Analyze generated data or responses to extract necessary information. Supports integration with frameworks like Langchain, llamaindex, litellm Offer enterprise plans for SSO support Flowise also has an advanced interface to build agent flows. Tracing Flowise open source repository has a built-in telemetry that collects anonymous usage information. Marketplace Flowise has a large number of templates available that can be useful as a starter template for complex agents. Langflow: Langflow is an OSS framework for building multi-agent and RAG applications. It Python-powered, fully customizable, and LLM and vector store agnostic. Agent Development Canvas Langflow provides a canvas that can easily connect different components, such as prompts, language models, and data sources to help build agentic applications. Each component in a flow is a node that performs a specific task, like an AI model or a data source. Each component has a Configuration menu. Code pane shows a component's underlying Python code. Components are connected with edges to form flows. Components Langflow 1.1 introduced a new agent component, designed to support complex orchestration with built-in model selection, chat memory, and traceable intermediate steps for reasoning and tool-calling actions. Playground Langflow provides a dynamic interface designed for real-time interaction with LLMs, allowing users to chat, access memories, and monitor inputs and outputs. Here, users can directly prototype their models, making adjustments and observing different outcomes API Langflow provides an API pane for code templates to call flows from applications. Starter templates: Langflow has a library of pre-built templates categorized by use case and methodology. Langflow Store Langflow has integration and custom connectors for Flows and components that can be downloaded and imported to the workflows. Feature Comparison: Feature Autogen Studio Semantic Workbench Letta License CC-BY-4.0, MIT licenses MIT license Apache-2.0 license Canvas for chat / agent flow dev Canvas available, limited visual / low-code capabilities, Pro-code. Canvas available, limited visual/low-code capabilities. Pro-code. Limited capabilities. Limited local dev interface post 0.5.0 version. Chat sessions / test flows Available Available Available Templates Tracing Available Available Available Add-in connectors Limited / no options by default. Can be custom built Limited / no options by default. Can be custom built Provides memory tools by default. Deploy agents Available Available Currently on limited preview. Feature Langflow flowise Rivet License MIT license Apache-2.0 license MIT license Canvas for chat / agent flow dev Canvas with rich UI / UX capabilities Canvas with rich UI / UX capabilities Playground available, better UI / UX for agent creation Chat sessions / test flows Available Available Available Templates Tracing Available Available Available Add-in connectors Wide range of connectors available Wide range of connectors available Wide range of built-in connectors Deploy agents Available Available Available References How to develop AI Apps and Agents in Azure - A Visual Guide | All things Azure AutoGen Studio — AutoGen Semantic Workbench for Agentic AI Development microsoft/semanticworkbench: A versatile tool designed to help prototype intelligent assistants, agents and multi-agentic systems Ironclad/rivet: The open-source visual AI programming environment and TypeScript library Introduction to Rivet | Rivet https://github.com/Ironclad/rivet letta-ai/letta: Letta (formerly MemGPT) is a framework for creating LLM services with memory. Agent Development Environment (ADE) — Letta https://docs.flowiseai.com/ https://github.com/FlowiseAI/Flowise https://volcano-ice-cd6.notion.site/Introduction-to-Practical-Building-LLM-Applications-with-Flowise-LangChain-03d6d75bfd20495d96dfdae964bea5a5#eeeab3f52f4047aaa218317f9892aa26 https://github.com/langflow-ai/langflow
Numonix supercharges their value to clients with multimodality using Azure AI Content Understanding
Numonix is a compliance recording company that specializes in capturing, analyzing, and managing customer interactions across various modalities. They’re committed to revolutionizing how businesses extract value from customer interactions, offering solutions that empower businesses to make informed decisions while accelerating revenue growth, enhancing customer experiences, and maintaining regulatory compliance. By leveraging state-of-the-art technology, they provide powerful tools that help organizations ensure compliance, mitigate risk, and discover actionable insights from communications data. Numonix has many call center clients for whom regulatory compliance is crucial. They needed a way to help their clients monitor calls and customer interactions, solve call-compliance issues, and gather and extract valuable insights from their data. They were further challenged by the need to process and extract insights from different formats, including audio, images, video, and text, while improving data quality and accuracy, and streamlining workflows. However, the manual process to evaluate calls is cumbersome, inaccurate, inefficient, and resource intensive. Plus, their legacy media deployments required a lot of complex hardware on-premises, which hindered their ability to react quickly. If a client suddenly ran out of storage, Numonix had to scramble to quickly scale up in order to provide the needed storage. Plus, customers had to pay upfront for potential scale that they might want to reach in the future. To solve these issues, they partnered with Microsoft to leverage Azure’s flexible cloud services and updated their service to better manage multimodal content. To do this, they leveraged Azure AI Content Understanding, which helps businesses and developers create multimodal AI apps from varied data types, and helps to unify their separate workflows related to individual modalities. It offers prebuilt templates, a streamlined workflow, and opportunities to customize outputs for use-cases such as call center analytics, marketing automation, and content search, without needing specialized AI skills, all while maintaining robust security and accuracy. Now, Numonix has the ability to capture insights from all recorded call data in multiple modes, including audio, video, text, and images. They can transcribe and analyze content from calls and meetings, understand context, watch videos of call interactions, and ensure compliance across all conversations. With the transition to Azure, the challenges associated with on-premises server environments, including costs and ongoing maintenance, have been eliminated. Now, when it’s determined that extra space is needed, the service seamlessly scales to the increased volume of content being processed. “We have successfully delivered seamless scalability for our customers, including the capability to integrate their Azure Blob storage accounts with our platform,” said Michael Levy, Founder and CEO. “Our platform offers a combination of robust functionality, exceptional flexibility, and comprehensive security—key advantages that we are proud to provide to our customers.” “Adopting Azure has been a transformative decision, enabling us to deliver a cloud-native solution that facilitates faster deployment for our customers while ensuring long-term scalability, technology advancements, and robust security. Scaling from 1,000 to 10,000 users is now a seamless license adjustment, with no need for complex backend modifications or DevOps intervention." Empowered by Azure AI Content Understanding, Numonix now offers industry-leading quality management. They’ve been able to take their customers’ call coverage from roughly around 3% all the way up to 100%, and the cost is lower because it’s far less resource-intensive. “It’s been a productivity multiplier,” says Evan Kahan, CTO of Numonix. “Leveraging Azure AI Content Understanding across multiple modalities has allowed us to supercharge the value of recorded data Numonix captures on behalf of our customers.” At the same time, they’ve expanded their business capabilities to extend even more value to their clients. As a result of implementing Azure AI Content Understanding, they’ve grown from only offering audio, to also offering video, screen sharing, document sharing, chat, live interaction, and document intelligence. In addition, they can leverage multiple tools for their customers—tools like secure and compliant meeting insights, PII redactions, and automated risk alerts all come together to help clients gather and unlock their data's full potential, drive efficiency, and innovate. Says Evan Kahan: “We’re able to bring that all together…to make sure that the audio and video that you have provides the most value that you could possibly get out of it, which we really didn’t have access to before. Everything we got out of it before was a small piece of the picture. Now, with Azure AI Content Understanding, we’re really able to leverage all of these Microsoft tools to bring this full picture to our customers.” Get started: Learn more about Azure AI Content Understanding. Try Azure AI Content Understanding in Azure AI Foundry.` Our commitment to Trustworthy AI Organizations across industries are leveraging Azure AI and Copilot capabilities to drive growth, increase productivity, and create value-added experiences. We’re committed to helping organizations use and build AI that is trustworthy, meaning it is secure, private, and safe. We bring best practices and learnings from decades of researching and building AI products at scale to provide industry-leading commitments and capabilities that span our three pillars of security, privacy, and safety. Trustworthy AI is only possible when you combine our commitments, such as our Secure Future Initiative and our Responsible AI principles, with our product capabilities to unlock AI transformation with confidence.From Foundry to Fine-Tuning: Topics you Need to Know in Azure AI Services
With so many new features from Azure and newer ways of development, especially in generative AI, you must be wondering what all the different things you need to know are and where to start in Azure AI. Whether you're a developer or IT professional, this guide will help you understand the key features, use cases, and documentation links for each service. Let's explore how Azure AI can transform your projects and drive innovation in your organization. Stay tuned for more details! Term Description Use Case Azure Resource Azure AI Foundry A comprehensive platform for building, deploying, and managing AI-driven applications. Customizing, hosting, running, and managing AI applications. Azure AI Foundry AI Agent Within Azure AI Foundry, an AI Agent acts as a "smart" microservice that can be used to answer questions (RAG), perform actions, or completely automate workflows. can be used in a variety of applications to automate tasks, improve efficiency, and enhance user experiences. Link AutoGen An open-source framework designed for building and managing AI agents, supporting workflows with multiple agents. Developing complex AI applications with multiple agents. Autogen Multi-Agent AI Systems where multiple AI agents collaborate to solve complex tasks. Managing energy in smart grids, coordinating drones. Link Model as a Platform A business model leveraging digital infrastructure to facilitate interactions between user groups. Social media channels, online marketplaces, crowdsourcing websites. Link Azure OpenAI Service Provides access to OpenAI’s powerful language models integrated into the Azure platform. Text generation, summarization, translation, conversational AI. Azure OpenAI Service Azure AI Services A suite of APIs and services designed to add AI capabilities like image analysis, speech-to-text, and language understanding to applications. Image analysis, speech-to-text, language understanding. Link Azure Machine Learning (Azure ML) A cloud-based service for building, training, and deploying machine learning models. Creating models to predict sales, detect fraud. Azure Machine Learning Azure AI Search An AI-powered search service that enhances information to facilitate exploration. Enterprise search, e-commerce search, knowledge mining. Azure AI Search Azure Bot Service A platform for developing intelligent, enterprise-grade bots. Creating chatbots for customer service, virtual assistants. Azure Bot Service Deep Learning A subset of ML using neural networks with many layers to analyze complex data. Image and speech recognition, natural language processing. Link Multimodal AI AI that integrates and processes multiple types of data, such as text and images(including input & output). Describing images, answering questions about pictures. Azure OpenAI Service, Azure AI Services Unimodal AI AI that processes a single type of data, such as text or images (including input & output). Writing text, recognizing objects in photos. Azure OpenAI Service, Azure AI Services Fine-Tuning Models Adapting pre-trained models to specific tasks or datasets for improved performance. Customizing models for specific industries like healthcare. Azure Foundry Model Catalog A repository of pre-trained models available for use in AI projects. Discovering, evaluating, fine-tuning, and deploying models. Model Catalog Capacity & Quotas Limits and quotas for using Azure AI services, ensuring optimal resource allocation. Managing resource usage and scaling AI applications. Link Tokens Units of text processed by language models, affecting cost and performance. Managing and optimizing text processing tasks. Link TPM (Tokens per Minute) A measure of the rate at which tokens are processed, impacting throughput and performance. Allocating and managing processing capacity for AI models. Link PTU(provisioned throughput) provisioned throughput capability allows you to specify the amount of throughput you require in a deployment. Ensuring predictable performance for AI applications. LinkAnnouncing the General Availability of Document Intelligence v4.0 API
The Document Intelligence v4.0 API is now generally available! This latest version of Document Intelligence API brings new and updated capabilities across the entire product including updates to Read and Layout APIs for content extraction, prebuilt and custom extraction models for schema extraction from documents and classification models. Document Intelligence has all the tools to enable RAG and document automation solutions for structured and unstructured documents. Enhanced Layout capabilities This release brings significant updates to our Layout capabilities, making it the default choice for document ingestion with enhanced support for Retrieval-Augmented Generation (RAG) workflows. The Layout API now offers a markdown output format that provides a better representation of document elements such as headers, footers, sections, section headers and tables when working with Gen AI models. This structured output enables semantic chunking of content, making it easier to ingest documents into RAG workflows and generate more accurate results. Try Layout in the Document Intelligence Studio or use Layout as a skill in your RAG pipelines with Azure Search. Searchable PDF output Document Intelligence no longer outputs only JSON! With the 4.0 release, you can now generate a searchable PDF output from an input document. The recognized text is overlaid over the scanned text, making all the content in the documents instantly searchable. This feature enhances the accessibility and usability of your documents, allowing for quick and efficient information retrieval. Try the new searchable PDF output in the Studio or learn more. Searchable PDF is available as an output from the Read API at no additional cost. This release also includes several updates to the OCR model to better handle complex text recognition challenges. New and updated Prebuilt models Prebuilt models offer a simple API to extract a defined schema from known document types. The v4.0 release adds new prebuilt models for mortgage processing, bank document processing, paystub, credit/debit card, check, marriage certificate, and prebuilt models for processing variants of the 1095, W4, and 1099 tax forms for US tax processing scenarios. These models are ideal for extracting specific details from documents like bank statements, checks, paystubs, and various tax forms. With over 22 prebuilt model types, Document Intelligence has models for common documents in procurement, tax, mortgage and financial services. See models overview for a complete list of document types supported with prebuilt models. Query field add-on capability Query field is an add-on capability to extend the schema extracted from any prebuilt model. This add-on capability is ideal when you have simple fields that need to be extracted. Query field also work with Layout, so for simple documents, you don’t need to train a custom model and can just define the query fields to begin processing the document with no training. Query field supports a maximum of 20 fields per request. Try query field in the Document Intelligence Studio with Layout or any prebuilt model. Document classification model The custom classification models are updated to improve the classification process and now support multi-language documents and incremental training. This allows you to update the classifier model with additional samples or classes without needing the entire training dataset. Classifiers also support analyzing Office document types (.docx, .pptx, and .xls). Version 4.0 adds a classifier copy operation for copying your classifier across resources, regions or subscriptions making model management easier. This version also introduces some changes in the splitting behavior, by default, the custom classification model no longer splits documents during analysis. Learn more about the classification and splitting capabilities. Improvements to Custom Extraction models Custom extraction models now output confidence scores for tables, table rows, and cells. This makes the process of validating model results much easier and provides the tools to trigger human reviews. Custom model capabilities have also improved with the addition of signature detection to neural models and support for overlapping fields. Neural models now include a paid training tier for when you have a large dataset of labeled documents to train. Paid training enables longer training to ensure you have a model that performs better on the different variations in your training dataset. Learn more about improvements to custom extraction models. New implementation of model compose for greater flexibility With custom extraction models in the past, you could compose multiple models into a single composed model. When a document was analyzed with a composed model, the service picked the model best suited to process the document. With this version, the model compose introduces a new implementation requiring a classification model in addition to the extraction models. This enables processing multiple instances of the same document with splitting, conditional routing and more. Learn more about the new model compose implementation. Get started with the v4.0 API today The Document Intelligence v4.0 API is packed with many more updates. Start with the what’s new page to learn more. You can try all of the new and updated capabilities in the Document Intelligence Studio. Explore the new REST API or the language specific SDKs to start building our updating your document workflows.
Analyze complex documents with Azure Document Intelligence Markdown Output and Azure OpenAI
In today’s digital era, where data is the new gold, efficiently extracting and processing information from complex documents, including those with dynamic tables, is crucial for businesses. Microsoft’s Azure AI services offer robust solutions for tackling these challenges, especially through the Document Intelligence Layout model. In this blog post, we will explore how you can use markdown output to enhance the capabilities of Azure Document Intelligence Layout model and subsequently feed this refined data into Azure AI for comprehensive information extraction.Document Field Extraction with Generative AI
Adoption of Generative AI technologies is accelerating, driven by the transformative potential they offer across various industry sectors. Azure AI enables organizations to create interactive and responsive AI solutions customized to their requirements, playing a significant part helping businesses harness Generative AI effectively. With the new custom field extraction preview, you can leverage generative AI to efficiently extract fields from documents, ensuring standardized output and a repeatable process to support document automation workflows.
