Generative AI for Beginners - Full Videos Series Released!
With so many new technologies, tools and terms in the world of Generative AI, it can be hard to know where to start or what to learn next. "Generative AI for Beginners" is designed to help you on your learning journey no matter where you are now. We are happy announce that the "Generative AI for Beginners" course has received a major refresh - 18 new videos for each lesson.Building the Ultimate Nerdland Podcast Chatbot with RAG and LLM: Step-by-Step Guide
Large Language Models (LLMs) are popular in tech. In Belgium and the Netherlands, the podcast "Nerdland" is a favorite for tech and science fans. It covers topics like bioscience, space, robotics, and AI. With over 100 episodes, "Nerdland" is a goldmine of information. So, why not create a chatbot for "Nerdland" fans? This chatbot uses podcast content to engage and inform users. It allows the "Nerdland" community to interact with the content in new ways and makes the information accessible in many languages, thanks to LLMs' multi-language capabilities. This blog post explains the project's technical details, including the LLMs used, integration process, and deployment on Azure.RAG Deep Dive: 10-part live stream series
Our most popular RAG solution for Azure has now been deployed thousands of times by developers using it across myriad domains, like meeting transcripts, research papers, HR documents, and industry manuals. Based on feedback from the community (and often, thanks to pull requests from the community!), we've added the most hotly requested features: support for multiple document types, chat history with Cosmos DB, user account and login, data access control, multimodal media ingestion, private deployment, and more. This open-source RAG solution is powerful, but it can be intimidating to dive into the code yourself, especially now that it has so many optional features. That's why we're putting on a 10-part live series in January/February, diving deep into the solution and showing you all the ways you can use it. Register for the whole series on Reactor or scroll down to learn about each session and register for individual sessions. We look forward to seeing you in the live chat and hearing how you're using the RAG solution for your own domain. See you in the streams! 👋🏻 The RAG solution for Azure 13 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor Join us for the kick-off session, where we'll do a live demo of the RAG solution and explain how it all works. We'll step through the RAG flow from Azure AI Search to Azure OpenAI, deploy the app to Azure, and discuss the Azure architecture. Customizing our RAG solution 15 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor In our second session, we'll show you how to customize the RAG solution for your own domain - adding your own data, modifying the prompts, and personalizing the UI. Plus, we'll give you tips for local development for faster feature iteration. Optimal retrieval with Azure AI Search 20 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor Our RAG solution uses Azure AI Search to find matching documents, using state-of-the-art retrieval mechanisms. We'll dive into the mechanics of vector embeddings, hybrid search with RRF, and semantic ranking. We'll also discuss the data ingestion process, highlighting the differences between manual ingestion and integrated vectorization Multimedia data ingestion 22 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor Do your documents contain images or charts? Our RAG solution has two different approaches to handling multimedia documents, and we'll dive into both approaches in this session. The first approach is purely during ingestion time, where it replaces media in the documents with LLM-generated descriptions. The second approach stores images of the media alongside vector embeddings of the images, and sends both text and images to a multimodal LLM for question answering. Learn about both approaches in this session so that you can decide what to use for your app. User login and data access control 27 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor In our RAG flow, the app first searches a knowledge base for relevant matches to a user's query, then sends the results to the LLM along with the original question. What if you have documents that should only be accessed by a subset of your users, like a group or a single user? Then you need data access controls to ensure that document visibility is respected during the RAG flow. In this session, we'll show an approach using Azure AI Search with data access controls to only search the documents that can be seen by the logged in user. We'll also demonstrate a feature for user-uploaded documents that uses data access controls along with Azure Data Lake Storage Gen2. Storing chat history 29 January, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor Learn how we store chat history using either IndexedDB for client-side storage or Azure Cosmos DB for persistent storage. We'll discuss the API architecture and data schema choices, doing both a live demo of the app and a walkthrough of the code. Adding speech input and output 3 February, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor Our RAG solution includes optional features for speech input and output, powered either by the free browser SDKs or by the powerful Azure Speech API. We also offer a tight integration with the VoiceRAG solution, for those of you who want a real-time voice interface. Learn about all the ways you can add speech to your RAG chat in this session! Private deployment 5 February, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor To ensure that the RAG app can only be accessed within your enterprise network, you can deploy it to an Azure virtual network with private endpoints for each Azure service used. In this session, we'll show how to deploy the app to a virtual network that includes AI Search, OpenAI, Document Intelligence, and Blob storage. Then we'll log in to the virtual network using Azure Bastion with a virtual machine to demonstrate that we can access the RAG app from inside the network, and only inside the network. Evaluating RAG answer quality 10 February, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor How can you be sure that the RAG chat app answers are accurate, clear, and well formatted? Evaluation! In this session, we'll show you how to generate synthetic data and run bulk evaluations on your RAG app, using the azure-ai-evaluation SDK. Learn about GPT metrics like groundedness and fluency, and custom metrics like citation matching. Plus, discover how you can run evaluations on CI/CD, to easily verify that new changes don't introduce quality regressions. Monitoring and tracing LLM calls 12 February, 2025 | 11:30 PM UTC | 3:30 PM PT Register for the stream on Reactor When your RAG app is in production, observability is crucial. You need to know about performance issues, runtime errors, and LLM-specific issues like Content Safety filter violations. In this session, learn how to use Azure Monitor along with OpenTelemetry SDKs to monitor the RAG application.Building Retrieval Augmented Generation on VSCode & AI Toolkit
LLMs usually have limited knowledge about specific domains. Retrieval Augmented Generation (RAG) helps LLMs be more accurate and give relevant output to specific domains and datasets. We will see how we can do this for local models using AI Toolkit,Microsoft Build 2024: Essential Guide for AI Developers at Startups and Cloud-First Companies
Generative AI is advancing fast, with OpenAI’s GPT-4o leading the way. GPT-4o boasts improved multilingual understanding, faster responses, lower costs, and real-time processing of text, audio, and images. This boosts new Generative AI (GenAI) use cases. Explore cutting-edge solutions like models, frameworks, vector databases, and LLM observability platforms. Born-in-the-cloud companies are at the forefront of this AI revolution. Be part of the future at Microsoft Build 2024!Speed Up OpenAI Embedding By 4x With This Simple Trick!
In today’s fast-paced world of AI applications, optimizing performance should be one of your top priorities. This guide walks you through a simple yet powerful way to reduce OpenAI embedding response sizes by 75%—cutting them from 32 KB to just 8 KB per request. By switching from float32 to base64 encoding in your Retrieval-Augmented Generation (RAG) system, you can achieve a 4x efficiency boost, minimizing network overhead, saving costs and dramatically improving responsiveness. Let's consider the following scenario. Use Case: RAG Application Processing a 10-Page PDF A user interacts with a RAG-powered application that processes a 10-page PDF and uses OpenAI embedding models to make the document searchable from an LLM. The goal is to show how optimizing embedding response size impacts overall system performance. Step 1: Embedding Creation from the 10-Page PDF In a typical RAG system, the first step is to embed documents (in this case, a 10-page PDF) to store meaningful vectors that will later be retrieved for answering queries. The PDF is split into chunks. In our example, each chunk contains approximately 100 tokens (for the sake of simplicity), but the recommended chunk size varies based on the language and the embedding model. Assumptions for the PDF: - A 10-page PDF has approximately 3325 tokens (about 300 tokens per page). - You’ll split this document into 34 chunks (each containing 100 tokens). - Each chunk will then be sent to the embedding OpenAI API for processing. Step 2: The User Interacts with the RAG Application Once the embeddings for the PDF are created, the user interacts with the RAG application, querying it multiple times. Each query is processed by retrieving the most relevant pieces of the document using the previously created embeddings. For simplicity, let’s assume: - The user sends 10 queries, each containing 200 tokens. - Each query requires 2 embedding requests (since the query is split into 100-token chunks for embedding). - After embedding the query, the system performs retrieval and returns the most relevant documents (the RAG response). Embedding Response Size The OpenAI Embeddings models take an input of tokens (the text to embed) and return a list of numbers called a vector. This list of numbers represents the “embedding” of the input in the model so that it can be compared with another vector to measure similarity. In RAG, we use embedding models to quickly search for relevant data in a vector database. By default, embeddings are serialized as an array of floating-point values in a JSON document so each response from the embedding API is relatively large. The array values are 32-bit floating point numbers, or float32. Each float32 value occupies 4 bytes, and the embedding vector returned by models like OpenAI’s text-embedding-ada-002 typically consists of 1536-dimensional vectors. The challenge is the size of the embedding response: - Each response consists of 1536 float32 values (one per dimension). - 1536 float32 values result in 6144 bytes (1536 × 4 bytes). - When serialized as UTF-8 for transmission over the network, this results in approximately 32 KB per response due to additional serialization overhead (like delimiters). Optimizing Embedding Response Size One approach to optimize the embedding response size is to serialize the embedding as base64. This encoding reduces the overall size by compressing the data, while maintaining the integrity of the embedding information. This leads to a significant reduction in the size of the embedding response. With base64-encoded embeddings, the response size reduces from 32 KB to approximately 8 KB, as demonstrated below: base64 vs float32 Min (Bytes) Max (Bytes) Mean (Bytes) Min (+) Max (+) Mean (+) 100 tokens embeddings: text-embedding-3-small 32673.000 32751.000 32703.800 8192.000 (4.0x) (74.9%) 8192.000 (4.0x) (75.0%) 8192.000 (4.0x) (74.9%) 100 tokens embeddings: text-embedding-3-large 65757.000 65893.000 65810.200 16384.000 (4.0x) (75.1%) 16384.000 (4.0x) (75.1%) 16384.000 (4.0x) (75.1%) 100 tokens embeddings: text-embedding-ada-002 32882.000 32939.000 32909.000 8192.000 (4.0x) (75.1%) 8192.000 (4.0x) (75.2%) 8192.000 (4.0x) (75.1%) The source code of this benchmark can be found at: https://github.com/manekinekko/rich-bench-node (kudos to Anthony Shaw for creating the rich-bench python runner) Comparing the Two Scenarios Let’s break down and compare the total performance of the system in two scenarios: Scenario 1: Embeddings Serialized as float32 (32 KB per Response) Scenario 2: Embeddings Serialized as base64 (8 KB per Response) Scenario 1: Embeddings Serialized as Float32 In this scenario, the PDF embedding creation and user queries involve larger responses due to float32 serialization. Let’s compute the total response size for each phase: 1. Embedding Creation for the PDF: - 34 embedding requests (one per 100-token chunk). - 34 responses with 32 KB each. Total size for PDF embedding responses: 34 × 32 KB = 1088 KB = 1.088 MB 2. User Interactions with the RAG App: - Each user query consists of 200 tokens (which is split into 2 chunks of 100 tokens). - 10 user queries, requiring 2 embedding responses per query (for 2 chunks). - Each embedding response is 32 KB. Total size for user queries: Embedding responses: 20 × 32 KB = 640 KB. RAG responses: 10 × 32 KB = 320 KB. Total size for user interactions: 640 KB (embedding) + 320 KB (RAG) = 960 KB. 3. Total Size: Total size for embedding responses (PDF + user queries): 1088 KB + 640 KB = 1.728 MB Total size for RAG responses: 320 KB. Overall total size for all 10 responses: 1728 KB + 320 KB = 2048 KB = 2 MB Scenario 2: Embeddings Serialized as Base64 In this optimized scenario, the embedding response size is reduced to 8 KB by using base64 encoding. 1. Embedding Creation for the PDF: - 34 embedding requests. - 34 responses with 8 KB each. Total size for PDF embedding responses: 34 × 8 KB = 272 KB. 2. User Interactions with the RAG App: - Embedding responses for 10 queries, 2 responses per query. - Each embedding response is 8 KB. Total size for user queries: Embedding responses: 20 × 8 KB = 160 KB. RAG responses: 10 × 8 KB = 80 KB. Total size for user interactions: 160 KB (embedding) + 80 KB (RAG) = 240 KB 3. Total Size (Optimized Scenario): Total size for embedding responses (PDF + user queries): 272 KB + 160 KB = 432 KB. Total size for RAG responses: 80 KB. Overall total size for all 10 responses: 432 KB + 80 KB = 512 KB Performance Gain: Comparison Between Scenarios The optimized scenario (base64 encoding) is 4 times smaller than the original (float32 encoding): 2048 / 512 = 4 times smaller. The total size reduction between the two scenarios is: 2048 KB - 512 KB = 1536 KB = 1.536 MB. And the reduction in data size is: (1536 / 2048) × 100 = 75% reduction. How to Configure base64 encoding format When getting a vector representation of a given input that can be easily consumed by machine learning models and algorithms, as a developer, you usually call either the OpenAI API endpoint directly or use one of the official libraries for your programming language. Calling the OpenAI or Azure OpenAI APIs Using OpenAI endpoint: curl -X POST "https://api.openai.com/v1/embeddings" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "input": "The five boxing wizards jump quickly", "model": "text-embedding-ada-002", "encoding_format": "base64" }' Or, calling Azure OpenAI resources: curl -X POST "https://{endpoint}/openai/deployments/{deployment-id}/embeddings?api-version=2024-10-21" \ -H "Content-Type: application/json" \ -H "api-key: YOUR_API_KEY" \ -d '{ "input": ["The five boxing wizards jump quickly"], "encoding_format": "base64" }' Using OpenAI Libraries JavaScript/TypeScript const response = await client.embeddings.create({ input: "The five boxing wizards jump quickly", model: "text-embedding-3-small", encoding_format: "base64" }); A pull request has been sent to the openai SDK for Node.js repository to make base64 the default encoding when/if the user does not provide an encoding. Please feel free to give that PR a thumb up. Python embedding = client.embeddings.create( input="The five boxing wizards jump quickly", model="text-embedding-3-small", encoding_format="base64" ) NB: from 1.62 the openai SDK for Python will default to base64. Java EmbeddingCreateParams embeddingCreateParams = EmbeddingCreateParams .builder() .input("The five boxing wizards jump quickly") .encodingFormat(EncodingFormat.BASE64) .model("text-embedding-3-small") .build(); .NET The openai-dotnet library is already enforcing the base64 encoding, and does not allow setting encoding_format by the user (see). Conclusion By optimizing the embedding response serialization from float32 to base64, you achieved a 75% reduction in data size and improved performance by 4x. This reduction significantly enhances the efficiency of your RAG application, especially when processing large documents like PDFs and handling multiple user queries. For 1 million users sending 1,000 requests per month, the total size saved would be approximately 22.9 TB per month simply by using base64 encoded embeddings. As demonstrated, optimizing the size of the API responses is not only crucial for reducing network overhead but also for improving the overall responsiveness of your application. In a world where efficiency and scalability are key to delivering robust AI-powered solutions, this optimization can make a substantial difference in both performance and user experience. ■ Shoutout to my colleague Anthony Shaw for the the long and great discussions we had about embedding optimisations.RAG Time: Ultimate Guide to Mastering RAG!
RAG Time is a brand-new AI learning series designed to help developers unlock the full potential of Retrieval-Augmented Generation (RAG). If you’ve been looking for a way to build smarter, more efficient AI systems—join us in RAG Time, every Wednesday 9AM PT from March 5 through April 2 on Microsoft Developer YouTube. What's in RAG Time? RAG Time is a five-part learning journey, with new videos and blog posts releasing every week in March. The series features: 🔥 Expert-led discussions breaking down RAG fundamentals and best practices 🎤 Exclusive leadership interviews with AI leaders ⚡ Hands-on demos & real-world case studies showing RAG in action 🎨 Creative doodle summaries making complex concepts easier to grasp and remember 🛠 Samples & resources in the RAG Time repository so you can start building today What You’ll Learn The series is structured into five learning journeys, each tackling a crucial aspect of RAG-powered AI: 📌 March 5th, 9AM PT - Journey 1: RAG and Knowledge Retrieval Fundamentals – Start with the basics! Learn how RAG, search indexing, and vector search work together to create smarter AI retrieval systems. 📌 March 12th, 9AM PT - Journey 2: Build the Ultimate Retrieval System for RAG – Go beyond the fundamentals with hybrid search, semantic ranking, and relevance tuning to refine how AI retrieves the most relevant information. 📌 March 19th, 9AM PT - Journey 3: Optimize Your Vector Index for Scale – Learn how to scale vector search efficiently, optimize storage, and implement advanced techniques like quantization and Matryoshka learning for large-scale AI applications. 📌 March 26th, 9AM PT - Journey 4: RAG for All Your Data: Multimodal and Beyond – Move beyond text-based retrieval! Discover how to integrate images, audio, and structured data into your RAG workflows and leverage multimodal pipelines for next-level AI capabilities. 📌 April 2nd, 9AM PT - Journey 5: Hero Use Cases for RAG – Explore real-world implementations, industry-leading examples, and best practices, while diving into Responsible AI considerations to ensure ethical and impactful solutions. Why You Should Watch If you're a developer, data scientist, or AI enthusiast, this series is built for you! Whether you’re just getting started or looking to master enterprise-grade retrieval systems, RAG Time delivers practical knowledge, hands-on resources, and expert insights to help you stay ahead. Journey starts here 🚀 Start your journey from the RAG Time repo: https://aka.ms/rag-time. You'll find all the information about the video series, samples, documentation and doodles in the repo! Share your experience and feedback on GitHub discussions.
