The Future of AI: Harnessing AI for E-commerce - personalized shopping agents

Microsoft

Feb 28, 2025

The Future of AI blog series is an evolving collection of posts from the AI Futures team in collaboration with subject matter experts across Microsoft. In this series, we explore tools and technologies that will drive the next generation of AI. Explore more at: https://aka.ms/the-future-of-ai

Harnessing AI for eCommerce - personalized shopping agents

In the dynamic sector of eCommerce, personalization is crucial for enhancing user experience and driving engagement. Retailers are leveraging AI to create personalized shopping agents with multimodal capabilities.

Imagine being able to upload an image of a celebrity’s outfit and instantly receiving product recommendations that match the style. This seamless interaction is made possible by orchestrating specialized AI skills and agents. Let’s dive into how we built our personalized shopping agent to bring this vision to life.

How Our AI Shopping Agent Works

Our system is built using multiple services from Azure AI Foundry. At the center it has an orchestrator or a state machine and a couple of independent skills/agents with carefully crafted prompts, to deliver accurate and personalized recommendations. Watch our eCommerce agent demo.

Step 1: Query-preprocess: Content Moderation & Rephrasing

When a user uploads an image, the system first processes it through the Azure AI Content Safety service to ensure compliance and prevent inappropriate or harmful content from entering the system. Additionally, the query is rephrased to incorporate all relevant context from the conversation history, ensuring a more accurate and meaningful search experience.

Step 2: Image Analysis & Apparel Recognition

If the image passes moderation, the orchestrator invokes an image describer agent, which identifies key apparel elements. For example, if a celebrity is wearing a blue suit, the system detects this and provides structured metadata. The agent may then suggest complementary pieces, such as a matching button-down shirt.

Step 3: Smart Product Recommendations

With the context preserved, the recommender skill is triggered. It generates search queries tailored for searching the catalog index—queries that are more targeted to produce highly relevant and personalized results. Using these search queries, the search skill is triggered by the orchestrator to find relevant products.

The Technology Behind Our AI Shopping Agent

Data Ingestion and Indexing

Our ingestion service processes large volumes of product catalog data, which are then vectorized and indexed using Azure AI Search. This enables semantic search capabilities, allowing users to find products not just through keywords but through meaning-based searches.

Enhancing Product Descriptions with AI

We leverage Azure OpenAI Service to enrich product descriptions, standardize categories, and improve overall searchability. By doing so, we enhance product discoverability and ensure customers receive highly relevant results.

Session Management for Seamless Interactions

The session manager preserves user context across interactions, enabling the aggregation of intermediate results and delivering real-time responses to enhance user engagement throughout the shopping journey. To ensure efficient communication, a message queue between the orchestrator and the session manager facilitates seamless sharing of intermediate results.

AI Orchestration and Tailored Responses

The AI orchestrator and domain-specific skills use the Azure OpenAI Service chat completion API to generate conversational, context-aware responses. This creates a smooth and intuitive shopping experience.

Evaluating for High-Quality AI Performance

Evaluation plays a critical role in the AI development lifecycle. Using Azure AI Foundry evaluation tools, we rigorously tested our solution to ensure it meets the highest standards for accuracy, relevance, and responsiveness.

We're excited about the potential of personalized shopping concierge agents and look forward to seeing how they enhance e-commerce experience.