Optimize Retail Service in Travel & Transportation with Azure Percept

Have you ever traveled in an airplane? Do you know what effort it takes for the airline to ensure the correct food & beverages inventory is needed? Airline crews manually conduct onboard inventory. It is a tedious, mundane, and time-consuming activity prone to human error.

Our goal is to solve this problem efficiently, avoid human error, and reduce the time spent on this activity. We, a team from DXC Luxoft, took the use case of Gin counting (a subset of Beverage Inventory Counting as part of any airline). We demonstrated how Azure Percept could be used to solve it efficiently.

The benefit of augmenting this process goes beyond improving crew utilization. The augmentation will provide additional data that stakeholders can leverage to discover insights that could use to:

• Tailor inventory to the pattern of customer demand, driving improved throughput, and sales
• Optimization of Supply Chain and Stock management
• Identification of anomalies, e.g., Inconsistent Shrinkage

This blog post outlines how to get insights quickly from video and audio streams with no-code low-code using Azure Percept DK (Dev Kit).

Overview of the Solution

The solution we developed using Azure Percept was Gin Counting Solution within the context of a retail inventory management system. We developed a custom model trained with Customvision.ai, deployed it to Azure Percept DK, and then created a custom Gin Counting IoT (Internet of Things) Edge module that runs on Azure Percept DK.

A brief demo of the solution can be found in the following YouTube video:

We had three varieties of Gin – Bombay, Hendricks, and Tanquery. We started training the model with 20 images for each Gin Type. And we went up to 8 iterations with over 100 images for each type of Gin to get better Precision & Recall. Below is the architecture diagram of the solution highlighting various components and Azure services.

Solution Implementation

We implemented the solution with the following 5 steps:

Step 1: Used CustomVision.ai project and trained the model with three labels

We trained the model with three labels, a.k.a. Gin Types – Bombay, Hendricks, Tanquery. We did eight model iterations to get good precision and recall, and the model performed with high accuracy when exported to iOS mobile phones.

Step 2: Deployed Trained Model to Azure Percept DK

After we trained the to our satisfaction based on precision and recall, we deployed it to Azure Percept DK using Azure Percept Studio's 'Deploy the custom project' feature. With this feature, the developer's life becomes easy as there is no code to be written for this step.

Step 3: Developed a Custom IoT Edge Module for Gin Counting

Once the model is deployed to Azure Percept DK, the Azureeyemodule starts processing the Gin bottles and showing them with bounding boxes and confidence scores. We developed a custom module to count the Gin types for each label, and this module was written in Python with VS (Visual Studio) Code using the Azure IoT Edge Module template.

Step 4: Deployed Gin Counting Custom IoT Edge Module to Azure Percept DK

After developing the custom module, we directly deployed it to Azure Percept DK from VS Code. The IoT Edge Module template provides 'deployment.template.json' and 'module.json.' We need to change the configuration parameters so that the module can be packaged as a container and load it to Azure Container Registry. The Azure Edge agent downloads it and deploys it to the device.

Step 5: Process the Video Stream

Now, the Azure Percept DK sends the video stream to the Gin Counting custom module, and the Gin counting module processes it, creates a summarized data, and sends it to the Azure IoT hub. We created two consumer groups for Stream Analytics and another for Time Series Insights, and stream Analytics sends the data to Power BI (Business Intelligence) for visualization.

Closing Remarks

Azure Percept provides an effortless way to get insights from video and audio streams. As an immediate next step, we plan to evolve the solution with the following steps:

1. Train the model with Semantic Segmentation instead of bounding boxes: Our current model includes only bounding boxes to find the Gin bottles. Due to this, it delivers low accuracy in finding the Gin bottles. To classify the objects more precisely, we like to use Semantic Segmentation in the future.
2. Find ways to take inventory snapshots instead of real-time video stream: Our use case demands to count the inventory only when the crew wants. So, instead of the real-time video stream, we adapt it to as and when needed snapshots.
3. Take trade-offs on Power BI, Stream Analytics, and Time Series Insights: Our goal is to make the insights a pattern to use as a standard offering for our customers. We evaluate the cost models for enterprise workloads and decide on the common pattern that helps our customers to get an optimized cost structure for the solution.
4. Fix all known issues: We experienced a few challenges and brought them to the Azure Percept team for resolution. For example, a few schema compatibility issues exist with custom IoT Edge solutions from VS code.

Also, our Go-To-Market strategy involves:

• Partner with the Azure Percept team to position the Azure Percept to our enterprise clients.
• Develop customer-ready use cases.
• Deliver an effective production-ready solution with an optimized cost for enterprise customer scenarios.

You can get hands-on with the Azure Percept DK and deliver on customer scenarios. To learn more about Azure Percept features, tutorials, samples, check out the links below.

Learn about Azure Percept 

• Azure Percept | Edge Computing Solution
• The cutting edge of AI: Discover the possibilities with Azure Percept
• Azure Percept Documentation overview  
• Azure Percept product overview
• Explore pre-built AI models
• Azure Percept YouTube
• Azure Percept DK and Azure Percept Audio now available in more regions!