Threat detection in airports, railways, and retail with Azure Percept

Copper Contributor

Jun 13, 2022

Conceptual image of data flow.

During the current uncertain times, providing peace-of-mind for both customers and passengers in airports, malls, and bus/train stations is paramount. Enterprises are in search of a comprehensive and flexible threat detection and management solution curated to meet their needs.

As part of the Microsoft Azure Percept Bootcamp, a team from HCL, leveraging the capabilities of Azure Percept, developed a threat detection solution to identify anomalies such as suspicious objects and generate alerts to ensure proper actions are taken in a timely manner.

In this article, we can see how easy it is to build a versatile vision solution with Azure Percept catering to the relevant enterprise verticals. We’ll go through the end-to-end solution encompassing the architecture and implementation, including the various subcomponents using Azure services.

Overview of the solution

The solution depicts a scenario involving a fictious airport called Thrimain Airport, where security is a concern. Thrimain needs a streamlined automated process to detect threats and send an alert upon such identification. We identified three sample objects to demonstrate the different threats. Once Azure Percept has trained the AI model, we could observe the live video stream at the edge as well as the inferenced data sent to our management engine developed in HCL and hosted in Azure. We then mapped the detections to corresponding threat levels and had a central dashboard to display all the threats and their associated time stamps and alerts generation.

A brief demo of the end-to-end solution can be found in the following YouTube video:

Solution Architecture

The following diagram shows the overall architecture for the solution:

A diagram of the overall architecture for the solution.

The solution uses Azure Percept to detect the presence of threats. For proof-of-concept simplification purposes, we used the following objects for threats:

1 Weapon: threat level = 8

1 Unattended bag: threat level = 5

1 Apple: threat level = 1 (a tripping hazard)

The solution leverages the Azure Cognitive Services object detection model. After training the model, we deployed it over the air to Azure Percept. The latter processes the real time video stream at the edge and channels the telemetry data to Azure IoT Hub. The event data is then sent to Azure Stream Analytics for processing and filtering, and the data insights are forwarded to either Power BI or Management Engine via Service Bus for mapping of threats, aggregation over time periods, and visualization.

Solution Implementation  

The threat detection solution is created as a proof of concept using the detection of basic objects in lieu of threats in airports, stations, and malls. The details of the project implementation are described below.

Azure Percept Studio: Custom Vision

The first step is to use Custom Vision, part of the Azure Cognitive Services, to load and capture images, annotate the data, train the model, and deploy the model to Azure Percept. In this project, we used a minimum of fifty images per object to ensure better prediction via having varied environmental conditions, quantity, and background.

Sample Custom Vision project:

A sample of a custom vision project.

Below is a sample output of the video stream once the model is deployed along with the inferenced data:

A sample output of the video stream.

Stream Analytics

We created the Azure Stream Analytics job to derive telemetry data and process it to channel to Power BI or Management Engine via Service Bus Queue:

An image of the Azure Stream Analytics job to derive and process telemetry data.

The query extracts relevant insights from telemetry data and outputs to either Power BI or Management portal via the Service Bus.

Power BI Dashboard

We configured Power BI to receive the data stream from Stream Analytics so we can visualize the timestamp and associated threat detection results.

An image for visualizing the timestamp and associated threat detection results.

Service Bus to Management Engine

An alternate output method is to develop a custom Management Engine to receive input from the Service Bus.

An image of an alternate output method from  Service Bus to Management Engine.

The Management Engine extracts the real-time data insights received from the service bus, maps the different threat level corresponding to the objects, aggregates data samples over a fixed time interval, and displays the data and alerts generated based on different threat levels.

Sample code for detection to threat mapping:

Sample code for detection to threat mapping.

Below is the dashboard for the visualization of real time detection and the corresponding mapped threat alerts.

An image of dashboard for the visualization of real time detection and mapped threat alerts.

Final Thoughts

Azure Percept is a complete edge computing solution that powers real-time insights through hardware accelerators integrated with Azure AI and ML, which is very easy to use. This proof of concept is a simplified demonstration of a real-world threat detection solution. It can be a starting point to a more comprehensive threat detection solution. Augments such as combining audio inputs, incremental threat level based on the time interval of unattended objects, as well as more comprehensive alerting and management center can be the next roadmap items. Furthermore, this solution can be integrated with 5G MEC and AI edge application suite.

To learn more about Azure Percept, visit these resources: