The security benefits of structuring your Azure OpenAI calls – The System Role

In the rapidly evolving landscape of GenAI usage by companies, ensuring the security and integrity of interactions is paramount. A key aspect is managing the different conversational roles—namely system, user, and assistant. By clearly defining and separating these roles, you can maintain clarity and context while enhancing security. In this blog post, we explore the benefits of structuring your Azure OpenAI calls properly, focusing especially on the system prompt. A misconfigured system prompt can create a potential security risk for your application, and we’ll explain why and how to avoid it. 

The Different Roles in an AI-Based Chat Application 

Any AI chat application, regardless of the domain, is based on the interaction between two primary players, the user and the assistant. 

• The user provides input or queries. 

• The assistant generates contextually appropriate and coherent responses. 

Another important but sometimes overlooked player is the designer or developer of the application. This individual determines the purpose, flow, and tone of the application. Usually, this player is referred to as the system. 

• The system provides the initial instructions and behavioral guidelines for the model. 

Microsoft Defender for Cloud’s researchers identified emerging anti-pattern 

Microsoft Defender for Cloud (MDC) offers security posture management and threat detection capabilities across clouds and has recently released a new set of features to help organizations build secure enterprise-ready gen-AI apps in the cloud, helping them build securely and stay secure.  MDC’s research experts continuously track the development patterns to enhance the offering but also to promote secure practices to their customers and the wider tech community. They are also primary contributors to the OWASP Top 10 threats for  LLM (Idan Hen, research team manager). 

Recently, MDC's research experts identified a common anti-pattern in AI application development is emerging – appending the system to the user prompt. Mixing these sections is easy and tempting – developers often use it because it’s slightly faster while building and also allows them to maintain context through long conversations. But this practice is harmful – it introduces detrimental security risks that could easily result in 'game over' – exposing sensitive data, getting your computer abused, or making your system vulnerable to Jailbreak attacks. 

Diving deeper: How system prompts evaluation keeps your application secure 

Separate system, user and assistant prompts with Azure OpenAI ChatCompletion API

Azure OpenAI Service's Chat Completion API is a powerful tool designed to facilitate rich and interactive conversational experiences. Leveraging the capabilities of advanced language models, this API enables developers to create human-like chat interactions within their applications.  

By structuring conversations with distinct roles—system, user, and assistant—the API ensures clarity and context throughout the dialogue: 

[{"role": "system", "content": [Developer’s instructions]}, 

{"role": "user", "content”: [User’s request]}, 

{"role": "assistant", "content": [Model’s response] } ] 

This structured interaction model allows for enhanced user engagement across various use cases such as customer support, virtual assistants, and interactive storytelling. By understanding and predicting the flow of conversation, the Chat Completion API helps create not only natural and engaging user experiences but securer applications, driving innovation in communication technology. 

Anti-pattern explained 

When developers append their instructions to the user prompt. The model receives single input composed by two different sources: developer and user: 

{"role": "user", "content”: [Developer’s instructions] + [User’s request]} 

{"role": "assistant", "content": [Model’s response] } 

When developer instructions are mingled with user input, detection and content filtering systems often struggle to distinguish between the two.  

Anti-pattern resulting in less secured application 

This blurring of input roles can facilitate easier manipulation through both direct and indirect prompt injections, thereby increasing the risk of misuse and harmful content not being detected properly by security and safety systems.  

Developer instructions frequently contain security-related content, such as forbidden requests and responses, as well as lists of do's and don'ts. If these instructions are not conveyed using the system role, this important method for restricting model usage becomes less effective. Additionally, customers have reported that protection systems may misinterpret these instructions as malicious behavior, leading to a high rate of false positive alerts and the unwarranted blocking of benign content. In one case, a customer described forbidden behavior and appended it to the user role. The threat detection system then flagged it as malicious user activity.    

Moreover, developer instructions may contain private content and information related to the application's inner workings, such as available data sources and tools, their descriptions, and legitimate and illegitimate operations. Although it is not recommended, these instructions may also include information about the logged-in user, connected data sources and information related to the application's operation. Content within the system role enjoys higher privacy; a model can be instructed not to reveal it to the user, and a system prompt leak is considered a security vulnerability. When developer instructions are inserted together with user instructions, the probability of a system prompt leak is much higher, thereby putting our application at risk. 

Figure 1: Good Protection vs Poor Protection

Why do developers mingle their instructions with user input? 

In many cases, recurring instructions improve the overall user experience. During lengthy interactions, the model tends to forget earlier conversations, including the developer instructions provided in the system role. For example, a model instructed to role-play in an English teaching application or act as a medical assistant in a hospital support application may forget its assigned role by the end of the conversation. This can lead to poor user experience and potential confusion. To mitigate this issue, it is crucial to find methods to remind the model of its role and instructions throughout the interaction. One incorrect approach is to append the developer's instructions to user input by adding them to the User role. Although it keeps developers’ instructions fresh in the model's 'memory,' this practice can significantly impact security, as we saw earlier. 

Enjoy both user experience and secured application 

To enjoy both quality detection and filtering capabilities along with a maximal user experience throughout the entire conversation, one option is to refeed developer instructions using the system role several times as the conversation continues: 

{"role": "system", "content": [Developer’s instructions]}, 

{"role": "user", "content”: [User’s request 1]} 

{"role": "assistant", "content": [Model’s response 1] } 

{"role": "system", "content": [Developer’s instructions]}, 

{"role": "user", "content”: [User’s request 2]} 

{"role": "assistant", "content": [Model’s response 2] } 

By doing so, we achieve the best of both worlds: maintaining the best practice of separating developer instructions from user requests using the Chat Completion API, while keeping the instructions fresh in the model's memory. This approach ensures that detection and filtering systems function effectively, our instructions get the model's full attention, and our system prompt remains secure, all without compromising the user experience. 

To further enhance the protection of your AI applications and maximize detection and filtering capabilities, it is recommended to provide contextual information regarding the end user and the relevant application. Additionally, it is crucial to identify and mark various input sources and involved entities, such as grounding data, tools, and plugins. By doing so, our system can achieve a higher level of accuracy and efficacy in safeguarding your AI application. In our upcoming blog post, we will delve deeper into these critical aspects, offering detailed insights and strategies to further optimize the protection of your AI applications. 

Start secure and stay secure when building Gen-AI apps with Microsoft Defender for Cloud 
Structuring your prompts securely is the best-practice when designing chatbots. There are other lines of defense that must be put in place to fully secure your environment. 
 

• Sign up and Enable the new Defender for cloud threat protection for AI for active threat detection (preview). 

• Enable posture management to cover all your cloud security risks, including new AI posture features. 

Further Reading 

• Microsoft Defender for Cloud (MDC). 
• AI protection using MDC. 
• Chat Completion API. 
• Security challenges related to GenAI. 
• How to craft effective System Prompt. 
• The role of System Prompt in Chat Completion API. 
• Responsible AI practices for Azure OpenAI models. 

Asaf Harari, Data Scientist, Microsoft Threat Protection Research.
Shiran Horev, Principal Product Manager, Microsoft Defender for Cloud. 
Slava Reznitsky, Principal Architect, Microsoft Defender for Cloud.