Blog Post

Azure Data Factory Blog
2 MIN READ

Securing outbound traffic with Azure Data Factory's outbound network rules

Abhishek Narain's avatar
Abhishek Narain
Icon for Microsoft rankMicrosoft
Jun 12, 2023

Data security is paramount in today's digital world. With an increasing number of cyber threats, organizations are always on the lookout for robust solutions to enhance their security posture. In this blog, we delve into a critical feature provided by Azure Data Factory – Outbound Rules – that allows users to control and restrict outbound traffic to specific Fully Qualified Domain Names (FQDN).

 

Understanding Outbound Allow listing in Azure Data Factory

Outbound allow listing of FQDN is a network security practice that allows organizations to control outbound traffic from their networks to specific, approved domain names. Outbound rules in Azure Data Factory apply to pipeline activities, such as Copy, Dataflows, Web, Webhook, and Azure Function activities and authoring scenarios like data preview and test connection.

 

Note:

  • This feature is in Preview.
  • SSIS Integration runtime and Managed Airflow Integration runtime currently do not support the outbound rules.
  • This feature is independent of Managed VNet and applies to all supported activities running on SHIR, Azure IR (including AutoResolve IR), and Azure IR in Managed VNet. However, we suggest using Managed VNet for higher levels of compute isolation in conjunction with outbound allowlist capability to prevent data exfiltration.

 

These rules help organizations create a secure and exfiltration-proof data integration solution. What's more, Azure Policy enforces these rules, thereby boosting governance.

As it uses Azure Policy, these outbound rules can be enforced at different management levels based on the organization’s needs.

  • Management Group
  • Subscription
  • Resource Group
  • Resource (UI within Data Factory for this assignment is coming soon, but you can use REST API/ SDK to achieve this today)

Note: While in preview, the compliance for this policy is not reported

 

Steps to enable Azure Policy for outbound rules

  1. Assign the outbound Policy with the desired scope.

 

  1. Configure the parameters of the policy specifying the allowed domain names. Create the policy.
    Note: Regex is not supported hence the domains should exactly be the same as used in the linked services. To update the outbound url list, please update the policy parameter.

 

  1. Enable the feature in ADF studio.

 

The Outbound Rules feature in Azure Data Factory allows organizations to exercise granular control over outbound traffic, thereby strengthening network security during data integration. By integrating with Azure Policy, this feature also improves overall governance.

 

Resources:

If you have any questions or feedback, please post them in the comments below. 

 

Updated Jun 12, 2023
Version 1.0
Azure Data Factory
Azure Data Integration
Azure Integration Runtime
Data Security
  • mickhanneyIRL's avatar
    mickhanneyIRL
    Copper Contributor
    Sep 10, 2024

    Hi,

    I am trying to get line of sight on when this policy will move from Preview to Generally Available?  Its been more than a year since original post. 

    Thanks

    M

  • MaxvPutten's avatar
    MaxvPutten
    Copper Contributor
    Aug 11, 2023

    Abhishek Narain Mayank_Bansal Thank you for showcasing the new feature. Do you have an indication on when this preview will be rolled out to be used in an ARM/BICEP template? 

     

    Thank you in advance. 

  • andy205's avatar
    andy205
    Copper Contributor
    Jul 31, 2023

    Does the ability to enable/disable the outbound rule feature, map to an action within the Microsoft.DataFactory resource provider ?

     

    We would like to enable the feature but our users are generally ADF contributors of their own subscriptions, so from what I understand, we can enable Azure policy containing the allowed FQDNS on their Sub/Resource group/ADF, but then its optional to enable or disable the setting inside the ADF portal. 

    If users are contributors, they could enable/disable the feature to get around the restriction. 

     

    I was thinking a custom RBAC policy would do it, containing most of the contributor roles, but disalowing the enable/disablement of the feature.... but cant see anything in the list of published actions that might be one we wanted to restrict. 

  • Mayank_Bansal's avatar
    Mayank_Bansal
    Icon for Microsoft rankMicrosoft
    Jun 20, 2023

    shaunsatt we don't cache the evaluation results today, so each request is checked against policy
    As you can see the throttling is applied at two levels, at subscription & at resource (Data Factory), & if the resource hits it's limit then it will get throttled, meaning even if you only have 1 subscription, & only 1 Data Factory, then the effective throttling for your data factory is 1000 request / 5mins
    However if you have 1 subscription, & 10 Data Factory, then for each Data Factory you would have 1000 requests / 5mins. 

  • shaunsatt's avatar
    shaunsatt
    Copper Contributor
    Jun 20, 2023

    Mayank_Bansal This works as expected now, thank you.

     

    With regards to your throttling limits, is there any caching involved or is each request checked against Policy, also if I have only 1 ADF in a subscription can it consume 50,000 requests / 5 minutes or is it limited to the 1000 limit?

  • Mayank_Bansal's avatar
    Mayank_Bansal
    Icon for Microsoft rankMicrosoft
    Jun 19, 2023

    shaunsatt we have deployed the fix to all regions, could you please give it another try & let us know if that solved the issue for you?

  • shaunsatt's avatar
    shaunsatt
    Copper Contributor
    Jun 14, 2023

    Abhishek Narain I have setup 2 linked services, 1 to a source storage account the other to a destination storage account. Copying a file from one to the other using managed identities works fine when not enabled. Then when I enable outbound rules and copy the FQDN's of both storage accounts to the policy in an array I get a DENY error message for both storage accounts within ADF saying it's unauthorised.

     

    "Connector or activity name: blob_destination, connector or activity type: AzureBlobStorage, error: Calling partner RP EvaluatePolicyAsync returned an invalid status code 'Unauthorized', ReasonPhrase Unauthorized"]'

  • Abhishek Narain's avatar
    Abhishek Narain
    Icon for Microsoft rankMicrosoft
    Jun 14, 2023

    shaunsatt you need to provide the exact FQDN used in the linked service. It does not support regex yet; hence, it must be the exact FQDN, not just the domain. 
    Also, if there is any other policy applied at the same or higher levels (say another assignment at the resource group level or subscription or management group level, then all policies should allow the same FQDNs).
    Let me know if you still face the problem.

  • shaunsatt's avatar
    shaunsatt
    Copper Contributor
    Jun 14, 2023

    Abhishek Narain is there some nuance to this as I am unable to get it working, as soon as I enable Outbound rules using Azure Policy in ADF it blocks everything even if there is no policy, with policy but disabled and with FQDN entries in the policy. Can it take specific FQDN's rather than just a domain name?