Forum Discussion

m0rkal's avatar
m0rkal
Copper Contributor
Jan 15, 2023

Connecting Azure Databricks VMs to external VNet for Log Analytics ingestion

Hello,

 

We run an ADB cluster using cluster pools. I'm looking for a way connect the ADB cluster pool VMs which reside on the automatically generated Resource Group databricks-rg-<xyz> to another Resource Group and VNet where Log Analytics resource reside. The communication needs to be over private IPs since we block public ingestion in the Log Analytics workspace for security propose. The VMs will report OMS metrics for performance proposes. See this link for more details: https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md#Appendix-A)

 

See following diagram:

 

I found this link: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link but I'm not sure it's for this propose. When trying to create the private-link on the ADB resource I have the following options to pick from: "databricks_ui_api" and "browser_authentication", which both seems not relevant for my need.

 

Appreciate your help on this matter.

Thank you!

 

 

  • Varun_Ghildiyal's avatar
    Varun_Ghildiyal
    Brass Contributor
    Mar 08, 2023
    It sounds like you want to enable private communication between your ADB cluster pool VMs and your Log Analytics workspace. To do this, you can use Azure Private Link, which allows you to access Azure services over a private endpoint within your own virtual network.

    Here's how you can set up Private Link between your ADB cluster pool and Log Analytics workspace:

    Create a new virtual network in the same region as your ADB cluster pool. This virtual network will be used to host the Private Link endpoint for Log Analytics.

    Create a Private Link service for your Log Analytics workspace. This will allow you to connect to your workspace over a private endpoint within your virtual network.

    In your virtual network, create a Private Link endpoint for Log Analytics. This will allow your ADB cluster pool VMs to connect to Log Analytics over a private IP address.

    Create a virtual network peering between your ADB cluster pool virtual network and your Log Analytics virtual network. This will allow your ADB cluster pool VMs to communicate with Log Analytics over the Private Link endpoint.

    Configure your ADB cluster pool VMs to send their OMS metrics to the Log Analytics Private Link endpoint.

    By following these steps, you can ensure that all communication between your ADB cluster pool and Log Analytics workspace is secure and private, and that no data is transmitted over the public internet.

Resources