Sending Messages to Confluent Cloud topic using Logic App
With Logic Apps, we can create workflows that connect to various services and systems, allowing us to automate tasks and streamline the business operations. In this blog post, we will explore how to use Azure Logic Apps to send messages to Kafka Confluent topic. Currently, there is no out-of-box Kafka Confluent connector in logic app. We found that Kafka Confluent provides REST API Confluent Cloud API Reference Documentation. This sample shows how to use HTTP action in workflow to call the Kafka Confluent API which produce record to topic. Prerequisites Azure Account and access to Azure Portal. Access to Confluent Cloud. Confluent Cloud is a fully managed pay-as-you-go Kafka service. You can get a free trial here. Setup the Kafka cluster and topic If you are new to Confluent Kafka, you can check their tutorials: Quick Start for Confluent Cloud | Confluent Documentation. Create a new Kafka cluster on Confluent Cloud. Navigate to the cluster and click Cluster settings. Note the REST endpoint. We will use following endpoint in this example: Create a new Kafka Topic called "LAtest" using the default topic settings. Create a new API Key and Secret. Navigate to the cluster and from the left menu, select Cluster Overview -> API Keys. Click Create key and follow the prompts to create a Global access API key. Note down the value of key and secret. To communicate with the REST API, we need to use this API key ID and corresponding secret to create the base64 encoded string in the authorization header that will be included in the REST calls. To learn more, see Authentication part which describes Cloud and Cluster API keys and base64 encoding. Create Logic App workflow To produce message to a topic, we need to provide JSON data and a base64-encoded API key and secret to the REST Produce endpoint: /kafka/v3/clusters/<cluster-id>/topics/<topic-name>/records. Below is a sample REST code snippet (Non-streaming mode): curl \ -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Basic <base64-encoded API key and secret>" \ https://xxx.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-mxpx52/topics/LAtest/records \ -d '{"value":{"type":"JSON","data":"Hello World!"}}' In the Logic App workflow, we can add a "When a HTTP request is received trigger" to fire the workflow for test. Then we add a HTTP action with following configurations: Run the workflow We can send the body message to target topic successfully. View the Messages tab of Latest topic on the Confluent Cloud UI:
Deploy Logic App Standard with Application Routing Feature Based on Terraform and Azure Pipeline
Due to Terraform's cross-cloud compatibility, automation, and efficient execution, among many other advantages, more and more customers use it to deploy integration solutions based on Azure Logic App standard. However, despite the extensive contributions from the community and individual contributors providing Terraform templates and supporting VNET integration solutions for Logic App standards, there are still very few terraform templates covering the "Application routing" and "Configuration routing" settings: This article shared a mature plan to deploy logic app standard then set the mentioned routing features automatically. It's based on Terraform template and Azure DevOps Pipeline. Code Reference: https://github.com/serenaliqing/LAStandardTerraformDeployment/tree/main/Terraform-Deployment-Demo About Terraform Template: Please kindly find the the template in directory Terraform/LAStandard.tf, it includes the terraform definitions for logic app standard, the backend storage account, application insights, virtual network and VNET integration settings. About VNET Routing Configuration Because there is no terraform examples available for VNET routing, we add VNET Settings by invoking "Patch" request to ARM RESTful API endpoint for interacting with logic app standard site: https://management.azure.com/subscriptions/<Your subscription id>/resourceGroups/$(deployRG)/providers/Microsoft.Web/sites/$(deployLA)?api-version=2022-03-01 We figured out the required request body in network trace as the following format: { "properties": { "vnetContentShareEnabled": false, "vnetImagePullEnabled": true, "vnetRouteAllEnabled": false, "vnetBackupRestoreEnabled": false } } Please find the YAML file in TerraformPipeline/logicappstandard-terraform.yml. Within the Yaml file , the "AzureCLI@2" task is used to send the request by Azure CLI command. task to send the patch request. Special Tips: To use the terraform task during Azure pipeline run, it's required to install terraform extension (which you can find in the following link): https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks Terraform tasks: Reference: Deploy Logic App Standard with Terraform and Azure DevOps pipelines https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service https://azure.microsoft.com/en-us/products/devops/pipelines
