Forum Discussion
I need an idea of how to make an architecture in Azure
Hello, I need help with a project. I need to upload an Excel file to my data lake and create an automated process that converts it to CSV without the need for user intervention or debugging. Can someone assist me with this? It is very important for me to complete this process successfully.
You may consider this:
1. Azure Blob Storage or Data Lake Storage
- Use Azure Data Lake Storage Gen2 to store the uploaded Excel files. This will act as the central repository for your raw data.
2. Azure Logic Apps or Azure Functions
- Set up an Azure Logic App or Azure Function to trigger whenever a new file is uploaded to the storage.
- The trigger can initiate the process of converting the Excel file to CSV.
3. Excel to CSV Conversion
- Use Azure Functions with a script (e.g., Python or .NET) to read the Excel file and convert it to CSV. Libraries like pandas (Python) or EPPlus (.NET) can handle this conversion efficiently.
4. Automated Workflow
- Once the file is converted, save the CSV file back to the same or a different folder in the Data Lake.
5. Monitoring and Logging
- Implement monitoring using Azure Monitor or Application Insights to track the success or failure of the process.
- Log all activities for debugging and auditing purposes.
6. Optional Enhancements
- Use Azure Data Factory for more complex workflows, such as integrating with other data sources or performing additional transformations.
- Add Azure Key Vault to securely manage any credentials or secrets required for the process.
