Recent Discussions
Accessing data from an outsourced third party service company
Hi , Can someone please throw some light on this ? We have some services outsourced to a third-party company but, we realised the data which goes into their process is very valuable and want to access it for different purposes. we are currently downloading certain reports from their web pages and going forward they are planning to provide API end points to give us the data required in the form of multiple pre-defined reports. But, I am just wondering, if there is any other secure and feasible method with which we can get the current state of their entire database filtered for our company, which gets automatically refreshes every few hours? Thank you, KPAutomating Geneva MDM & MDS queries
Here in Hybrid Networking we need to implement data collection and management for network MDM metrics and MDS log data that extend what is available in Geneva. For Data Science projects we need to automate scraping selected metrics & logs to create longitudinal datasets suitable for detailed model building. As such we have some adhoc connectors using available C# APIs to MDM / MDS to periodically scrape data and archive it in ADL2, to make it accessible for processing in SPARK. However, a standard solution would be preferable. The ADL2 - SPARK (e.g. DataBricks) combination I believe has clear advantages over use of Cosmos (not CosmosDB, but the legacy batch job system) for which a wider range of connectors are available. Has anyone else implemented automation tools for such a data pipeline?
Recent Blogs
- In Azure Data Factory, continuous integration and delivery (CI/CD) involves transferring Data Factory pipelines across different environments such as development, test, UAT and production. This proce...Nov 13, 2024
- 4 MIN READIntegrating database development into continuous integration and continuous deployment (CI/CD) workflows is the best practice for Azure SQL managed instance database projects. Automating the process ...Nov 07, 2024
