Automating Geneva MDM & MDS queries

 Here in Hybrid Networking we need to implement data collection and management for network MDM metrics and MDS log data that extend what is available in Geneva. For Data Science projects we need to automate scraping selected metrics & logs to create longitudinal datasets suitable for detailed model building.  As such we have some adhoc connectors using available C# APIs to MDM / MDS to periodically scrape data and archive it in ADL2, to make it accessible for processing in SPARK.  However, a standard solution would be preferable. The ADL2 - SPARK (e.g. DataBricks) combination I believe has clear advantages over use of Cosmos (not CosmosDB, but the legacy batch job system) for which a wider range of connectors are available. 

Has anyone else implemented automation tools for such a data pipeline?