Blog Post

Microsoft Mission Critical Blog
3 MIN READ

A Practical Guide to Implement End-to-End Data Warehousing PoC Using Microsoft Fabric-Part2

Dharmendra_Keshari's avatar
Dec 26, 2024

Microsoft Fabric provides robust platform for building scalable data warehousing for customers to make informed decisions. In this 5-blog series, we'll guide you through creating a Proof of Concept (PoC) for an end-to-end data warehouse using Microsoft Fabric.

In this continuation (part 2) of the previous blog, you will create the required Fabric items within a workspace and create the tables in Lakehouse and Data Warehouse.

Task1: Create a Microsoft Fabric Workspace, Data Warehouse and Lakehouse

Before you can start building the solution, you'll first need to create a workspace where you'll create the rest of the items.

Follow the steps below to create the workspace.

  1. Sign in to https://app.fabric.microsoft.com/
  2. Select Workspaces > New Workspace.

 

 

  1. Fill out the Create a workspace dialog as follows:
    • Name: LevelUpFabric1 (or a name of choice). Name should be unique.
  1. Expand the Advanced section.
  2. In the License Mode choose Trial or Fabric Capacity.

 

  1. Click Apply. The workspace will be created and opened.
  2. In the upper left corner, select New Item> search and click on Warehouse to create Data Warehouse and name it “DW_Levelup”. This will create the Data Warehouse and open it.
  3. Click on the workspace icon the left navigation to navigate to the workspace page.

     

  4. In the upper left corner of the Workspace, select New Item > search and click on Lakehouse to create a Lakehouse and name it, “LH_Levelup”. This will create the Lakehouse and open it.

Task2: Create table Shortcuts in the Lakehouse from ADLS Gen2

We will now create shortcuts in the Lakehouse pointing to the two delta folders (Sales_SalesOrderHeader and Sales_SalesOrderDetail) which you had uploaded into the ADLS Gen2 store account in the pre-requisite steps.

  1. In the Lakehouse explorer, hover over the Tables folder (not Files folder) and click on the ellipsis and click “New Shortcut”.
  1. In the New Shortcut dialog box, select ADLS Gen2 under External sources.
  2. In the next screen, provide the required details as shown in the below snip. You may refer to your storage account settings for details. It is discouraged to use Account key for authentication but is done so for ease of use.
  3. The URL should be similar to: https://xxxxx.dfs.core.windows.net/levelup
  1. Ensure the shortcut name “Sales_SalesOrderDetail” and “Sales_SalesOrderHeader” matches the delta folder name.

Important: Please use Sales_SalesOrderHeader and Sales_SalesOrderDetail as shortcut names. These names are used in the stored procedures. If you choose to use different shortcut names in step 5, you may need to update the SPs that refer to it, to avoid errors.

  1. The shortcuts should show up within the tables section of the Lakehouse like the below. We will use these shortcuts later in the solution. You may have to refresh the tables section for the shortcuts to appear (you may also have to refresh the tables section if the shortcuts appear as under “Unidentified” too. If it continues to be unidentified, there could be some steps above which were not followed correctly.)

Task3: Create the required Tables and SPs in the Fabric Data Warehouse 

  • Now, go to Fabric workspace “LevelUpFabric1” and Navigate to the Data Warehouse and open a new query window and run below scripts.

Note: The above scripts are available in the “FabricLevelUp_Files” folder that was downloaded during the prerequisite steps within “TSQL_Script” folder.

Scripts

Details

01_create_AllStgDimTbls.sql

Creates staging (stg) schema and all the staging related tables.

02_TablesToCopy.sql

Creates metadata tables to dynamic full load activity.

03_del_AllRecordsOfStgDimTbls.sql

Creates stored procedure to delete all the records of the staging dim tables.

04_crt_Prd_Sch_Tbl_Sp.sql

Create production (prd) schema, fact table, and SP for incremental update to fact table.

05_crt_Prd_Dim_Tbls.sql

Create stored procedure to create all the production dim tables.

06_upd_prd_dim_Customer.sql

Creates SP to update incremental stg.DimCustomer  records to prd.DimCustomer tables.

So far, we have created workspace, all the required items in the Lakehouse and Data Warehouse which will be used to load the Data Warehouse.

Continue to the next blog...

Updated Dec 26, 2024
Version 2.0
No CommentsBe the first to comment