Following are the steps the script performs to create Azure Resources and downloads data from cms.gov and loads data into Synapse Tables:
- Prompts User for Resource Group Name and Password
- Presets various variables like SQL Server Name, Azure Data Factory Name, Storage Name, etc. using Resource Group Name provided in Step 1.
- Creates SQL Server
- Creates Synaspe (SQL Pool)
- Gets your Public IP so it can configured in the Synapse Firewall
- Sets the Firewall Rules for Synaspe (SQL Pool)
- Creates Azure Data Factory
- Creates Azure Data Lake Storage Account
- Creates Storage Containers for the CMS data
- Creates parameters file for the Azure Data Factory ARM Template
- Deploys Azure Data Factory ARM Template that contains,pipelines, datasets and dataflows
- Gets Storage Access Key
- Gets Connection String for the Synaspe (SQL Pool)
- Creates Tables and Views in Synapse using script synapseCMSddls.sql
- Downloads CMS PartD data from website https://www.cms.gov/ and saves into the Storage
- Executes Azure Data Factory pipelines that reads CMS Data from Storage and loads into Synaspe (SQL Pool)