Did you think that it’s straightforward? I mean, did you think you can simply export the ARM template from Azure Data Factory and import it into Azure Synapse? Well, let me break the bad news for you! There is no provision to import an ARM template into Azure Synapse.
Okay! Okay! Hold on! That’s too fast-paced. Hence, let’s introduce the characters here. Firstly comes the Azure Data Factory. It is Microsoft Azure’s primary ETL/ELT tool. Next comes Azure Synapse Family comprising two members viz. dedicated SQL pool (Formerly MPP SQLDW) and Synapse workspace. The integration service of Synapse workspace is nothing but the ADF equivalent of the Synapse family.
You may wonder why does this topic deserves an article? Can’t we export the ARM template from ADF and import it to Synapse? Well, the answer is NO. So, wherein lies the answer? It’s in two steps:
- Download ADF pipeline support files.
- Deploying them using PowerShell.
Download ADF support files
I won’t reinvent the wheel by repeating the steps that have been shown succinctly on this blog:
Follow the steps mentioned and download the support files on your machine.
Deploying pipelines to Azure Synapse using PowerShell
Before pipeline deployment, we need to understand the way ADF is organized. The most fundamental entity of ADF is a Linked Service. Linked Services set up a connection between ADF and the datastores (sources and sink). On top of linked services, we have datasets. A dataset is a logical view on which data processing activities are built. Finally, the activities combine to create a pipeline. The below diagram is an excellent summary.
Hence, the deployment/migration of ADF happens in the following order:
- Linked Service
- Datasets
- Pipelines
Now let’s go through the steps for migration of ADF to synapse:
1 – Establish a connection
Connect to your Azure tenant:
Connect-AzAccount
Make sure the current subscription content is correctly set by running the following command:
$context = Get-AzContext $context
If not, run the following command.
$context = Select-AzSubscription -SubscriptionName "<subscription name>" $context
However, make sure that you have the Azure Synapse PowerShell module installed before you go further, using these commands:
Install-Module Az.Synapse Import-Module Az.Synapse
2 – Import a linked service
Create linked services using the following command:
Set-AzSynapseLinkedService -WorkspaceName <synapse workspace name> -Name <name of linked service name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context
3 – Import a dataset
Create dataset using the following command:
Set-AzSynapseDataSet -WorkspaceName <synapse workspace name> -Name <dataset name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context
4 – Import a pipeline
Finally, import the pipeline using the following command:
Set-AzSynapsePipeline -WorkspaceName <synapse workspace name> -Name <pipeline name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context
Reference: https://github.com/solliancenet/azure-synapse-analytics-ga-content-packs/blob/main/hands-on-labs/lab-03/README.md
