Pre-requisite
In the first part of this series, we focused on moving a single file from one blob location to another using ADF. Here is the link for the first part: Move Files with Azure Data Factory- Part I. The second part dealt with the aspect of moving multiple files. In the process, we introduced two important activities in Azure Data Factory viz. Get Metadata activity and the ForEach activity. Here comes the link to the second part: Move Files with Azure Data Factory- Part II.
The first two parts were based on a fundamental premise that files are present in the source location. In this part, we will focus on a scenario that occurs frequently in real-life i.e. empty source location. In this process, we will introduce an important activity in ADF named If Condition activity. Furthermore, we will introduce the Execute Pipeline activity to invoke the main pipeline. Lastly, a web activity will be introduced in order to trigger a Logic App that sends an email notification, in case no files are found in the source location.
Pipeline Overview
Here is a high-level block diagram of the pipeline we are going to build.
As far as implementation is concerned, we need a Get Metadata activity to get the Items. Further, we use an If Condition activity, which checks the presence of files based on a specific condition.
Pipeline Name: Move Files if Found
The If Condition activity in ADF is based on the principle of conditional execution. It consists of two sets of activities viz. If True Activities and If False Activities.
Once you click Edit If True/False Activities, you are taken to another pane in which you can define your inner activities. However, we need to define a condition which gives a boolean result i.e. True/False. Go to settings in the above image and use the following expression to check for the presence of files in the source location.
@empty(activity('Get FileList').output.childItems)
Edit If True Activities: The web activity and Logic App
Clicking on the Edit If True Activities, we will see the following Web activity
This Web Activity in ADF takes certain parameters:
- URL: The static URL of Logic App, which will send an Email.
- Method: The HTTP method to be used. We will use the POST method to send a post request.
- Headers: We need to set the Content-Type as application/json.
- Body: Finally, we define a request body. We will send a blank request body in this demonstration.
To retrieve the static URL of the Logic App, open the Logic App and copy the HTTP POST URL in the picture below.
Also, the Send an Email action of the Logic app looks like this.
Edit If False Activities: The Execute Pipeline Activity and Move Files pipeline
Clicking on the Edit If False Activities, we will see the following Execute Pipeline activity.
The Move Files pipeline looks like this. Details of this pipeline here: Move Files with Azure Data Factory- Part II.
One might be tempted to try and place the Move Files pipeline within the If Condition. However, we cannot place a ForEach activity as a part of If Condition.
Running the Pipeline
Let us set up for the demonstration. We have three files in the source location.
Now, we will run the pipeline Move Files if Found. It is evident that in the first run, we can see that the Move Files pipeline has run since files were present in the source location.
Now, we will rerun the pipeline without the source files.
Conclusion
Here we complete this series. Hope you find it helpful. Reach out to me in comments if you have any questions.
Related Article: Azure Data Factory Webhook activity
Disclaimer: The articles and code snippets on data4v are for general information purposes only. We make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose.
