Motivating Webhook
Before getting into details of a webhook activity, we should analyse its need in ADF. Apart from being an ETL/ELT tool, Azure Data Factory(ADF) has evolved into an orchestration tool in Microsoft cloud offering. Furthermore, seamless integration with another orchestration tool Azure Logic app adds to its beauty. A common use case for this coupling is to send email notifications from ADF since, unlike its predecessor viz. SSIS, ADF lacks a Send Email Task.
Azure Data Factory communicates with Logic App using REST API calls through an activity named Web Activity, the father of Webhook activity. Let us walk through an example based on Web Activity, so that we can be in a better position to appreciate the successor.
Send an Email with Web Activity
Creating the Logic App
We begin our demo by creating a Logic App. First, we have to set up an HTTP listener in the Logic App, to receive requests from Azure Data Factory. In the new Logic App, search for HTTP in the search bar and select HTTP request. Save the logic app to generate the URL.
For demonstration purpose, we won’t pass anything in the request body.
Next, add a Send an email action with the following data. This could be an Outlook action or Gmail one. We will use Outlook action here. Add your email id in the To section.
Now we are done with Logic App creation.
Web Activity
Once the Logic App is created, we have to send a POST request on the POST URL of the HTTP listener in Logic App. Hence we use ADF Web activity with the POST method. Copy the POST URL from the HTTP listener and paste it in the URL field of the Web Activity. Do not forget to fill all the fields appropriately.
Now your setup is ready to send email notifications with ADF and Logic App. However, there is one caveat. Web activity is asynchronous with Logic App i.e. it triggers the Logic App and marks the success status, thus moving on with the next activity in the pipeline. Furthermore, any failure in the Logic App cannot be traced unless a response is sent back to ADF. If you are someone who is not familiar with Request-Response concepts, you can use the Webhook Activity.
Send an Email with Webhook Activity
Before delving into the implementation, let us understand the mechanism through which ADF synchronizes with Logic App using Webhook. While sending a request to the HTTP listener in Logic App via Webhook activity, ADF appends a callBackUri to the request body. Post that, the ADF waits for the callBackUri to be invoked(until timeout) and it treats callBackUri invocation as the success signal. An example here would be more intuitive.
Creating the Logic App
In this implementation, the design remains the same except for the fact that an additional step is added to invoke the callBackUri. Add an HTTP action and write this expression to retrieve the callBackUri sent by the ADF in the request body.
triggerBody().callBackUri
Webhook Activity
On the ADF front, the Web activity is replaced by a Webhook activity. Configure it in a similar fashion and run it. Here, you will see the key difference in the sense that if the callBackUri is not invoked, the ADF remains in the hung state, while it gives the completion signal when the same is invoked.
Now, for demonstration purpose, let us remove the callBackUri invocation from the Logic app. It shows the complete signal.
Also, expanding the HTTP listener, we can see the callBackUri in the request body.
However, the ADF is in a hung state.
Next, after invoking the callBackUri, we see the following results where along with Logic App, ADF is completely successful!
P.S.
Please note that we can dodge the Webhook using Web activity with the HTTP response. We will cover this aspect in another article.
Also read Azure Databricks tutorial: an end to end analytics.
Disclaimer: The articles and code snippets on data4v are for general information purposes only. We make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose.
