Joshcarlisle.io is a blog written by Josh Carlisle based out of Raleigh North Carolina where I explore, Share, and sometimes abuse the modern software development landscape. 

 

Building Office 365 Solutions with Azure - Webhooks (Part 1)

Building Office 365 Solutions with Azure - Webhooks (Part 1)

Office 365 is a fantastically full-featured platform that has many ready to use features for many very common business scenarios directly out of the box. Additionally, the platform offers a degree of flexibility through configurable customizations to help achieve your business goals. However, sometimes it becomes necessary to go beyond what is out of the box and what is configurable and for these occasions, Azure can be a great fit.

This is the first part of a multi-part series we'll be walking through a complete Azure solution for Office 365.

Building Office 365 Solutions with Azure Part 1 - Webhooks
Building Office 365 Solutions with Azure Part 2 - Handling Notifications with Azure Functions
Building Office 365 Solutions with Azure Part 3 - Working with Cognitive Services
Building Office 365 Solutions with Azure Part 4 - Updating SharePoint with Azure Functions

The Scenario

It is a very common business requirement to execute some logic after a file or item is updated in a SharePoint list.  In our case, we have a client requirement for us to automatically tag a photo and provide additional metadata when a photo is uploaded or updated.  We don't want to have to have the user be troubled and we want to ensure both accuracy and consistency so instead, we'd like to make use of some intelligent services to accurately provide us our metadata.   Now the challenge lies in how to make all this happen!

The Solution

Core to our requirement is an intelligent way to automatically get metadata from a photo. Luckily for us Microsoft has a fantastic set of services called Microsoft Cognitive Services and specifically one for photo analysis called Computer Vision that makes that process super simple. Well simple for us there is some really impressive machine learning going on behind the scenes! But how best do we communicate and process that data from the Computer Vision REST API?

In this type of scenario, one option might be a simple SharePoint workflow but we'll eliminate that for now as we likely have some complexity in our solution that rules out SharePoint workflows.  We also now have access to the new Microsoft Flow but being pretty new we're also worried we might have some of the same restrictions as traditional SharePoint workflows. Although our solution sounds like a great candidate for being part of a Microsoft Flow it's likely not the best solution to build it with.  We realize what we really need is to execute some good old fashion code which requires us to get out of Office 365 and into an environment that Office 365 can communicate with but allows us to execute custom code. No old school full trust solutions available here. Luckily we've got this great cloud solution from Microsoft called Azure to work with.

This still leaves us with figuring out how we let our external solution know that somebody did something in SharePoint we want to know about. Ideally, we want to get notified when a new photos has been added or existing one updated.  Before this year we essentially had a couple options including Remote Event Receivers based on dated WCF\SOAP architecture or some external application that worked to poll\track for changed.  Both still viable solutions but as of January 2017, we now have one more tool in our developer toolbelt - SharePoint Webhooks. Conceptually having some high-level similarities to Remote Event Receivers,  SharePoint Webhooks allow us to register standard REST endpoints to be notified when a change occurs.  Sounds like just the ticket!

The remaining task is how we want to host this REST endpoint. We have quite a few options on Azure.  One good choice might be using services deployed to a an API Application but that’s seems a little bit of an overkill for a single REST endpoint. Luckily for us Azure has a great service called Azure Functions that are intended for discrete units of work, and can potentially come in at a very low cost.  Sounds like just what we need! 

From a high level we now have a clear picture of how things are going to work. Now time to dig into the details and as we always know the devil is in the details!

SharePoint Webhooks Notification Cycle

Webhooks at its core is simply a notification system. Webhooks allow us to avoid the need for a constant polling for changes. They instead allow us to subscribe to get notifications when a change occurs with the details being pushed to us instead. This push comes in the form of an HTTP post to our REST end point.  Webhooks are not unique to SharePoint and are implemented in many other web based platforms such as Git, Slack, Box, Dropbox and even Facebook and Instagram.  What is unique however, and different for every platform, is what takes place during that notification. There is no established contracts. Each platform governs what content is part of the notification.  This is where we will dig into some detail. We need to understand how SharePoint Webhooks work in order to implement our Azure Function REST endpoints correctly.

Implementation wise I had previously made the comparison to Remote Event Receiver. They have some commonality in the general sense of responding to events within a list but after those things start to diverge.  Unlike Remote Event Receivers, this notification could be for one or more events.  Also instead of being provided with references to the new or updated items we're simply given a change date so we're responsible for finding out what items actually changed after that.  This allows for Webhooks to scale very well but requires a different approach for handling changes to your code.  Finally, we only have asynchronous events available to us so we'll always be responding to some activity after the fact.

SharePoint Webhooks will send notifications for the following events:

  • ItemAdded, ItemUpdated, ItemDeleted
  • ItemCheckedOut, ItemCheckedIn, ItemUncheckedOut
  • ItemAttachmentAdded, ItemAttachmentDeleted
  • ItemFileMoed
  • ItemVersionDeleted
  • ItemFileConverted

Creating SharePoint Webhooks

The first step is creating a new subscription with a SharePoint List or Library through the SharePoint REST API. The subscription process requires three pieces of information:

  • Resource EndPoint - The SharePoint List URL that you are subscribing to
  • Server Notification Url - Our custom service REST endpoint URL
  • Expiration Date - When our subscript ends. Must be less than 6 months (the default is 6 months)
  • Client State - This is some optional data that gets passed along with the validation request. This only has meaning in the context of your service.

Your resulting JSON object will look similar to this:

 

The subscription API is currently only available through the SharePoint REST API but Microsoft is considering also adding subscription functionality to the Microsoft Graph as well.  Creating a new subscription requires us to submit a HTTP POST method to /_api/web/lists('[List Guid]')/subscriptions with the subscription object in the payload.  Subsequently, SharePoint will then validate your HTTP Endpoint specified in the Server Notification URL. Validation is accomplished by returning back a ValidationToken within 5 seconds of the request that was a parameter on the initial validation request.  If all works correctly you'll get back a HTTP 200 OK - any other response indicates a failure.

The entire process looks like this:

Where can I administer SharePoint Webhooks?

Despite the fact we have full API support for managing subscriptions there is currently not any UI to manage your SharePoint Webhooks out of the box.  To help fill in that hopefully temporary gap for my projects I decided to build a SharePoint Webhooks Administration Webpart.  This webpart, based on the SharePoint Framework, and built with React and the Office UI Fabric Components for React to provide a simple user interface to manage and create subscriptions.

With this user interface we can manage our subscriptions much easier. We have the ability to:

Review List Subscriptions and Subscription Count Summary

Review List Subscriptions and Subscription Count Summary

Add New Subscriptions

Add New Subscriptions

View & Delete existing subscriptions

View & Delete existing subscriptions

The source code is currently available up on Github and can be found at https://github.com/joshdcar/spwebhookadmin .

DISCLAIMER: This code base is a work in progress. Issue reports and contributions are welcomed.  The source code for this solution is provided as-is and I don't provide any warranty or guarantees.  Use at your own risk.

So what's next?

Now that we have a solid understanding of what our solution looks like and how SharePoint List Webhooks work we can move onto the next step of building our service with Azure Functions which we will be covering in Part #2 of this series - Building Office 365 Solutions with Azure.

Visual Studio 2017 Tools for Azure Functions and Continuous Integration with VSTS

Visual Studio 2017 Tools for Azure Functions and Continuous Integration with VSTS

Building a REST Service with Node.JS, DocumentDb, and TypeScript

Building a REST Service with Node.JS, DocumentDb, and TypeScript