Joshcarlisle.io is a blog written by Josh Carlisle based out of Raleigh North Carolina where I explore, Share, and sometimes abuse the modern software development landscape. 

 

Introducing Azure Function Extensions for Cognitive Services

Introducing Azure Function Extensions for Cognitive Services

A few weeks ago I gave a sneak peak on a project I was working on that improved the developer story for working with Cognitive Services and Azure Functions. I'm excited today to announce the first public preview of Azure Functions Extensions for Cognitive Services, a collection of Azure Function Bindings that greatly simplify working with Microsoft's Cognitive Services.

For those not familiar, Cognitive Services are a set of machine learning algorithms that Microsoft has developed to solve problems in the field of Artificial Intelligence (AI).  Cognitive Services provide a set of REST APIs that provide intelligent analysis for Images, Speech, Language, and Knowledge. The Cognitive Services bindings greatly simplify the process of working with the Cognitive Services APIs within Azure Functions and also takes care of frequently needed supporting operations such as automatic retries.

The preview features include:

  • Support for Computer Vision APIs including:
    • Image Analysis
    • Optical Character Recognition (OCR)
    • Handwriting
    • Celebrity and Landmark Analysis
    • Thumbnail Generation
  • Intuitive, strongly typed, easy to use API
  • Support for automatic retry policies for throttled requests (via Polly.Net)
  • Simplify the execution of asynchronous Cognitive Services APIs such as Handwriting analysis
  • Support for Key Vault storage of Cognitive Services Keys
  • Additional Exception, Warning, Information, and Metrics Logging

Getting Started

All source code, samples, getting started guides, package locations can be found on the Github project site located at  https://github.com/joshdcar/azure-functions-extensions-cognitive-services. The project is covered under the MIT License and these extensions are free to use and incorporate into your own solutions.

The Azure Function Extensions for Cognitive Services currently  supports Azure Function 2.0. Please post any questions, issues, feature requests, etc at  https://github.com/joshdcar/azure-functions-extensions-cognitive-services/issues and we can continue the conversation there!

Working with these custom bindings is as straight forward as adding the Nuget Package to your Azure Functions 2.0 project (see github repository link above for more details) and adding references to the various bindings available within your function.  The bindings require either the file to be processed by cognitive services OR a valid URL that is accessible to Cognitive Services. Unless your URLs are publicly accessible you will likely be working directly with files.   The bindings work especially well when used in conjunction with Blob Triggers and bindings.

public static async Task Run(
   [BlobTrigger("visionrequest/{name}")]Stream storageBlob,
   [VisionAnalysis(Key = "%VisionKey%", Url = "%VisionUrl%")]VisionAnalysisClient visionclient,
   string name,
   TraceWriter log){
            var result = await visionclient.AnalyzeAsync(new VisionAnalysisRequest(storageBlob));
            log.Info($"Analysis Results:{result.ToString()}");
 }

Core to working with the extensions are the Attribute Bindings provided for each of the Cognitive Services API such as the VisionAnalysis and their corresponding client class instantiated by the binding.  Many of the connection related requirements such as the key and url can be bound through the various attributes in addition to some default request options that support values that attributes support (strings).  If you need to assign or customize these settings they can also be set through the various request classes such as the VisionAnalysisRequest class above allowing the extensions to support both simple and more complex options logic.  

What's Next

In the coming weeks I will be improving the samples, getting started guides, general documentation alongside working hard to make any required stability updates and implement the remainder of my 1.0 feature targets including:

  • Support for the remainder of the Vision APIs within Cognitive Services
  • Support for Language and Knowledge APIs within Cognitive Services
  • Automatic Image Resize to better support Cognitive Services image size limits
  • Providing additional language support and guidance for Javascript (along with other languages popular with Azure Functions)

Additionally I will be investigating other binding options that may allow for more automatic operations independent of explicit requests to the client classes.  More to come with that soon.

You may have noticed that we have a lot of preview dependencies with this project which itself is in Preview. Azure Functions 2.0 is in preview, several Cognitive Services are still in preview, the native .net core image library ImageSharp that I am relying on for image sizing is also in preview.  Despite all the preview tags my goal is to reach a GA status and make these extensions production ready alongside the Azure Functions 2.0 GA Release.

Like many projects stemming from a simple concept this has been more work than I initially expected as I worked through establishing a pattern I was happy with in regards to the bindings architecture.  I'm very excited with this initial release but I'm even more excited to see how this progresses in the coming weeks and months as I work on improvements.

I do plan to support community contributions in the very near future if you are interested in contributing to the project. In the mean time I value any feedback and suggestions!

Are you getting "KeyNotFoundException: The given key was not present in the dictionary." with your VSTS Builds?

Are you getting "KeyNotFoundException: The given key was not present in the dictionary." with your VSTS Builds?

2018 Community Talks Portfolio

2018 Community Talks Portfolio