Joshcarlisle.io is a blog written by Josh Carlisle based out of Raleigh North Carolina where I explore, Share, and sometimes abuse the modern software development landscape. 

 

Building a REST Service with Node.JS, DocumentDb, and TypeScript

Building a REST Service with Node.JS, DocumentDb, and TypeScript

REST Services are commonly the backbone for modern applications.  They provide the interface to the backend logic, storage, security, and other services that are vital for an application whether it is a web app, a mobile app, or both!  

For me, as recently as a years ago, my solutions were typically built with the .NET Stack consisting of ASP.NET Web API based services written in c#, various middleware plugins, and backend storage that usually involved SQL Server. Those applications could then be packaged up and deployed to IIS on premise or somewhere in the cloud such as Microsoft Azure. It's a pretty common architecture and has been a fairly successful recipe for me. 

Like many developers however in recent years I've spent an increasing amount of time with JavaScript. My traditional web application development has slowly moved from .NET server side solutions such as ASP.NET Forms & ASP.NET MVC to client side JavaScript solutions using popular JavaScript frameworks such as Angular. As I began working with the beta of Angular 2 I was first introduced to TypeScript and quickly grew to appreciate JavaScript even further. 

Spending so much time in JavaScript though I was intrigued with the idea of a unified development stack based entirely on JavaScript that wasn't necessarily tied directly to any particular platform.  I also started spending much more time with Microsoft Azure solutions and the combination of a REST based services built on Node.JS, Express, TypeScript, and DocumentDB seemed very attractive.  In my journey with that goal in mind I found I couldn't find a single all-inclusive resource that provided me what I was looking for, especially with DocumentDB,  so I worked through a sample project of my own which I'm sharing in this blog post to hopefully benefit some others on the same path.

A quick message on Azure DocumentDB

One of the foundations of this solution is Azure's DocumentDB service.  DocumentDB is one of Azure's Schema-Free NoSQL \ JSON Database offerings.  If you've had experience with MongoDB you will find DocumentDB very familiar.  In fact DocumentDB introduced protocol compatibility with MongoDB so your existing MongoDB apps can be quickly migrated to DocumentDB.   In addition to all the benefits, you might expect from a NoSQL solution you also get the high availability, highly scalable, low latency benefits of the Azure platform.  You can learn more about DocumentDB over at https://docs.microsoft.com/en-us/azure/documentdb/documentdb-introduction.

Prerequisites

Before getting started there are a couple prerequisites I suggest for the best experience while working with the source code of this project.

Visual Studio Code

If you're not using Visual Studio Code I highly encourage you to check it out. It's a fantastic free code editor from Microsoft. Grab yourself a copy at https://code.visualstudio.com/ . If you're already using another editor such as Atom or Sublime you're still good to go but you will need to make adjustments for your own debugging and development workflow for that editor.  If you're using Visual Studio Code you should have a solid "f5" debugging experience with the current configuration.

Azure DocumentDB Emulator

This project is pre-configured to use the Azure DocumentDB Emulator so no Azure Subscription is required!  The emulator is still in preview but is pretty solid and saves a lot of money for those just wishing to evaluate DocumentDB without any costs or subscriptions. More details on the emulator and links to download can be found at https://docs.microsoft.com/en-us/azure/documentdb/documentdb-nosql-local-emulator

If you'd like to you can also use a live instance of DocumentDB with your azure subscription but please make sure you understand the cost structure of DocumentDB as creating additional collections within a DocumentDB database has a cost involved.

Getting Started

For this sample project we'll be building a hypothetical photo location information storage service API. Our API will accept requests to create, update, delete, get photo location information.  In a future post we'll spend some more time with DocumentDB's Geospatial query features but for now we'll just keep it to simple CRUD operations.

The entire project can be found over at https://github.com/joshdcar/azure-documentdb-node-typescript .  Feel free to clone\review the code there. 

For this project we'll be making use of the following stack:

  • NodeJS - 6.10.0 (Runtime)
  • Express - 4.14.1 (Host)
  • TypeScript - 2.2.1 (Language)
  • Mocha\Chai - 2.2.39\3.4.35 (Testing)
  • Azure DocumentDB Emulator - 1.11.136.2 (Storage)

Project Setup

There are some folks with some pretty strong opinions in regards to a project's folder structure but I tend to side more with consistency than any particular ideology.

-- dist
-- src
+ -- data
 + -- LocationData.ts
 + -- LocationDataConfig.ts
 + -- LocationDocument.ts
+ -- routes
 + -- PhotoLocationRouter.ts
+ -- app.ts
+ -- index.ts
-- test
+ -- photolocation.test.ts
-- gulpfile.js
-- package.json
-- tsconfig.json

Let's break down some of these project elements:

  • dist - build destination for compiled javascript
  • src  - project source code containing our typescript. We'll be breaking down each of the source files further down in the post
  • test - test scripts
  • gulpfile.js - our build tasks
  • package.json - project metadata & NPM dependencies
  • tsconfig.json - typescript configuration file

NPM Packages - Packages.json

One of my pet peeves with sample projects, especially when I’m new to a technology,  are unexplained dependencies. With the multitude of open source tools and libraries I often find myself looking up modules to find out what they do and if I need them.  This muddies the waters for those not as familiar with the stack so I find it helpful keeping them to a minimum in my projects and explaining each of them and their purpose.

  • @types/*  - these are typescript definition files so our typescript code can interact with external libraries in a defined manner (key to intellisense in many editors)
  • Typescript - core typescript module
  • gulp/del - javascript based task runner we use for typescript builds and any extra needs. Del is a module that deletes files for us
  • gulp-sourcemaps/gulp-typescript - helper modules to allow us to use gulp to compile our typescript during builds
  • Mocha - A javascript testing framework and test runner
  • chai/chai-http - a testing assertion framework we use for creating tests. We're specifically testing out http REST requests
  • Gulp-mocha - A gulp task to help us run mocha tests in gulp (NOTE: running into issues with this one so it remains while I sort it out but we'll be running tests form npm scripts - more on this further in the post)
  • Ts-node - A typescript helper module used by Mocha for tests written in TypeScript
  • Documentdb: Azure's DocumentDB Javascript Module
  • Express - our service host
  • Body-parser - a module that helps express parse JSON parameters automatically for us.

Setting up and working with Typescript

Typescript is fairly straight forward especially for developers who are comfortable working with compiled languages.  Essentially TypeScript provides us with advanced language features that aren't yet available in the current version of Javascript. It does this by compiling typescript down to a targeted version of Javascript. For browsers application, this is typically ES5 but for Node.JS based applications we can reliably target ES6 which is newer but generally not available in most browsers.  The end result of that process though is always standard JavaScript. Typescript is not actually a runtime environment.  For .net developers, this is very much a kin to c# being compiled down to IL.   Additionally, to making debugging easier we have the concept of sourcemaps which map our generated JavaScript with the original lines of typescript code.

Where the confusion often occurs is where\when\how to compile. There are lots of options for us. For front-end UI developers webpack is a common tools. For Node.js projects, such as this, another common approach, and one we make use of in this project, is gulp. 

Configuring Typescript and Gulp

Getting Typescript to compile is typically pretty straight forward, but getting it to compile properly for debugging can be another story entirely and typically the pain point for many developers.

The first file we'll be working with is the tsconfig.json which provides compiler options to the typescript compiler.

In the case of our project we wanted to target JavaScript ES6, use commonjs for our module format, and specify which directories we want the compiler to include and exclude.

@Types - working with external libraries

Type definition files in Typescript allow TypeScript to work with external libraries that were not developed originally in typescript. Most popular JavaScript libraries have had types defined by either the project or contributions from the typescript community at large. They define the interfaces, types, functions, data types that the library exposes. The way in which TypeScript references and works with Types changed from 1.x - 2.x of typescript so you still may see references to the old way. Ensure that you're using @types/* to get the latest versions of your types from NPM and that they are in sync with the version of the library you pulled down from NPM if your library doesn't already included types. 

Gulp

Gulp is a JavaScript based task runner. It has a very large community of plugins to help execute practically every conceivable task you could think of that you might need for a build. To go back to a .NET comparison Gulp is akin to the features you may find in MSBuild.

Gulp tasks are defined as JavaScript functions and can optionally have dependent other tasks that are executed first which is syntactically represented by the option array of functions as the second argument for a task function. In our case we have a "Compile" task that executes the typescript gulp plugin and works in conjunction with the source maps plugin to compile and output our Javascript to the dist directly.  Note that we have the "dist" directory both within the gulp task and the tsconfig task. The gulp plugin uses the base settings from the tsconfig to execute the typescript compilation process but also supports Gulps "stream" concept to output files to a desired location.

TAKE NOTE:  If you change the commands within the Compile function you may not have full debugging support within Visual Studio Code. The settings provided ensure that the sourcemaps have the correct references and paths used by Visual Studio to allow for breakpoints to be used as expected. These few lines of code took me several hours of research, trial & error for all the stars to line up. If you have problems the first place to start looking is your generate *.map files and ensure the paths provided are what you expect.

Configuring Express and your API Routes

Express is a web application framework for Node.js. Express will host our REST services and provide all the middleware plumbing we need for our service.

Index.ts

Index.ts is essentially responsible for wiring our express based application into the http pipeline within node.js. It's also the entry point for our application where we can configure options such as port to listen to. 

NOTE: I need to give some credit to another developer for Index.ts because I know at some point many aspects of this module were copy\paste from somewhere but unfortunately I can't recall who that was but if I do find the original source I'll make sure to update the source and this post with the developer's name.

App.ts

App.ts sets up an instance of express and provides us a location to configure our middleware with some additional plumbing such as parsing JSON, urlencoding and some basic logging.  Also key to our application is the configuration of the routes used by our API and the code that is going to handle the routes. Other possible middleware components could be authentication, authorization, and caching just to name a couple.  

this.express.use('/api/v1/photolocations', PhotoLocationRouter);

PhotoLocationRouter.ts

The core of how our service handles request at a particular end point is managed within the router.  We define the functions that execute on a given protocol and any appropriate parameters:

this.router.get("/:id", this.GetPhotoLocation),
this.router.post("/", this.AddPhotoLocation),
this.router.put("/",this.UpdatePhotoLocation),
this.router.delete("/:id",this.DeletePhotoLocation)

For more complex applications it can be benficial to follow this seperate of routes from the app.  In the case of our sample project we're following REST standards of using Get/Post/Put/Delete appropriately for different CRUD operations.

public GetPhotoLocation(req:Request, res: Response){

let query:string = req.params.id;
var data:LocationData = new LocationData();

data.GetLocationAsync(query).then( requestResult => {
res.status(200).send(requestResult);

}).catch( e => {

res.status(404).send({
message: e.message,
status: res.status});

});

}

Each operation has a Request and Response object to interact with. In the request object we can access the parameters from either the URL or from the body. Our body parser middleware neatly packs up our body data into a JSON object (given the correct content-type header) and any URL paramters also get neatly extracted from the URL and packed into a property name based on the pattern provided in the route. In the above example we're able to access the "id" parameter.  The response object alllows us to respond with specific\appropriate http codes and any appropriate payload of data. 

NOTE: I dislike peppering my blog posts with disclaimers but this configuration should be intended for development use only. Additional configuration options can\should be applied for production applications, especially with security and threat hardening. I say this because of a recent Express vulnerability that left a lot of Node.JS sites unnecessarily exposed. For a good place to start checkout Helmet over at  https://www.npmjs.com/package/helmet

 

Working with DocumentDB

DocumentDB is a NoSQL solution that at its core stores JSON Documents. There are various best practices around working with DocumentDB but sticking with the basic concepts we'll be creating, updating, deleting, and querying those JSON documents.  Having worked with DocumentDB recently within the .NET framework using Linq this was a bit of a departure but luckily DocumentDB supports standard SQL queries so my years of working with relational databases paid off once again!  This was great foresight from Microsoft to make use of SQL instead of implementing yet another query language (I'm pointing at you SharePoint CAML!) 

DATABASE AND COLLECTION CREATION:  This sample project has a DocumentDB database and Collection.  The code does NOT create those for you so the expectation is that you will do this step up front. The database is named photolocations and the collection locations. Due to the fact that DocumentDB Collections and Databases have a direct cost involved with them (outside the emulator of course) I'm not a fan of code that generates these entities for you. I rather have the process be explicit. 

Defining our document

As an initial step we're going to define that JSON document within our project so we can work with it in a consistent way. In addition to our own properties DocumentDBadditionally has some standard properties that are part of every document. We can think of them as system fields.  To better support these fields the DocumentDB framework requires us to implement these interfaces for new documents (such as the id) and for retrieved documents which have additional fields such as etag for optimistic concurrancy checks.

ADVICE:  To keep things simple we're exposing the DocumentDb JSON document directly from the REST Service.  There are many scenarios where you may want to have a separate model that has different\fewer properties that is returned from the REST calls based on your applications needs.

DocumentDB operations

The core heavy lifting class within DocumentDB is DocumentClient. All operations such as queries, updates, inserts, deletes are all done through the DocumentClient class.  These functions make heavy use of callbacks so to make our own data layer functions more friendly to work against we wrapped all of our calls within Promises.  Note that many of the classes and interfaces we are working with are all imported in from the DocumentDB module and provided to us through those all-important interfaces.

Unit Testing

Cards on the table I'm very new with Javascript testing frameworks but I found the test-first approach really reduced the debugging and development cycles greatly. I created the test plans first calling the service and letting them fail.  I defined all my expectations (assertions) for the service and then I developed the service until it passed all the tests. I found using Mocha \ Chai pretty straight forward and I'm looking forward to spending more time with the framework in the coming months and sharing that experience.

NOTE: I would not use my test plans for this project as a model for your own test plans. They work but they require more teardown and cleanup then I would like and I suspect introducing the concepts of mocking while working against the DocumentDB would be beneficial. For now the tests are actually performing operations against DocumentDB. I place these firmly in the "place to start" category.

Development & Debugging

Throughout the development process I made heavy use of a Chrome Plugin called postman.  Postman allows me to test various REST calls and adjust the body\headers\etc as needed. I know there are other tools such as CURL that provide the same set of features but I've found the features and user interface of Postman superior to most other solutions I've worked with adding a great bit of efficiency to my development cycle.

TIP:  Chrome has this nasty habit of automatically redirecting localhost traffic to SSL if anything else runs SSL on localhost -it's a feature called HSTS.  In our case the emulator runs under SSL on localhost so I battled this "feature" constantly. This is internal to Chrome and must be reset by going to a settings page within Chrome chrome://net-internals/#hsts and entering in "localhost" to delete the domain. Even worse this property doesn't stick and routinely gets added back in. The "fix" for this is to either run your service under SSL as well,  add a host header for your app locally so it's on a different host then localhost, or use another browser such as Firefox or IE for testing.  I know this is a safety feature but it's very annoying for us developers and I wish there was a way to disable it permanently for given domains. 

Parting thoughts

So we've learned the basics of building a simple REST service based on Node.JS, DocumentDB, and TypeScript. In an upcoming post I'll be building on this solution to demonstrate more features of DocumentDB, especially the GeoSpatial query features, and we're also explore adding oauth authentication to our service to safeguard access to our data.

 
Building Office 365 Solutions with Azure - Webhooks (Part 1)

Building Office 365 Solutions with Azure - Webhooks (Part 1)

Azure Functions & Office 365 - Match Made in Heaven

Azure Functions & Office 365 - Match Made in Heaven