Ein Kommentar

How to create a microservice with Node.js

As you are reading this, I assume that you are already aware of the big avantages of using microservices. Furthermore I assume, that you already chose Node.js and Typescript (or were instructed to do so) so I don’t need to convince you to. In the following article, we will learn how to write a microservice, that will provide a RESTful API to CRUD [stuff].

In the next articles, we will handle:

Prerequisites

Folder and file structure

For me it was very hard to find a „good structure“ for my project as there are a lot of different examples out there in the internet and there are for sure aspects of my structure, that can be improved. After the tutorial, our project will look like this. Entries in italics will be created automatically. For a better overview the tree does not contain files or folders that will not be submitted into source control like packages or the compiler output.

  • .vscode
  • src
    • controllers
      • stuff.ts
    • errors
      • bad-request.ts
      • not-found.ts
      • response.ts
      • server.ts
    • models
      • stuff.ts
    • routes
      • health.ts
    • app.ts
    • index.ts
  • test
    • health.ts
    • stuff.ts
  • .gitignore
  • .gitlab-ci.yml
  • nodemon.json
  • package.json
  • package-lock.json
  • service.env
  • README.MD
  • swagger.yml
  • tsconfig.json
  • tslint.json

Node.js init

Let us start by initializing a new Node.js project by browsing to the projects directory and running:

npm init

This will provide a small text based wizzard, that will result in a package.json file in your project directory. Working with npm will also create the package-lock.json file.

Typescript

You can simply use npm to install Typescript to your project.

npm i typescript ts-node

We now need to add type information for the basic node types by running:

npm i -D @types/node

To start using typescript, you will need a tsconfig.json file. You can create one by running

./node_modules/.bin/tsc --init

Afterwards open the newly created file tsconfig.json and uncomment the lines with the following keys and change the values like this:

{
  "compilerOptions": {
    "target": "es2018",
    "module": "commonjs",
    "strict": true,
    "strictNullChecks": true,
    "esModuleInterop": true
  }
}

Visual Studio Code

Start your Visual Studio Code and add the newly created project directory to your workspace. To improve the performance, we will ignore some file in VS Code. To do so press [Ctrl + Shift + p] and type „Folder Settings“. Now use the arrow key to select „Preferences: Open Folder Settings“ and confirm with [Enter]. Select your project folder from the list and confirm with [Enter]. In the settings Dialog, search for „exclude files“. On the left side click ob [Text Editor > Files] and then click the „Add Pattern“ button to add these patterns

  • **/node_modules
  • /dist
  • /coverage
  • /.nyc_output

VS Code will automatically create a folder „.vscode“ that contains a file „settings.json“. When later comitting the project to git, you should check this file in.

Now we press [Ctrl + Shift + e] to open the explorer view. In the explorer view we can create the folder structure for the project.

  • src
    • controllers
    • models
    • routes
  • test

Ignore files for git

After ignoring the node_modules in VS Code we should also ignore them in git, so they will no spam the list of added files or they might even accidently get added to the repo. To do se we create a file called „.gitignore“ in the root directory of the project and add this content:

node_modules/
dist/
coverage/
.nyc_output/

Linting the code

Linting is important to improve the readability of the code. We want to use tslint for this. In the prerequisites we already installed the tslint extension for VS Code and now we need to install tslint and some default configs to our project.

npm i -D tslint tslint-config-standard tslint-no-unused-expression-chai

The next thing we need is a tslint config. This command can be used to create a default one:

./node_modules/.bin/tslint --init

Linting is mostly a matter of taste. I personally use „tslint-config-standard“ to cover the basics and „tslint-no-unused-expression-chai“ to avoid false positives in my test files when using chai. This can be easily achieved by putting those ids in the extends part of the tslint config file.

{
  "extends": [
    "tslint-config-standard",
    "tslint-no-unused-expression-chai"
  ]
}

You can alter any specific rule by adding an entry to the rules object. I usually use this [list of available rules](https://palantir.github.io/tslint/rules/) if i want to add or change a rule. If we wanted to enforce alphabetical sorted imports, we can simply add it to the rules:

{
"rules": {
  "ordered-imports": true
}

Also go to your VS Code project settings (.vscode/settings.json) and add this lines to make sure your changes a being autocorrected when saving a file in VS Code. Also VS Code is supposed to use 2 spaces for indentation (default is 4).

{
{
"editor.codeActionsOnSave": {
"source.fixAll.tslint": true
},
"editor.tabSize": 2
}

Installing the packages we need to run the webserver

Let’s install the required packages for the webserver and also install the types for those packages

npm i express morgan body-parser cors http-status-codes
npm i -D @types/express @types/morgan @types/body-parser @types/cors

What do these packages do?

  • express is our web framework
  • morgan is the logger
  • body-parser will be used to get objects from the bodies of the HTTP requests
  • cors will be used to support cross origin resource sharing when the api and the website are not running on the same domain.
  • http-status-codes will be used to avoid *magic numbers* as HTTP status codes
  • dotenv will be used to prepare the environment on our development system to make the app executable without its container

Environment

Before we can start, we need to define how to communicate with the outside world. We will use environment variables for our configuration. If we just want to create a Node.js service, this is a little over the top, but it will be very useful when we put the service in a Docker container. Dotenv will help us to modify the environment to fit our needs.

npm i dotenv
$ npm i -D @types/dotenv

Next we create a file „service.env“ in the root directory of the project and enter some configuration. We will later load the key value pairs from this file into our environment.

PORT=8080
DB_HOST=localhost
DB_PORT=27017
DB_DATABASE=things
DB_SSL=false

The PORT defines on which port the service will listen. The DB_ entries describe the connection to the MongoDB server.

Basic Configuration

In the next step we will do the basic configuration of our express app. Let’s create a file called api.ts directly in the src directory.

import * as bodyParser from 'body-parser'
import cors from 'cors'
import express from 'express'
import * as httpStatusCodes from 'http-status-codes'
import logger from 'morgan'

// this prefix will later be used for all api calls that will be available from the outside
const prefix = '/api/v1/stuff/'

class App {
  public express: express.Application

  constructor () {
    this.express = express()
    this.middleware()
    this.routes()
  }

  private middleware (): void {
    this.express.use(logger('[asset-manager] :method :url :status :response-time ms - :res[content-length]'))
    this.express.use(bodyParser.json())
    this.express.use(bodyParser.urlencoded({ extended: false }))
    this.express.use(cors({ optionsSuccessStatus: httpStatusCodes.OK }))
  }

  private routes (): void {
    // ToDo add your routes here

    this.express.use(this.errorHandler.bind(this))
  }

  private errorHandler (err: any, req: express.Request, res: express.Response, next: express.NextFunction) {
    // ToDo bring the error into our format
  }
}

const app = new App()
export default app.express

The app class is being used to describe how the webserver is supposed to behave but not how it is supposed to be reached(listen to ports etc). Separating these two aspects makes unit testing much easier.

Start the Server

To make our project startable, we create a file called „index.ts“ in the root directory.

import dotenv from 'dotenv'
dotenv.config({ path: 'service.env' })
import * as http from 'http'
import App from './api'

const port = process.env.PORT
const server = http.createServer(App)
server.listen(port)
server.on('listening', onListening)

function onListening (): void {
  let addr = server.address()
  let bind = (typeof addr === 'string') ? `pipe ${addr}` : `port ${addr.port}`
  console.info(`Listening on ${bind}`)
}

This file is pretty lightweight. It simply imports the app we just created and hands it over to a webserver that listens to the port configured in the enviroment. To make sure a port is configured on development systems, it imports the key value paris from service.env using dotenv. Typically this file is rarely being touched when continuing development. We should now be able to compile and start the server.

./node_modules/.bin/tsc
node ./dist/index.js

When opening http://localhost:8080/api/v1/stuff the server should return 404 NOT_FOUND.

Ts-node and Nodemon

Compiling and Restarting the app after every change is a little annoying. To improve our comfort we will use ts-node and nodemon. Ts-node can directly start typescript projects without compiling them first and also error messages will no longer point to lines in the .js files but to the .ts file instead. Nodemon will watch the src directory and restart the server, every time we change the code.

First we need to install the packages.

npm i -D ts-node nodemon

Then we need a nodemon config so we create a new nodemon.json in the root folder.

{
"watch": ["src"],
"ext": "ts",
"exec": "./node_modules/.bin/ts-node ./src/index.ts"
}

To start the server, we start the watcher.

./node_modules/.bin/nodemon

Test runner etc

Automated tests are very important in modern software development, so of course we will write some, too. As most steps in this tutorial, we will start by installing some packages. We also need to install types for mocha as they are not automatically included.

npm i -D mocha @types/mocha chai @types/chai chai-http nyc

What do these packages do?

  • mocha is the test framework
  • chai is the assertion library
  • chai-http is the http plugin for chai assertions
  • nyc will analyze the code coverage of the tests

Health endpoint

To prepare the service for the deployment to kubernetes (we will do that in one of the next articles) we need a possibility to check if the service is healthy and ready. Kubernetes supports health checks via HTTP, which obviously is the best choice for a service that provide an API via HTTP.

We will start by creating some files.

In the routes directory we create a file called „health.ts“. In this file we will provide a router that returns { „status“: „OK“ } and status code 200 for GET requests on /liveliness and /readiness.
This does not do much so far, but later on we can add logic to check the db connection etc.

import express from 'express'
import { Router } from 'express-serve-static-core'

export function GetRouter (): Router {
  const router = express.Router()

  router.get('/liveliness', (req: express.Request, res: express.Response, next: express.NextFunction) => res.json({ status: 'OK' }))
  router.get('/readiness', (req: express.Request, res: express.Response, next: express.NextFunction) => res.json({ status: 'OK' }))

  return router
}

Now we need to add this router to our app. To do so, we open the existing api.ts and import the router.

import { GetRouter as getHealthRouter } from './routes/health'

Now we can use the router in the routes() method.

private routes (): void {
  this.express.use('/health/', getHealthRouter())
  // ToDo add your routes here
  this.express.use(this.errorHandler.bind(this))
}

Asuming the watcher is running, we can now browse to http://localhost:8080/health/liveliness to get this response

{"status":"OK"}

Creating the first tests cases

As the api finally provides some information, we can write our first test cases. To do so, we create a file called health.ts in the test directory.

import { expect, request, use as chai_use } from 'chai'
import chaiHttp = require('chai-http')
import * as HttpStatusCodes from 'http-status-codes'
import app from '../src/api'

chai_use(chaiHttp)

const apiPrefix = '/health'

describe('/health', () => {
  it('get liveliness', async () => {
    const res = await request(app).get(`${apiPrefix}/liveliness`)
    expect(res).to.have.status(HttpStatusCodes.OK)
    expect(res).to.be.json
    expect(res.body.status).to.equal('OK')
  })

  it('get readiness', async () => {
    const res = await request(app).get(`${apiPrefix}/readiness`)
    expect(res).to.have.status(HttpStatusCodes.OK)
    expect(res).to.be.json
    expect(res.body.status).to.equal('OK')
  })
})

Now we should execute those tests using the mocha test runner and again ts-node.

./node_modules/.bin/mocha -r ts-node/register test/**/*.ts

The output should look like this

  /health
[asset-manager] GET /health/liveliness 200 2.906 ms - 15
    √ get liveliness
[asset-manager] GET /health/readiness 200 0.410 ms - 15
    √ get readiness

VS Code debugging

Running the app is nice, but if we want to debug it, we need to be able to set breakpoints and step through the code. It will just take a little configuration to be able to use VS Code for that so let’s get to work.
[Ctrl + Shift + d] will open the debug panel. On top of it we can see a small gear wheel icon. If we click it, VS Code will create a launch.json file in the .vscode directory and open it for us.

I usually use these configurations. The first starts the whole server in debug mode and allows me to set breakpoints while regularly running the webserver. The second configuration will run all tests with a #dev flag in debug mode. If there are a couple hundred tests in the project, you want to be able to select which cases to run. You can add the #dev tag to any „describe“ or „it“ description in your tests and mocha will do the rest for you as we are using „–grep #dev“ as mocha parameter. If you submit this tag to source control (and this will happen eventually!) it is not a problem as the regular test runner will print it to the output but otherwise ignore it.

  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Service",
      "args": [
        "./src/index.ts"
      ],
      "runtimeArgs": [
        "--nolazy",
        "-r",
        "ts-node/register"
      ],
      "sourceMaps": true,
      "cwd": "${workspaceRoot}",
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Tests with #dev",
      "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
      "args": [
        "-r",
        "ts-node/register",
        "--timeout",
        "999999",
        "--colors",
        "${workspaceFolder}/test/**/*.ts",
        "--grep",
        "#dev"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "protocol": "inspector"
    }
  ]

npm scripts

As the project is almost ready now and we are kind of lazy (every developer should be) we should shorten the commands, that we need to run on a regular basis.
We simply use npm for that. Open your package.json and search for the „scripts“ part. We will change it to look like this:

"scripts": {
  "build": "tsc",
  "start": "node ./dist/index.js",
  "watch": "nodemon",
  "test": "nyc -r lcov -e .ts -x \"test/*.ts\" mocha --timeout=30000 -r ts-node/register test/**/*.ts && nyc report"
},

Now we can run all of those scripts by running npm run script. For „start“ and „test“ we do not even need the „run“ keyword.

As you can see there is one thing called „nyc“ we did not talk about so far. It is the coverage tool for our tests. Let’s see what it does by running

npm test

You will see that the tests are executed like before and afterwards you will see a small text based table in your console that shows information about code coverage. Y ou will also find a directory called coverage in your project directory. It contains a directory „lcov-report“ which contains a report in html format. Just open the index.html and click through it. You will see that the only code, that was not covered to far is the errorHandler in api.ts. This is because we did neither finish nor use it.

Error handling

Most things we have done so far was using packages to do what we needed. For error handling I could not find anything like it, so we need to develop something ourselves.
Let’s create a directory called „errors“ in our source directory and place a file called „response.ts“ there. This will be our abstract base error class.

export abstract class ResponseError extends Error {
  constructor (message: string, code: number) {
    super(message)
    this.code = code
  }
  code: number
}

Now we create our first non abstract error class in a file called „bad-request.ts“

import * as HttpStatusCodes from 'http-status-codes'
import { ResponseError } from './response'

export class NotFoundError extends ResponseError {
  constructor (message: string) {
    super(message, HttpStatusCodes.NOT_FOUND)
  }
}

We can also create some more errors like the examples below now or whenever we need them.

 

File Error name Status code
server.ts ServerError HttpStatusCodes.INTERNAL_SERVER_ERROR
bad-request.ts BadRequestError HttpStatusCodes.BAD_REQUEST

Now we need to go to api.ts and change the code of our errorHandler()

private errorHandler (err: any, req: express.Request, res: express.Response, next: express.NextFunction) {
  // bring the error into our format
  // istanbul ignore if
  if (!(err instanceof ResponseError)) {
    err = new ServerError(err.message)
  }
  res.statusCode = err.code
  res.json({ code: err.code, time: (new Date()).toISOString(), message: err.message })
}

As we are using code coverage analysis and we try to get a good coverage, we need to ignore those lines that are supposed to handle unexpected errors. // istanbul ignore if is being used to remove the „if part“ of the statement from the coverage analysis.
To use this error code we need to make sure that every request that results in an error calls next(error). As we do not have any request yet that we expect to return any error this will need to wait before we can use it.

Now we have created some kind of default template we can use for every nodejs microservice that is supposed to provide a RESTful API.
I have uploaded the template from this tutorial here.

 

Ein Kommentar

Kommentar schreiben

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.