Auto incrementing IDs for MongoDB

If you’re familiar with relational databases like MySQL or PostgreSQL, you’re probably also familiar with auto incrementing IDs. You select a primary key for a table and make it auto incrementing. Every row you insert afterwards, each of them gets a new ID, automatically incremented from the last one. We don’t have to keep track of what number comes next or ensure the atomic nature of this operation (what happens if two different client wants to insert a new row at the very same time? do they both get the same id?). This can be very useful where sequential, numeric IDs are essential. For example, let’s say we’re building a url shortener. We can base62 encode the ID of the url id to quickly generate a short slug for that long url.

Fast forward to MongoDB, the popular NoSQL database doesn’t have any equivalent to sequential IDs. It’s true that you can insert anything unique as the required _id¬†field of a mongodb document, so you can take things to your hand and try to insert unique ids yourselves. But you have to ensure the uniqueness and atomicity of the operation.

A very popular work around to this is to create a separate mongodb collection. Then maintain documents with a numeric value to keep track of your auto incrementing IDs. Now, every time we want to insert a new document that needs a unique ID, we come back to this collection, use the $inc operator to atomically increment this number and then use the incremented number as the unique id for our new document.

Let me give an example, say we have an messages collection. Each new message needs a new, sequential ID. We create a new collection named sequences. Each document in this sequences collection will hold the last used ID for a collection. So, for tracking the unique ID in the messages collection, we create a new document in the sequences collection like this:

{
    "_id" : "messages",
    "value" : 0
}

Next, we will write a function that can give us the next sequential ID for a collection by it’s name. The code is in Python, using PyMongo library.

def get_sequence(name):
    collection = db.sequences
    document = collection.find_one_and_update({"_id": name}, {"$inc": {"value": 1}}, return_document=True)

    return document["value"]

If we need the next auto incrementing ID for the messages collection, we can call it like this:

{"_id": get_sequence("messages")}
Find and Modify – Deprecated

If you have searched on Google, you might have come across many StackOverflow answers as well as individual blog posts which refer to findAndModify()¬†call (find_and_modify¬†in Pymongo). This was the way to do things. But it’s deprecated now, so please use the new find_one_and_update¬†function now.

(How) Does this scale?

We would only call the get_sequence function before inserting a new mongo document. The function uses the $inc operator which is atomic in nature. Mongo guarantees this. So even if 100s of different clients trying to increment the value for the same document, they will be all applied one after one. So each value they get will be unique, new IDs.

I personally haven’t been able to test this strategy at a larger scale but according to people on StackOverflow and other forums, people have scaled this to thousands and millions of users. So I guess it’s pretty safe.

REST API with KoaJS and MongoDB (Part – 3)

In Part -1¬†of this series, we saw how we can get started with KoaJS and in Part – 2¬†we built CRUD endpoints with MongoDB. In this part, we’re going to work with authentication. We will be using JSON Web Tokens aka JWT for the auth part. We have written detailed pieces on JWT before. You can read¬†Understanding JWT¬†to check out the basics and read our tutorial on JWT with Flask¬†or JWT with Django to see how other frameworks like Flask uses JWT.

JWT with KoaJS

To implement JSON Web Tokens with KoaJS, we would be using two packages Рkoa-jwt and jsonwebtoken. The second package (jsonwebtoken) provides useful helper functions to generate and verify JWTs. Where as koa-jwt provides an easy to use middleware that we can use with KoaJS.

Let’s go ahead and install these packages:

npm i -S koa-jwt jsonwebtoken

That should install the dependencies and save them in our package.json.

Securing Routes with JWT

We have the required packages installed. So we can now start securing our routes with JWT. We can just require the koa-jwt package directly and use it. But we want to customize some aspects. For that we would create our own module named jwt.js and put the custom stuff in there.

const jwt = require("koa-jwt");
const SECRET = "S3cRET~!";
const jwtInstance = jwt({secret: SECRET});

module.exports = jwtInstance;

Now in our index.js file, we would add the middleware to the app.

app.use(require("./jwt"));

If we try to visit¬†http://localhost:3000/, we will get a plain text error message saying “Authentication Error”. While the message is clear and concise, we want to output JSON, not a plain text error message. For that, we will write a custom middleware.

function JWTErrorHandler(ctx, next) {
    return next().catch((err) => {
        if (401 == err.status) {
            ctx.status = 401;
            ctx.body = {
                "error": "Not authorized"
            };
        } else {
            throw err;
        }
    });
};

The code for this middleware is pretty simple. It invokes the next middleware and if it catches an error, it checks if it’s 401, if so, it sets a nice detailed JSON as the output. If you’re familiar with how middlewares work in express / koa, this should make sense. If it doesn’t make sense, don’t worry, you will get it over time.

Now we need to export this function from our jwt.js¬†module. Let’s change the exports a little bit.

module.exports.jwt = () => jwtInstance;
module.exports.errorHandler = () => JWTErrorHandler;

Now we’re exporting two functions, which, when called will return the specific middlewares. We also need to change our imports in index.js¬†–

const jwt = require("./jwt");
app.use(jwt.errorHandler()).use(jwt.jwt());

Please note the order of the middleware we used. The error handler must come before the JWT middleware itself, so it can call next() and check for the 401 error.

If we try to browse the API now, we should get a nice JSON like this:

{"error":"Not authorized"}

Secured Routes and Router

We used the middleware directly on the koa app. That means all our routes are now secure. All the routes would now check for the Authorization¬†header value and try to verify it’s value as a JSON Web Token. That’s good but there’s a slight problem. If we can’t access any of the routes without a token, which route do we access to get the token in the first place? And what token do we use for that? Yeah, we need to have at least one route which is not secured with JWT which will accept login details and issue the JWTs to the users. Besides, there could be other API end points which we can keep open to everyone, we don’t need authentication on those routes. How do we achieve that?

Luckily, Koa allows us to use multiple routers and each router can have their own set of middlewares. We will keep our current router open and add the routes to obtain the JWT. We will create a separate route which will use the middleware and be secured. We will call this one the “secured router” and the routes would be “secured routes”.

// Create a new securedRouter
const router = new Router();
const securedRouter = new Router();

// Add the securedRouter to our app as well
app.use(router.routes()).use(router.allowedMethods());
app.use(securedRouter.routes()).use(securedRouter.allowedMethods());

We modified our existing codes. We now have two routers and we added them both to the app. Let’s now move our old CRUD routes to the secured router and apply the JWT middleware to just the secured router.

// Apply JWT middleware to secured router only
securedRouter.use(jwt.errorHandler()).use(jwt.jwt());

// List all people
securedRouter.get("/people", async (ctx) => {
    ctx.body = await ctx.app.people.find().toArray();
});

// Create new person
securedRouter.post("/people", async (ctx) => {
    ctx.body = await ctx.app.people.insert(ctx.request.body);
});

// Get one
securedRouter.get("/people/:id", async (ctx) => {
    ctx.body = await ctx.app.people.findOne({"_id": ObjectID(ctx.params.id)});
});

// Update one
securedRouter.put("/people/:id", async (ctx) => {
    let documentQuery = {"_id": ObjectID(ctx.params.id)}; // Used to find the document
    let valuesToUpdate = ctx.request.body;
    ctx.body = await ctx.app.people.updateOne(documentQuery, valuesToUpdate);
});

// Delete one
securedRouter.delete("/people/:id", async (ctx) => {
    let documentQuery = {"_id": ObjectID(ctx.params.id)}; // Used to find the document
    ctx.body = await ctx.app.people.deleteOne(documentQuery);
});


We removed the previously setup JWT middleware from the app and used it on securedRouter instead. Remember, the JWT middleware must be setup before we setup the routes themselves. Ordering of middleware matters.

If we try to visit “http://localhost:3000/”, we will no longer get the auth error, rather will see “not found” (we didn’t define any routes for the root url). However, if we try to visit “http://localhost:3000/people”, we will get the authentication error again. Exactly what we wanted.

Issuing JWTs

We now need to create the route to issue JWTs to our users. We will be accepting their login (username and password) and if they’re valid, we will issue them JWTs which they can use to further access our APIs.

The koa-jwt package no longer supports issuing tokens. We have to use the jsonwebtoken package for that instead. Personally, I like to create a helper function in my custom jwt.js module like this:

// Import jsonwebtoken
const jsonwebtoken = require("jsonwebtoken");

// helper function
module.exports.issue =  (payload) => {
    return jsonwebtoken.sign(payload, SECRET);
};

Then we can write a new route on our public router like this:

router.post("/auth", async (ctx) => {
    let username = ctx.request.body.username;
    let password = ctx.request.body.password;

    if (username === "user" && password === "pwd") {
        ctx.body = {
            token: jwt.issue({
                user: "user",
                role: "admin"
            })
        }
    } else {
        ctx.status = 401;
        ctx.body = {error: "Invalid login"}
    }
});

We have hardcoded the username and password here. In production environments, we would store the details in a database and we would hash the password. No one in their right mind should store password in plain text.

In this view, we are accepting a JSON payload and checking the username and password. And then if the details match, we are issuing the token. To test if it’s working, we can make a curl request and checkout the response:

curl -X POST \
  http://localhost:3000/auth \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{"username": "user", "password": "pwd"}'

If it worked, we will get a JSON back with a `token` value containing the JWT.

Using the JWT

We made a request and got the following response:

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoidXNlciIsInJvbGUiOiJhZG1pbiIsImlhdCI6MTUwMjI3MTM0Nn0.GWtjeECIHFQr7vI_MphfUle06Pav_zx4sLmSrd3HE8g"}

That is our token. Now we can start using it in the Authorization header. The format should be like:Authorization: Bearer <Token>¬†. We can make a request to our secured “/people” resource using curl with this header:

curl -X GET \
  http://localhost:3000/people \
  -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoidXNlciIsInJvbGUiOiJhZG1pbiIsImlhdCI6MTUwMjI2OTg4MX0.Ugbh4UwN9tRwhIQEQUHoo-affUf5CAsCztzAXncBYt4' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \

We will now get back the list of people we have stored in our mongodb.

REST API with KoaJS and MongoDB (Part – 2)

In our last post about REST API with KoaJS and MongoDB, we got started with KoaJS and learned to create simple views. We also saw how we can access the query strings and incoming JSON payloads. In this tutorial, we are going to go ahead and implement the RESTful routes and CRUD operations with MongoDB.

In case you’re new to REST API development, you might also want to check out the REST API Concepts¬†and our REST API Tutorials with Flask and Django REST Framework.

Installing MongoDB and NodeJS Driver

I am assuming you have installed MongoDB already on your system. You can install MongoDB using their official installer on Windows or Homebrew on OS X. On Linux systems, you can use the package manager that ships with your distro.

If you don’t have MongoDB installed locally, don’t worry, you can also use a free third party mongo hosting service like mLab.

Once we have MongoDB setup and available, we’re going to install the NodeJS driver for MongoDB next.

npm i -S mongodb

Connecting to MongoDB

We will create a separate file named mongo.js and put the following codes in it:

const MongoClient = require('mongodb').MongoClient;
const MONGO_URL = "mongodb://localhost:27017/polyglot_ninja";


module.exports = function (app) {
    MongoClient.connect(MONGO_URL)
        .then((connection) => {
            app.people = connection.collection("people");
            console.log("Database connection established")
        })
        .catch((err) => console.error(err))

};

Our module exports just one function which takes the app object as the only parameter. Once called, the function connects to our mongodb instance and once connected, sets the people property on our app instance. The people property would actually be a reference to the people collection on our database. So whenever we will have access to the app instance, we will be just using the app.people property to access the collection from within our app. If the connection fails, we will have the error message printed on our terminal.

We have used promises instead of callback. Which makes the code a bit cleaner. Now in our index.js file, we will call the exported function like this:

require("./mongo")(app);

That should import the function and invoke it. Assuming everything worked fine, you should see the message saying database connection established when you run the app next time.

Please Note:¬†We didn’t create the mongodb database or the collection ourselves. MongoDB is smart enough to figure out that we used the names of non existing database / collection and create them for us. If anything with that name exists already, just uses them.

Inserting Records Manually

Before we can start writing our actual code, let’s connect to our mongo database and insert some entries manually so we can play with those data. You can use the command line tool or a mongodb GUI to do so. I will use the command line tool.

$ mongo
MongoDB shell version: 3.2.7
connecting to: test
> use polyglot_ninja
switched to db polyglot_ninja
> db
polyglot_ninja
> db.people.insert({"name": "masnun", "email": "[email protected]"})
WriteResult({ "nInserted" : 1 })
> db.people.find()
{ "_id" : ObjectId("597ef404b5256ba58d26ac53"), "name" : "masnun", "email" : "[email protected]" }
>

I inserted a document with my name and email address in the people collection of the polyglot_ninja db.

Implementing The Routes

Now we will go ahead and implement the routes needed for our REST API.

Please note: Too keep the actual code short, we will skip – validation, error handling and sending proper http status codes. But these are very important in real life and must be dealt with proper care. I repeat – these things are skipped intentionally in this tutorial but should never be skipped in a production app.

GET /people (List All)

This is going to be our root element for the people api. When someone makes a GET¬†request to /people, we should send them a list of documents we have. Let’s do that.

// List all people
router.get("/people", async (ctx) => {
    ctx.body = await ctx.app.people.find().toArray();
});

Now if we run our app and visit the url, we shall see the document we created manually listed there.

POST /people (Create New)

Since we already have the body parser middleware installed, we can now easily accept JSON requests. We will assume that the user sends us properly valid data (in real life you must validate and sanitize) and we will directly insert the incoming JSON into our mongo collection.

// Create new person
router.post("/people", async (ctx) => {
    ctx.body = await ctx.app.people.insert(ctx.request.body);
});

You can POST JSON to the /people endpoint to try it.

curl -X POST \
  http://localhost:3000/people \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{"name": "genji", "email": "[email protected]"}'

Now go back to the all people list and see if your new requests are appearing there. If everything worked, they should be there ūüôā

GET /people/:id (Get One)

To query by mongo IDs we can’t just use the string representation of the ID but we need to convert it to an ObjectID object first. So we will import ObjectID¬†in our index.js¬†file first:

const ObjectID = require("mongodb").ObjectID;

The rest of the code will be simple and straightforward:

// Get one
router.get("/people/:id", async (ctx) => {
    ctx.body = await ctx.app.people.findOne({"_id": ObjectID(ctx.params.id)});
});

PUT /people/:id (Update One)

We usually use PUT¬†when we want to replace the entire document. For single field updates, we prefer PATCH. In the following code example, we have used PUT¬†but the code is also valid for a PATCH¬†request since mongo’s updateOne¬†can update as many fields as you wish. It can update just one field or the entire document. So it would work for both PUT and PATCH methods.

Here’s the code:

// Update one
router.put("/people/:id", async (ctx) => {
    let documentQuery = {"_id": ObjectID(ctx.params.id)}; // Used to find the document
    let valuesToUpdate = ctx.request.body;
    ctx.body = await ctx.app.people.updateOne(documentQuery, valuesToUpdate);
});

The updateOne method requires a query as a matching criteria to find the target document. If it finds the document, it will update the fields passed in an object (dictionary) in the second argument.

Delete /people/:id (Delete One)

Deleting one is very simple. The deleteOne method works just like the updateOne method we saw earlier. It takes the query to match a document and deletes it.

// Delete one
router.delete("/people/:id", async (ctx) => {
    let documentQuery = {"_id": ObjectID(ctx.params.id)}; // Used to find the document
    ctx.body = await ctx.app.people.deleteOne(documentQuery);
});

What’s Next?

In this tutorial, we saw how we can implement RESTful routes and use MongoDB CRUD operations. We have finally created a very basic REST APIs. But we didn’t validate or sanitize incoming data. We also didn’t use proper http status codes. Please go through different resources on the internet or our earlier REST API tutorials to learn more about those.

In our future tutorials, we shall be covering authentication, serving static files and file uploads.

REST API with KoaJS and MongoDB (Part – 1)

We have previously written about REST API Development¬†in length including tutorials using the Flask Framework¬†and of course a multi part tutorial series using the Django REST Framework. In this tutorial series, we are going to focus more on JavaScript and see how we can build a very simple REST API using the KoaJS framework for NodeJS. We’re also going to use MongoDB as our database.

Learning JavaScript

Before we can proceed, we should make sure that we’re proficient using JavaScript. We wouldn’t need any extra ordinary JS mastery but we need to have a certain level of command over JavaScript. If you are new to JavaScript and looking for some resources to learn the language, we have you covered. Please do checkout our JavaScript Learning Guide.

NodeJS and KoaJS

NodeJS has made JavaScript popular outside the browser world. There was a time, JS was meant to be used only on the browser side. For anything else, we would be using other languages like Java, Python, PHP etc. But then NodeJS came and proved that JavaScript is an equally viable language. With the very rapid growth of the platform, NodeJS today is being used for many types of programming and software development. From web backend and REST APIs to IoT tools and Desktop applications – NodeJS is literally everywhere.

There are many free and open source frameworks available for NodeJS but Express is perhaps the most popular option. Koa is quite like Express but it’s light weight, flexible and very simple. I personally am a big fan of this framework. At work, I have built multiple services on top of KoaJS and have been very happy with the outcome. This is why I chose Koa.

MongoDB

MongoDB has become very popular as a NoSQL database in the recent times. MongoDB is not like the traditional relational databases we’re so used to. Rather it’s a document oriented database where the documents look just like JSON. If you’re new to MongoDB, don’t worry, they have a brilliant University¬†where you can take your courses free and master the database system. I have personally taken their courses and I would happily recommend the courses. They do cover some of the popular languages on the web. You will be learning to use MongoDB with Python / Java / NodeJS. Once you learn the concepts well, you should be able to transfer the knowledge to any other programming languages / platforms – so no worries if your favourite language is not available yet.

Getting Started

Before we can get started, make sure you have the latest NodeJS and npm installed. We would be using some of the modern JS syntax (ie. async / await), so please make sure you’re on the latest version of NodeJS (at the time of this writing, that would be Node 8). Please also make sure MongoDB is installed as well. On Windows you download the setup files and install them manually. On OS X, you can use a package manager like Homebrew. On Linux distros, you should be able to get them installed using the package manager available on your OS.

Creating The Project

Let’s first create a directory named “koa-example”.

mkdir koa-example
cd koa-example

Now we’re going to initialize the npm project.

npm init

The prompt will ask you a few questions. Once you complete answering them, we’ll be ready to start installing 3rd party dependencies.

Install KoaJS

Now we’re going to install KoaJS in our new project.

npm i -S koa

This should install KoaJS and save it as a project dependency in the package.json file.

A little more about Koa

Please note that Koa is being developed and maintained by the same people who work on Express. Koa is a much light weight version of Express except it follows a different path. Koa uses the modern JS features like async / await to make the code readable and maintainable. Although, in being light weight, Koa had to leave out many of the built in features in Express. Throughout our tutorial(s), we shall be installing third party packages, often to avail the same features which is prevalent in Express and other frameworks. Koa gives us a really minimal core and a very narrow, focused set of functionality. It lets us choose the components we need. That is in a way liberating and very flexible regardless of how inconvenient that might sound at the beginning.

One of the common Koa middleware I always install is koa-router which enables express like convenient routing on Koa.

npm i -S koa-router

Hello World!

Let’s start writing some codes, we shall of course start with the hello world example. We will first see the codes and then go through them.

const Koa = require("koa");
const Router = require("koa-router");

const app = new Koa();
const router = new Router();

router.get("/", async function (ctx) {
    ctx.body = {message: "Hello World!"}
});

app.use(router.routes()).use(router.allowedMethods());

app.listen(3000);

The code is pretty straightforward. We are importing Koa and Router and creating new instances of them. We are then adding a new route / view to the router. Please note the usage of async¬†functions. We mentioned earlier that Koa uses the latest JS features, notably the async¬†and await¬†features. If you are not familiar with the syntax, here’s a very in depth article that covers from the basics of promises, usages of generators and finally async and await. It would be a good read since Koa used to use generators to achieve similar results before async / await landed on Node.

The async function takes a parameter called ctx which is short for context. We can just set a body to the context and it will be rendered as response. If we pass certain types, for example a dictionary / object, it will be sent as JSON format. We can also access the request object using the context.

After the function is defined, we have used the routes middleware and launched the app on port 3000. If we browse the url http://localhost:3000 we should see the response we set.

Query Strings in KoaJS

Now that we have seen a basic hello world example, let’s see how we can access query strings in KoaJS. Query strings are appended to the url in the form – /hello?name=masnun¬†– here the part starting from the question mark is the query string. We separate different parameters using &¬†in the query string part. For example: name=masnun&country=bangladesh.

To access a parameter passed in the query string, we can use the ctx.request.query¬†object to access the parsed querystring as JS object (dictionary). Let’s change our function to access the name¬†from the query string and display a custom greeting.

router.get("/", async function (ctx) {
    let name = ctx.request.query.name || "World";
    ctx.body = {message: `Hello ${name}!`}
});

Now visit – `http://localhost:3000/?name=masnun` and see how it works!

Accepting JSON

We have by now got used to the idea that we need to plug in 3rd party middlewares to achieve common tasks. One of the common thing in building REST APIs would be to accept JSON payloads. And like most other things, we need to use an external package to handle incoming JSON requests. In this case, we will use the koa-bodyparser package.

npm i -S koa-bodyparser

Now let’s modify the previous code to accept a POST request and parse incoming JSON so we can take the value for `name` variable from the JSON data. Here’s our modified code, first take a look and then we will discuss what we did different here:

const Koa = require("koa");
const Router = require("koa-router");
const BodyParser = require("koa-bodyparser");

const app = new Koa();
const router = new Router();

// Use the bodyparser middlware
app.use(BodyParser());

router.post("/", async function (ctx) {
    let name = ctx.request.body.name || "World";
    ctx.body = {message: `Hello ${name}!`}
});

app.use(router.routes()).use(router.allowedMethods());

app.listen(3000);

The highlighted lines have been changed in this version. First we required the body parser package. Next we used it as a middleware. And finally, we accessed the value of¬†name¬†from the ctx.request.body object. Note, for query string it was request.query and now it’s request.body. The body parser parses the body of the request and sets the value there.

We can try our new code using the following request:

curl -X POST \
  http://localhost:3000/ \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{"name": "masnun"}'

(The cURL command line was generated by Postman, feel free to use your favorite GUI tool for testing the APIs)

Logging Requests on Console

Our current app doesn’t list the requests it serves on the console. If we could get a nice log of what requests are being accepted and handled, it would help us with the workflow. We will be using the koa-logger¬†package for just that.

npm i -S [email protected]

Now we can use it in our app:

const logger = require('koa-logger');
app.use(logger());

Now launch the app and make a few requests, you will notice nicely formatted output on your terminal.

Use Nodemon for Auto Reload

During development, it can be a tedious task to restart the app from time to time whenever you make some new changes. Nodemon is one of the tools which offer help with that. Install it globally and then just use nodemon . inside your current working directory. It will monitor file changes and restart the app as needed.

Learn more about Nodemon here –¬†https://nodemon.io/¬†(along with installation instructions and use cases).

Up Next!

In this tutorial, we just got started using KoaJS and Koa Router package to build our views. In the next tutorial we shall resume our journey forward, learning to use MongoDB as our database.