REST API with KoaJS and MongoDB (Part – 1)

We have previously written about REST API Development¬†in length including tutorials using the Flask Framework¬†and of course a multi part tutorial series using the Django REST Framework. In this tutorial series, we are going to focus more on JavaScript and see how we can build a very simple REST API using the KoaJS framework for NodeJS. We’re also going to use MongoDB as our database.

Learning JavaScript

Before we can proceed, we should make sure that we’re proficient using JavaScript. We wouldn’t need any extra ordinary JS mastery but we need to have a certain level of command over JavaScript. If you are new to JavaScript and looking for some resources to learn the language, we have you covered. Please do checkout our JavaScript Learning Guide.

NodeJS and KoaJS

NodeJS has made JavaScript popular outside the browser world. There was a time, JS was meant to be used only on the browser side. For anything else, we would be using other languages like Java, Python, PHP etc. But then NodeJS came and proved that JavaScript is an equally viable language. With the very rapid growth of the platform, NodeJS today is being used for many types of programming and software development. From web backend and REST APIs to IoT tools and Desktop applications – NodeJS is literally everywhere.

There are many free and open source frameworks available for NodeJS but Express is perhaps the most popular option. Koa is quite like Express but it’s light weight, flexible and very simple. I personally am a big fan of this framework. At work, I have built multiple services on top of KoaJS and have been very happy with the outcome. This is why I chose Koa.

MongoDB

MongoDB has become very popular as a NoSQL database in the recent times. MongoDB is not like the traditional relational databases we’re so used to. Rather it’s a document oriented database where the documents look just like JSON. If you’re new to MongoDB, don’t worry, they have a brilliant University¬†where you can take your courses free and master the database system. I have personally taken their courses and I would happily recommend the courses. They do cover some of the popular languages on the web. You will be learning to use MongoDB with Python / Java / NodeJS. Once you learn the concepts well, you should be able to transfer the knowledge to any other programming languages / platforms – so no worries if your favourite language is not available yet.

Getting Started

Before we can get started, make sure you have the latest NodeJS and npm installed. We would be using some of the modern JS syntax (ie. async / await), so please make sure you’re on the latest version of NodeJS (at the time of this writing, that would be Node 8). Please also make sure MongoDB is installed as well. On Windows you download the setup files and install them manually. On OS X, you can use a package manager like Homebrew. On Linux distros, you should be able to get them installed using the package manager available on your OS.

Creating The Project

Let’s first create a directory named “koa-example”.

mkdir koa-example
cd koa-example

Now we’re going to initialize the npm project.

npm init

The prompt will ask you a few questions. Once you complete answering them, we’ll be ready to start installing 3rd party dependencies.

Install KoaJS

Now we’re going to install KoaJS in our new project.

npm i -S koa

This should install KoaJS and save it as a project dependency in the package.json file.

A little more about Koa

Please note that Koa is being developed and maintained by the same people who work on Express. Koa is a much light weight version of Express except it follows a different path. Koa uses the modern JS features like async / await to make the code readable and maintainable. Although, in being light weight, Koa had to leave out many of the built in features in Express. Throughout our tutorial(s), we shall be installing third party packages, often to avail the same features which is prevalent in Express and other frameworks. Koa gives us a really minimal core and a very narrow, focused set of functionality. It lets us choose the components we need. That is in a way liberating and very flexible regardless of how inconvenient that might sound at the beginning.

One of the common Koa middleware I always install is koa-router which enables express like convenient routing on Koa.

npm i -S koa-router

Hello World!

Let’s start writing some codes, we shall of course start with the hello world example. We will first see the codes and then go through them.

const Koa = require("koa");
const Router = require("koa-router");

const app = new Koa();
const router = new Router();

router.get("/", async function (ctx) {
    ctx.body = {message: "Hello World!"}
});

app.use(router.routes()).use(router.allowedMethods());

app.listen(3000);

The code is pretty straightforward. We are importing Koa and Router and creating new instances of them. We are then adding a new route / view to the router. Please note the usage of async¬†functions. We mentioned earlier that Koa uses the latest JS features, notably the async¬†and await¬†features. If you are not familiar with the syntax, here’s a very in depth article that covers from the basics of promises, usages of generators and finally async and await. It would be a good read since Koa used to use generators to achieve similar results before async / await landed on Node.

The async function takes a parameter called ctx which is short for context. We can just set a body to the context and it will be rendered as response. If we pass certain types, for example a dictionary / object, it will be sent as JSON format. We can also access the request object using the context.

After the function is defined, we have used the routes middleware and launched the app on port 3000. If we browse the url http://localhost:3000 we should see the response we set.

Query Strings in KoaJS

Now that we have seen a basic hello world example, let’s see how we can access query strings in KoaJS. Query strings are appended to the url in the form – /hello?name=masnun¬†– here the part starting from the question mark is the query string. We separate different parameters using &¬†in the query string part. For example: name=masnun&country=bangladesh.

To access a parameter passed in the query string, we can use the ctx.request.query¬†object to access the parsed querystring as JS object (dictionary). Let’s change our function to access the name¬†from the query string and display a custom greeting.

router.get("/", async function (ctx) {
    let name = ctx.request.query.name || "World";
    ctx.body = {message: `Hello ${name}!`}
});

Now visit – `http://localhost:3000/?name=masnun` and see how it works!

Accepting JSON

We have by now got used to the idea that we need to plug in 3rd party middlewares to achieve common tasks. One of the common thing in building REST APIs would be to accept JSON payloads. And like most other things, we need to use an external package to handle incoming JSON requests. In this case, we will use the koa-bodyparser package.

npm i -S koa-bodyparser

Now let’s modify the previous code to accept a POST request and parse incoming JSON so we can take the value for `name` variable from the JSON data. Here’s our modified code, first take a look and then we will discuss what we did different here:

const Koa = require("koa");
const Router = require("koa-router");
const BodyParser = require("koa-bodyparser");

const app = new Koa();
const router = new Router();

// Use the bodyparser middlware
app.use(BodyParser());

router.post("/", async function (ctx) {
    let name = ctx.request.body.name || "World";
    ctx.body = {message: `Hello ${name}!`}
});

app.use(router.routes()).use(router.allowedMethods());

app.listen(3000);

The highlighted lines have been changed in this version. First we required the body parser package. Next we used it as a middleware. And finally, we accessed the value of¬†name¬†from the ctx.request.body object. Note, for query string it was request.query and now it’s request.body. The body parser parses the body of the request and sets the value there.

We can try our new code using the following request:

curl -X POST \
  http://localhost:3000/ \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{"name": "masnun"}'

(The cURL command line was generated by Postman, feel free to use your favorite GUI tool for testing the APIs)

Logging Requests on Console

Our current app doesn’t list the requests it serves on the console. If we could get a nice log of what requests are being accepted and handled, it would help us with the workflow. We will be using the koa-logger¬†package for just that.

npm i -S [email protected]

Now we can use it in our app:

const logger = require('koa-logger');
app.use(logger());

Now launch the app and make a few requests, you will notice nicely formatted output on your terminal.

Use Nodemon for Auto Reload

During development, it can be a tedious task to restart the app from time to time whenever you make some new changes. Nodemon is one of the tools which offer help with that. Install it globally and then just use nodemon . inside your current working directory. It will monitor file changes and restart the app as needed.

Learn more about Nodemon here –¬†https://nodemon.io/¬†(along with installation instructions and use cases).

Up Next!

In this tutorial, we just got started using KoaJS and Koa Router package to build our views. In the next tutorial we shall resume our journey forward, learning to use MongoDB as our database.

Learning JavaScript

You really don’t have to check where JavaScript stands in the TIOBE index, it is undoubtedly the most popular programming language in today’s world. JavaScript started big in the browser land but with the inception and rise of NodeJS, it has come out of it’s browser context and now being used in all sorts of tools and toys. From modern progressive web apps to desktop apps, from command line tools to IoT – JavaScript is literally everywhere. The language has it’s quirks, it’s not perfect and it gets a lot hatred from time to time. Despite the backlash, JavaScript continues to grow and get more and more mind share. Specially if you’re interested in web development, sooner or later you would need some level of skills in the language. So in this blog post, I will go ahead and list some of the resources that can help beginners in the JS world.

Learning The Language

JavaScript, the language can be used in the browser or on the server side through a framework like NodeJS. You can also use JS in different other places for scripting purposes. Depending on where we’re using the language, the libraries or the APIs can vary. But the language – it remains more or less the same. So before you want to start learning the next modern front end framework or the latest MVC framework in the Node land, you probably want to learn the language first. You should learn basics very well and have a solid grasp of the language before you choose your platform / stack.

  • Speaking JavaScript¬†is an excellent book to get started with ES5. The same author brings you Exploring ES6.¬†¬†And that’s not all, he also has another book – Exploring ES2016 and ES2017. You can probably already guess – the author, Dr. Axel Rauschmayer is pretty awesome. Reading these books will definitely get your basics strong. You can read these books free online.
  • Eloquent Javascript¬†is also free online¬†and should help you learn both the language and how it works in the browser.
  • JavaScript: The Good Parts is another excellent book. The book is not very big in terms of volume but it covers the language pretty well.
  • You Don’t Know JS¬†is the title of a series of books written by Kyle Simpson. You can read the books free online. This series is pretty awesome!
  • JavaScript Allong√©, the “Six” Edition – can also be read free online and packs lots of excellent content.

You may consult multiple books to understand the same topic. But it would be a good idea to start one book and stick to it.

Practising JavaScript

When reading through the concepts, please do try out the codes yourself. Just reading through won’t help much if you don’t practise. You can practise JavaScript in these places:

  • HackerRank – solve programming problems with JS here.
  • You can also train JS at CodeWars.
  • How about learning JS while playing some games? CodeCombat has you covered!
  • You can solve problems at Project Euler too!

What’s Next?

Once you have learned the language, you can choose to work on the front end or the backend or you can choose to be a full stack JS developer! The choice is yours. Do try both and see which ones you like. Having a solid understanding of the language will make it easier for you to pick up any stack / framework. So put extra focus on learning the concepts well and taking your JS skills to intermediate level!

Golang: Working with JSON

We previously wrote an introductory blog post on Golang describing why and how the language is gaining traction quickly. Go aka Golang is being used by many large scale systems for various purposes. One thing common in most complex systems is that we have to communicate with other systems / services. JSON is a very popular data interchange format for this kind of scenarios. In fact, it is so popular that you may even go ahead and call it the de-facto data interchange format on the internet.  If you have followed any of our REST API tutorials (Django / Flask), you might have also noticed they all output JSON. In this blog post, we will see the different ways we can work with JSON in Golang.

Creating JSON

We can use the encoding/json¬†package to easily create JSON from our Go data structures. There’s a few things to consider while creating JSON in Go. One of them is the choice of data structure.

From Structs

Structs are very convenient to use and often come as cheaper compared to map or other complex structures. We can pass any instance of a struct to the json.Marshal function and get back the JSON as a slice of bytes ([]byte).

package main

import (
	"encoding/json"
	"fmt"
)

type Person struct {
	Name   string
	Age    int
	Emails []string
}

func main() {
	masnun := Person{
		Name:   "Abu Ashraf Masnun",
		Age:    27,
		Emails: []string{"[email protected]", "[email protected]"},
	}

	json_bytes, _ := json.Marshal(masnun)
	fmt.Printf("%s",json_bytes)
}

Output should be:

{"Name":"Abu Ashraf Masnun","Age":27,"Emails":["[email protected]","[email protected]"]}

Things to note here:

  • The field name has to start with a capital letter – this is important and will be discussed soon
  • You can nest other structures, in this example our Emails¬†field contain a list of strings
  • The json.Marshal¬†function returns the JSON and error, don’t forget to handle the error. You may not often get errors from the Marshal¬†function but in some cases where some types can not be converted into JSON, you will get errors. So look out for it.
  • The returned JSON is in the form of bytes list. So if you want to use it as string, you will need to convert it.

Choosing Field Names

If you have seen the output, the keys/fields in the created JSON structure all start with a capital letter (our struct fields were similar, so this is no surprise). If you have a curious and adventurous mind, you might have tried to go ahead and convert the struct fields into lower caps. But that doesn’t work, does it? Check it out –¬†https://play.golang.org/p/93eDoFSjnW¬†– because if a field / member name does not start with a capital letter, it is not exported. External code can not access it. This is why our name¬†and age¬†fields are not part of the generated JSON structure.

But don’t worry, there’s a simple way of “tagging” our struct fields so we can describe how to marshal our structs. Let’s modify our codes to look like this:

type Person struct {
	Name   string   `json:"name"`
	Age    int      `json:"age"`
	Emails []string `json:"emails"`
}

Next to each field, we have provided tags to describe how this field should be marshalled or unmarshalled. Now we can see you expected results here:¬†https://play.golang.org/p/xlcjU1_VSE¬†ūüôā

If you don’t understand the tags, don’t worry, try reading this answer on StackOverflow –¬†https://stackoverflow.com/questions/10858787/what-are-the-uses-for-tags-in-go. We will understand the concepts more when we use structs for various purposes (for example mapping to database tables, but for now, let’s not worry).

Omitting Empty Fields

What if some of the fields are empty? Let’s try that.

func main() {
	masnun := Person{}

	json_bytes, _ := json.Marshal(masnun)
	fmt.Printf("%s", json_bytes)
}

The output would be:

{"name":"","age":0,"emails":null}

If a field is empty, we might want to omit that from our JSON. We can do so by using the omitempty flag in our json field tags.

package main

import (
	"encoding/json"
	"fmt"
)

type Person struct {
	Name   string      `json:"name,omitempty"`
	Age    int         `json:"age,omitempty"`
	Emails []string    `json:"emails,omitempty"`
}

func main() {
	masnun := Person{
		Name: "Abu Ashraf Masnun",
	}

	json_bytes, _ := json.Marshal(masnun)
	fmt.Printf("%s", json_bytes)
}

Now if we check the output again:

{"name":"Abu Ashraf Masnun"}

Nice, no?

Skipping Fields

Let’s say in our struct, we need to keep the Age field. But we don’t want it to be a part of the produced JSON. We can use the -¬†flag in the json tag for that particular field.

package main

import (
	"encoding/json"
	"fmt"
)

type Person struct {
	Name   string      `json:"name,omitempty"`
	Age    int         `json:"-"`
	Emails []string    `json:"emails,omitempty"`
}

func main() {
	masnun := Person{
		Name: "Abu Ashraf Masnun",
		Age:  27,
	}

	json_bytes, _ := json.Marshal(masnun)
	fmt.Printf("%s", json_bytes)
}

The output would be:

{"name":"Abu Ashraf Masnun"}

Even though the struct had the Age field set, it wasn’t included in the output. This comes very handy in some cases, for example when a¬†User¬†struct has a Password¬†field that we don’t want to serialize into JSON.

Using Maps and Slices

So far we have used structs. We can also use maps and slices instead. Here’s a quick code example:

package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	masnun := map[string]interface{}{

		"name": "Abu Asharf Masnun",
		"age":  27,
	}

	json_bytes, _ := json.Marshal(masnun)
	fmt.Printf("%s", json_bytes)
}

And using slices:

package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	emails := []string{"[email protected]", "[email protected]"}
	json_bytes, _ := json.Marshal(emails)
	fmt.Printf("%s", json_bytes)
}

They both work as expected. But in most codebases I have come across, structs are more widely used.

 

Parsing JSON

We have so far seen how we can generate JSON from our go data. Now we will see the opposite. We will be parsing JSON into Go data structures.

Into Structs

We will first see how we can parse JSON data into structs. It’s quite similar to what we did earlier. We will be using the Unmarshal¬†function which takes bytes and pointer to any interface{}¬†type. It reads through the JSON and stores the data in the struct we pass as the second parameter. Let’s see an example:

package main

import (
	"encoding/json"
	"fmt"
)

type Person struct {
	Name   string   `json:"name"`
	Age    int      `json:"age"`
	Emails []string `json:"emails"`
}

func main() {

	json_bytes := []byte(`
		{
			"Name":"Abu Ashraf Masnun",
			"Age":27,
			"Emails":["[email protected]","[email protected]"]
		}
	`)

	masnun := Person{}
	err := json.Unmarshal(json_bytes, &masnun)
	if err != nil {
		panic(err)
	}

	fmt.Println(masnun.Name, masnun.Age, masnun.Emails)

}

Here json_bytes hold the JSON we want to process. We already have a Person type with tagged fields. We just need to pass this json_bytes and a pointer to an instance of Person to the Unmarshal function. Please note the pointer is important. We have to pass a pointer otherwise the parser would not be able to write to the struct.

If the struct doesn’t have some fields which are present in the JSON, those will be silently ignored. In the same way, if the struct has fields which are not available in the JSON, they will be ignored too.

package main

import (
	"encoding/json"
	"fmt"
)

type Person struct {
	Name    string   `json:"name"`
	Age     int      `json:"age"`
	Emails  []string `json:"emails"`
	Address string
}

func main() {

	json_bytes := []byte(`
		{
			"Name":"Abu Ashraf Masnun",
			"Age":27,
			"Emails":["[email protected]","[email protected]"],
			"Score":97
		}
	`)

	var masnun Person
	err := json.Unmarshal(json_bytes, &masnun)
	if err != nil {
		panic(err)
	}

	fmt.Println(masnun.Address)

}

In the above example, the struct has a field named Address¬†which the JSON doesn’t provide. On the other hand, the JSON has the Score¬†key which the struct knows nothing about. In this case, masnun.Address¬†will be empty string.

Into Maps / Slices

We have previously mentioned how structs are cheaper and more widely used than maps. But there’s these use cases where we can not be certain about the structure of the JSON data we want to parse. In such cases, maps can be very useful. Let’s see:

package main

import (
	"encoding/json"
	"fmt"
)

func main() {

	json_bytes := []byte(`
		{
			"Name":"Abu Ashraf Masnun",
			"Age":27,
			"Emails":["[email protected]","[email protected]"],
			"Score":97
		}
	`)

	var masnun map[string]interface{}
	err := json.Unmarshal(json_bytes, &masnun)
	if err != nil {
		panic(err)
	}

	fmt.Println(masnun["Name"], masnun["Age"], masnun["Emails"], masnun["Score"])

}

See? We have passed map[string]interface{} and received all the data in JSON. But please remember, the values to each key in the map will be of type interface{}. If we want to extract part of the data, for example, one of the emails and then use it as a string, I will have to manually convert it to a string.

For example this code will fail:

package main

import (
	"encoding/json"
	"fmt"
	"strings"
)

func main() {

	json_bytes := []byte(`
		{
			"Name":"Abu Ashraf Masnun",
			"Age":27,
			"Emails":["[email protected]","[email protected]"],
			"Score":97
		}
	`)

	var masnun map[string]interface{}
	err := json.Unmarshal(json_bytes, &masnun)
	if err != nil {
		panic(err)
	}

	var firstEmail string
	firstEmail = masnun["Emails"][0]

	fmt.Println(strings.ToUpper(firstEmail))

}

We will get an error:

# command-line-arguments
./main.go:27: invalid operation: masnun["Emails"][0] (type interface {} does not support indexing)

That’s what I was trying to explain just above. The Emails¬†key has a value of interface{}¬†type. Let’s cast it to a list of interface{} first. Then we can take an element (which will be again interface{}¬†type). We further cast it to a string.

package main

import (
	"encoding/json"
	"fmt"
	"strings"
)

func main() {

	json_bytes := []byte(`
		{
			"Name":"Abu Ashraf Masnun",
			"Age":27,
			"Emails":["[email protected]","[email protected]"],
			"Score":97
		}
	`)

	var masnun map[string]interface{}
	err := json.Unmarshal(json_bytes, &masnun)
	if err != nil {
		panic(err)
	}

	var emails []interface{}
	var firstEmail string
	emails = masnun["Emails"].([]interface{})
	firstEmail = emails[0].(string)

	fmt.Println(strings.ToUpper(firstEmail))

}

You may be wondering why couldn’t we just get¬†Emails¬†as¬†[]string? Well, Go doesn’t know the types of the values in our JSON. So it uses interface{}¬†. That is when it stores Emails, it stores it as a list of unknown types or a list of interface{}. That is why we first need to get it as a list and then we can take individual items and further convert them to the type we want.

Now it works fine ūüôā

Streaming JSON Encoding and Decoding

The json package offers NewEncoder and NewDecoder functions which would get us Encoder and Decoder types. These types can work with other objects that support io.Reader and io.Writer interfaces to offer streaming support.

Streaming JSON from a File

We can open a JSON file using the os.Open¬†function and stream it using the json.NewDecoder¬†function. Here’s a quick example:

package main

import (
	"encoding/json"
	"fmt"
	"os"
)

func main() {

	fileReader, _ := os.Open("masnun.json")
	var masnun map[string]interface{}
	json.NewDecoder(fileReader).Decode(&masnun)
	fmt.Println(masnun)

}

We opened a file which implements the io.Reader¬†interface. So we can use it with our Decoder¬†type. We created a decoder with the file reader and then called the Decode¬†method on it. That’s all we needed ūüôā

Streaming JSON into a File

Writing JSON is also very similar. We need to open a file in write mode, grab the io.Writer and pass it to json.NewEncoder Рthen we can pass our data to the Encode method to stream the json into the file.

package main

import (
	"encoding/json"
	"os"
)

type Person struct {
	Name   string
	Age    int
	Emails []string
}

func main() {
	masnun := Person{
		Name:   "Abu Ashraf Masnun",
		Age:    27,
		Emails: []string{"[email protected]", "[email protected]"},
	}

	fileWriter, _ := os.Create("output.json")
	json.NewEncoder(fileWriter).Encode(masnun)
}

 

Custom Marshal / Unmarshal

If we want to change how our own types are marshalled or unmarshalled, we can implement the json.Marshaler¬†and json.Unmarshaler¬†interfaces. It’s actually simple. We need to define the MarshalJSON¬†and UnmarshalJSON¬†methods on our structs and we’re done. Here’s an example from the official documentation:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type Animal int

const (
	Unknown Animal = iota
	Gopher
	Zebra
)

func (a *Animal) UnmarshalJSON(b []byte) error {
	var s string
	if err := json.Unmarshal(b, &s); err != nil {
		return err
	}
	switch strings.ToLower(s) {
	default:
		*a = Unknown
	case "gopher":
		*a = Gopher
	case "zebra":
		*a = Zebra
	}

	return nil
}

func (a Animal) MarshalJSON() ([]byte, error) {
	var s string
	switch a {
	default:
		s = "unknown"
	case Gopher:
		s = "gopher"
	case Zebra:
		s = "zebra"
	}

	return json.Marshal(s)
}

func main() {
	blob := `["gopher","armadillo","zebra","unknown","gopher","bee","gopher","zebra"]`
	var zoo []Animal
	if err := json.Unmarshal([]byte(blob), &zoo); err != nil {
		log.Fatal(err)
	}

	census := make(map[Animal]int)
	for _, animal := range zoo {
		census[animal] += 1
	}

	fmt.Printf("Zoo Census:\n* Gophers: %d\n* Zebras:  %d\n* Unknown: %d\n",
		census[Gopher], census[Zebra], census[Unknown])

}

Pretty nice, no?

Python: Iterators

If you have written some Python code and used the for loop, you have already used iterators behind the scene but you probably didn’t know about it. Iterators are objects that we can iterate over one by one.¬†They are practically everywhere in a Python codebase. Understanding the concepts of iterators and how they work can help us write better, more efficient code from time to time. In this post, we will discuss iterators and other related concepts.

How does iteration work?

Before we can dive into iterators, we first need to understand how iteration works in Python. When we do the for loop, how does Python fetch one item at a time? How does this process work?

There are two functions that come into play – iter¬†and next. The iter¬†function gets an iterator from an object. It actually calls the __iter__¬†special method on the object to get the iterator. So if an object wants to allow iteration, it has to implement the __iter__¬†method. Once it gets the iterator object, it continues to call next¬†on the iterator. The next function in turn calls the __next__¬†method on the iterator object. Let’s see a quick example:

>>> l = [1, 2, 3]

>>> i = iter(l)

>>> type(l)
<class 'list'>

>>> type(i)
<class 'list_iterator'>

>>> next(i)
1

>>> next(i)
2

>>> next(i)
3

>>> next(i)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


>>>

Let’s see. We first create a list named l¬†with 3 elements. We then call iter()¬†on it. The type of l¬†is list¬†but look at the type of i¬†– it’s list_iterator¬†– interesting! Now we keep calling next¬†on i¬†and it keeps giving us the values we saw in the list, one by one, until there’s a StopIteration¬†exception.

Here the list is an iterable because we can get an iterator from it to iterate over the list. The list_iterator¬†object we got is an iterator, it’s an object that we can actually iterate over. When we loop over a list, this is what happens:

l = [1, 2, 3]

iterator = iter(l)

while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        break

Makes sens? The for loop actually gets the iterator and keeps looping over until a StopIteration exception is encountered.

Iterator

The iterator is an object which implements __next__¬†method so we can call next¬†on it repeatedly to get the items. Let’s write an iterator that keeps us giving us the next integer, without ever stopping. Let’s name it InfiniteIterator.

class InfiniteIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        self.__int += 1
        return self.__int

If we keep calling next on it, we will keep getting the integers, starting from one.

>>> inf_iter = InfiniteIterator()
>>> next(inf_iter)
1
>>> next(inf_iter)
2
>>> next(inf_iter)
3
>>> next(inf_iter)
4
>>>

Iterable

What if we wanted to create an InfiniteNumbers iterable? It would be such that when we use the for loop on it, it never stops. It keeps producing the next integer in each loop. What would we do? Well, we have an InfiniteIterator. All we need is to define an __iter__ method that returns a new instance of InfiniteIterator.

class InfiniteNumbers:
    def __iter__(self):
        return InfiniteIterator()


infinite_numbers = InfiniteNumbers()

for x in infinite_numbers:
    print(x)

    if x > 99:
        break

If you remove the break statement and the if block, you will notice, it keeps running Рlike forever.

Using StopIteration

Instead of breaking out from our code ourselves, we could use the StopIteration exception in our iterator so it stops after giving us the 100 numbers.

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


class InfiniteNumbers:
    def __iter__(self):
        return HundredIterator()


one_hundred = InfiniteNumbers()

for x in one_hundred:
    print(x)

Iterators must also implement __iter__

We saw that the __next__¬†method does it’s work just fine. But we also need to implement the __iter__¬†method on an iterator (just like we did in iterable). Why is this required? Let me quote from the official docs:

Iterators are required to have an__iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted.

If we tried to use the for loop over our iterator, it would fail:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

We will get the following exception:

Traceback (most recent call last):
  File "iter.py", line 15, in <module>
    for x in one_hundred:
TypeError: 'HundredIterator' object is not iterable

That kind of makes sense because we saw that the for loop runs the iter¬†function on an object to get an iterator from it. Then calls next¬†on the iterator. That’s the problem, we don’t have an __iter__¬†method. The official documentation suggests that every iterator should be a proper iterable too. That is, it should implement the __iter__¬†method and just return an instance of itself. Let’s do that:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

Now the code works fine ūüôā

The Iterator Protocol

The iterator protocol defines the special methods that an object must implement to allow iteration. We can summarize the protocol in this way:

  • Any object that can be iterated over needs to implement the __iter__¬†method which should return an iterator object. Any object that returns an iterator is an iterable.
  • An iterator must implement the __next__¬†method which returns the next item when called. When all items are exhausted (read retrieved), it must raise the StopIteration¬†exception.
  • An iterator must also implement the __iter__¬†method to behave like an iterable.

Why do we need Iterables?

In our last example, we saw that it’s possible for an object to implement a __next__¬†method and an __iter__¬†method that returns self. In this way, an iterator behaves just like an iterable alright. Then why do we need Iterables? Why can’t we just keep using Iterators which refer to itself?

Let’s get back to our HundredIterator¬†example. Once you have iterated over the items once, try to iterate again. What happens? No numbers are output on the screen. Why? Well, because the iterator objects store “state”. Once it has reached StopIteration, it has reached the end line. It’s now exhausted.¬†Every time you call iter¬†on it, it returns the same instace (self) which has nothing more to output.

This is why Iterables are useful. You can just return a fresh instance of an iterator every time the iterable is looped over. This is actually what many built in types like list does.

Why is Iterators so important?

Iterators allow us to consume data each item at a time. Just imagine, if there’s a one GB file and we tried to load it all in memory, it would require huge memory. But what if we implemented an iterator that reads the file one line at a time? We could then just store that one line in memory and do necessary processing before moving on to the next item. This allow us to write really efficient programs ūüôā

This all seems very confusing

If you find the concepts very confusing and hard to grasp, don’t worry. Give it a few tries, write the codes by hand and see the output. Tinker with the examples. Inspect the code, try to see what happens when you modify part of it. All things become easier when you practise more and more. Try writing your own iterables and iterators – perhaps try to clone the built in containers’ functionalities? May be write your own list implementation? Don’t worry, it will come to you in time.

 

Django REST Framework: JSON Web Tokens (JWT)

(This post is a part of a tutorial series on Building REST APIs in Django)

Our last post was about Authentication and Permissions and we covered the available methods of authentication in Django REST Framework. In that post, we learned how to use the built in Token based authentication in DRF. In this post, we will learn more about JSON Web Tokens aka JWT and we will see if JWT can be a better authentication mechanism for securing our REST APIs.

Understanding JSON Web Tokens (JWTs)

We have actually written a detailed blog post about JSON Web Tokens earlier. In case you have missed it, you probably should read it first. We have also described how to use JWT with Flask – reading that one might also help better understand how things work. And of course, we will briefly cover the idea of JWT in this post as well.

If we want to put it simply – you take some data in JSON format, you hash it with a secret and you get a string that you use as a token. You (your web app actually) pass this token to the user when s/he logs in. The user takes the token and on subsequent requests, passes it back in the “Authorization” header. The web app now takes this token back, “decodes” it back to the original JSON payload. It can now read the stored data (identity of the user, token expiry and other data which was embedded in the JSON). While decoding, the same secret is used, so third party attackers can’t just forge a JWT. We would want our token to be small in size, so the JSON payload is usually intentionally kept small. And of course, it should not contain any sensitive information like user password.

JWT vs DRF’s Token Based Authentication

So in our last blog post, we saw Django REST Framework includes a token based authentication system which can generate a token for the user. That works fine, right? Why would we want to switch to JSON Web Tokens instead of that?

Let’s first see how DRF generates the tokens:

    def generate_key(self):
        return binascii.hexlify(os.urandom(20)).decode()

It’s just¬†random. The token generated can not be anyway related to the user that it belongs to. So how does it associate a token with an user? It stores the token and a reference to the user in a table in database. Here comes the first point – while using DRF’s token based auth, we need to query database on every request (unless of course we have cached that token which). But what if we have multiple application servers?¬†Now we need all our application servers to connect to the same database or same cache server. How will that scale when the project gets really really big? What if we want to provide single sign on across multiple services? We will need to maintain a central auth service where other services request to verify a token. Can JWT simplify these¬†for us?

JWT is just an encoded (read – hashed / signed) JSON data. As long as any webservice has access to the secret used in signing the data, it can also decode and read the embedded data. It doesn’t need any database calls. You can generate the token from one service and other services can read and verify it just fine. It’s more efficient and simply scales better.

JWT in Django REST Framework

DRF does not directly support JWTs out of the box. But there’s an excellent package that adds support for it. Let’s see how easily we can integrate JWT in our REST APIs.

Install and Configure

Let’s first install the package using pip¬†–

pip install djangorestframework-jwt

That should install the package. Now we need to add rest_framework_jwt.authentication.JSONWebTokenAuthentication to the default authentication classes in REST Framework settings.

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    )
}

We added it to the top of the list. Next, we just have to add it’s built in view to our urlpatterns.

from rest_framework_jwt.views import obtain_jwt_token

urlpatterns = router.urls + [
    url(r'^jwt-auth/', obtain_jwt_token),
]
Obtain a Token

The obtain_jwt_token¬†view will check the user credentials and provide a JWT if everything goes alright. Let’s try it.

$ curl --request POST \
  --url http://localhost:8000/api/jwt-auth/ \
  --header 'content-type: application/json' \
  --data '{"username": "test_user", "password": "awesomepwd"}'

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0"}

Awesome, everything worked just fine. We have got our token too. What do we do next? We use this token to access a secured resource.

Using the obtained JWT

We need to pass the token in the form of JWT <token>¬†as the value of the Authorization¬†header. Here’s a sample curl request:

$ curl -H "Content-Type: application/json" -H "Authorization: JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0" -X GET  http://localhost:8000/api/subscribers/

[{"id":1,"name":"Abu Ashraf Masnun","age":29,"email":"[email protected]"},{"id":2,"name":"Abu Ashraf Masnun","age":29,"email":"[email protected]"},{"id":3,"name":"Abu Ashraf Masnun","age":29,"email":"[email protected]"},{"id":4,"name":"Abu Ashraf Masnun","age":29,"email":"[email protected]"}]

So our token worked fine! Cool!

Where to go next?

Now that you have seen how simple and easy it is to add JSON Web Token based authentication to Django REST Framework, you probably should dive deeper into the package documentation. Specially these topics might be interesting –

  • Refresh Tokens: If you enable JWT token refreshing, you can exchange your current token with a new, fresh one before¬†the existing one expires. The new token will of course have a renewed expiry time set.
  • Verify Token: If you just share the secret, all services can verify the user on their own. However, in modern micro service based architecture, you may want to provide an API end point that other services can use to verify a JWT they received from the user. This can be useful for those scenarios.
  • And of course look at the settings options available¬†and see how you can customize the token generation process.

In the future, we shall try to cover more about Django, Django REST Framework and Python in general. If you liked the content, please subscribe to the mailing list so we can notify you when we post new contents.

Kotlin: An Introduction

In Google I/O 2017, Kotlin was announced as an officially supported language for android development. The news created a massive buzz among the developers community. Google is a big name and so is Android. So that level of attention was only¬†natural. But many of those who have followed the language since¬†it’s inception, didn’t need to wait this long to fall in love with the language. Kotlin is a brilliant language. It’s statically typed yet quite productive. The syntax does not feel rigid or restrictive, rather it feels rather expressive and enjoyable. Writing code in Kotlin is really fun. You don’t have to trust my word for that,¬†just give it a try.

History of Kotlin

In case you didn’t know, Kotlin is named after the Kotlin Island near Saint Petersburg, Russia. You can read more about the Island in Wikipedia. So another programming language named after an Island, but why? Well, Kotlin is developed by those awesome guys at Jetbrains¬†¬†who build our favorite IDEs. Apparently Kotlin was developed in their Saint Petersburg office and the developers named it after the Island.

Jetbrains was using Java to build their IDEs but why did they decide to create a language of their own? The answer is probably obvious to you by now. Every new language is created out of frustration from the existing ones. We already saw a similar story in our post about Golang. The same thing happened here too. Jetbrains lead Dmitry Jemerov mentioned they were looking for features not offered by Java or most other languages. They liked Scala but the compile time was a bugger. So they came up with Kotlin and thanks to them that they did!

Work on Kotlin started in 2010 and Jetbrains announced the language publicly in July 2011. They made the project open source under the Apache 2 License in 2012. They continued developing the language by accepting and reviewing user’s feedback. After a lot of improvements and iteration, the language reached version 1.0 in February 2016, the first officially stable release with commitment towards backward compatibility. That was just a year go. Kotlin became quickly popular and developers seemed to love it. Many large companies started adopting it even before it became officially supported on Android. But the official support didn’t take long either, in Google I/O 2017, first class support came for Kotlin on Android.

Write once, Run Everywhere

Do you recognize the words – “Write once, run anywhere / everywhere”? If you’re a Java developer, you might recognize them. Kotlin takes things a little bit further than that. Kotlin started off as a language based on the JVM. You could run Kotlin anywhere you could run Java. But that’s not all, Kotlin code now can be transpiled into Javascript too! You already know it works on Android. The guys are also working on a native version based on LLVM. Once that happens, you can truly write code once and run them anywhere and everywhere. Of course you can’t fully port all codes from one platform to another, for example if you’re using JavaFX on the JVM, those will not work on the browser. But the common business logic can be ported without much¬†efforts in most cases.

Kotlin is fully interoperable with Java and can take advantage of the JVM. So the plethora of existing Java libraries and packages are at your disposal. This opens up enormous possibilities and a large eco system to the enterprising Kotlin developer.

Most Loved Kotlin Features

So what makes Kotlin great? Why do people love it? Here are some of the features people love in the language:

  • Full interoperability with Java (and the JVM)
  • Multi platform – JVM, Android, Browser and Native (coming soon)
  • The syntax is quite similar to other popular programming languages. Expressive and productive.
  • Type inference is another cool feature. You can just write val number = 23¬†and it an identify that it’s an Int. You can of course declare the type if you wish ūüôā
  • The when¬†block can cut the need of complex if/else block or switch statements and makes things simpler.
  • Libraries like Anko¬†not only makes android development faster, it also makes if more enjoyable. Also the android extensions for Kotlin makes many pain points easier to handle.
  • Functions are fun, literally ūüėČ fun meaningOfLife() = 42¬†– see? fun¬†– it is! ūüėÄ
  • Simple one line functions paired with string interpolation is pretty useful¬†– fun fullName() = "${first} ${last}"
  • The idea of data class is to provide easy POJOs in a very short, concise manner.
  • Operator overloading is quite simple.
  • You can add extension functions to classes, even built in ones, easily extending their functionality.
  • Kotlin provides null safety by forcing you to check nullable types for null first. No more NullPointerException¬†for you! ūüėČ
  • Lambdas are super fun. Expressive, powerful and of course productive.
  • Easy to use and useful ranges.
  • Easy to add custom get¬†and set¬†methods to public fields. Pair that with simple functions. You have got properties without much boilerplate code.
  • The default arguments and named arguments are pretty cool
  • You can use ==¬†for checking equality, no need for equals¬†calls.
  • The is¬†operator provides automatic smart casts. When you write if (param is String) { // code }, the param¬†is cast into a string for you in the block. So no more instance checks.
  • Super expressive maps, filters with lambdas
  • Ability to create custom DSLs
  • Coroutines
  • Awesome tooling support. IntelliJ Idea lets you convert Java code to Kotlin in just a click. You know their IDE is the best, right? But there’s also support for other popular IDEs like Eclipse.

Learning Kotlin

You can also visit us on the Facebook, we have a group named – Kotlin Ninja¬†where some Kotlin enthusiasts learn and share together ūüôā

Django REST Framework: Authentication and Permissions

(This post is a part of a tutorial series on Building REST APIs in Django)

In our last post about ViewSet, ModelViewSet and Router, we saw how easily we can create REST APIs with the awesome Django REST Framework. In this blog post, we would see how we can secure our endpoints with user authentication and permissions. Authentication will help us identify which user is currently logged in and permissions will decide which user(s) can access which resources.

Authentication

The idea of authentication is pretty simple. When a new incoming request comes, we have to check the request and see if we can identify any user credentials along with it. If you have read the Flask HTTP Auth tutorial or the one about JWT, you might remember how we were checking the authorization header to authenticate our users. We might also receive the user login data via a POST request (form submission) or the user may already be logged in and we can identify using the session data.

We can see that the authentication mechanism can largely vary. Django REST Framework is very flexible in accommodating them. We can give DRF a list of classes, DRF will run the authenticate method on those classes. As soon as a class successfully authenticates the user, the return values from the call is set to request.user and request.auth. If none of the classes manage to authenticate the user, then the user is set to django.contrib.auth.models.AnonymousUser .

We can set these classes using the DEFAULT_AUTHENTICATION_CLASSES¬†settings under the DRF settings. Here’s an example:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    )
}

In the example above we used BasicAuthentication and SessionAuthentication Рtwo of the built in classes from Django REST Framework. We will look at how they work and we will also check how we can write our own class for our custom authentication.

(PS: Here we set the authentication policy globally, for all views / paths / resources – if we want, we can also use different authentication mechanism for each one, individually but that is usually not done in most cases).

Basic Authentication

In our example before, we mentioned the BasicAuthentication class. This class first checks the http authorization header (HTTP_AUTHORIZATION in request.META ). If the header contains appropriate string (something like Basic <Base64 Encoded Login>), it will decode the string, split the username, password and try to authenticate the user.

Basic Authentication is very simple, easy to setup and might be quite convenient for testing / debugging but I would highly discourage using this method on production.

Session Authentication

If you have used Django, you already know about session based authentication. In fact, Django itself handles the session based auth and sets the user as part of the request¬†object (an instance of HttpRequest¬†object. DRF just reads the user data from the request and checks for CSRF. That’s it.

Session Authentication works very well if your users are interacting with your API on the web, perhaps using ajax calls? In such cases, if the user is once logged in, his/her auth¬†is stored in the session and we can depend on those data while making requests from our web app. However, this will not work well if the client doesn’t or can not accept cookies (apps on different domains, mobile or desktop apps, other micro services etc).

Token Authentication

If you understand JWT, this one will feel similar, except in this case, the token will be just a “token”, no JSON or no signing. The user logs in and gets a token. On subsequent requests, this token must be passed as part of the authorization header.

To use token based auth, we first need to add the rest_framework.authtoken app to the INSTALLED_APPS list in your settings.py file. And then run the migration to create the related tables.

python manage.py migrate

We also need to add the TokenAuthentication class to our DRF auth class list:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    )
}

Now let’s create a view to issue tokens to user.

from django.contrib.auth import authenticate
from rest_framework.decorators import api_view
from rest_framework.response import Response
from rest_framework.status import HTTP_401_UNAUTHORIZED
from rest_framework.authtoken.models import Token


@api_view(["POST"])
def login(request):
    username = request.data.get("username")
    password = request.data.get("password")

    user = authenticate(username=username, password=password)
    if not user:
        return Response({"error": "Login failed"}, status=HTTP_401_UNAUTHORIZED)

    token, _ = Token.objects.get_or_create(user=user)
    return Response({"token": token.key})

The code here should be self explanatory. We take username and password. We then try to authenticate the user using Django’s default authentication (checking username and password against what’s stored in the database). If the authentication fails, we return error message along with http status code 401.¬†If the authentication succeeds, we issue a token for the user and pass it in the response.

We need to add this view to our urlpatterns next:

url(r'^login', login)

Now let’s try it out:

$ curl --request POST \
  --url http://localhost:8000/api/login \
  --header 'content-type: application/json' \
  --data '{"username": "test_user", "password": "awesomepwd"}'

{"token":"5e2effff34c85c11a8720a597b96d73a4634c9ad"}%

So we’re getting the tokens successfully. Now to access a secured resource, we need to pass it as part of the authorization header. But how do we make a resource available only to a logged in user? Well, permissions come into play here.

Permissions

While authentication tells us which user is logged in (or not), it’s our responsibility to check if the current user (a valid logged in user or a guest, not logged in visitor) has access to the resource. Permissions¬†can help us deal with that. Just like authentication, we can also set a class of permissions globally or on each resource individually. Let’s start with the IsAuthenticated¬†permission first. Let’s add this to our SubscriberViewSet.

from rest_framework.permissions import IsAuthenticated

class SubscriberViewSet(ModelViewSet):
    serializer_class = SubscriberSerializer
    queryset = Subscriber.objects.all()
    permission_classes = (IsAuthenticated,)

If we try to access subscribers without any authentication, we will get an error message now:

{
  "detail": "Authentication credentials were not provided."
}

So let’s provide authentication using the token we got.

$ curl -H "Content-Type: application/json" -H "Authorization: Token 5e2effff34c85c11a8720a597b96d73a4634c9ad" http://localhost:8000/api/subscribers/

Now it works fine! There are many useful, already provided permission classes with Django REST Framework. You can find a list of them here http://www.django-rest-framework.org/api-guide/permissions/#api-reference.

Custom Authentication and Permissions

The authentication and permission classes which come with DRF are quite enough for many cases. But what if we needed to create our own? Let’s see how we can do that.

Writing a custom authentication class is very simple. You define your custom authenticate method which would receive the request object. You will have to return an instance of the default User model if authentication succeeds, otherwise raise an exception. You can also return an optional value for the auth object to be set on request. If our authentication method can not be used for this request, we should return None so other classes are tried.

Here’s an example from DRF docs:

from django.contrib.auth.models import User
from rest_framework import authentication
from rest_framework import exceptions

class ExampleAuthentication(authentication.BaseAuthentication):
    def authenticate(self, request):
        username = request.META.get('X_USERNAME')
        if not username:
            return None

        try:
            user = User.objects.get(username=username)
        except User.DoesNotExist:
            raise exceptions.AuthenticationFailed('No such user')

        return (user, None)

In this example, the username is being retrieved from a custom header (X_USERNAME) and the rest is quite easy to understand.

Next, let’s see how we can create our custom permission class. For permissions, we can have two types of permissions – global permission or per object permission. Here’s an example of global permission from DRF docs:

from rest_framework import permissions

class BlacklistPermission(permissions.BasePermission):
    """
    Global permission check for blacklisted IPs.
    """

    def has_permission(self, request, view):
        ip_addr = request.META['REMOTE_ADDR']
        blacklisted = Blacklist.objects.filter(ip_addr=ip_addr).exists()
        return not blacklisted

If the has_permission¬†method returns True¬†then the user has permission, otherwise not. Let’s see the example for per object permission:

class IsOwnerOrReadOnly(permissions.BasePermission):
    """
    Object-level permission to only allow owners of an object to edit it.
    Assumes the model instance has an `owner` attribute.
    """

    def has_object_permission(self, request, view, obj):
        # Read permissions are allowed to any request,
        # so we'll always allow GET, HEAD or OPTIONS requests.
        if request.method in permissions.SAFE_METHODS:
            return True

        # Instance must have an attribute named `owner`.
        return obj.owner == request.user

For dealing with per object permission, we can override the has_object_permission method. It can take the request, the view and the obj. We have to check if the current user can access the obj in question. Just like before, we need to return True or False to allow or deny the request.

In this blog post, we learned the basics of authentication and permissions. We now know how we can secure our API endpoints with DRF. While the token based authentication was very useful, we kind of like JWT. So in our next post, we will be using a third party package to implement JWT for Django REST Framework.

Django REST Framework: ViewSet, ModelViewSet and Router

(This post is a part of a tutorial series on Building REST APIs in Django)

In our last blog post on ModelSerializer and Generic Views, we demonstrated how we can use the generic views along with ModelSerializer classes to rapidly develop our REST APIs. In this post, we will discuss about ViewSet and ModelViewset and see how they can speed up building APIs even further. At the same time we will take a look at the concepts of Routers which allow us to manage our api routes in a cleaner fashion.

Understanding ViewSet

So far we have learned how we can use the generic views. We can now use them to create the two kinds of resources – “collections” and “elements”. It works very well but we need to write at least two different classes to handle them properly. But if we think about it, both resources focus on the same entity, the same model – “Subscriber”. Wouldn’t it make more sense if we could have all the actions related to a Subscriber in a single class? Wouldn’t that be more convenient if we can put all the Subscriber related logic in a single place?

ViewSets come to the rescue. A ViewSet¬†is, as the name suggests, a class that provides the functionality of a set of views which are closely related. It’s one class but provides a set of views. We can handle both “collection” and “element” type of resources from the same class. And not just those, we can add even other¬†related actions.

Since a ViewSet handles both type of resources, we can no longer think in terms of the http verbs. Because both /api/subscriber and /api/subscriber/1 can respond to  GET requests and should produce different types of response. So we can no longer work with the get, post, put etc methods. We need to think more along the actions we can take on the entity (Subscriber) as a whole. A ViewSet works with these methods instead:

  • list¬†– list all elements, serves GET¬†to /api/subscriber
  • create¬†– create a new element, serves POST¬†to /api/subscriber
  • retrieve¬†– retrieves one element, serves GET¬†to /api/subscriber/1
  • update¬†and partial_update – updates single element, handles PUT/PATCH¬†to /api/subscriber/1
  • destroy¬†– deletes single element, handles DELETE¬†to /api/subscriber/1

Instead of our old get¬†and post¬†methods in separate classes, we can now define these methods in one single class and be done with it. But with general ViewSet, we have to provide logic / code for these views. That would¬†require more time and efforts. What if, we could generate these methods from our models / querysets? Well, we can! ūüėÄ

The ModelViewSet

The ModelViewSet¬†would only ask for the serializer class and the queryset. And then it will provide all the functionality of the different¬†ViewSet¬†methods. Let’s see the example:

from rest_framework.viewsets import ModelViewSet

from .serializers import SubscriberSerializer
from .models import Subscriber


class SubscriberViewSet(ModelViewSet):
    serializer_class = SubscriberSerializer
    queryset = Subscriber.objects.all()

Our SubscriberViewSet now extends the ModelViewSet and we provided the queryset and the serializer_class. Done. Really!

But there’s one slight problem, the ViewSet¬†and ModelViewSet¬†both handle at least two distincts url paths – /api/subscriber¬†(the collection path) and /api/subscriber/1¬†– the elements path. How do we tell Django’s URLConf to route requests to both urls to our single ViewSet? Well, we have to declare those paths ourselves. We can use the as_view¬†to both paths, defining which http verb should be routed to which methods:

SubscriberViewSet.as_view({'get': 'list', 'post':'create'}) # For: /api/subscriber
SubscriberViewSet.as_view({'get': 'retrieve', 'put':'update'}) # For: /api/subscriber/1

Or we can simply use a Router.

Using Routers

A Router¬†instance let’s us register¬†our ViewSet¬†to it and then we can just add the router.urls¬†to our urlpatterns. It will keep track of the view sets we register and the urls¬†property will generate all the url patterns required for those view sets to work. ¬†Let’s add our newly created SubscriberViewSet¬†to a router and see for ourselves. Open the api/urls.py¬†file and create the router there and register the viewset.

from rest_framework.routers import SimpleRouter

from .views import SubscriberViewSet

router = SimpleRouter()
router.register("subscribers", SubscriberViewSet)

urlpatterns = router.urls

That’s all that is required. Now try visiting – http://localhost:8000/api/subscribers or¬†http://localhost:8000/api/subscribers/1 – it all works.

Fantastic, so we just generated a full RESTful API from a model with¬†a surprisingly short amount of code! Isn’t that wonderful? Just don’t trust my words, go and look at your SubscriberViewSet¬†or the SubscriberSerializer¬†or the above defined router.

With the help of ModelSerializer, ModelViewSet and a router instance, we can build elegant CRUD APIs from our Django models insanely fast. At the same time, if we need, we can just override one of those methods (list, retrieve, create etc) to alter the default behavior with our own.

What’s Next?

We have crafted a nice, functional REST API. The next stop would be securing it. In our next post, we will be discussing Authentication and Permissions.

In the mean time, I would request you to subscribe to the mailing list so I can keep you posted about new exciting contents on this site. If you liked the post, please do share with your friends!

 

Django REST Framework: ModelSerializer and Generic Views

(This post is a part of a tutorial series on Building REST APIs in Django)

In our last post on Serializers, we learned how to use Serializers with APIViews. In this post we will discuss how ModelSerializer and the Generic views can take things even further.

Model + Serializer = ModelSerializer

That one line equation is probably enough to explain the concepts of ModelSerializers. In our example, we had to define similar fields on both Serializer and the Model class. We had to write codes for the same fields twice. That is 2x the efforts. ModelSerializers can help solve that problem. A ModelSerializer might remind us of ModelForm. The idea is the same. We extend ModelSerializer and pass it the model. The serializer inspects the model and knows what fields to use and what their types are.

Let’s refactor our old serializer to be a ModelSerializer:

from rest_framework import serializers

from .models import Subscriber


class SubscriberSerializer(serializers.ModelSerializer):
    class Meta:
        model = Subscriber
        fields = "__all__"

That’s all we need – our new SubscriberSerializer¬†now infers the fields from the model (Subscriber) we passed to it. We can however choose which fields should be used while serializing / deserializing. In this example we pass the special value “__all__” which means we want to use all fields. But in many cases we would want to selectively use some fields. For example, for Django’s default¬†User¬†model, we don’t want to leak the password¬†data to public, so we will exclude the password¬†field on our User model serializer.

If we try out our API, we would notice everything is working just like before. Except our serializer class is now shorter and more concise. Awesome, right? Let’s move on to our next topic – generic views.

Generic Views

If you have decent amount of experience with Django, you probably have already come across and used the built in generic views. They provide useful functionality around common database operations. For example, we can use the generic list view and provide it with a queryset and the template, it will do the rest for us.

In the same way, we can pass a queryset and serializer to a ListAPIView¬†and it will create the list view for us. Same way we can use CreateAPIView¬†to implement the create view. What’s even better, since we’re using class based views, we can use both of them together, cool, no? Let’s refactor old code to use these two classes.

from rest_framework.generics import ListAPIView, CreateAPIView

from .serializers import SubscriberSerializer
from .models import Subscriber


class SubscriberView(ListAPIView, CreateAPIView):
    serializer_class = SubscriberSerializer
    queryset = Subscriber.objects.all()

Please note how we no longer need to provide the get and post methods but the API still works. The generic views just know how to deal with those.

There is actually a ListCreateAPIView which is the combination of these two as you can probably understand from the class name. We can just use that one.

from rest_framework.generics import ListCreateAPIView

from .serializers import SubscriberSerializer
from .models import Subscriber


class SubscriberView(ListCreateAPIView):
    serializer_class = SubscriberSerializer
    queryset = Subscriber.objects.all()

As you can see this generic view is quite helpful for generating “collection” resources (/api/subscriber) easily. There’s also RetrieveUpdateDestroyAPIView which we can use to generate “element” resources. As you can guess from the name, they provide get, put, patch¬†and delete¬†handler for single items (/api/subscriber/12).

Generic views are very handy for quickly generating resources from our models (querysets actually). This allows rapid API development.

What’s Next?

We have seen how we can very quickly create working REST APIs using model serializers and generic views.¬†Things will be even more amazing when we learn about ViewSets, specially ModelViewSets. You will have to wait for our next post for those ūüôā

In case you didn’t notice, we have a mailing list where you can subscribe to get latest updates. We don’t spam, only contents we post. Also if you enjoyed reading the post and found it informative, shouldn’t you share it with your friends? ūüėÄ

Django REST Framework: Serializers

(This post is a part of a tutorial series on Building REST APIs in Django)

In our last blog post, Getting started with Django REST Framework, we saw how we could use the APIView and accept inputs from users using request.data. In our example, we dealt with string, so it was pretty straightforward. But consider the case of age or account_balance Рone has to be integer, the other has to be float / decimal. How do we properly validate the incoming data?

We can manually check every input field and send an error if the field type doesn’t match. But soon we’re going to have a problem at our hand – when the number of inputs will grow, we can’t just keep doing this kind of manual validation. In Django, we would probably use Django Forms for validation. Does DRF provide us with something similar? Yes, it does. The solution to our problem is Serializers.

What can a Serializer do for us?

Have you ever tried JSON serializing a Django model? Or a queryset? You can’t directly because they are not JSON serializable. So what do we do instead? We convert them to Python’s native data structures which could be serialized into JSON. We can serialize querysets into lists and model instances to dictionaries. But doing that by hand is cumbersome.

On the other hand, we saw how we can get incoming data from request.data¬†– we get this data¬†as key value pairs. We can’t just store them in database directly – we have to transform them into Django’s data structures like models and querysets. Doing that by hand is also cumbersome.

Serializers can help us with that. It can serialize complex types into Python natives types and then again deserialize native types into those complex types. Besides that, it also does basic validation based on the serializer field types. If a field is defined as an integer field, it will raise an error if we pass a string to that field. If we need more advanced validation rules, we can plug in the built in Validators¬†or even write our own. Let’s see code examples to understand the use case better.

Defining a Serializer

Create a file named serializers.py inside the api app directory. Put the following codes into it.

from rest_framework import serializers


class HelloWorldSerializer(serializers.Serializer):
    name = serializers.CharField(required=True, max_length=6)
    age = serializers.IntegerField(required=False, min_value=10, default=10)

We’re creating a HelloWorldSerializer¬†which extends serializers.Serializer. We’re defining two fields on this serializer –

  • name¬†is a CharField¬†so it accepts string. It has a max_length¬†of 6.
  • age¬†is an optional integer field. The value must be at least 10 if provided. If not provided, default value will be 10.

With this serializer setup, let’s modify our view to use it.

from .serializers import HelloWorldSerializer


class HelloWorldView(APIView):
    def get(self, request):
        return Response({"message": "Hello World!"})

    def post(self, request):
        serializer = HelloWorldSerializer(data=request.data)
        if serializer.is_valid():
            valid_data = serializer.data

            name = valid_data.get("name")
            age = valid_data.get("age")

            return Response({"message": "Hello {}, you're {} years old".format(name, age)})
        else:
            return Response({"errors": serializer.errors})

We pass the request.data as the data parameter to HelloWorldSerializer so it can read all the request data and parse them. Then we check if the serializer is valid. If you have used Django Forms, this will feel very similar. If the serializer is valid, that means we have a valid set of data available. So we can take the value of name and age and show a pretty message. On the other hand, if the serializer is not valid, we can pass the serializer.errors back to the client, which will contain elaborate error messages.

Let’s try out the API to see what happens. Let’s first send an empty request:

$ curl -H "Content-Type: application/json" -X POST http://localhost:8000/api/hello

{"errors":{"name":["This field is required."]}}

The errors say the name¬†field is required. Of course it is! Let’s pass the name.

$ curl -H "Content-Type: application/json" -X POST -d '{"name": "masnun"}' http://localhost:8000/api/hello

{"message":"Hello masnun, you're 10 years old"}

We just passed the name but didn’t pass the age. Since it is not required and has a default value set, we get the default value. But what if we set a low value?

$ curl -H "Content-Type: application/json" -X POST -d '{"name": "masnun", "age": '8'}' http://localhost:8000/api/hello

{"errors":{"age":["Ensure this value is greater than or equal to 10."]}}

So we passed 8¬†and it’s not happy about that. Please note we passed the 8 as a string but DRF doesn’t mind as long as it can convert it to an integer successfully. What if we pass a value that is no number?

$ curl -H "Content-Type: application/json" -X POST -d '{"name": "masnun", "age": "ten"}' http://localhost:8000/api/hello

{"errors":{"age":["A valid integer is required."]}}

That works too! Cool, okay then let’s give it a rest and pass a valid value.

$ curl -H "Content-Type: application/json" -X POST -d '{"name": "masnun", "age": 12}' http://localhost:8000/api/hello

{"message":"Hello masnun, you're 12 years old"}

 Serializer with Model

How does Serializers help us in working with models? To understand that, let’s first create one model.

Creating the Subscriber Model

Open api/models.py and add the Subscriber model like this:

class Subscriber(models.Model):
    name = models.CharField("Name", max_length=50)
    age = models.IntegerField("Age")
    email = models.EmailField("Email")

Now create and run the migration.

python manage.py makemigrations
python manage.py migrate

That should setup the table for our new model.

Update The Serializer

We added an email field to our model, also the max length for name is now 50 chars. Let’s update our serializer to match these constraints. Also rename it as SubscriberSerializer.

from rest_framework import serializers


class SubscriberSerializer(serializers.Serializer):
    name = serializers.CharField(max_length=50)
    age = serializers.IntegerField()
    email = serializers.EmailField()
Update The View

And now let’s refactor our view.

from .serializers import SubscriberSerializer
from .models import Subscriber


class SubscriberView(APIView):
    def get(self, request):
        return Response({"message": "Hello World!"})

    def post(self, request):
        serializer = SubscriberSerializer(data=request.data)
        if serializer.is_valid():
            subscriber_instance = Subscriber.objects.create(**serializer.data)
            return Response({"message": "Created subscriber {}".format(subscriber_instance.id)})
        else:
            return Response({"errors": serializer.errors})

The code is very simple and straightforward. If the serializer validation succeeds, we create a new subscriber out of the validated data.

Update URLConf

Let’s update the urls.py¬†to update our url end point.

url(r'^subscriber', SubscriberView.as_view(), name="subscriber")

Now let’s try it out. We will post the following JSON using curl or postman:

{"name": "Abu Ashraf Masnun", "email": "[email protected]", "age": 29}

And we will get back the following response:

{
  "message": "Created subscriber 1"
}

With the serializer, we needed so much less codes. And we did it in a very clean way.

List All Subscribers

According to the REST Best Practices, the GET¬†call to a resource route (/api/subscriber) should return a list of all the items (subscribers). So let’s refactor the get¬†method to return the subscribers list.

    def get(self, request):
        all_subscribers = Subscriber.objects.all()
        serialized_subscribers = SubscriberSerializer(all_subscribers, many=True)
        return Response(serialized_subscribers.data)

We are fetching all subscribers and then passing the queryset to the serializer constructor. Since we’re passing a query set (not just a single model instance rather a list of model instances), we need to set ¬†many=True¬†. Also note,¬†we don’t need to call is_valid¬†– the data is coming from database, they’re already valid. In fact, we can’t call is_valid¬†unless we’re passing some value to the data¬†parameter (SubscriberSerializer(data=request.data)). When we pass queryset or model, the data is automatically available as serializer.data.

What’s Next?

We have so far learned how to use APIView¬†and Serializer¬†to build beautiful APIs with clean code. But we still have some duplication of efforts, for example when we had to define fields on both the model and the serializer. Also, we have to implement both collection and element resources. So that’s like 5 method implementation. Wouldn’t it be great if, by some mechanism we could¬†make things simpler, shorter and cleaner?

We’ll see, in our next blog post ūüôā Please subscribe to our mailing list for email updates. Also if you liked the content, please don’t forget to share with your friends!