Abu Ashraf Masnun

Python

Python asyncio: Future, Task and the Event Loop

Post author By masnun
Post date November 20, 2015
8 Comments on Python asyncio: Future, Task and the Event Loop

Event Loop

On any platform, when we want to do something asynchronously, it usually involves an event loop. An event loop is a loop that can register tasks to be executed, execute them, delay or even cancel them and handle different events related to these operations. Generally, we schedule multiple async functions to the event loop. The loop runs one function, while that function waits for IO, it pauses it and runs another. When the first function completes IO, it is resumed. Thus two or more functions can co-operatively run together. This the main goal of an event loop.

The event loop can also pass resource intensive functions to a thread pool for processing. The internals of the event loop is quite complex and we don’t need to worry much about it right away. We just need to remember that the event loop is the mechanism through which we can schedule our async functions and get them executed.

Futures / Tasks

If you are into Javascript too, you probably know about Promise. In Python we have similar concepts – Future/Task. A Future is an object that is supposed to have a result in the future. A Task is a subclass of Future that wraps a coroutine. When the coroutine finishes, the result of the Task is realized.

Coroutines

We discussed Coroutines in our last blog post. It’s a way of pausing a function and returning a series of values periodically. A coroutine can pause the execution of the function by using the yield yield from or await (python 3.5+) keywords in an expression. The function is paused until the yield statement actually gets a value.

Fitting Event Loop and Future/Task Together

It’s simple. We need an event loop and we need to register our future/task objects with the event loop. The loop will schedule and run them. We can add callbacks to our future/task objects so that we can be notified when a future has it’s results.

Very often we choose to use coroutines for our work. We wrap a coroutine in Future and get a Task object. When a coroutine yields, it is paused. When it has a value, it is resumed. When it returns, the Task has completed and gets a value. Any associated callback is run. If the coroutine raises an exception, the Task fails and not resolved.

So let’s move ahead and see example codes.

import asyncio


@asyncio.coroutine
def slow_operation():
    # yield from suspends execution until
    # there's some result from asyncio.sleep

    yield from asyncio.sleep(1)

    # our task is done, here's the result
    return 'Future is done!'


def got_result(future):
    print(future.result())


# Our main event loop
loop = asyncio.get_event_loop()

# We create a task from a coroutine
task = loop.create_task(slow_operation())

# Please notify us when the task is complete
task.add_done_callback(got_result)

# The loop will close when the task has resolved
loop.run_until_complete(task)

import asyncio

@asyncio.coroutine

def slow_operation():

# yield from suspends execution until

# there's some result from asyncio.sleep

yield from asyncio.sleep(1)

# our task is done, here's the result

return 'Future is done!'

def got_result(future):

print(future.result())

# Our main event loop

loop = asyncio.get_event_loop()

# We create a task from a coroutine

task = loop.create_task(slow_operation())

# Please notify us when the task is complete

task.add_done_callback(got_result)

# The loop will close when the task has resolved

loop.run_until_complete(task)

As you can see already:

@asyncio.coroutine declares it as a coroutine
loop.create_task(slow_operation()) creates a task from the coroutine returned by slow_operation()
task.add_done_callback(got_result) adds a callback to our task
loop.run_until_complete(task) runs the event loop until the task is realized. As soon as it has value, the loop terminates

The run_until_complete function is a nice way to manage the loop. Of course we could do this:

import asyncio


async def slow_operation():
    await asyncio.sleep(1)
    return 'Future is done!'


def got_result(future):
    print(future.result())

    # We have result, so let's stop
    loop.stop()


loop = asyncio.get_event_loop()
task = loop.create_task(slow_operation())
task.add_done_callback(got_result)

# We run forever
loop.run_forever()

import asyncio

async def slow_operation():

await asyncio.sleep(1)

return 'Future is done!'

def got_result(future):

print(future.result())

# We have result, so let's stop

loop.stop()

loop = asyncio.get_event_loop()

task = loop.create_task(slow_operation())

task.add_done_callback(got_result)

# We run forever

loop.run_forever()

Here we make the loop run forever and from our callback, we explicitly shut it down when the future has resolved.

Python

Python: Generators, Coroutines, Native Coroutines and async/await

Post author By masnun
Post date November 13, 2015
21 Comments on Python: Generators, Coroutines, Native Coroutines and async/await

NOTE: This post discusses features which were mostly introduced with Python 3.4. And the native coroutines and async/await syntax came in Python 3.5. So I recommend you to use Python 3.5 to try the codes, if you don’t know how to update python, make sure to visit this website.

Generators

Generators are functions that generates values. A function usually returns a value and then the underlying scope is destroyed. When we call again, the function is started from scratch. It’s one time execution. But a generator function can yield a value and pause the execution of the function. The control is returned to the calling scope. Then we can again resume the execution when we want and get another value (if any). Let’s look at this example:

def simple_gen():
    yield "Hello"
    yield "World"


gen = simple_gen()
print(next(gen))
print(next(gen))

def simple_gen():

yield "Hello"

yield "World"

gen = simple_gen()

print(next(gen))

Please notice, a generator function doesn’t directly return any values but when we call it, we get a generator object which is like an iterable. So we can call next() on a generator object to iterate over the values. Or run a for loop.

So how’s generators useful? Let’s say your boss has asked you to write a function to generate a sequence of number up to 100 (a super secret simplified version of range()). You wrote it. You took an empty list and kept adding the numbers to it and then returned the list with the numbers. But then the requirement changes and it needs to generate up to 10 million numbers. If you store these numbers in a list, you will soon run out of memory. In such situations generators come into aid. You can generate these numbers without storing them in a list. Just like this:

def generate_nums():
    num = 0
    while True:
        yield num
        num = num + 1


nums = generate_nums()

for x in nums:
    print(x)

    if x > 9:
        break

def generate_nums():

num = 0

while True:

yield num

num = num + 1

nums = generate_nums()

for x in nums:

print(x)

if x > 9:

break

We didn’t dare run after the number hit 9. But if you try it on console, you will see how it keeps generating numbers one after one. And it does so by pausing the execution and resuming – back and forth into the function context.

Summary: A generator function is a function that can pause execution and generate multiple values instead of just returning one value. When called, it gives us a generator object which acts like an iterable. We can use (iterate over) the iterable to get the values one by one.

Coroutines

In the last section we have seen that using generators we can pull data from a function context (and pause execution). What if we wanted to push some data too? That’s where coroutines comes into play. The yield keyword we use to pull values can also be used as an expression (on the right side of “=”) inside the function. We can use the send() method on a generator object to pass values back into the function. This is called “generator based coroutines”. Here’s an example:

def coro():
    hello = yield "Hello"
    yield hello


c = coro()
print(next(c))
print(c.send("World"))

def coro():

hello = yield "Hello"

yield hello

c = coro()

print(next(c))

print(c.send("World"))

OK, so what’s happening here? We first take the value as usual – using the next() function. This comes to yield "Hello" and we get “Hello”. Then we send in a value using the send() method. It resumes the function and assigns the value we send to hello and moves on up to the next line and executes the statement. So we get “World” as a return value of the send() method.

When we’re using generator based coroutines, by the terms “generator” and “coroutine” we usually mean the same thing. Though they are not exactly the same thing, it is very often used interchangeably in such cases. However, with Python 3.5 we have async/await keywords along with native coroutines. We will discuss those later in this post.

Async I/O and the `asyncio` module

From Python 3.4, we have the new asyncio module which provides nice APIs for general async programming. We can use coroutines with the asyncio module to easily do async io. Here’s an example from the official docs:

import asyncio
import datetime
import random


@asyncio.coroutine
def display_date(num, loop):
    end_time = loop.time() + 50.0
    while True:
        print("Loop: {} Time: {}".format(num, datetime.datetime.now()))
        if (loop.time() + 1.0) >= end_time:
            break
        yield from asyncio.sleep(random.randint(0, 5))


loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))
asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

import asyncio

import datetime

import random

@asyncio.coroutine

def display_date(num, loop):

end_time = loop.time() + 50.0

while True:

print("Loop: {} Time: {}".format(num, datetime.datetime.now()))

if (loop.time() + 1.0) >= end_time:

break

yield from asyncio.sleep(random.randint(0, 5))

loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))

asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

The code is pretty self explanatory. We create a coroutine display_date(num, loop) which takes an identifier (number) and an event loop and continues to print the current time. Then it used the yield from keyword to await results from asyncio.sleep() function call. The function is a coroutine which completes after a given seconds. So we pass random seconds to it. Then we use asyncio.ensure_future() to schedule the execution of the coroutine in the default event loop. Then we ask the loop to keep running.

If we see the output, we shall see that the two coroutines are executed concurrently. When we use yield from, the event loop knows that it’s going to be busy for a while so it pauses execution of the coroutine and runs another. Thus two coroutines run concurrently (but not in parallel since the event loop is single threaded).

Just so you know, yield from is a nice syntactic sugar for for x in asyncio.sleep(random.randint(0, 5)): yield x making async codes cleaner.

Native Coroutines and `async`/`await`

Remember, we’re still using generator based coroutines? In Python 3.5 we got the new native coroutines which uses the async/await syntax. The previous function can be written this way:

import asyncio
import datetime
import random


async def display_date(num, loop, ):
    end_time = loop.time() + 50.0
    while True:
        print("Loop: {} Time: {}".format(num, datetime.datetime.now()))
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(random.randint(0, 5))


loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))
asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

import asyncio

import datetime

import random

async def display_date(num, loop, ):

end_time = loop.time() + 50.0

while True:

print("Loop: {} Time: {}".format(num, datetime.datetime.now()))

if (loop.time() + 1.0) >= end_time:

break

await asyncio.sleep(random.randint(0, 5))

loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))

asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

Take a look at the highlighted lines. We must define a native coroutine with the async keyword before the def keyword. Inside a native coroutine, we use the await keyword instead of yield from.

Native vs Generator Based Coroutines: Interoperability

There’s no functional differences between the Native and Generator based coroutines except the differences in the syntax. It is not permitted to mix the syntaxes. So we can not use await inside a generator based coroutine or yield/yield from inside a native coroutine.

Despite the differences, we can interoperate between those. We just need to add @types.coroutine decorator to old generator based ones. Then we can use one from inside the other type. That is we can await from generator based coroutines inside a native coroutine and yield from native coroutines when we are inside generator based coroutines. Here’s an example:

import asyncio
import datetime
import random
import types


@types.coroutine
def my_sleep_func():
    yield from asyncio.sleep(random.randint(0, 5))


async def display_date(num, loop, ):
    end_time = loop.time() + 50.0
    while True:
        print("Loop: {} Time: {}".format(num, datetime.datetime.now()))
        if (loop.time() + 1.0) >= end_time:
            break
        await my_sleep_func()


loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))
asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

import asyncio

import datetime

import random

import types

@types.coroutine

def my_sleep_func():

yield from asyncio.sleep(random.randint(0, 5))

async def display_date(num, loop, ):

end_time = loop.time() + 50.0

while True:

print("Loop: {} Time: {}".format(num, datetime.datetime.now()))

if (loop.time() + 1.0) >= end_time:

break

await my_sleep_func()

loop = asyncio.get_event_loop()

asyncio.ensure_future(display_date(1, loop))

asyncio.ensure_future(display_date(2, loop))

loop.run_forever()

Javascript NodeJS

Using ES7 async/await today with Babel

Post author By masnun
Post date November 11, 2015
15 Comments on Using ES7 async/await today with Babel

Let’s take a code snippet that contains the demonstration of async/await — https://gist.github.com/patrickarlt/8c56a789e5f185eb9722 – our objective is to transpile this piece of code to ES5 (current day Javascript) so we can run it with today’s version of NodeJS.

You will notice a command on top of the snippet which no longer works because Babel JS has changed. I am going to describe how we can do it with the latest version of babel as of this writing (6.1.4 (babel-core 6.1.4)).

Install Babel and Plugins

The new Babel depends on individual plugins to transform and parse codes. To transform async functions, we shall use the transform-regenerator plugin. We also need to add the syntax plugin to recognize the async/await syntax. Otherwise Babel won’t recognize those. Apart from that, we also install the ES2015 preset which includes a sane set of plugins for transforming ES6 to ES5. We will keep those so we can use other ES6 goodies.

First we install babel-cli globally:

npm install -g babel-cli

1	npm install -g babel-cli

Here’s our package.json file so you can just do npm install:

{
  "name": "awesome-async",
  "version": "1.0.0",
  "description": "",
  "main": "github.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "babel-plugin-syntax-async-functions": "^6.1.4",
    "babel-plugin-transform-regenerator": "^6.1.4",
    "babel-polyfill": "^6.1.4",
    "babel-preset-es2015": "^6.1.4",
    "request": "^2.65.0"
  }
}

{

"name": "awesome-async",

"version": "1.0.0",

"description": "",

"main": "github.js",

"scripts": {

"test": "echo \"Error: no test specified\" && exit 1"

"author": "",

"license": "ISC",

"dependencies": {

"babel-plugin-syntax-async-functions": "^6.1.4",

"babel-plugin-transform-regenerator": "^6.1.4",

"babel-polyfill": "^6.1.4",

"babel-preset-es2015": "^6.1.4",

"request": "^2.65.0"

}

Configuring Babel

Here’s the .babelrc file I put in the same directory:

{
  "presets": ["es2015"],
  "plugins": ["syntax-async-functions","transform-regenerator"]
}

{

"presets": ["es2015"],

"plugins": ["syntax-async-functions","transform-regenerator"]

}

The file tells babel how to transform your code.

Transpile & Run

Once we have installed the dependencies, we can then start transpiling the codes to JS:

babel github.es6 -o github.js
node github.js

1 2	babel github.es6 -o github.js node github.js

You might run into a problem like this:

/Users/masnun/Projects/awesome-async/github.js:34
  return regeneratorRuntime.async(function printPublicGists$(_context) {
         ^
ReferenceError: regeneratorRuntime is not defined
    at printPublicGists (/Users/masnun/Projects/node/github.js:34:10)
    at Object.<anonymous> (/Users/masnun/Projects/node/github.js:63:1)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at startup (node.js:129:16)
    at node.js:814:3

/Users/masnun/Projects/awesome-async/github.js:34

return regeneratorRuntime.async(function printPublicGists$(_context) {

ReferenceError: regeneratorRuntime is not defined

at printPublicGists (/Users/masnun/Projects/node/github.js:34:10)

at Object.<anonymous> (/Users/masnun/Projects/node/github.js:63:1)

at Module._compile (module.js:460:26)

at Object.Module._extensions..js (module.js:478:10)

at Module.load (module.js:355:32)

at Function.Module._load (module.js:310:12)

at Function.Module.runMain (module.js:501:10)

at startup (node.js:129:16)

at node.js:814:3

It’s because we need to include the regenerator run time. The runtime is packed in the babel-polyfill we have already installed. We just need to include it in our source code. So the final github.es6 file would look like this:

require("babel-polyfill");

import request from "request";

// promise returning function
function get (url){
  return new Promise(function(resolve, reject){
    request({
      method: 'GET',
      url: url,
      json: true,
      headers: {
        'User-Agent': 'request'
      }
    }, function(err, resp, body){
      if(err){
        reject(err);
      } else {
        resolve(body);
      }
    });
  });
}


// create a new "async" function so we can use the "await" keyword
async function printPublicGists(){
  // "await" resolution or rejection of the promise
  // use try/catch for error handling
  try {
    var gists = await get('https://api.github.com/gists/public');
    
    // now you can write this like syncronous code!
    gists.forEach(function(gist){
      console.log(gist.description);
    });
  } catch (e) {
    // promise was rejected and we can handle errors with try/catch!
  }
}


printPublicGists();

require("babel-polyfill");

import request from "request";

// promise returning function

function get (url){

return new Promise(function(resolve, reject){

request({

method: 'GET',

url: url,

json: true,

headers: {

'User-Agent': 'request'

}

}, function(err, resp, body){

if(err){

reject(err);

} else {

resolve(body);

}

});

}

// create a new "async" function so we can use the "await" keyword

async function printPublicGists(){

// "await" resolution or rejection of the promise

// use try/catch for error handling

try {

var gists = await get('https://api.github.com/gists/public');

// now you can write this like syncronous code!

gists.forEach(function(gist){

console.log(gist.description);

});

} catch (e) {

// promise was rejected and we can handle errors with try/catch!

}

printPublicGists();

Now if we transpile and run again, it should work fine.

Recent Posts

Recent Comments

Archives

Categories

Event Loop

Futures / Tasks

Coroutines

Fitting Event Loop and Future/Task Together

Generators

Coroutines

Async I/O and the asyncio module

Native Coroutines and async/await

Native vs Generator Based Coroutines: Interoperability

Install Babel and Plugins

Configuring Babel

Transpile & Run

Async I/O and the `asyncio` module

Native Coroutines and `async`/`await`