Async Python Tricks

3 minutes read
programming • tutorials
Linux • Python • Asynchronous Python

Starting an Async Program

In Python 3.7 and above, you can use asyncio.run to easily start your async program:

async def async_main():
    await asyncio.sleep(1)
    print("Async")

def main():
    asyncio.run(async_main(), debug=True)

if __name__ == "__main__":
    main()

Alternatives

To run an async program in other ways, you need to use the loop directly.

loop = asyncio.get_running_loop()
  1. loop.run_until_complete if you have one function you need to run
  2. loop.run_forever if you never want to close the async loop

Running blocking synchronous functions after Python 3.9

You should avoid calling blocking I/O functions inside async code. Instead, try to run the code using asyncio.to_thread.

await asyncio.to_thread(blocking_function)

This is available in Python 3.9 and above.

If your blocking functions share state or require some sort of synchronization, you will have to make sure that running them concurrently on multiple threads does not affect them.

Running all blocking code in a single thread

To avoid having to synchronize the threads started by asyncio.to_thread, you can use run_in_executor with a thread pool with a single thread (max_workers=1) so your blocking tasks are all run in the same thread.

Instead of

await asyncio.to_thread(blocking_function)

use:

async_executor = ThreadPoolExecutor(
  max_workers=1, thread_name_prefix="async_worker_thread"
)
await asyncio.get_running_loop().run_in_executor(
  async_executor, blocking_function
)

For debugging purposes, give the executor a reasonable name related to the blocking functions you’ll run on it. Simply replace "async_worker_thread".

This should work in versions before Python 3.9.

Internally, asyncio.to_thread uses loop.run_in_executor in a very similar way.

The problem is the blocking functions must wait for other blocking functions to finish. You could use multiple worker pools to group similar blocking tasks, e.g. one worker pool for redis connections.

Running blocking synchronous functions before Python 3.9

If you don’t have asyncio.to_thread or you don’t want to run tasks in a single thread, you can use run_in_executor with the default executor:

await loop.run_in_executor(executor=None, func=blocking_func)

Starting a coroutine from a thread

If you are running code in a thread but need to start an asynchronous function, you can use asyncio.run_coroutine_threadsafe to request that the async loop run the function.

You need access to the loop.

asyncio.run_coroutine_threadsafe(coro, loop)

This is available in Python 3.5.1 and above.

Wait for result

You can use the optional timeout parameter to specify how long to wait.

asyncio.run_coroutine_threadsafe(coro, loop).result(timeout)

Optimization: Use a faster async loop

Use uvloop instead of the built-in loop for a potential performance increase.

Call uvloop.install() before calling asyncio.run() or manually creating an asyncio event loop.

What can run asynchronously?

Built-in

External libraries

There is support for many types of network protocols, and other I/O bound tasks. Many projects have official libraries that support asyncio.

See awesome-asyncio.