github twitter keybase instagram spotify

asyncio: We Did It Wrong

| updated

These series of posts are an accompaniment to a couple of talks given: Advanced asyncio: Solving Real-world Production Problems at EuroPython 2019 & PyCon US 2019; and asyncio in Practice: We Did It Wrong at EuroPython 2018 and PiterPy 2018. Slides can be found here and the PyCon US 2019 talk recording here.

Example code can be found on GitHub. All code in this series of blog posts is licensed under MIT.

asyncio. “The concurrent Python programmer’s dream”, the answer to everyone’s asynchronous prayers. The asyncio module has various layers of abstraction allowing developers as much control as they need and are comfortable with.

Simple “Hello, World”-like examples show how it can be so effortless; look at that!

Python 3.7.0 (default, Jul  6 2018, 11:30:06)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio, datetime
>>> async def hello():
...   print(f'[{}] Hello...')
...   await asyncio.sleep(1)  # some I/O-intensive work
...   print(f'[{}] ...World!')
[2018-07-07 10:45:55.559856] Hello...
[2018-07-07 10:45:56.568737] ...World!

But it’s easy to get lulled into a false sense of security. This ain’t helpful. We’re led to believe that we’re able to do a lot with the structured async/await API layer. Some tutorials, while great for the developer getting their toes wet, try to illustrate real world examples, but are actually just beefed-up “hello, world”s. Some even misuse parts of asyncio’s interface, allowing one to easily fall into the depths of callback hell. Some get you easily up and running with asyncio, but then you may not realize it’s not correct or exactly what you want, or only gets you part of the way there. While there are tutorials that do to improve upon the basic Hello, World “use” case, often times, it doesn’t go far enough. It’s often still just a web crawler. I’m not sure about others, but I’m not building web crawlers at Spotify.

It’s not the fault of anyone though; asynchronous programming is difficult. Whether you use asyncio, Twisted, Tornado, or Golang, Erlang, Haskell, whatever, it’s just difficult. I myself have even fallen into this false sense of ease that the asyncio community builds where once I import it, everything will just fall into place as they should. I do believe asyncio is quite user-friendly, but I did underestimate the inherit complexity concurrent programming brings.

The past couple of services we built at Spotify were perfect use cases for asyncio: a chaos-monkey-like service for restarting instances in Google Cloud, and an event-driven host name generation service for DNS. Sure, we needed to make a lot of HTTP requests that should be non-blocking much like web crawlers. But these services also had to react to messages from a pub/sub, measure the progress of actions initiated from those messages, handle any incomplete actions or other external errors, take care of pub/sub message lease management, measure SLIs, and send metrics. And we needed to use non-asyncio-friendly dependencies as well. This quickly got difficult.

The Tutorial: Mayhem Mandrill

So having lived through that and survived, allow me to provide you a real-world example that actually comes from the real world. As I mentioned, one of the asyncio services we built is similar to a chaos monkey to do periodic hard restarts of our entire fleet of instances. We’ll build a simplified version, dubbing it “Mayhem Mandrill” which will listen for a pub/sub message as a trigger to go ahead and restart a host based off of that message. As we build this service, I’ll point out potential traps to avoid. This will essentially become the type of resource that past Lynn would have wanted a year or two ago.

Tutorial Contents

Part 0: Initial Setup
Part 1: True Concurrency
Part 2: Graceful Shutdowns
Part 3: Exception Handling
Part 4: Working with Synchronous & Threaded Code
Part 5: Testing asyncio Code
Part 6: Debugging asyncio Code
Part 7: Profiling asyncio Code

Has this article been helpful for you? Consider expressing your gratitude!
Need some help? I'm available for tutoring, mentoring, and interview prep!

comments powered by Disqus