.NET async/await in a single picture

It is said that picture is worth a thousand words, and I agree. That’s why I like preparing technical drawings to explain various concepts. So, here it is – a short story of how async/await works in .NET.

The main power behind async/await is that while we “await” on an ongoing I/O operation, the calling thread may be released for doing other work. And this provides a great thread re-usability. Thus, better scalability – much smaller number of threads is able to handle the same amount of operations comparing to asynchronous/waiting approach.

The main role here plays so-called overlapped I/O (in case of Windows) which allows to asynchronously delegate the I/O operation to the operating system, and only after completion the provided callback will notify us about the result. The main workforce here is so-called I/O completion port (IOCP).

I/O completion ports are a very efficient mechanisms for asynchronous I/O in Windows, allowing to observe huge amount of ongoing operations by only one port. So indeed, in case of .NET all this is baked by only a single IOCP. Then, a few I/O completion threads (managed by a ThreadPool) are observing this single IOCP. Typical number of IOCP threads is about equal to number of logical processors, but ThreadPool may create up to 1000 by default (we can change this maximum).

The callback of I/O operation is executed on one of the IOCP threads – it sets the result of the operation and decides whether to:

• execute continuation in-place on the IOCP thread itself – which is beneficial because it does not incur any more context switches and avoids cache trashing, but obviously has the risk of polluting (an important) IOCP thread with the user code (which may lead to threads starvation).
• or schedule it back to “the other place” – which may be pointed by SynchronizationContext/TaskScheduler, so quite typically it will be just queued for execution by worker threads

Typically most continuations are scheduled to the worker threads. One of the most significant exceptions are HttpClient continuations on Windows (since .NET Core 2.1).

Note. In case of Linux epoll mechanism is used instead of IOCPs. There is/are epoll thread(s), listening to events, that schedule the continuation to the worker threads (no inlining may happen). And ongoing work to rebuild ThreadPool for Linux using AIO and io_uring is being made.

All the dots are connected by a state machine instance, keeping track of the status, continuation to execute, the whole context ans so on.

Drawing this reminded me famous “There is no thread” article from Stephen Cleary where he explains that there are no “user code executing” threads consumed when an asynchronous I/O operations is performed. And indeed it is true, although as people also noticed in the comments, there are IOCP threads blocked for processing I/O completion port notifications – so, there is (almost) no thread.

Similar picture about more generic relationship between ThreadPool and I/O completion port drawn pretty big attention yesterday, so you may be also interested:

.NET ThreadPool and an overlapped I/O – this will be heavily reused in explaining async/await too pic.twitter.com/kaC2FTxbkI

— Konrad Kokosa (@konradkokosa) May 25, 2020

All this is just a work for a much bigger project, which is https://asyncexpert.com/ on-line course about asynchronous and concurrent programming in .NET. If you found it interesting, stay tuned by subscribing to the newsletter on the above-mentioned page!