Bug report
Bug description:
On Windows, BaseProactorEventLoop wakes itself through a self-pipe, which is a loopback TCP socketpair created by socket.socketpair(). If that connection reaches a clean EOF while the loop is running -- for example the OS tears the idle loopback connection down across a power/session state change -- the loop spins at 100% of one core forever, with no exception raised and nothing logged.
Lib/asyncio/proactor_events.py:
def _loop_self_reading(self, f=None):
try:
if f is not None:
f.result() # may raise
if self._self_reading_future is not f:
return
f = self._proactor.recv(self._ssock, 4096)
except exceptions.CancelledError:
return
except (SystemExit, KeyboardInterrupt):
raise
except BaseException as exc:
self.call_exception_handler({
'message': 'Error on reading from the event loop self pipe',
'exception': exc,
'loop': self,
})
else:
self._self_reading_future = f
f.add_done_callback(self._loop_self_reading)
At EOF, f.result() returns b''. That is not an exception, so no except branch runs and control falls through to else, which re-arms recv() on a socket that is at EOF. That recv completes immediately, whose callback is _loop_self_reading, which re-arms again -- a tight infinite loop.
The loop is otherwise completely idle. py-spy shows MainThread as active+gil with:
_loop_self_reading (asyncio/proactor_events.py:804)
recv (asyncio/windows_events.py:493)
_register (asyncio/windows_events.py:720)
and _run_once locals sched_count: 0, ntodo: 1, timeout: None -- nothing is scheduled, one handle re-runs forever.
Related precedent
IocpProactor.recv() in Lib/asyncio/windows_events.py returns b'' in this situation, while recv_into() was changed to return 0 in bpo-41467 -- "asyncio: recv_into() must not return b'' if the socket/pipe is closed". The same reasoning applies to recv(). Independently of that, _loop_self_reading has no EOF handling at all, so a dead self-pipe can never be recovered from.
Reproducer
Deterministic -- 12/12 runs on both interpreters tested.
import asyncio
import socket
import time
async def main():
loop = asyncio.get_running_loop()
# Let the loop arm its self-pipe read first.
await asyncio.sleep(0.1)
# Graceful half-close: the read half sees a clean EOF, which is what an OS
# teardown of the loopback connection looks like.
loop._csock.shutdown(socket.SHUT_WR)
start = time.process_time()
await asyncio.sleep(3)
print(f"CPU consumed while sleeping 3s: {time.process_time() - start:.2f}s")
asyncio.run(main()) # Windows default is ProactorEventLoop
Expected: roughly 0.00s -- the process is asleep.
Actual: roughly 2.90s -- a full core burned for a three second sleep.
Important: it must be a graceful half-close. An abortive close() surfaces as an exception, which the existing except BaseException branch handles, so it does not reproduce reliably (~80% of runs, and only when the recv is issued after the socket is already gone).
How this was hit in production
Three unrelated long-running Python programs on the same Windows 11 machine -- two independent MCP servers plus a minimal control program written purely to isolate this -- all began pinning a core at the same instant, roughly 69 minutes into their lifetime. They had accumulated near-identical CPU time (5465s, 5465s, 5485.7s, 5482.7s over ~9638s of uptime), which is what pointed at a shared external trigger rather than three independent bugs.
The trigger turned out to be waking the screen after the machine had been locked -- not going idle. With the display off the machine was silent; moving the mouse lit the screen and every asyncio process immediately pinned a core.
Socket state confirms the mechanism:
|
sockets |
CPU |
| healthy process |
127.0.0.1:A->B ESTABLISHED + 127.0.0.1:B->A ESTABLISHED (plus vestigial 0.0.0.0:A BOUND) |
0% |
| spinning process |
both ESTABLISHED halves gone, only BOUND left |
97-99% |
Nothing is logged, no exception is raised, and the affected processes never recover. From a user's point of view the machine simply starts running hot after every unlock, with several cores pinned, and the only remedy is killing the processes.
The machine has no Modern Standby (powercfg /a reports S3 only), so this is an ordinary session/display transition on a fully awake system, not a sleep/resume cycle.
Environment
- Windows 11
- Python 3.12.10 (
tags/v3.12.10:0cc8128, MSC v.1943 64 bit) -- affected
- Python 3.13.12 (
main, MSC v.1944 64 bit) -- affected
- The code path is unchanged on
main as of this writing.
Suggested fix
Treat a zero-length result as the self-pipe being gone, and rebuild it instead of re-arming a read that can never block again:
def _loop_self_reading(self, f=None):
try:
if f is not None:
data = f.result() # may raise
if not data:
# The self-pipe reached EOF: the socketpair is gone (this can
# happen when the OS tears down the loopback connection across a
# power or session state change). Re-arming here would spin the
# CPU forever, so rebuild the pipe instead.
self._self_reading_future = None
self._close_self_pipe()
self._make_self_pipe()
return
...
A more conservative variant would be to leave the rebuild out and only stop re-arming, which at least turns an invisible 100% CPU spin into a loop that can no longer be woken -- but rebuilding keeps the loop functional, which seems strictly better.
Making IocpProactor.recv() return a length rather than b'', matching what bpo-41467 did for recv_into(), would additionally make this class of bug harder to reintroduce.
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Windows
Bug report
Bug description:
On Windows,
BaseProactorEventLoopwakes itself through a self-pipe, which is a loopback TCP socketpair created bysocket.socketpair(). If that connection reaches a clean EOF while the loop is running -- for example the OS tears the idle loopback connection down across a power/session state change -- the loop spins at 100% of one core forever, with no exception raised and nothing logged.Lib/asyncio/proactor_events.py:At EOF,
f.result()returnsb''. That is not an exception, so noexceptbranch runs and control falls through toelse, which re-armsrecv()on a socket that is at EOF. Thatrecvcompletes immediately, whose callback is_loop_self_reading, which re-arms again -- a tight infinite loop.The loop is otherwise completely idle.
py-spyshowsMainThreadasactive+gilwith:and
_run_oncelocalssched_count: 0,ntodo: 1,timeout: None-- nothing is scheduled, one handle re-runs forever.Related precedent
IocpProactor.recv()inLib/asyncio/windows_events.pyreturnsb''in this situation, whilerecv_into()was changed to return0in bpo-41467 -- "asyncio: recv_into() must not return b'' if the socket/pipe is closed". The same reasoning applies torecv(). Independently of that,_loop_self_readinghas no EOF handling at all, so a dead self-pipe can never be recovered from.Reproducer
Deterministic -- 12/12 runs on both interpreters tested.
Expected: roughly
0.00s-- the process is asleep.Actual: roughly
2.90s-- a full core burned for a three second sleep.Important: it must be a graceful half-close. An abortive
close()surfaces as an exception, which the existingexcept BaseExceptionbranch handles, so it does not reproduce reliably (~80% of runs, and only when therecvis issued after the socket is already gone).How this was hit in production
Three unrelated long-running Python programs on the same Windows 11 machine -- two independent MCP servers plus a minimal control program written purely to isolate this -- all began pinning a core at the same instant, roughly 69 minutes into their lifetime. They had accumulated near-identical CPU time (5465s, 5465s, 5485.7s, 5482.7s over ~9638s of uptime), which is what pointed at a shared external trigger rather than three independent bugs.
The trigger turned out to be waking the screen after the machine had been locked -- not going idle. With the display off the machine was silent; moving the mouse lit the screen and every asyncio process immediately pinned a core.
Socket state confirms the mechanism:
127.0.0.1:A->B ESTABLISHED+127.0.0.1:B->A ESTABLISHED(plus vestigial0.0.0.0:A BOUND)BOUNDleftNothing is logged, no exception is raised, and the affected processes never recover. From a user's point of view the machine simply starts running hot after every unlock, with several cores pinned, and the only remedy is killing the processes.
The machine has no Modern Standby (
powercfg /areports S3 only), so this is an ordinary session/display transition on a fully awake system, not a sleep/resume cycle.Environment
tags/v3.12.10:0cc8128, MSC v.1943 64 bit) -- affectedmain, MSC v.1944 64 bit) -- affectedmainas of this writing.Suggested fix
Treat a zero-length result as the self-pipe being gone, and rebuild it instead of re-arming a read that can never block again:
A more conservative variant would be to leave the rebuild out and only stop re-arming, which at least turns an invisible 100% CPU spin into a loop that can no longer be woken -- but rebuilding keeps the loop functional, which seems strictly better.
Making
IocpProactor.recv()return a length rather thanb'', matching what bpo-41467 did forrecv_into(), would additionally make this class of bug harder to reintroduce.CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Windows