Implement CPython 3.14 subinterpreter APIs - #8605
Conversation
Apply the patch baseline for the low-level subinterpreter modules:
- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package
Assisted-by: Claude:claude-opus-5
Argument matching: - Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and route every module function of both modules through it. This enforces keyword-only parameters, rejects unexpected keywords, accepts keyword forms that were previously positional-only (`whence(id=)`, `get_config(id=)`, `set___main___attrs(id=, updates=)`, `capture_exception(exc=)`, `_register_end_types(send=, recv=)`), and reproduces the arity messages. - `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None, and `_PyArg_BadArgument` renders None as "None" rather than "NoneType". - `new_config` takes at most one positional `str`; `list_all` of `_interpchannels` is argument-less. - Convert arguments in `kwlist` order, so an argument's own error is raised ahead of the checks that follow it. _interpreters: - Add the `CrossInterpreterBufferView` type and build the received memoryview on it, keeping the sending interpreter's exporter out of the destination. - Build `excinfo.errdisplay` from `traceback.TracebackException`. Feature flags: - `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`. - `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter without `allow_threads`. Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings and comments. Assisted-by: Claude:claude-opus-5
…t CPython 3.14 verify_stateless_function follows _PyFunction_VerifyStateless: it rejects non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__, and code whose LOAD_GLOBAL names are held by the function's globals or absent from its builtins. code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator, coroutine and async-generator code is rejected up front, the instruction walk skips inline caches and maps specialized and instrumented opcodes back, and the LOAD_CONST preceding each RETURN_VALUE is compared against the index of None in co_consts. ExcInfo::capture keeps an empty exception message instead of dropping it, and builds `formatted` as `module.qualname: msg`, leaving out the builtins and __main__ modules. _interpreters.call packs the callable through _PyFunction_GetXIData before pickle and re-raises the stateless-check failure when both fail. A func, args or kwargs that cannot be rebuilt in the target, and a result without cross-interpreter data, now raise NotShareableError with the snapshot as the cause rather than being returned as excinfo. _interpreters.capture_exception() without an argument returns None. pickle_loads and _PyFunction_GetXIData attach the underlying failure as __cause__. channel_send converts the object to cross-interpreter data after the channel's `closing` check. ChannelID rich comparison returns the result of comparing its id with the other number instead of coercing that result to bool. PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is required, not 'X'" for an object that is not a buffer. Assisted-by: Claude:claude-opus-5
Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.
Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.
Guard each tuple item conversion with with_recursion("while sharing a
tuple").
Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.
Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.
parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.
Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.
Assisted-by: Claude:claude-opus-5
_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.
channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.
Assisted-by: Claude:claude-opus-5
_interpqueues holds a process-wide queue table and exposes create, destroy, list_all, put, get, bind, release, get_maxsize, get_queue_defaults, is_full, get_count and _register_heap_types, raising QueueError and QueueNotFoundError. QueueEmpty and QueueFull, which subclass queue.Empty and queue.Full, are registered per interpreter by concurrent.interpreters._queues. Cross-interpreter data carries a queue as a refcounted queue ID that binds when the data is captured and releases when it is dropped, and is_shareable reports a registered Queue as shareable. Queue and channel items owned by an interpreter are now cleared from Interpreter::finalize, after module finalization, rather than from destroy_owned_interpreter, so values sent by an atexit callback also become unbound. is_shareable gates memoryview on the same registration flag as the getdata lookup. Assisted-by: Claude:claude-opus-5
Interpreter, ExecutionFailed and the queue aliases, copied verbatim from CPython 3.14. Assisted-by: Claude:claude-opus-5
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [ ] lib: cpython/Lib/difflib.py dependencies:
dependent tests: (243 tests)
[x] lib: cpython/Lib/io.py dependencies:
dependent tests: (108 tests)
[ ] lib: cpython/Lib/concurrent dependencies:
dependent tests: (17 tests)
[ ] lib: cpython/Lib/glob.py dependencies:
dependent tests: (58 tests)
[ ] lib: cpython/Lib/email dependencies:
dependent tests: (53 tests)
[x] lib: cpython/Lib/ast.py dependencies:
dependent tests: (149 tests)
[ ] lib: cpython/Lib/compression dependencies:
dependent tests: (100 tests)
[ ] lib: cpython/Lib/json dependencies:
dependent tests: (13 tests)
[ ] lib: cpython/Lib/mimetypes.py dependencies:
dependent tests: (42 tests)
[ ] lib: cpython/Lib/dataclasses.py dependencies:
dependent tests: (98 tests)
[ ] lib: cpython/Lib/html dependencies:
dependent tests: (14 tests)
[ ] lib: cpython/Lib/urllib dependencies:
dependent tests: (80 tests)
[x] lib: cpython/Lib/getopt.py dependencies:
dependent tests: (42 tests)
[ ] lib: cpython/Lib/csv.py dependencies:
dependent tests: (4 tests)
[x] lib: cpython/Lib/calendar.py dependencies:
dependent tests: (29 tests)
[ ] test: cpython/Lib/test/test_str.py (TODO: 5) dependencies: dependent tests: (no tests depend on str) [x] lib: cpython/Lib/datetime.py dependencies:
dependent tests: (67 tests)
[ ] lib: cpython/Lib/base64.py dependencies:
dependent tests: (54 tests)
[ ] test: cpython/Lib/test/test_unicodedata.py (TODO: 13) dependencies: dependent tests: (no tests depend on unicode) [x] lib: cpython/Lib/colorsys.py dependencies:
dependent tests: (1 tests)
[x] lib: cpython/Lib/os.py dependencies:
dependent tests: (187 tests)
[ ] test: cpython/Lib/test/test_isinstance.py dependencies: dependent tests: (no tests depend on isinstance) [ ] lib: cpython/Lib/socket.py dependencies:
dependent tests: (101 tests)
[ ] test: cpython/Lib/test/test_exceptions.py (TODO: 22) dependencies: dependent tests: (no tests depend on exception) [ ] lib: cpython/Lib/random.py dependencies:
dependent tests: (140 tests)
[ ] lib: cpython/Lib/collections dependencies:
dependent tests: (331 tests)
[ ] test: cpython/Lib/test/test_syntax.py (TODO: 65) dependencies: dependent tests: (no tests depend on syntax) [ ] lib: cpython/Lib/argparse.py dependencies:
dependent tests: (258 tests)
[x] lib: cpython/Lib/weakref.py dependencies:
dependent tests: (222 tests)
[ ] test: cpython/Lib/test/test_dict.py (TODO: 4) dependencies: dependent tests: (no tests depend on dict) [ ] lib: cpython/Lib/pydoc.py dependencies:
dependent tests: (5 tests)
[x] lib: cpython/Lib/shlex.py dependencies:
dependent tests: (10 tests)
[x] lib: cpython/Lib/copy.py dependencies:
dependent tests: (321 tests)
[ ] test: cpython/Lib/test/test_atexit.py (TODO: 1) dependencies: dependent tests: (177 tests)
[x] lib: cpython/Lib/struct.py dependencies:
dependent tests: (179 tests)
[ ] lib: cpython/Lib/locale.py dependencies:
dependent tests: (104 tests)
[ ] test: cpython/Lib/test/test_list.py (TODO: 3) dependencies: dependent tests: (no tests depend on list) [ ] lib: cpython/Lib/test/support dependencies:
dependent tests: (2 tests)
[x] lib: cpython/Lib/code.py dependencies:
dependent tests: (2 tests) [x] lib: cpython/Lib/compileall.py dependencies:
dependent tests: (1 tests)
[ ] lib: cpython/Lib/string dependencies:
dependent tests: (75 tests)
[x] lib: cpython/Lib/dis.py dependencies:
dependent tests: (77 tests)
[ ] lib: cpython/Lib/ntpath.py dependencies:
dependent tests: (53 tests)
[ ] lib: cpython/Lib/configparser.py dependencies:
dependent tests: (2 tests)
[ ] test: cpython/Lib/test/test_compile.py dependencies: dependent tests: (no tests depend on compile) [ ] test: cpython/Lib/test/test_bigmem.py (TODO: 1) dependencies: dependent tests: (no tests depend on bigmem) [x] lib: cpython/Lib/pickle.py dependencies:
dependent tests: (102 tests)
[ ] test: cpython/Lib/test/test_array.py (TODO: 3) dependencies: dependent tests: (102 tests)
[ ] test: cpython/Lib/test/test_math.py (TODO: 1) dependencies: dependent tests: (245 tests)
[ ] test: cpython/Lib/test/test_enumerate.py dependencies: dependent tests: (no tests depend on enumerate) [ ] test: cpython/Lib/test/test_descr.py (TODO: 32) dependencies: dependent tests: (no tests depend on descr) [ ] lib: cpython/Lib/inspect.py dependencies:
dependent tests: (96 tests)
[x] test: cpython/Lib/test/test_thread.py (TODO: 3) dependencies: dependent tests: (14 tests) [x] lib: cpython/Lib/py_compile.py dependencies:
dependent tests: (44 tests)
[x] lib: cpython/Lib/codeop.py dependencies:
dependent tests: (104 tests)
[x] test: cpython/Lib/test/test_marshal.py (TODO: 5) dependencies: dependent tests: (25 tests)
Legend:
|
Summary
_interpreters,_interpchannels, and_interpqueuesconcurrent.interpretershigh-level API, including queuesCPython compatibility review
The implementation was compared against the CPython 3.14 sources, including argument parsing, error shapes, shareability checks, interpreter ID handling, channel defaults, queue semantics, and interpreter finalization order.
In particular, channel and queue cleanup now happens after module and
atexitfinalization. This preserves values written by finalizers and exposes them asUNBOUND, matching CPython behavior after the owning interpreter exits. The high-levelconcurrent.interpretersfiles are synchronized with CPython 3.14.7.Validation
prek run --from-ref 9edd97ec4 --to-ref HEADcargo clippycargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi(cd crates/capi && cargo test)— 102 passedUNBOUNDhandling, and interpreter destructionupstream/mainAI assistance
Claude (claude-opus-5) assisted with implementation. OpenAI Codex (GPT-5) assisted with CPython source comparison, lifecycle review, validation, and PR preparation. The corresponding implementation commits include the required
Assisted-bytrailers.