Skip to content

Add max_tasks_per_child param to parallel - #961

Merged
jph00 merged 1 commit into
mainfrom
fresh-workers
Nov 5, 2026
Merged

jph00 merged 1 commit into
mainfrom
fresh-workers

Conversation

@jackhogan

Copy link
Copy Markdown
Contributor

Setting max_tasks_per_child=1 ensures every item in ProcessPoolExecutor runs in a fresh process.

Currently, a pool reuses its workers, so whatever one item changes in the worker's process is still there for the next item; e.g. a module it imported, an attribute it set on a class, os.environ, a global cache. Any f that relies on starting clean gets different results depending on which items share a worker. For example, in nbdev, each notebook is one item, so a notebook's imports and monkeypatches leak into the next notebook that the same worker runs, and nbdev-test --save stores outputs built from another notebook's state.

The argument needs Python 3.11 and a non-fork start method, so when it is set and method isn't, parallel uses forkserver (spawn on Windows). f then has to be importable, not defined in __main__. Default is None, so existing calls are unaffected.

@jackhogan jackhogan self-assigned this Nov 5, 2026
@jackhogan jackhogan added the enhancement New feature or request label Nov 5, 2026
@jackhogan
jackhogan requested a review from jph00 November 5, 2026 16:18
@jph00
jph00 merged commit f08ec21 into main Nov 5, 2026
4 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants