Repository navigation
fix: bound retries and time limits on content/search Celery tasks [backport to verawood] - #39414
blarghmatey wants to merge 1 commit into
Conversation
…enedx#39042) Every task in the module was declared with autoretry_for and nothing else, so each inherited Celery's defaults: 3 retries, a flat unjittered 180s apart, and no time limit. rebuild_index_incremental was the only exception. That matters most for the tasks handlers.py runs with .apply() so the index is updated before the Authoring MFE refetches. Under .apply() the retry is eager (Task.apply re-invokes the signature immediately, ignoring the countdown), so the whole attempt sequence runs inline in the CMS web worker, inside a Studio HTTP request. On mitxonline a single upsert_library_block_index_doc reached 95-115s against a 60s gateway timeout, surfacing as a 504 on V2 library component creation that leaves an orphaned empty component (mitodl/hq#13014). Three policies, applied by call pattern: request-path tasks the handlers run inline get 2 retries and a 10s soft limit; background tasks reached only via .delay() get 3 retries with jittered backoff capped at 60s; whole-context walks that load every block in a course or library get a 30 minute budget. rebuild_index_incremental keeps no time limit, since a full rebuild runs for hours, but gains a backoff cap and jitter. Note the split in what each knob covers: soft_time_limit and time_limit are enforced by the worker's prefork pool and do nothing under .apply(), so on the inline path only max_retries bounds the wall time. The unbounded poll loop in api._wait_for_meili_task is the other half of that path and is left alone here. Backport to release/verawood. Verawood still decorates every task with @set_code_owner_attribute (master dropped it in openedx#39047), so the decorator is kept on each task. upsert_library_block_index_doc keeps its verawood signature, which has no wait_timeout argument. (cherry picked from commit f73b7ae) Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FGZebGbsk1ihYzf2PamV3P
|
Thanks for the pull request, @blarghmatey! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
Description
Backport of #39042 to
release/verawood. Every task incontent/search/tasks.pyinherits Celery's retry defaults (3 retries, a flat unjittered 180s apart, no time limit), and the tasks the handlers run with.apply()replay those attempts inline inside a Studio request. This applies the same three retry and time-limit policies as #39042. See #39042 for the full description.It is a cherry-pick of the merge commit with two adaptations:
@set_code_owner_attribute(master removed it in chore: remove set_code_owner_attribute decorator from Celery tasks #39047), so each task keeps it.upsert_library_block_index_dockeeps its Verawood signature. Thewait_timeoutargument it has on master is not on this branch.The handlers on Verawood dispatch the same tasks with
.apply()and.delay()as master does, so each task gets the same policy as on master.Testing instructions
Same as #39042.
pytest openedx/core/djangoapps/content/searchpasses on this branch: 128 tests.https://claude.ai/code/session_01FGZebGbsk1ihYzf2PamV3P