Skip to content

refactor: drop scipy from the course quality API - #39410

Merged
ormsbee merged 1 commit into
openedx:masterfrom
ormsbee:ormsbee/remove-scipy-stats-mode
Nov 20, 2026
Merged

ormsbee merged 1 commit into
openedx:masterfrom
ormsbee:ormsbee/remove-scipy-stats-mode

Conversation

@ormsbee

@ormsbee ormsbee commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

The course quality API (GET /api/courses/v1/quality/{course_id}/) was the only place edx-platform imported scipy, and it only used scipy.stats.mode(). This swaps it for min(statistics.multimode(data)) from the standard library.

scipy.stats.mode breaks ties by returning the smallest of the most common values, and taking min() of multimode() keeps that behavior. The existing test_quality.py expectations rely on it: the subsection block-type counts are a 1/2 tie, and the expected mode is 1.

The JSON response doesn't change. The old code returned a numpy scalar, which DRF serialized with .tolist(). The new code returns the same value as a plain Python int or float.

This doesn't remove scipy from the requirements. It's still installed through chem, and capa problem code expects it to be importable. But I'm planning to remove it from chem via:

And I hope the recent removal of local Codejail means that we can get rid of it from platform install entirely (and leave it to the codejail service):

Testing instructions

  • pytest cms/djangoapps/contentstore/api/tests/test_quality.py
  • I checked min(statistics.multimode(x)) against scipy.stats.mode(x, axis=None)[0] (scipy 1.17.1) on about 20k random int and float lists, plus some hand-picked ties. They matched every time.

🤖 Generated with Claude Code

The course quality view only used scipy for stats.mode(). Replace it
with min(statistics.multimode(...)) from the standard library, which
keeps the same tie-breaking (the smallest of the most common values).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ormsbee
ormsbee marked this pull request as ready for review November 20, 2026 02:19

@kdmccormick kdmccormick left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 🪓

@ormsbee
ormsbee enabled auto-merge (rebase) November 20, 2026 02:48
@ormsbee
ormsbee merged commit acc27d1 into openedx:master Nov 20, 2026
42 checks passed
ormsbee added a commit to openedx/openedx-chem that referenced this pull request Nov 20, 2026
chemcalc used nltk's context-free grammar chart parser to parse chemical
formulas. Importing nltk is expensive because it also imports
scipy.stats, and the LMS imports chemcalc in every worker process. Each
codejail run that uses chemcalc also pays for it in a fresh process.

The grammar is now written in pyparsing, which chemcalc already used for
tokenizing. Its parse actions build the same trees the old parser
produced after its cleanup pass. A small Tree class replaces
nltk.tree.Tree with the same equality and ordering, so the rendering and
comparison code is unchanged.

The parse actions take pyparsing's full (string, location, tokens)
arguments. With fewer, pyparsing works out how many an action takes by
trial and error the first time it's called, and that isn't thread-safe:
threads making the first parse in a process at the same time could
leave an action raising TypeError on every later call. A problem with
two chemical equation inputs asks for both previews as the page loads,
so a threaded server can hit this.

scipy is no longer a dependency either. Nothing in chem has imported it
since 2013, when an unused import was removed. nltk imports scipy.stats
only when scipy is installed, so it was the one thing still using it.
Anything that got scipy only by installing chem now has to list it
itself. openedx-platform was in that position, but it no longer imports
scipy as of openedx/openedx-platform#39410. Its sandbox requirements
list scipy directly, so problem authors' code in codejail still has it.

Behavior changes:

* Input containing { or } is now a parse error. nltk raised ValueError
  for it, so render_to_html and chemical_equations_equal crashed
  instead of reporting invalid input.
* Brackets nested more than 20 deep (MAX_NESTING) are a parse error.
  The old parser accepted about 245 levels, then raised RecursionError.
  The limit is checked before parsing, so it doesn't depend on how deep
  the caller's stack already is.

Baseline from chem 2.0.1:

Course content written while chemcalc parsed with nltk depends on how
it rendered and compared formulas. chemcalc_baseline.json records what
chem 2.0.1, the last nltk release, returned for about 500 expressions,
630 expression pairs and 320 equation pairs. test_chemcalc_baseline
checks the current code against it.

The baseline stores public results (render_to_html,
divide_chemical_expression and chemical_equations_equal), not parse
trees, so internals can change as long as rendering and grading don't.
The pairs include near misses, where one element, subscript, charge,
bracket or phase differs. Those catch structural differences that a
rendered string can't.

generate_chemcalc_baseline.py made the file. It refuses to run unless
it is importing chem 2.0.1, so the baseline can't be regenerated from
the current code by mistake.

Compatibility quirks:

Some behavior the baseline pins is probably wrong, but is kept so that
existing answers are rendered and graded the same way:

* charge-starting-with-1: ^12+ is read as ^+.
* space-inside-number: "1 0H2" is read as 10H2.
* float-factors: ratios are worked out with floats, so H2O / 3H2O isn't
  exactly 1/3, and proportional fractional counts can compare unequal.
* zero-count-raises: a count of 0 in the second expression raises
  ZeroDivisionError, even from chemical_equations_equal.

Each quirk has its own test in test_chemcalc_quirks.py, and a comment
with the same name in chemcalc.py where it comes from.

Baseline entries whose results depend on a quirk are tagged with it and
checked in a separate class, TestChemcalcBaselineQuirks, so a mismatch
in TestChemcalcBaseline always means a regression. For the first and
third quirks, the generator finds those entries by rerunning chem 2.0.1
with the quirk fixed. For the other two, it uses the input or result,
which is exact for those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ormsbee added a commit to openedx/openedx-chem that referenced this pull request Nov 20, 2026
Importing nltk is slow, mostly because it imports scipy.stats. Every
LMS worker pays that cost, and so does each codejail run that uses
chemcalc.

The grammar is now written in pyparsing, which chemcalc already used
for tokenizing. Parse actions build the same trees as before, in a
small Tree class that keeps nltk.tree.Tree's equality and ordering.
The rendering and comparison code is unchanged.

Parse actions take all three of pyparsing's arguments. With fewer,
pyparsing works out the count on the first call, which isn't
thread-safe and can leave an action raising TypeError from then on.

Also drops scipy, which chem hasn't imported since 2013.
openedx-platform stopped importing it in
openedx/openedx-platform#39410.

Behavior changes:

* { or } in the input is a parse error. It used to raise ValueError.
* Brackets nested more than 20 deep (MAX_NESTING) are a parse error.
  The old parser took about 245 levels, then raised RecursionError.

Tests:

* chemcalc_baseline.json records what chem 2.0.1, the last nltk
  release, returned for about 1,460 expressions and pairs.
  test_chemcalc_baseline checks the current code against it.
* Four quirks that are probably bugs are kept so that grading doesn't
  change: charge-starting-with-1, space-inside-number, float-factors
  and zero-count-raises. Each has a test in test_chemcalc_quirks.py.
  Baseline entries that depend on one are tagged and checked
  separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants