Skip to content

Keep the Python object of a __namespace__ subclass of a .NET exception - #167

Open
quantconnect-kronos[bot] wants to merge 5 commits into
masterfrom
kronos/bug-namespaced-exception-subclass-construction
Open

quantconnect-kronos[bot] wants to merge 5 commits into
masterfrom
kronos/bug-namespaced-exception-subclass-construction

Conversation

@quantconnect-kronos

@quantconnect-kronos quantconnect-kronos Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

What changed and why

A Python subclass of a .NET exception that declares __namespace__ gets a generated .NET type, but ClassManager reflected that type as a plain exception (ExceptionClassObject) before checking whether it was a Python-derived type. So:

  • Its Python wrapper never stored itself in the generated object's __pyobj__ field (ClassDerivedObject.NewObjectToPython does that, ExceptionClassObject doesn't). When the generated constructor ran, PythonDerivedType.InvokeCtor found __pyobj__ empty, created a temporary wrapper, stored its pointer and released it. Later calls through the .NET object (an overridden virtual method, the object coming back from .NET through ClassDerivedObject.ToPython) used a freed object, and the process could crash with SIGSEGV. A subclass without __init__ crashed on construction.
  • A subclass whose __init__ doesn't call the base __init__ never ran a constructor, so field initializers stayed null and the constructor's defaults (such as the message) weren't set.

Generated exception types now get ExceptionClassDerivedObject, a ClassDerivedObject with the exception tp_repr, tp_str and __init__ handling (args and __cause__) of ExceptionClassObject. They behave like any other __namespace__ subclass: the wrapper is set on the object, the base parameterless constructor runs before __init__, and the Python object's lifetime follows the .NET object's. Reflected exceptions that aren't generated, and generated types that aren't exceptions, are handled as before.

With the wrapper now valid, a Python override of ToString on such a subclass is reached from .NET, so str() of the exception returns it, as it already does for ordinary __namespace__ subclasses.

Making these types derived types exposed three more problems, fixed here too:

  • Release of a chain of __namespace__ subclasses. Each generated level has its own Finalize, and each calls PythonDerivedType.PyFinalize on the same object. PyFinalize cleared only a copy of __pyobj__, so the same Python object was queued, and freed, once per level, and the process crashed. PyFinalize now clears the field once it has queued the object and skips an object already queued. Chains of __namespace__ subclasses of System.Object crashed the same way on master; this fixes them too. The generated Finalize now runs its base class finalizer before PyFinalize, so a .NET base finalizer that calls a virtual or abstract member still reaches the Python override. The innermost generated level queues the Python object once the finalizer chain has run, and the outer levels skip it.
  • Exception cycles. ClassDerivedObject.tp_clear is a no-op, since the .NET finalizer frees the object, so an exception referencing itself through args, __cause__, __context__ or __traceback__ was never collected. ExceptionClassDerivedObject.tp_clear clears those fields (the native exception base's tp_clear), and leaves the wrapper and its handle for the .NET object.
  • Python overrides in the exception slots. The exception slots that run managed code (tp_str, tp_repr and __init__) now run all their work inside a guard that raises an error to the caller, instead of letting it escape the slot and abort the process. tp_str runs ToString(), tp_repr reads Message, and __init__ runs Exceptions.SetArgsAndCause, which reads Message. The error is raised with its __cause__. A Python override of the member throws its PythonException directly, and that is raised as it is: its InnerException is the Python error's cause, not a wrapper. A member the Python subclass inherits, such as Exception.ToString, runs through reflection (PythonDerivedType.InvokeMethod), which wraps what it throws, including a Python get_Message override's error, in a TargetInvocationException; only that wrapper is unwrapped. A None returned by a Python ToString override reads as an empty message. The __init__ of reflected and generated exception types shares one guarded ExceptionClassObject.SetArgsAndCause. ClassBase.tp_str unwraps the cause the same way for other derived types; that predates this change and is str() of a Python subclass of a .NET type raises the cause of its ToString override's error instead of the error #170.

Tests

Both tests run their script in a subprocess, so a native crash fails the test instead of aborting the test run.

  • test_namespaced_exception_subclass_construction covers three __namespace__ subclasses of ExtendedException: one without __init__, one whose __init__ doesn't call the base (with a ToString override), and one calling super().__init__("message"). For each it checks that the field initializer ran, that the object comes back from a .NET List[Object] as the same Python object, that it can be raised and caught, its str() and args, and that collection afterwards is clean. Without the runtime change it fails with return code -11 (SIGSEGV).
  • test_namespaced_exception_subclass_release releases ten instances of a two-level __namespace__ exception chain and asserts that no instance of 30 exceptions referencing themselves through __cause__, __context__ or args survives collection. It also releases one- and two-level __namespace__ subclasses of new fixtures whose finalizer calls an abstract or virtual Callback, and asserts that the finalizer got the Python override's value. Releasing runs Python's and .NET's GC, then calls into a Python override from .NET 300 times, which drains the queue of Python objects to release. It fails with -11 without the PyFinalize change, leaves the cycles uncollected without tp_clear, and aborts (-6, NotImplementedException on the finalizer thread) with the previous finalizer order.
  • test_namespaced_exception_subclass_override_errors constructs an exception whose ToString override returns None and asserts str() is '' and repr() shows the Message override. It then makes both overrides raise ValueError("outer") from KeyError("inner") and asserts that str(), repr() and a constructor calling super().__init__() each raise that ValueError with its KeyError cause. It checks the same for str() of a subclass that inherits ToString and overrides only get_Message. Disabling any one of the three guards aborts the subprocess (-6). Restoring the cause unwrapping makes the KeyError escape, and dropping the TargetInvocationException unwrapping fails the inherited case. The earlier head d282ec1 aborts on the None case.

The CI steps, on Python 3.11 / .NET 10:

  • dotnet test src/embed_tests/ --filter "FullyQualifiedName!~SetPythonPath": 1049 passed, 8 skipped
  • dotnet test src/embed_tests/ --filter "FullyQualifiedName~SetPythonPath": 1 passed
  • pytest --runtime netcore tests: 489 passed, 23 skipped
  • dotnet test src/python_tests_runner/: 2 passed

Also tested end to end in Lean Python backtests:

  • An algorithm defining the same three kinds of subclass of System.ArgumentException constructs, round-trips and raises each, and fails if any result is wrong. It passed with this change: the base constructor's message was set, the same object came back from .NET, the ToString override was used, and args matched. The same algorithm crashed with a segmentation fault on current master.
  • A second algorithm releases a two-level chain, checks that self-referencing exceptions are collected (0 remaining), and checks that a raising ToString override surfaces as a catchable ValueError. It passed. A third algorithm defines an exception whose ToString override returns None, and whose ToString and get_Message overrides then raise a chained ValueError. It checks that str() is '', and that str(), repr() and construction each raise the ValueError with its KeyError cause. It passed.
  • A fourth algorithm, with this repository's test fixtures loaded, checks that a base finalizer reaches the Python Callback override of one- and two-level __namespace__ subclasses (abstract and virtual), and that str() of a subclass inheriting ToString raises its get_Message override's ValueError with its KeyError cause. It passed.

Related, predating this change: #170 (ClassBase.tp_str raises a chained error's cause) and #172 (a __namespace__ subclass of a __namespace__ subclass doesn't reach its parent's Python overrides from .NET).

A Python subclass of System.Exception, or of any exception type, that
declares __namespace__ gets a generated .NET type, but that type was
reflected as a plain exception rather than as a Python-derived type. Its
Python wrapper never stored itself on the .NET object, so the generated
constructor created a temporary wrapper, kept its pointer and released it:
later calls through the .NET object, such as an overridden virtual method
or the object coming back from .NET, used a freed object and could crash
the process. An __init__ that didn't call the base __init__ also left the
base constructor unrun, so field initializers stayed null.

Generated exception types now get the derived-type handling of ordinary
__namespace__ subclasses (the wrapper set on the object, the base
constructor run before __init__, the object's lifetime tied to the .NET
object) together with the exception repr, str and args.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:working develop: in progress kronos:review Request review labels Nov 17, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 17, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review November 17, 2026 18:56

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the root cause: ClassManager picked ExceptionClassObject before checking for the generated __pyobj__ field, so generated exception types never got ClassDerivedObject.NewObjectToPython or its tp_dealloc/tp_clear lifetime handling. The fix moves the derived-type check first, so it only affects generated types. For every other type, GetPyObjField returns null and the type falls through to the same branch as before. The new ExceptionClassDerivedObject keeps the exception tp_repr/tp_str and the args/__cause__ refresh after __init__. Slot lookup walks the implementing type's hierarchy from the most derived class, so the new static slots win over ClassBase's, as the comment says.

What I verified

  • CI is green on all jobs, including the Lean regression and unit test runs. The new test ran in every Python job (test_exceptions.py shows 30 passed, none skipped).

  • Locally, master installed in its own venv vs this branch (Python 3.11, .NET 10). The new test's script segfaults (rc 139) on master and passes on the branch. I also probed namespaced subclasses of System.Exception on both:

    • a plain one constructed with a message
    • a Python subclass of a namespaced one, raised and caught as itself
    • one thrown from .NET with ExceptionDispatchInfo.Throw and caught in Python as the same object
    • 20k raise/catch iterations with Python and .NET GC: stable memory
    • a traceback whose frame locals reference other objects: those objects are freed after GC

    All of these crash on master and pass on the branch.

  • End to end: I ran a Lean Python backtest with this build. Its algorithm defines __namespace__ subclasses of System.ArgumentException without __init__, with an __init__ that skips super() plus a ToString override, and with super().__init__("message"). For each, it checks the round-trip through List[Object], identity on raise, str() and args. All checks passed.

Notes (non-blocking, not caused by this change)

  • A Python __str__/__repr__ on a namespaced exception subclass is ignored: str() returns the .NET message. Namespaced subclasses of System.Object behave the same way on this branch, so this matches the other derived types.
  • When a namespaced exception is the InnerException of a .NET exception converted to Python, __cause__ is a second wrapper: outer.__cause__ is inner is False. That comes from Exceptions.SetArgsAndCause using CLRObject.GetReference instead of Converter.ToPython. Because the second wrapper now gets ClassDerivedObject.tp_dealloc, which never frees it, each such conversion keeps roughly 200 bytes. Master crashes before it gets this far, so this change doesn't make it worse. A follow-up could switch that call to Converter.ToPython.

Other repositories: Lean picks this up through its pythonnet package reference, so it needs a version bump after release. No other repository needs a code change.

Risk: I set risk:low. The change is small and easy to revert. It only changes behaviour for generated (__namespace__) exception types, which crash on master. A regression test fails without the fix, CI is green including Lean's suites, and the end-to-end run passed. There's no public API change.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The construction fix addresses the reported failure, but I reproduced regressions in cases that current master handles without crashing. Please address both findings in the inline comments before merging:

  1. Derived lifetime handling is not safe for exceptions: two generated inheritance levels can enqueue and free the same Python wrapper twice (SIGSEGV), and the inherited no-op tp_clear leaves native exception-field cycles permanently retained.
  2. A Python ToString() failure escapes tp_str: str(error) aborts the process rather than producing a catchable Python exception. This also reproduced end to end in a Lean Python backtest.

Verification: all nine CI checks are green. Locally on Python 3.11 / .NET 10, the full Python suite passed (487 passed, 23 skipped), embedding tests passed (1049 passed, 8 skipped, plus the isolated SetPythonPath test), and the managed Python runner passed (2/2). Its first run failed because an unrelated installed tests package shadowed this repository; it passed after correcting the module search path. The new subprocess regression crashes on master and passes on this head. The additional minimal probes instead demonstrate master exiting normally versus this head aborting during formatting or segfaulting during multilevel finalization; cause-cycle collection leaves 0 exceptions on master versus 100 on this head. Green existing tests therefore do not cover the failing paths.

Follow-up: .NET-created generated exceptions also retain pre-constructor args and omit their cause. Master already aborts on that construction/return path, so I filed #168 separately rather than treating it as another regression blocker.

Affected repositories: Lean needs its pythonnet package references updated after a corrected release; no caller-side API change is needed for these fixes.

Risk: I set risk:high, replacing risk:low, because the new lifetime/formatting paths introduce verified native process crashes and unreclaimed exceptions. Passing subprocess tests for these cases, including draining the derived-finalization queue and asserting collection rather than only survival, plus an end-to-end run without the formatting abort, would materially lower that risk.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot removed the kronos:review:working review: in progress label Nov 17, 2026
- A __namespace__ subclass of another one has a generated Finalize at each
  level, all calling PythonDerivedType.PyFinalize on the same object, which
  queued the same Python object once per level for release: freed twice,
  the process crashed. PyFinalize now clears the object's __pyobj__ field
  once it queued it, and skips an object it already queued. This also
  fixes chains of __namespace__ subclasses of System.Object.
- The derived types' tp_clear is a no-op, so an exception referencing
  itself through args, __cause__, __context__ or __traceback__ was never
  collected. A __namespace__ exception subclass now clears those fields,
  keeping its wrapper for the .NET object.
- A Python ToString override that raised escaped the exception str() slot
  and aborted the process; its error is now raised to str()'s caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Author

Addressed the requested changes in e491af1 (description updated):

  • Chain of __namespace__ subclasses released once: PyFinalize clears __pyobj__ after queuing and skips an object already queued. This also fixes the same crash for chains of System.Object subclasses, which happens on master too.
  • Exception cycles collected: ExceptionClassDerivedObject.tp_clear clears the native exception fields and keeps the wrapper for the .NET object.
  • Raising ToString override: ExceptionClassObject.tp_str raises the error to str()'s caller instead of aborting.
  • New subprocess test test_namespaced_exception_subclass_release: it fails with each fix removed in turn.

Full suite: embedding tests 1049 passed, 8 skipped, plus 1 passed (SetPythonPath); pytest 488 passed, 23 skipped; Python test runner 2 passed. An end-to-end Lean Python backtest covering all three cases passed.

The .NET-created case reported in #168 stays separate. So does the second __cause__ wrapper from SetArgsAndCause mentioned in the approving review, which happens on master too.

@quantconnect-kronos
quantconnect-kronos Bot marked this pull request as draft November 17, 2026 19:35
@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 17, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 17, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The two fixes for the earlier findings work, but the cause behind the ToString finding is still there in two more slots.

What changed since my last review (e491af1)

  • PyFinalize now queues each object once. Every generated level declares its own private __pyobj__, but GetPyObjField(obj.GetType()) always resolves the most derived type's field, so all levels read and clear the same field. AddDerivedFinalizedObject copies the pointer before SetPyObj clears the field, and nothing reads the field after that. The guard also prevents the ArgumentNullException that an empty field used to throw on the finalizer thread. Resolved.
  • ExceptionClassDerivedObject.tp_clear breaks args/__cause__/__context__ cycles. Resolved. One trade-off: it also clears __dict__ on an exception that .NET still holds. Namespaced System.Object subclasses already lose __dict__ the same way on master, so this matches them rather than making things worse. I opened #169 for that, and it doesn't block this change.
  • ExceptionClassObject.tp_str now raises the ToString error to str()'s caller, the same way ClassBase.tp_str does. Resolved.

Finding (blocking): Python overrides reached from the other exception slots can still abort the process. The wrapper is valid now, so the slots that ExceptionClassDerivedObject takes from ExceptionClassObject also reach Python overrides of virtual members. The previous commit guarded only tp_str. tp_repr reads e.Message, and Init calls Exceptions.SetArgsAndCause, which reads e.Message too. Neither catches errors, so a Python get_Message that raises escapes the native slot. I tested both on Python 3.11 / .NET 10, master vs this head:

  • repr(error) for a subclass with def __init__(self): pass and a raising get_Message: master prints the base message and exits 0. This head aborts with exit 134 (Unhandled exception. Python.Runtime.PythonException).
  • The same subclass with super().__init__(): on master, construction succeeds. On this head it aborts with exit 134.

These have the same cause as the ToString finding, so guard every exception slot that runs managed code, not one slot per review. In practice, wrap the managed calls in ExceptionClassObject.tp_repr and ExceptionClassDerivedObject.Init in the try/catch that tp_str now uses, and extend test_namespaced_exception_subclass_release to check that the ValueError reaches the caller of repr() and of the constructor. Details inline.

Verification: all nine CI checks are green on e491af1, including both Lean jobs. I built master and this head in separate environments and ran the probes above. I also checked the cycle and chain fixes against the code paths described above.

Other repositories: after release, Lean needs its pythonnet package reference bumped. No other code changes are needed.

Risk: stays risk:high. This head still aborts the process in cases that exit normally on master: repr() and construction with a raising get_Message override. With both guarded and covered by the subprocess test, the remaining evidence (green CI, regression tests that fail without each fix, the end-to-end runs) would support a lower rating.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. ab439ce fixes both findings from the last review of d282ec1, and I found nothing new that blocks.

What changed since my last review (d282ec1)

  • None from a ToString override: tp_str and tp_repr now do all their work inside the guard, including the prefix stripping and the string conversion. e.ToString() ?? string.Empty handles a None result. Resolved.
  • Chained errors: all three guards now call Exceptions.SetError(error) without unwrapping. These slots call ToString/Message directly, so the only wrapper to remove is the PythonException itself. SetError restores it as the original Python error with its __cause__. Resolved.
  • The repr test now reaches repr: test_namespaced_exception_subclass_override_errors constructs the exception while its overrides succeed, then turns them on to fail. I rebuilt with only tp_repr's InnerException ?? error restored, and that test then fails (the KeyError escapes). Resolved.
  • tp_repr now checks IsNullOrEmpty, so a get_Message override returning None shows E() instead of E(''). This only affects overrides, since a .NET Message is never null.

What I verified

  • All nine CI checks pass on ab439ce, including the Lean regression and unit-test jobs.
  • Locally on Python 3.11 / .NET 10 with this head built: tests/test_exceptions.py passed (32 tests).
  • Extra probes on a namespaced System.Exception subclass all exit normally:
    • a ToString override returning an int raises a catchable InvalidCastException from str()
    • a get_Message override returning None gives repr() = E()
    • reflected exceptions keep their str()/repr() ('abc', InvalidOperationException('abc'), InvalidOperationException())

Non-blocking note: a .NET exception raised inside a Python override, such as raise InvalidOperationException(...) in ToString, reaches the caller of str()/repr() as ClrBubbledException, not as InvalidOperationException. Calling the override through a .NET method (for example String.Format("{0}", obj)) gives the same result today. ClassBase.tp_str unwraps it instead, which is the inconsistency #170 tracks. On master, this case crashed, so the change doesn't make it worse.

Other repositories: after release, Lean needs its pythonnet package reference bumped. No other repository needs a code change.

Risk: I set risk:low. The change is small and easy to revert, and adds no public API. Its behaviour changes are limited to two areas: generated __namespace__ exception types, which crash on master, and the PyFinalize once-only guard, which fixes the double release for chains. The slot guards only replace process aborts with catchable Python errors. Each fix has a regression test that fails without it. CI is green, including Lean's suites, and the description reports end-to-end runs of every case that passed. The other reviewer's open findings are on d282ec1, and they still need to review this head.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest direct-override fixes work: None from ToString is handled, direct Python errors keep their cause, and the revised regression really reaches repr(). However, two findings still block this head:

  1. Finalizer deduplication invalidates the Python wrapper too early. Generated finalizers call PyFinalize before the managed base finalizer. Clearing __pyobj__ there disables Python virtual dispatch while that base finalizer is still running. A concrete cleanup hook silently runs the base implementation; an abstract hook throws an unhandled NotImplementedException and terminates the host. This affects ordinary generated subclasses too, not just exceptions. Details and a reproduction are inline.
  2. The exception-boundary fix still loses the original error on inherited formatting. If ToString is inherited and calls a Python get_Message override, the bridge invokes the inherited implementation through reflection. Its failure reaches the new catch as TargetInvocationException, which is now raised unchanged instead of restoring the original ValueError and its cause. The new test overrides ToString directly and misses this path. This is the same error-translation finding as before, not a request for a wider API change.

Verification: all nine CI checks pass. Locally on Python 3.11 / .NET 10: Python suite 489 passed, 23 skipped; embedding tests 1049 passed, 8 skipped, plus 1 passed in the isolated SetPythonPath run; managed Python runner 2 passed. The finalizer probes use the default finalizer settings: current master calls the Python hook and exits normally; this head skips the concrete hook or aborts with exit 134 for an abstract hook. For inherited formatting, d282ec1 restores ValueError with its KeyError cause; this head raises TargetInvocationException. Current master returns the base message without failing.

End to end: the null-return and three direct chained-error assertions pass. The inherited-formatting assertion fails with the wrong error type. A separate finalizer backtest aborts during initialization on this head, while the same algorithm on master executes the Python callback and reaches algorithm completion. The base control then encounters a separate pre-existing shutdown error; I am not counting that as a clean process exit.

Follow-ups: I filed #171 for the clrproperty Message construction abort, reproduced equally on master, so it does not block this change. #168–#170 remain separate. After release, Lean needs its pinned pythonnet package references updated; no caller-side API change is needed.

Risk: I set risk:high, replacing risk:low, because the finalizer change introduces a verified host-terminating regression beyond exception subclasses, and error translation still changes which exception callers catch. Preserve callbacks through base finalization, cover inherited formatting as well as direct overrides, and rerun the subprocess and end-to-end checks before lowering the label.

Comment thread src/runtime/Types/ClassDerived.cs Resolved
catch (Exception error)
{
message = message.Substring(fullTypeName.Length);
Exceptions.SetError(error);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Distinguish an actual reflection wrapper from a Python error's cause. Passing the caught error unchanged fixes direct Python overrides, but inherited ToString is not always a direct call: PythonDerivedType.InvokeMethod falls back through Type.InvokeMember (ClassDerived.cs:760–764). If inherited Exception.ToString calls a failing Python get_Message, this catch receives a real TargetInvocationException. Exceptions.SetError only restores a direct PythonException, so an except ValueError no longer catches the original error.

Reproduction after loading CLR:

class Error(System.Exception):
    __namespace__ = 'Synthetic.InheritedFormatting'
    def __init__(self):
        self.failing = False
    def get_Message(self):
        if self.failing:
            raise ValueError('outer') from KeyError('inner')
        return 'message'

error = Error()
error.failing = True
try:
    str(error)
except ValueError as raised:
    assert isinstance(raised.__cause__, KeyError)

This head instead raises System.Reflection.TargetInvocationException; d282ec1 catches the ValueError correctly, with its cause. Master exits normally with its base message. The wrong error also reproduces end to end.

Complete the same boundary fix by unwrapping an actual invocation wrapper when present, without unwrapping PythonException.InnerException, and extend the new regression to an inherited ToString calling the override. The description's claim that these paths have no invocation wrapper needs correcting too.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2c5dc3b. The three exception slots now raise through one helper, which unwraps only an actual TargetInvocationException (to its InnerException) and passes anything else to Exceptions.SetError unchanged. A PythonException is therefore restored as itself, with its cause, whether it arrives directly from an override or wrapped by the reflection fallback for an inherited member. test_namespaced_exception_subclass_override_errors now also checks str() of a subclass that inherits ToString and overrides only get_Message with your chained raise: it gets ValueError with the KeyError cause. Without the unwrapping that case fails. I corrected the description's claim about invocation wrappers, and the same case passed end to end.

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot removed the kronos:review:working review: in progress label Nov 17, 2026
- A generated type's Finalize released the Python object before calling
  its base class finalizer, and since releasing became once only it
  clears __pyobj__: a base finalizer calling a virtual member then no
  longer reached the Python override (the .NET implementation ran, or an
  abstract one threw and terminated the process). Finalize now runs the
  base finalizer first: the innermost generated type queues the Python
  object once the finalizer chain ran, and the outer ones skip it.
- An inherited ToString runs through reflection, which wraps the error of
  a Python Message override it calls in a TargetInvocationException. The
  exception slots now raise that wrapper's inner error, and still raise a
  PythonException as it is, with its cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos

Copy link
Copy Markdown
Author

Addressed the requested changes in 2c5dc3b (description updated):

  • Finalizer callbacks: the generated Finalize runs the base class finalizer before releasing the Python object. Python overrides stay reachable from .NET base finalizers, and release stays once-only across levels.
  • Inherited formatting: the exception slots unwrap a real TargetInvocationException from the reflection fallback, and still raise a PythonException as it is, with its cause.
  • Tests: new finalizer-callback fixtures and assertions (abstract and virtual, one and two levels) in test_namespaced_exception_subclass_release, and an inherited-ToString case in test_namespaced_exception_subclass_override_errors. Each fails without its fix.
  • Filed A __namespace__ subclass of a __namespace__ subclass doesn't reach its parent's Python overrides from .NET #172: from .NET, a __namespace__ subclass of a __namespace__ subclass doesn't reach its parent's Python overrides. Master behaves the same.

Full suite: embedding tests 1049 passed, 8 skipped, plus 1 passed (SetPythonPath); pytest 489 passed, 23 skipped; Python test runner 2 passed. A Lean Python backtest covering the finalizer callbacks and the inherited-formatting case passed.

@quantconnect-kronos
quantconnect-kronos Bot marked this pull request as draft November 17, 2026 22:39
@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 17, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 17, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review November 17, 2026 23:02

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. 2c5dc3b fixes both findings from the other review of ab439ce, and I found nothing new that blocks.

What changed since my last review (ab439ce)

  • Finalizer order: the generated Finalize now calls the base class Finalize first and PyFinalize after it. A .NET base finalizer therefore still finds __pyobj__ set and reaches the Python override. In a chain, the innermost generated level queues the Python object after the .NET base finalizer has run, and the outer levels find the field cleared and skip it, so the object is still released only once. Master also called PyFinalize first, but it never cleared the field, so a base finalizer could call into a Python object that was already queued for release. The new order removes that window too. A base finalizer that throws now skips PyFinalize, but an unhandled exception on the finalizer thread ends the process either way, so that changes nothing in practice. Only __namespace__/__assembly__ subclasses get a generated Finalize (MetaType.cs:161).
  • Inherited formatting: the three slots share ExceptionClassObject.SetError, which unwraps only a real TargetInvocationException and passes a PythonException through unchanged, so its cause is kept. The new inherited-ToString case covers the reflection path.

What I verified

  • All nine CI checks pass on 2c5dc3b, including the Lean regression and unit-test jobs.
  • Locally on Python 3.11 / .NET 10 with this head built: tests/test_exceptions.py 32 passed with nothing skipped, and the full Python suite 488 passed, 24 skipped. The extra skip compared with the description is the numpy test, since numpy isn't installed here.
  • I rebuilt with only the old Finalize order restored. test_namespaced_exception_subclass_release then aborts with -6: NotImplementedException from AbstractFinalizerCallbackTest.Finalize on the finalizer thread. So the new fixtures do cover the regression.

Non-blocking: the description's Tests section says "Both tests run their script in a subprocess", but it now lists three.

Other repositories: after release, Lean needs its pythonnet package reference bumped (currently 2.0.70). Lean itself declares no __namespace__ subclasses, so the finalizer change only reaches user code that does. No other repository needs a code change.

Risk: I set risk:low. The change is small and easy to revert, and adds no public API. The finalizer reorder affects every generated __namespace__ type, but base finalizers still reach Python overrides as they do on master, and the Python object is now released only after they have run. The other behaviour changes are limited to generated exception types, which crash on master, and to the slot guards, which turn process aborts into catchable Python errors. Each fix has a subprocess regression test that fails without it, CI is green including Lean's suites, and the description reports end-to-end runs of every case that passed. The other reviewer's three open threads are on ab439ce, so they still need to review this head.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:low Low risk and removed risk:high High risk kronos:review:working review: in progress labels Nov 17, 2026
@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 17, 2026

@quantconnect-kronos-reviewer-2 quantconnect-kronos-reviewer-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The finalizer-order fix resolves my previous callback regression: managed base finalizers can reach the Python override before the wrapper is queued, and multilevel release remains once-only. I will resolve that thread. The direct formatting cases and the single-level inherited case also pass. However, the earlier exception-boundary/error-restoration finding is still only partially resolved.

Changes requested: restore the original exception through the complete leading reflection-wrapper chain, not just one generated inheritance level. With two generated exception levels, inherited ToString adds two TargetInvocationException wrappers; the helper removes one and raises the other. Consequently an except ValueError does not catch the get_Message override's error. Three levels leave two wrappers. This is the same unresolved restoration finding, not a request to widen the API or change parent-override dispatch. Details and a minimal reproduction are inline. Add multilevel subprocess coverage that preserves the original error and its KeyError cause, and include that case in the end-to-end check.

Verification: all nine CI checks pass. Locally on Python 3.11 / .NET 10: Python suite 489 passed, 23 skipped; embedding suite 1049 passed, 8 skipped, plus 1 passed in the isolated SetPythonPath run; managed Python runner 2 passed. Independent head/master probes confirm that this head raises the remaining reflection wrapper for two- and three-level inherited formatting, while current master returns its default message without failing. repr() correctly restores ValueError at both depths. The child explicitly defines both __init__ and get_Message, so this is not #172.

End to end: the construction, release/cycle, all four finalizer callback, direct override, and single-level inherited-formatting assertions pass with the reviewed runtime. The added two-level assertion fails during algorithm initialization with TargetInvocationException instead of ValueError. The process later also encounters the previously established baseline shutdown failure; that is separate from the assertion failure and is not counted as a regression of this change.

Follow-ups: #168–#172 remain separate. I filed #173 for managed-member access from a finalizer callback, reproduced equally on master, so it does not block this change. After release, Lean needs its pinned pythonnet package references updated; no caller-side API change is needed.

Risk: I set risk:high, replacing risk:low, because the newly enabled formatting path still raises a different exception from the one user code expects to catch, and the committed regression covers only one generated level. Depth-independent restoration, multilevel regression coverage, and an end-to-end run covering that case would lower this risk.

static void SetError(Exception error)
{
if (!base.Init(obj, args, kw)) return false;
var raised = error is TargetInvocationException { InnerException: { } invoked } ? invoked : error;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Restore the error through every generated inheritance level. This is still the earlier reflection-error restoration finding: each generated superclass traversed by inherited ToString adds an invocation wrapper (ClassDerived.cs:375–395, :761–765), but this expression removes exactly one. After loading CLR:

from System import Exception as ClrException

class Parent(ClrException):
    __namespace__ = 'Synthetic.Errors'
    def __init__(self):
        pass

class Child(Parent):
    __namespace__ = 'Synthetic.Errors'
    def __init__(self):
        self.failing = False
    def get_Message(self):
        if self.failing:
            raise ValueError('outer') from KeyError('inner')
        return 'message'

error = Child()
error.failing = True
try:
    str(error)
except ValueError as raised:
    assert isinstance(raised.__cause__, KeyError)

On this head, the handler is bypassed by System.Reflection.TargetInvocationException; its inner exception is the original PythonException(ValueError). A third generated level leaves two invocation wrappers. repr(error) restores the correct ValueError, and master completes this source normally with its default managed message. I reproduced the two-level wrong error end to end as well. The child defines its own get_Message, so this is not the inherited-Python-override issue #172.

Complete the boundary normalization for the leading chain of actual invocation wrappers, stopping before PythonException so its InnerException remains the explicit cause. Extend the subprocess test with two- and three-level inherited ToString cases; the new case at tests/test_exceptions.py:670–681 has only one generated level and cannot expose this failure.

@quantconnect-kronos

Copy link
Copy Markdown
Author

Reviews asked for changes 5 times since a person from QuantConnect last commented or reviewed, so the developer stops here for a person to decide how to go on, such as what the change should still cover. A comment or review from one restarts it, and so does adding kronos:develop or kronos:review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:idle develop: paused risk:high High risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants