Repository navigation
Keep the Python object of a __namespace__ subclass of a .NET exception - #167
quantconnect-kronos[bot] wants to merge 5 commits into
Conversation
A Python subclass of System.Exception, or of any exception type, that declares __namespace__ gets a generated .NET type, but that type was reflected as a plain exception rather than as a Python-derived type. Its Python wrapper never stored itself on the .NET object, so the generated constructor created a temporary wrapper, kept its pointer and released it: later calls through the .NET object, such as an overridden virtual method or the object coming back from .NET, used a freed object and could crash the process. An __init__ that didn't call the base __init__ also left the base constructor unrun, so field initializers stayed null. Generated exception types now get the derived-type handling of ordinary __namespace__ subclasses (the wrapper set on the object, the base constructor run before __init__, the object's lifetime tied to the .NET object) together with the exception repr, str and args. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Approving. I checked the root cause: ClassManager picked ExceptionClassObject before checking for the generated __pyobj__ field, so generated exception types never got ClassDerivedObject.NewObjectToPython or its tp_dealloc/tp_clear lifetime handling. The fix moves the derived-type check first, so it only affects generated types. For every other type, GetPyObjField returns null and the type falls through to the same branch as before. The new ExceptionClassDerivedObject keeps the exception tp_repr/tp_str and the args/__cause__ refresh after __init__. Slot lookup walks the implementing type's hierarchy from the most derived class, so the new static slots win over ClassBase's, as the comment says.
What I verified
-
CI is green on all jobs, including the Lean regression and unit test runs. The new test ran in every Python job (
test_exceptions.pyshows 30 passed, none skipped). -
Locally, master installed in its own venv vs this branch (Python 3.11, .NET 10). The new test's script segfaults (rc 139) on master and passes on the branch. I also probed namespaced subclasses of
System.Exceptionon both:- a plain one constructed with a message
- a Python subclass of a namespaced one, raised and caught as itself
- one thrown from .NET with
ExceptionDispatchInfo.Throwand caught in Python as the same object - 20k raise/catch iterations with Python and .NET GC: stable memory
- a traceback whose frame locals reference other objects: those objects are freed after GC
All of these crash on master and pass on the branch.
-
End to end: I ran a Lean Python backtest with this build. Its algorithm defines
__namespace__subclasses ofSystem.ArgumentExceptionwithout__init__, with an__init__that skipssuper()plus aToStringoverride, and withsuper().__init__("message"). For each, it checks the round-trip throughList[Object], identity onraise,str()andargs. All checks passed.
Notes (non-blocking, not caused by this change)
- A Python
__str__/__repr__on a namespaced exception subclass is ignored:str()returns the .NET message. Namespaced subclasses ofSystem.Objectbehave the same way on this branch, so this matches the other derived types. - When a namespaced exception is the
InnerExceptionof a .NET exception converted to Python,__cause__is a second wrapper:outer.__cause__ is innerisFalse. That comes fromExceptions.SetArgsAndCauseusingCLRObject.GetReferenceinstead ofConverter.ToPython. Because the second wrapper now getsClassDerivedObject.tp_dealloc, which never frees it, each such conversion keeps roughly 200 bytes. Master crashes before it gets this far, so this change doesn't make it worse. A follow-up could switch that call toConverter.ToPython.
Other repositories: Lean picks this up through its pythonnet package reference, so it needs a version bump after release. No other repository needs a code change.
Risk: I set risk:low. The change is small and easy to revert. It only changes behaviour for generated (__namespace__) exception types, which crash on master. A regression test fails without the fix, CI is green including Lean's suites, and the end-to-end run passed. There's no public API change.
There was a problem hiding this comment.
The construction fix addresses the reported failure, but I reproduced regressions in cases that current master handles without crashing. Please address both findings in the inline comments before merging:
- Derived lifetime handling is not safe for exceptions: two generated inheritance levels can enqueue and free the same Python wrapper twice (SIGSEGV), and the inherited no-op
tp_clearleaves native exception-field cycles permanently retained. - A Python
ToString()failure escapestp_str:str(error)aborts the process rather than producing a catchable Python exception. This also reproduced end to end in a Lean Python backtest.
Verification: all nine CI checks are green. Locally on Python 3.11 / .NET 10, the full Python suite passed (487 passed, 23 skipped), embedding tests passed (1049 passed, 8 skipped, plus the isolated SetPythonPath test), and the managed Python runner passed (2/2). Its first run failed because an unrelated installed tests package shadowed this repository; it passed after correcting the module search path. The new subprocess regression crashes on master and passes on this head. The additional minimal probes instead demonstrate master exiting normally versus this head aborting during formatting or segfaulting during multilevel finalization; cause-cycle collection leaves 0 exceptions on master versus 100 on this head. Green existing tests therefore do not cover the failing paths.
Follow-up: .NET-created generated exceptions also retain pre-constructor args and omit their cause. Master already aborts on that construction/return path, so I filed #168 separately rather than treating it as another regression blocker.
Affected repositories: Lean needs its pythonnet package references updated after a corrected release; no caller-side API change is needed for these fixes.
Risk: I set risk:high, replacing risk:low, because the new lifetime/formatting paths introduce verified native process crashes and unreclaimed exceptions. Passing subprocess tests for these cases, including draining the derived-finalization queue and asserting collection rather than only survival, plus an end-to-end run without the formatting abort, would materially lower that risk.
- A __namespace__ subclass of another one has a generated Finalize at each level, all calling PythonDerivedType.PyFinalize on the same object, which queued the same Python object once per level for release: freed twice, the process crashed. PyFinalize now clears the object's __pyobj__ field once it queued it, and skips an object it already queued. This also fixes chains of __namespace__ subclasses of System.Object. - The derived types' tp_clear is a no-op, so an exception referencing itself through args, __cause__, __context__ or __traceback__ was never collected. A __namespace__ exception subclass now clears those fields, keeping its wrapper for the .NET object. - A Python ToString override that raised escaped the exception str() slot and aborted the process; its error is now raised to str()'s caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the requested changes in e491af1 (description updated):
Full suite: embedding tests 1049 passed, 8 skipped, plus 1 passed (SetPythonPath); pytest 488 passed, 23 skipped; Python test runner 2 passed. An end-to-end Lean Python backtest covering all three cases passed. The .NET-created case reported in #168 stays separate. So does the second |
There was a problem hiding this comment.
Requesting changes. The two fixes for the earlier findings work, but the cause behind the ToString finding is still there in two more slots.
What changed since my last review (e491af1)
PyFinalizenow queues each object once. Every generated level declares its own private__pyobj__, butGetPyObjField(obj.GetType())always resolves the most derived type's field, so all levels read and clear the same field.AddDerivedFinalizedObjectcopies the pointer beforeSetPyObjclears the field, and nothing reads the field after that. The guard also prevents theArgumentNullExceptionthat an empty field used to throw on the finalizer thread. Resolved.ExceptionClassDerivedObject.tp_clearbreaksargs/__cause__/__context__cycles. Resolved. One trade-off: it also clears__dict__on an exception that .NET still holds. NamespacedSystem.Objectsubclasses already lose__dict__the same way on master, so this matches them rather than making things worse. I opened #169 for that, and it doesn't block this change.ExceptionClassObject.tp_strnow raises theToStringerror tostr()'s caller, the same wayClassBase.tp_strdoes. Resolved.
Finding (blocking): Python overrides reached from the other exception slots can still abort the process. The wrapper is valid now, so the slots that ExceptionClassDerivedObject takes from ExceptionClassObject also reach Python overrides of virtual members. The previous commit guarded only tp_str. tp_repr reads e.Message, and Init calls Exceptions.SetArgsAndCause, which reads e.Message too. Neither catches errors, so a Python get_Message that raises escapes the native slot. I tested both on Python 3.11 / .NET 10, master vs this head:
repr(error)for a subclass withdef __init__(self): passand a raisingget_Message: master prints the base message and exits 0. This head aborts with exit 134 (Unhandled exception. Python.Runtime.PythonException).- The same subclass with
super().__init__(): on master, construction succeeds. On this head it aborts with exit 134.
These have the same cause as the ToString finding, so guard every exception slot that runs managed code, not one slot per review. In practice, wrap the managed calls in ExceptionClassObject.tp_repr and ExceptionClassDerivedObject.Init in the try/catch that tp_str now uses, and extend test_namespaced_exception_subclass_release to check that the ValueError reaches the caller of repr() and of the constructor. Details inline.
Verification: all nine CI checks are green on e491af1, including both Lean jobs. I built master and this head in separate environments and ran the probes above. I also checked the cycle and chain fixes against the code paths described above.
Other repositories: after release, Lean needs its pythonnet package reference bumped. No other code changes are needed.
Risk: stays risk:high. This head still aborts the process in cases that exit normally on master: repr() and construction with a raising get_Message override. With both guarded and covered by the subprocess test, the remaining evidence (green CI, regression tests that fail without each fix, the end-to-end runs) would support a lower rating.
There was a problem hiding this comment.
Approving. ab439ce fixes both findings from the last review of d282ec1, and I found nothing new that blocks.
What changed since my last review (d282ec1)
Nonefrom aToStringoverride:tp_strandtp_reprnow do all their work inside the guard, including the prefix stripping and the string conversion.e.ToString() ?? string.Emptyhandles aNoneresult. Resolved.- Chained errors: all three guards now call
Exceptions.SetError(error)without unwrapping. These slots callToString/Messagedirectly, so the only wrapper to remove is thePythonExceptionitself.SetErrorrestores it as the original Python error with its__cause__. Resolved. - The
reprtest now reachesrepr:test_namespaced_exception_subclass_override_errorsconstructs the exception while its overrides succeed, then turns them on to fail. I rebuilt with onlytp_repr'sInnerException ?? errorrestored, and that test then fails (theKeyErrorescapes). Resolved. tp_reprnow checksIsNullOrEmpty, so aget_Messageoverride returningNoneshowsE()instead ofE(''). This only affects overrides, since a .NETMessageis never null.
What I verified
- All nine CI checks pass on ab439ce, including the Lean regression and unit-test jobs.
- Locally on Python 3.11 / .NET 10 with this head built:
tests/test_exceptions.pypassed (32 tests). - Extra probes on a namespaced
System.Exceptionsubclass all exit normally:- a
ToStringoverride returning anintraises a catchableInvalidCastExceptionfromstr() - a
get_Messageoverride returningNonegivesrepr()=E() - reflected exceptions keep their
str()/repr()('abc',InvalidOperationException('abc'),InvalidOperationException())
- a
Non-blocking note: a .NET exception raised inside a Python override, such as raise InvalidOperationException(...) in ToString, reaches the caller of str()/repr() as ClrBubbledException, not as InvalidOperationException. Calling the override through a .NET method (for example String.Format("{0}", obj)) gives the same result today. ClassBase.tp_str unwraps it instead, which is the inconsistency #170 tracks. On master, this case crashed, so the change doesn't make it worse.
Other repositories: after release, Lean needs its pythonnet package reference bumped. No other repository needs a code change.
Risk: I set risk:low. The change is small and easy to revert, and adds no public API. Its behaviour changes are limited to two areas: generated __namespace__ exception types, which crash on master, and the PyFinalize once-only guard, which fixes the double release for chains. The slot guards only replace process aborts with catchable Python errors. Each fix has a regression test that fails without it. CI is green, including Lean's suites, and the description reports end-to-end runs of every case that passed. The other reviewer's open findings are on d282ec1, and they still need to review this head.
There was a problem hiding this comment.
The latest direct-override fixes work: None from ToString is handled, direct Python errors keep their cause, and the revised regression really reaches repr(). However, two findings still block this head:
- Finalizer deduplication invalidates the Python wrapper too early. Generated finalizers call
PyFinalizebefore the managed base finalizer. Clearing__pyobj__there disables Python virtual dispatch while that base finalizer is still running. A concrete cleanup hook silently runs the base implementation; an abstract hook throws an unhandledNotImplementedExceptionand terminates the host. This affects ordinary generated subclasses too, not just exceptions. Details and a reproduction are inline. - The exception-boundary fix still loses the original error on inherited formatting. If
ToStringis inherited and calls a Pythonget_Messageoverride, the bridge invokes the inherited implementation through reflection. Its failure reaches the new catch asTargetInvocationException, which is now raised unchanged instead of restoring the originalValueErrorand its cause. The new test overridesToStringdirectly and misses this path. This is the same error-translation finding as before, not a request for a wider API change.
Verification: all nine CI checks pass. Locally on Python 3.11 / .NET 10: Python suite 489 passed, 23 skipped; embedding tests 1049 passed, 8 skipped, plus 1 passed in the isolated SetPythonPath run; managed Python runner 2 passed. The finalizer probes use the default finalizer settings: current master calls the Python hook and exits normally; this head skips the concrete hook or aborts with exit 134 for an abstract hook. For inherited formatting, d282ec1 restores ValueError with its KeyError cause; this head raises TargetInvocationException. Current master returns the base message without failing.
End to end: the null-return and three direct chained-error assertions pass. The inherited-formatting assertion fails with the wrong error type. A separate finalizer backtest aborts during initialization on this head, while the same algorithm on master executes the Python callback and reaches algorithm completion. The base control then encounters a separate pre-existing shutdown error; I am not counting that as a clean process exit.
Follow-ups: I filed #171 for the clrproperty Message construction abort, reproduced equally on master, so it does not block this change. #168–#170 remain separate. After release, Lean needs its pinned pythonnet package references updated; no caller-side API change is needed.
Risk: I set risk:high, replacing risk:low, because the finalizer change introduces a verified host-terminating regression beyond exception subclasses, and error translation still changes which exception callers catch. Preserve callbacks through base finalization, cover inherited formatting as well as direct overrides, and rerun the subprocess and end-to-end checks before lowering the label.
| catch (Exception error) | ||
| { | ||
| message = message.Substring(fullTypeName.Length); | ||
| Exceptions.SetError(error); |
There was a problem hiding this comment.
[P2] Distinguish an actual reflection wrapper from a Python error's cause. Passing the caught error unchanged fixes direct Python overrides, but inherited ToString is not always a direct call: PythonDerivedType.InvokeMethod falls back through Type.InvokeMember (ClassDerived.cs:760–764). If inherited Exception.ToString calls a failing Python get_Message, this catch receives a real TargetInvocationException. Exceptions.SetError only restores a direct PythonException, so an except ValueError no longer catches the original error.
Reproduction after loading CLR:
class Error(System.Exception):
__namespace__ = 'Synthetic.InheritedFormatting'
def __init__(self):
self.failing = False
def get_Message(self):
if self.failing:
raise ValueError('outer') from KeyError('inner')
return 'message'
error = Error()
error.failing = True
try:
str(error)
except ValueError as raised:
assert isinstance(raised.__cause__, KeyError)This head instead raises System.Reflection.TargetInvocationException; d282ec1 catches the ValueError correctly, with its cause. Master exits normally with its base message. The wrong error also reproduces end to end.
Complete the same boundary fix by unwrapping an actual invocation wrapper when present, without unwrapping PythonException.InnerException, and extend the new regression to an inherited ToString calling the override. The description's claim that these paths have no invocation wrapper needs correcting too.
There was a problem hiding this comment.
Fixed in 2c5dc3b. The three exception slots now raise through one helper, which unwraps only an actual TargetInvocationException (to its InnerException) and passes anything else to Exceptions.SetError unchanged. A PythonException is therefore restored as itself, with its cause, whether it arrives directly from an override or wrapped by the reflection fallback for an inherited member. test_namespaced_exception_subclass_override_errors now also checks str() of a subclass that inherits ToString and overrides only get_Message with your chained raise: it gets ValueError with the KeyError cause. Without the unwrapping that case fails. I corrected the description's claim about invocation wrappers, and the same case passed end to end.
- A generated type's Finalize released the Python object before calling its base class finalizer, and since releasing became once only it clears __pyobj__: a base finalizer calling a virtual member then no longer reached the Python override (the .NET implementation ran, or an abstract one threw and terminated the process). Finalize now runs the base finalizer first: the innermost generated type queues the Python object once the finalizer chain ran, and the outer ones skip it. - An inherited ToString runs through reflection, which wraps the error of a Python Message override it calls in a TargetInvocationException. The exception slots now raise that wrapper's inner error, and still raise a PythonException as it is, with its cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the requested changes in 2c5dc3b (description updated):
Full suite: embedding tests 1049 passed, 8 skipped, plus 1 passed (SetPythonPath); pytest 489 passed, 23 skipped; Python test runner 2 passed. A Lean Python backtest covering the finalizer callbacks and the inherited-formatting case passed. |
There was a problem hiding this comment.
Approving. 2c5dc3b fixes both findings from the other review of ab439ce, and I found nothing new that blocks.
What changed since my last review (ab439ce)
- Finalizer order: the generated
Finalizenow calls the base classFinalizefirst andPyFinalizeafter it. A .NET base finalizer therefore still finds__pyobj__set and reaches the Python override. In a chain, the innermost generated level queues the Python object after the .NET base finalizer has run, and the outer levels find the field cleared and skip it, so the object is still released only once. Master also calledPyFinalizefirst, but it never cleared the field, so a base finalizer could call into a Python object that was already queued for release. The new order removes that window too. A base finalizer that throws now skipsPyFinalize, but an unhandled exception on the finalizer thread ends the process either way, so that changes nothing in practice. Only__namespace__/__assembly__subclasses get a generatedFinalize(MetaType.cs:161). - Inherited formatting: the three slots share
ExceptionClassObject.SetError, which unwraps only a realTargetInvocationExceptionand passes aPythonExceptionthrough unchanged, so its cause is kept. The new inherited-ToStringcase covers the reflection path.
What I verified
- All nine CI checks pass on 2c5dc3b, including the Lean regression and unit-test jobs.
- Locally on Python 3.11 / .NET 10 with this head built:
tests/test_exceptions.py32 passed with nothing skipped, and the full Python suite 488 passed, 24 skipped. The extra skip compared with the description is the numpy test, since numpy isn't installed here. - I rebuilt with only the old
Finalizeorder restored.test_namespaced_exception_subclass_releasethen aborts with -6:NotImplementedExceptionfromAbstractFinalizerCallbackTest.Finalizeon the finalizer thread. So the new fixtures do cover the regression.
Non-blocking: the description's Tests section says "Both tests run their script in a subprocess", but it now lists three.
Other repositories: after release, Lean needs its pythonnet package reference bumped (currently 2.0.70). Lean itself declares no __namespace__ subclasses, so the finalizer change only reaches user code that does. No other repository needs a code change.
Risk: I set risk:low. The change is small and easy to revert, and adds no public API. The finalizer reorder affects every generated __namespace__ type, but base finalizers still reach Python overrides as they do on master, and the Python object is now released only after they have run. The other behaviour changes are limited to generated exception types, which crash on master, and to the slot guards, which turn process aborts into catchable Python errors. Each fix has a subprocess regression test that fails without it, CI is green including Lean's suites, and the description reports end-to-end runs of every case that passed. The other reviewer's three open threads are on ab439ce, so they still need to review this head.
There was a problem hiding this comment.
The finalizer-order fix resolves my previous callback regression: managed base finalizers can reach the Python override before the wrapper is queued, and multilevel release remains once-only. I will resolve that thread. The direct formatting cases and the single-level inherited case also pass. However, the earlier exception-boundary/error-restoration finding is still only partially resolved.
Changes requested: restore the original exception through the complete leading reflection-wrapper chain, not just one generated inheritance level. With two generated exception levels, inherited ToString adds two TargetInvocationException wrappers; the helper removes one and raises the other. Consequently an except ValueError does not catch the get_Message override's error. Three levels leave two wrappers. This is the same unresolved restoration finding, not a request to widen the API or change parent-override dispatch. Details and a minimal reproduction are inline. Add multilevel subprocess coverage that preserves the original error and its KeyError cause, and include that case in the end-to-end check.
Verification: all nine CI checks pass. Locally on Python 3.11 / .NET 10: Python suite 489 passed, 23 skipped; embedding suite 1049 passed, 8 skipped, plus 1 passed in the isolated SetPythonPath run; managed Python runner 2 passed. Independent head/master probes confirm that this head raises the remaining reflection wrapper for two- and three-level inherited formatting, while current master returns its default message without failing. repr() correctly restores ValueError at both depths. The child explicitly defines both __init__ and get_Message, so this is not #172.
End to end: the construction, release/cycle, all four finalizer callback, direct override, and single-level inherited-formatting assertions pass with the reviewed runtime. The added two-level assertion fails during algorithm initialization with TargetInvocationException instead of ValueError. The process later also encounters the previously established baseline shutdown failure; that is separate from the assertion failure and is not counted as a regression of this change.
Follow-ups: #168–#172 remain separate. I filed #173 for managed-member access from a finalizer callback, reproduced equally on master, so it does not block this change. After release, Lean needs its pinned pythonnet package references updated; no caller-side API change is needed.
Risk: I set risk:high, replacing risk:low, because the newly enabled formatting path still raises a different exception from the one user code expects to catch, and the committed regression covers only one generated level. Depth-independent restoration, multilevel regression coverage, and an end-to-end run covering that case would lower this risk.
| static void SetError(Exception error) | ||
| { | ||
| if (!base.Init(obj, args, kw)) return false; | ||
| var raised = error is TargetInvocationException { InnerException: { } invoked } ? invoked : error; |
There was a problem hiding this comment.
[P2] Restore the error through every generated inheritance level. This is still the earlier reflection-error restoration finding: each generated superclass traversed by inherited ToString adds an invocation wrapper (ClassDerived.cs:375–395, :761–765), but this expression removes exactly one. After loading CLR:
from System import Exception as ClrException
class Parent(ClrException):
__namespace__ = 'Synthetic.Errors'
def __init__(self):
pass
class Child(Parent):
__namespace__ = 'Synthetic.Errors'
def __init__(self):
self.failing = False
def get_Message(self):
if self.failing:
raise ValueError('outer') from KeyError('inner')
return 'message'
error = Child()
error.failing = True
try:
str(error)
except ValueError as raised:
assert isinstance(raised.__cause__, KeyError)On this head, the handler is bypassed by System.Reflection.TargetInvocationException; its inner exception is the original PythonException(ValueError). A third generated level leaves two invocation wrappers. repr(error) restores the correct ValueError, and master completes this source normally with its default managed message. I reproduced the two-level wrong error end to end as well. The child defines its own get_Message, so this is not the inherited-Python-override issue #172.
Complete the boundary normalization for the leading chain of actual invocation wrappers, stopping before PythonException so its InnerException remains the explicit cause. Extend the subprocess test with two- and three-level inherited ToString cases; the new case at tests/test_exceptions.py:670–681 has only one generated level and cannot expose this failure.
|
Reviews asked for changes 5 times since a person from QuantConnect last commented or reviewed, so the developer stops here for a person to decide how to go on, such as what the change should still cover. A comment or review from one restarts it, and so does adding |
What changed and why
A Python subclass of a .NET exception that declares
__namespace__gets a generated .NET type, butClassManagerreflected that type as a plain exception (ExceptionClassObject) before checking whether it was a Python-derived type. So:__pyobj__field (ClassDerivedObject.NewObjectToPythondoes that,ExceptionClassObjectdoesn't). When the generated constructor ran,PythonDerivedType.InvokeCtorfound__pyobj__empty, created a temporary wrapper, stored its pointer and released it. Later calls through the .NET object (an overridden virtual method, the object coming back from .NET throughClassDerivedObject.ToPython) used a freed object, and the process could crash with SIGSEGV. A subclass without__init__crashed on construction.__init__doesn't call the base__init__never ran a constructor, so field initializers stayed null and the constructor's defaults (such as the message) weren't set.Generated exception types now get
ExceptionClassDerivedObject, aClassDerivedObjectwith the exceptiontp_repr,tp_strand__init__handling (argsand__cause__) ofExceptionClassObject. They behave like any other__namespace__subclass: the wrapper is set on the object, the base parameterless constructor runs before__init__, and the Python object's lifetime follows the .NET object's. Reflected exceptions that aren't generated, and generated types that aren't exceptions, are handled as before.With the wrapper now valid, a Python override of
ToStringon such a subclass is reached from .NET, sostr()of the exception returns it, as it already does for ordinary__namespace__subclasses.Making these types derived types exposed three more problems, fixed here too:
__namespace__subclasses. Each generated level has its ownFinalize, and each callsPythonDerivedType.PyFinalizeon the same object.PyFinalizecleared only a copy of__pyobj__, so the same Python object was queued, and freed, once per level, and the process crashed.PyFinalizenow clears the field once it has queued the object and skips an object already queued. Chains of__namespace__subclasses ofSystem.Objectcrashed the same way on master; this fixes them too. The generatedFinalizenow runs its base class finalizer beforePyFinalize, so a .NET base finalizer that calls a virtual or abstract member still reaches the Python override. The innermost generated level queues the Python object once the finalizer chain has run, and the outer levels skip it.ClassDerivedObject.tp_clearis a no-op, since the .NET finalizer frees the object, so an exception referencing itself throughargs,__cause__,__context__or__traceback__was never collected.ExceptionClassDerivedObject.tp_clearclears those fields (the native exception base'stp_clear), and leaves the wrapper and its handle for the .NET object.tp_str,tp_reprand__init__) now run all their work inside a guard that raises an error to the caller, instead of letting it escape the slot and abort the process.tp_strrunsToString(),tp_reprreadsMessage, and__init__runsExceptions.SetArgsAndCause, which readsMessage. The error is raised with its__cause__. A Python override of the member throws itsPythonExceptiondirectly, and that is raised as it is: itsInnerExceptionis the Python error's cause, not a wrapper. A member the Python subclass inherits, such asException.ToString, runs through reflection (PythonDerivedType.InvokeMethod), which wraps what it throws, including a Pythonget_Messageoverride's error, in aTargetInvocationException; only that wrapper is unwrapped. ANonereturned by a PythonToStringoverride reads as an empty message. The__init__of reflected and generated exception types shares one guardedExceptionClassObject.SetArgsAndCause.ClassBase.tp_strunwraps the cause the same way for other derived types; that predates this change and is str() of a Python subclass of a .NET type raises the cause of its ToString override's error instead of the error #170.Tests
Both tests run their script in a subprocess, so a native crash fails the test instead of aborting the test run.
test_namespaced_exception_subclass_constructioncovers three__namespace__subclasses ofExtendedException: one without__init__, one whose__init__doesn't call the base (with aToStringoverride), and one callingsuper().__init__("message"). For each it checks that the field initializer ran, that the object comes back from a .NETList[Object]as the same Python object, that it can be raised and caught, itsstr()andargs, and that collection afterwards is clean. Without the runtime change it fails with return code -11 (SIGSEGV).test_namespaced_exception_subclass_releasereleases ten instances of a two-level__namespace__exception chain and asserts that no instance of 30 exceptions referencing themselves through__cause__,__context__orargssurvives collection. It also releases one- and two-level__namespace__subclasses of new fixtures whose finalizer calls an abstract or virtualCallback, and asserts that the finalizer got the Python override's value. Releasing runs Python's and .NET's GC, then calls into a Python override from .NET 300 times, which drains the queue of Python objects to release. It fails with -11 without thePyFinalizechange, leaves the cycles uncollected withouttp_clear, and aborts (-6,NotImplementedExceptionon the finalizer thread) with the previous finalizer order.test_namespaced_exception_subclass_override_errorsconstructs an exception whoseToStringoverride returnsNoneand assertsstr()is''andrepr()shows theMessageoverride. It then makes both overrides raiseValueError("outer") from KeyError("inner")and asserts thatstr(),repr()and a constructor callingsuper().__init__()each raise thatValueErrorwith itsKeyErrorcause. It checks the same forstr()of a subclass that inheritsToStringand overrides onlyget_Message. Disabling any one of the three guards aborts the subprocess (-6). Restoring the cause unwrapping makes theKeyErrorescape, and dropping theTargetInvocationExceptionunwrapping fails the inherited case. The earlier head d282ec1 aborts on theNonecase.The CI steps, on Python 3.11 / .NET 10:
dotnet test src/embed_tests/ --filter "FullyQualifiedName!~SetPythonPath": 1049 passed, 8 skippeddotnet test src/embed_tests/ --filter "FullyQualifiedName~SetPythonPath": 1 passedpytest --runtime netcore tests: 489 passed, 23 skippeddotnet test src/python_tests_runner/: 2 passedAlso tested end to end in Lean Python backtests:
System.ArgumentExceptionconstructs, round-trips and raises each, and fails if any result is wrong. It passed with this change: the base constructor's message was set, the same object came back from .NET, theToStringoverride was used, andargsmatched. The same algorithm crashed with a segmentation fault on current master.ToStringoverride surfaces as a catchableValueError. It passed. A third algorithm defines an exception whoseToStringoverride returnsNone, and whoseToStringandget_Messageoverrides then raise a chainedValueError. It checks thatstr()is'', and thatstr(),repr()and construction each raise theValueErrorwith itsKeyErrorcause. It passed.Callbackoverride of one- and two-level__namespace__subclasses (abstract and virtual), and thatstr()of a subclass inheritingToStringraises itsget_Messageoverride'sValueErrorwith itsKeyErrorcause. It passed.Related, predating this change: #170 (
ClassBase.tp_strraises a chained error's cause) and #172 (a__namespace__subclass of a__namespace__subclass doesn't reach its parent's Python overrides from .NET).