Skip to content

Reset delayed rewards on automatic episode restarts - #384

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/delayed-reward-auto-reset
Open

sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/delayed-reward-auto-reset

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #383.

Failure

A dm_env environment may return a FIRST timestep from step() when the previous episode has ended, without an explicit call to reset(). DelayedRewardWrapper currently handles clearing accumulated rewards only inside its own reset().

As a result, after a terminal or truncated episode, DelayedRewardWrapper.step() attempts to add the next episode's initial reward to the old accumulator. A normal dm_env.restart() has reward=None. With scalar rewards this raises a type error; with nested reward dictionaries, it raises a tree-structure mismatch. The accumulation counter also incorrectly continues across the episode boundary.

Change

Handle timestep.first() immediately after stepping the wrapped environment. Clear the delayed reward buffer and accumulation counter, then return the new episode's FIRST timestep unchanged. This matches the existing explicit-reset semantics and prevents both the exception and cross-episode reward carryover.

The normal MID/LAST reward accumulation logic is unchanged, including the flush on episode termination or truncation. This is independent of the FrameStackingWrapper automatic-reset work in #376 / #377.

Validation

  • Four new automatic-reset regression cases failed against unchanged master.
  • Seven focused tests pass with the fix: scalar and nested rewards, terminal and non-zero-discount truncated episodes, three successive episode transitions, and explicit-reset controls for accumulation periods 1, 2 and None.
  • The regression runs the production DelayedRewardWrapper against a deterministic dm_env.Environment using real dm-env, NumPy and dm-tree.
  • Ruff correctness checks (E4, E7, E9, F), Python compilation and git diff --check pass.

Scope and limitations

This changes only how an unexpected FIRST timestep is handled in step(). The repository's TensorFlow/Reverb-based full test environment is not installed locally; the focused tests were run with an isolated import harness to avoid those unrelated legacy dependencies. No global training or integration run is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DelayedRewardWrapper crashes when step() returns an automatic reset timestep

1 participant