Skip to content

Return clean FIRST timesteps after random no-op starts - #386

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/noop-starts-clean-first-timestep
Open

sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/noop-starts-clean-first-timestep

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #385.

Problem

NoopStartsWrapper.reset() executes a random number of no-op actions to randomize the initial state of an otherwise deterministic environment. It currently takes the final timestep and replaces only its step_type with FIRST.

When the final no-op action produces an ordinary MID timestep, its reward and discount remain populated. The synthetic FIRST is therefore internally inconsistent: it carries a reward and discount for an initialization-only step. Agents or episode collectors treating FIRST as the beginning of a fresh episode may incorrectly consume those fields. The same situation arises after the wrapper resets an episode that terminated during the no-op sequence and then continues stepping.

Fix

Return dm_env.restart(timestep.observation) after the initialization steps. This preserves the actual final observation, including the updated state reached after the no-op actions, but correctly returns reward=None and discount=None on FIRST. The number of no-ops sampled, their action values and handling of early terminal timesteps are unchanged.

Reproduction and testing

  • Three new regression scenarios fail on unchanged master: one no-op, three no-ops with an internal episode restart, and repeated short episodes during initialization. In each, FIRST incorrectly carries the previous step's reward.
  • Six focused tests pass after the fix. They cover 0, 1, 2, 3 and 5 no-op actions, the observation after early termination, correct action forwarding, and the next normal environment step following reset.
  • Tests execute the production NoopStartsWrapper against a deterministic dm_env.Environment with actual dm_env timesteps, without relying on a mocked return value from step().
  • Ruff correctness checks (E4, E7, E9, F), Python compilation and git diff --check pass.

Scope and limitations

The change only corrects the synthetic FIRST timestep returned by reset(). No-op selection, random seeds and underlying episode control are unchanged. The complete TensorFlow/Reverb-based Acme integration environment is not installed locally; the focused regression was run using an isolated import harness to avoid those unrelated legacy dependencies. This is independent of the DelayedRewardWrapper fix in #384.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NoopStartsWrapper returns FIRST timesteps with stale rewards and discounts

1 participant