Skip to content

test(validator): make provider input contract tests pass on Windows - #555

Open
mogul wants to merge 2 commits into
fix/validator-windows-exec-bitfrom
test/windows-harness-portability
Open

mogul wants to merge 2 commits into
fix/validator-windows-exec-bitfrom
test/windows-harness-portability

Conversation

@mogul

@mogul mogul commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Makes scripts/tests/test_provider_input_contract.py portable to Windows by launching normalizers through the interpreter the runtime uses, instead of executing shebang scripts directly. Windows' CreateProcess only loads PE images, so subprocess.run([str(script_path), ...]) fails with WinError 193 there and only ever worked on POSIX.

This PR is stacked on #547 and targets fix/validator-windows-exec-bit (a07c3d5). When #547 merges into main, this PR retargets to main automatically.

The symlink gating in this file now comes from the base branch, not from this PR. #547 at a07c3d5 already defines _symlink_skip_reason() and requires_symlink_support and already decorates test_windows_contract_keeps_symlink_and_inside_root_checks. This PR reuses that decorator and does not define a second copy; it only applies it to the two symlink tests the base does not yet cover.

Tests only. The diff touches exactly one file, scripts/tests/test_provider_input_contract.py (+26/−2). No product code changed, no assertion relaxed.

Evidence chain

Measured on a real Windows 11 host (native python.org 3.12.10, os.name == "nt"), running just this test file:

Base Result
origin/main 7dd2ddc 19 failed, 185 passed, 0 skipped
#547 @ 9af3707 5 failed, 252 passed, 2 skipped
#547 @ a07c3d5 (current base) 4 failed, 257 passed, 3 skipped
This branch (3ac3458) 2 failed, 257 passed, 5 skipped

At main, 15 of those 19 failures are the validator exec-bit defect that #547 fixes (st_mode & 0o111 is always 0 on Windows, so every bundle is rejected with normalizerPath must be an executable file). At 9af3707 the remaining harness failures were 2 × WinError 193 and 3 × WinError 1314. At a07c3d5, the base's own gate already removes one WinError 1314 failure; the 4 left are 2 × WinError 193 and 2 × WinError 1314 — both classes are removed on this branch.

Originating validation: #547 (comment)

What changed (this PR's contribution)

  1. _normalizer_argv() — one shared helper that builds argv the way the runtime launches a normalizer: .mjs through node, .py through sys.executable. A missing node is a skip with an explicit reason, never a silent pass.
  2. Its use in the two direct-invocation tests — test_usai_normalizer_output_satisfies_runtime_contract and test_fixture_normalizers_are_directly_invocable. Their contract assertions are unchanged.
  3. @requires_symlink_support on the two symlink tests the base does not decorate — test_runtime_validator_rejects_artifact_symlink and test_runtime_validator_rejects_normalizer_symlink. The decorator itself is the base's.

Explicitly not included: any second copy of _symlink_skip_reason() / requires_symlink_support, and any decoration of test_windows_contract_keeps_symlink_and_inside_root_checks (the base already does it, a second decorator would be redundant).

The 2 remaining failures

Both are product/fixture defects, and both now surface as real assertion/contract failures rather than WinError 193 — the normalizers actually execute, which is the point of this PR. WinError 193 appears zero times in the run.

Test Actual error Cause Owner
test_usai_normalizer_output_satisfies_runtime_contract ContractError: … catalog.json: cannot read JSON: [Errno 2] No such file or directory normalize-catalog.mjs guards on import.meta.url === `file://${process.argv[1]}` , which is file:///C:/… vs file://C:\… on Windows, so main() never runs: exit 0, no output written. Silent no-op. #556
test_fixture_normalizers_are_directly_invocable AssertionError: b'{\r\n …' != b'{\n …' (at index 1: b'\r' != b'\n') Fixture normalize.py writes with Path.write_text(...), which translates \n to CRLF on Windows while the committed snapshot is LF. JSON is equal; only bytes differ. #556

#556 (fix/windows-mjs-entrypoints) touches both normalize-catalog.mjs and the two fixture normalize.py files (adding newline="\n"), so it is the finish line for these two.

Skip list

Test Skip reason Runs on a Developer-Mode Windows runner?
test_runtime_validator_rejects_artifact_symlink SeCreateSymbolicLinkPrivilege not held / no symlink-capable filesystem Yes
test_runtime_validator_rejects_normalizer_symlink same Yes
test_windows_contract_keeps_symlink_and_inside_root_checks same (gated by the base, not by this PR) Yes
test_runtime_validator_rejects_non_executable_normalizer_on_posix POSIX mode bits are not reported on Windows No — permanently POSIX-only (pre-existing, preserved)
test_runtime_validator_ignores_shebang_when_posix_mode_bits_missing POSIX mode bits are not reported on Windows No — permanently POSIX-only (pre-existing, preserved)

GitHub's Windows runners have Developer Mode enabled, so the three symlink tests execute there. On this reporting host they skip because the account cannot create symlinks at all (WinError 1314).

Are the security assertions weakened?

No. All three symlink test bodies are untouched; the only change is a decorator, and the gate is capability-based (a real symlink attempt in a temporary directory), not os.name-based. Because this host cannot create symlinks at all, the gate's behaviour was verified by simulating the probe: a capable host returns the test unchanged (no skip); an incapable host attaches the skip with its reason; and if the probe wrongly claimed capability the test fails loudly with OSError winerror=1314 rather than passing. The assertions were not exercised on a symlink-capable host locally — CI is authoritative there.

Validation

# this branch, Windows 11, python 3.12.10 (os.name == nt)
python -m pytest scripts/tests/test_provider_input_contract.py -q
2 failed, 257 passed, 5 skipped
# WinError 193 occurrences in that run: 0

# tests-only scope
git diff --name-only a07c3d5
scripts/tests/test_provider_input_contract.py

# lint
ruff check scripts          -> All checks passed!
ruff format --check scripts -> 41 files already formatted
  • make is not on PATH on this host, so make validate could not be run as written. Its equivalent, python scripts/validate_repo.py, fails here for a reason unrelated to this change: skills is a tracked symlink (mode 120000 → .agents/skills) that git materialises as a plain text file when the account cannot create symlinks, so the pattern tree is unreachable and 17 reference errors are reported. That is a property of an unprivileged Windows checkout, not a regression from this PR.
  • CI note: ci.yml (and frontend-skills-qa.yml, zizmor.yml) trigger only on pull_request to main, so a PR based on fix/validator-windows-exec-bit gets only pr-lint. Linux behaviour is unchanged by this PR, and this branch's CI should be green once fix(validator): make normalizer exec-bit check Windows-aware #547 merges and this PR retargets to main; on POSIX the symlink tests run as before and the sys.executable / node prefix is a normal launch. I am not claiming local Linux results.

What this does not change

  • Validator behaviour in scripts/validate_provider_input_contract.py is untouched (that is fix(validator): make normalizer exec-bit check Windows-aware #547's subject).
  • Linux coverage is unchanged, and no assertions were relaxed.
  • No blanket os.name == "nt" skip; the only skip condition involved is the base's probed symlink capability.

AI Assistance Disclosure

AI-assisted (OpenCode [deepseek-v4.1-flash]); human review required.

  • AI tools were used in this contribution: OpenCode (deepseek-v4.1-flash) — diagnosed the Windows failure modes, drafted the test-only portability change, and ran it on a native Windows host.
  • I verified all AI-generated content: no fabricated/hallucinated APIs, dependencies, citations, or placeholder content.
  • Any inherited/reused third-party material retains its original license and is identified.

Eligibility & Accountability

  • I am a federal employee or contractor
  • I understand this change and can explain it in my own words
  • I attest I have the right to release this work under the repository's CC0-1.0 dedication

Left unchecked deliberately: eligibility and authorship attestation are for a human to make, not for the AI assistant that drafted this PR.

Type of Change

  • Infrastructure/tooling change
  • Bug fix

Testing

  • Ran make validate locally — make is absent on this Windows host; see Validation above for the equivalent command and its environment-specific result
  • Ran the test suite directly (pytest scripts/tests/test_provider_input_contract.py) on native Windows
  • Tested with at least one real scenario — real Windows 11 host, native CPython 3.12.10, os.name == "nt"
  • Commit messages follow Conventional Commits

Safety Checklist

  • No secrets, API keys, tokens, or passwords included
  • No PII
  • No CUI
  • No internal URLs, hostnames, or system paths
  • No customer data or proprietary information

Validation Results

python -m pytest scripts/tests/test_provider_input_contract.py -q
...
FAILED ...::test_usai_normalizer_output_satisfies_runtime_contract
FAILED ...::test_fixture_normalizers_are_directly_invocable
2 failed, 257 passed, 5 skipped

Related Issues

Relates to #547 (stacked base)
Relates to #556 (owner of the 2 remaining failures)

Additional Context

Sequence for "green on Windows": land #547, land this branch, land #556.

Worth noting for the wider effort: a default non-elevated Windows host cannot materialise this repository's skills git symlink at all, so make validate and part of the broader suite cannot run there regardless of this PR. That is a separate decision about whether Windows support requires Developer Mode.

Reviewer Notes

Reviewers should verify:

  • The diff is confined to scripts/tests/test_provider_input_contract.py and contains no product-code change
  • No second copy of _symlink_skip_reason() / requires_symlink_support is introduced — the base's is reused
  • test_windows_contract_keeps_symlink_and_inside_root_checks is decorated exactly once (by the base)
  • Both direct-invocation tests assert the same contract behaviour, now via the interpreter
  • The two POSIX-only mode-bit skips are untouched
  • The 2 residual failures are correctly attributed to fix(entrypoints): compare import.meta.url as a URL so mjs guards fire on Windows #556 and are no longer WinError 193

mogul added 2 commits October 30, 2026 14:19
Windows' CreateProcess only loads PE images, so executing a shebang
script directly fails with WinError 193. That made
test_usai_normalizer_output_satisfies_runtime_contract and
test_fixture_normalizers_are_directly_invocable POSIX-only despite
asserting platform-independent behaviour.

Launch normalizers the way the runtime does instead: .mjs through node
and .py through sys.executable, via one shared helper. A missing node is
a skip with an explicit reason rather than a silent pass. The contract
assertions are unchanged.
The base branch now provides _symlink_skip_reason and
requires_symlink_support and already decorates
test_windows_contract_keeps_symlink_and_inside_root_checks, so the copy
of that helper on this branch is redundant.

Drop the duplicate helper and apply the base decorator to the two
symlink tests the base does not cover:
test_runtime_validator_rejects_artifact_symlink and
test_runtime_validator_rejects_normalizer_symlink. Test bodies and
assertions are untouched.
@mogul
mogul force-pushed the test/windows-harness-portability branch from ecba5a5 to 3ac3458 Compare October 30, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant