Repository navigation
feat(repo): guidance:overlay arm and overlay reach check #338
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Added | ||
|
|
||
| - **`guidance:overlay` arm for per-agent AGENTS.md overlays** — `dokimi repo run --arm guidance:overlay` gives a cell the base `AGENTS.md` and the `AGENTS.<agent>.md` of the harness under test, and removes the overlays of other agents (agentsmd/agents.md#185). A harness profile names its agent with the new `overlay` key (`claude` for Claude Code, `codex`, `copilot`). The arm refuses to run without that key, or when the base has no overlay for the agent. `dokimi repo reach` now probes every `AGENTS.<name>.md`, and reports a foreign overlay that the harness loaded as a `finding:` line and in the `overlays` block of `guidance_reach.v1`. Its `own_loaded` field says whether the harness loads its own overlay at all. | ||
|
|
||
| ### Changed | ||
|
|
||
| - **`guidance:none` also removes the agents' overlays** — a `guidance:none` cell now deletes every `AGENTS.claude.md`, `AGENTS.codex.md` and `AGENTS.copilot.md` as well as the files the guidance globs match; a `guidance:as-skill` cell drops the same overlays and does not package them. The cell's execution id is unchanged, so on a repository with overlays a `guidance:none` or `guidance:as-skill` result is not comparable with one from an earlier version: start a new `--batch` (or re-run with `--force`) rather than resuming an old one. `repo tasks replay` now leaves overlays out of `change.patch` as it does guidance files; a replay task built earlier from a PR that edits an overlay fails to prepare under `guidance:none` and `guidance:as-skill`, and under `guidance:overlay` when the edited overlay is another agent's, until it is rebuilt with `--force`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -330,7 +330,7 @@ belongs to the repository's grader and is passed through unread, except `grader` | |
| 3. `git init` and commit (no history, so fixes cannot leak) | ||
| 4. apply `setup_patches` | ||
| 5. apply the arm: `guidance:none`, `guidance:raw`, `guidance:bridge`, | ||
| `guidance:overlay`, `guidance:rev:<rev>`, `patch:<file>`, `guidance:format=xml` (the guidance | ||
| files rendered as tagged sections, `dokimi/repos/guidance_format.py`) or | ||
| `guidance:as-skill` (the guidance files removed, their text packaged as one | ||
| skill, see [Guidance formats and packaging](benchmarking-and-comparison.md#guidance-formats-and-packaging)) | ||
|
|
@@ -346,13 +346,30 @@ Every `guidance:*` arm is made by the framework from `guidance.globs`, the | |
| harness profile and the source's own history; a repository supplies no arm | ||
| content: | ||
|
|
||
| - `none` deletes every file the globs match at the base and every overlay | ||
| `AGENTS.<agent>.md`, which is matched by its name, not by the globs; `as-skill` | ||
| drops the same files, packaging only the matched ones; `raw` keeps them. The `rev:` arm keeps the overlays: a repository with overlays that wants a | ||
| revision without them lists `**/AGENTS.*.md` in `guidance.globs` too. | ||
| - `bridge` exists only for a profile with a `bridge` key. It writes that file | ||
| (for Claude Code, a `CLAUDE.md` containing `@AGENTS.md`) next to every | ||
| matched `AGENTS.md`, not only at the root. Whether a harness needs it | ||
| depends on its version: Claude Code 2.1.286 loads `AGENTS.md` itself, but | ||
| only when no `CLAUDE.md` is in its start-up chain, so a repository-side | ||
| bridge arm would mean different things on different versions. | ||
| - `overlay` exists only for a profile with an `overlay` key (the agent name, for | ||
| Claude Code `claude`). It is the per-agent overlay of agentsmd/agents.md#185: | ||
| `AGENTS.<agent>.md` beside an `AGENTS.md`, at any depth. The workspace keeps | ||
| the base `AGENTS.md` and the `AGENTS.<overlay>.md` files and removes every | ||
| other `AGENTS.<name>.md` (not the harness's own names such as | ||
| `AGENTS.override.md`, nor files under `.claude`). The files are recognised by | ||
|
Contributor
There was a problem hiding this comment. With the usual On this PR's test fixture, Suggestion: have
Contributor
Author
There was a problem hiding this comment. Documented in aa9fc49:
Contributor
Author
There was a problem hiding this comment. Maintainer decision: keep. guidance:none and rev: match only guidance.globs, as the contract says. No change. |
||
| name, not by `guidance.globs`. The name must be the `overlay` key of a builtin | ||
| profile (`claude`, `codex`, `copilot`): `AGENTS.ja.md` is an ordinary file. A | ||
| profile loaded from a path can reuse one of those names (a copy of `claude-code.yaml` | ||
| keeps `overlay: claude`), but one whose `overlay` key is not a builtin name has no | ||
| recognised overlay, so its arm is refused. A base with no `AGENTS.md` at its root, or | ||
| with no `AGENTS.<overlay>.md`, is a preparation error, so a cell cannot be the base alone | ||
| or an overlay with no base by accident. The arm | ||
| does not make a harness load its overlay: the reach probe reports that. | ||
| - `rev:<rev>` keeps the base's code and replaces its guidance with that of | ||
| another revision of the source: the base's matched files are deleted and the | ||
| files the globs match at `<rev>` (outside `eval_dir`) are written. It answers | ||
|
|
@@ -909,8 +926,10 @@ that a commit on the first-parent path up to the reviewed revision answered (as | |
|
|
||
| **Guidance and `eval_dir` are left out.** Workspace preparation applies the | ||
| arm before the review patch (§3, steps 5 and 7): `guidance:none` deletes the | ||
| files the guidance globs match and every `AGENTS.<agent>.md` overlay, | ||
| `guidance:overlay` deletes the overlays of other agents and `guidance:rev:` | ||
| replaces the matched files, so a hunk on one would not apply, and a guidance | ||
| file the PR adds would reach every arm. Overlays count as guidance here. | ||
| `eval_dir` is never in a workspace. Files of either kind are left out of the | ||
| patch and listed in `left_out_files`, and their threads are left out of | ||
| `gold.json`. A pull request that changes nothing else gets no task. | ||
|
|
@@ -1025,6 +1044,7 @@ guidance: # DECLARED; verified by the reach probe, ne | |
| shadow_file: CLAUDE.local.md # written by `repo reach --scenarios shadow` (§7) | ||
| claude_displaces_agents: true # repo budget: a root CLAUDE* turns off every AGENTS.md, a nested one its own directory's | ||
| bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"} | ||
| overlay: claude # the AGENTS.claude.md the guidance:overlay arm keeps (agentsmd/agents.md#185) | ||
| args_if_present: # argv added after the "{tools}" element when the workspace file exists and the tool mode is `all` | ||
| .mcp.json: ["--mcp-config", ".mcp.json"] | ||
| ``` | ||
|
|
@@ -1274,6 +1294,25 @@ per scenario). | |
| It is not a tree file, so it has no `files` entry, no records and no | ||
| matrix row; the matrix prints one line per shadow scenario with its | ||
| class. | ||
| - Overlays (`overlays` in `guidance_reach.v1`, the design note of §3): every | ||
| `AGENTS.<name>.md` is probed, even where the globs miss it, so a foreign overlay | ||
| is tested rather than assumed absent. The expectation is that foreign overlays are | ||
| `not_loaded`. `cross_loaded` lists each foreign overlay some scenario reported | ||
| as `loaded`, `truncated` or `fetched`; the text report prints it as a `finding:` | ||
| line. `own_loaded` lists the harness's own overlays that loaded in some scenario; | ||
| when none did, the text report says so, but only if some own-overlay scenario | ||
| read as `not_loaded` and every own-overlay scenario gave a usable answer (none | ||
| errored, none was a refusal); otherwise it says loading could not be | ||
| determined. `own` is the profile's `overlay`. | ||
| `ambiguous` lists the names of a load that could have been read under another | ||
| name: a file with aliases when any harness-native guidance file is present in | ||
| the probe (it could import the target), or when the alias group holds two | ||
| overlay names. Those loads are neither own nor cross loads. The rule does not | ||
| depend on the profile's `guidance.imports`: codex and copilot (`imports: null`) | ||
| have it too, so with their `AGENTS.md` in the probe every overlay link is | ||
| ambiguous and never a `finding:`. The text | ||
| report prints no own-overlay note for them, and a foreign target with an own | ||
| alias is listed there too, not reported as a cross load. | ||
| - If payload capture is available, a `payload` field records whether each | ||
| token was in the request bytes. This settles cells where the model | ||
| under-reports, which is how a small model produces false `truncated` cells. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """The ``guidance:format=<fmt>`` and ``guidance:as-skill`` arms (workspace step 5, :mod:`.workspace`): the guidance | ||
| files rendered in place, or removed and packaged as one skill.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| from .arms import SKILLS_DIR, guidance_skill_name, tree_sha256 | ||
| from .errors import PrepError | ||
| from .guidance_format import FormatError, to_format | ||
| from .manifest import RepoManifest | ||
|
|
||
| #: The description of the skill ``guidance:as-skill`` packages the guidance into: the same for every repository. | ||
| GUIDANCE_SKILL_DESCRIPTION = ("Repository guidance for working in this codebase, its conventions, build and test " | ||
| "commands and rules. Load it before changing its code.") # no ": " (YAML plain scalar) | ||
|
|
||
|
|
||
| def _apply_format(ws: Path, matched: list[str], fmt: str) -> tuple[list[str], dict[str, Any]]: | ||
| """Each guidance file rewritten in ``fmt`` in place. A symlink is left alone: its target is rendered | ||
| once if it is a guidance file itself, and ``meta.json`` names the links that keep their raw bytes.""" | ||
| if not matched: | ||
| raise PrepError(f"guidance:format={fmt}: the base has no guidance file to render") | ||
| written, links = [], [] | ||
| for rel in matched: | ||
| if (ws / rel).is_symlink(): | ||
| _link_file(ws, rel, matched) # the link must reach a file that is rendered, not raw guidance | ||
| links.append(rel) | ||
| continue | ||
| try: | ||
| rendered = to_format(_guidance_text(ws, rel, f"guidance:format={fmt}"), fmt) | ||
| except FormatError as exc: | ||
| raise PrepError(f"guidance:format={fmt}: {rel}: {exc}") from exc | ||
| (ws / rel).write_bytes(rendered.encode("utf-8")) # no newline translation: the restore check holds on disk | ||
| written.append(rel) | ||
| return written, ({"guidance_links_kept": links} if links else {}) | ||
|
|
||
|
|
||
| def _link_file(ws: Path, rel: str, matched: list[str]) -> None: | ||
| """The regular file a guidance link of the workspace reaches, resolved as the filesystem resolves it (a linked | ||
| directory on the way counts). PrepError when the link dangles or cycles, or reaches a file outside the workspace | ||
| or outside the matched guidance files: that file would be dropped or left unrendered.""" | ||
| try: | ||
| target = (ws / rel).resolve(strict=True) | ||
| except (OSError, RuntimeError) as exc: # RuntimeError: a symlink loop, before Python 3.13 | ||
| raise PrepError(f"guidance link {rel} does not reach a file (a dangling link or a cycle): {exc}") from exc | ||
| root = ws.resolve() | ||
| if not target.is_relative_to(root) or not target.is_file(): | ||
| raise PrepError(f"guidance link {rel} reaches {target}, which is outside the workspace or not a regular file") | ||
| reached = target.relative_to(root).as_posix() | ||
| if reached not in matched: | ||
| raise PrepError(f"guidance link {rel} reaches {reached}, which is not a guidance file") | ||
|
|
||
|
|
||
| def _guidance_text(ws: Path, rel: str, arm: str) -> str: | ||
| """The text of a guidance file of the workspace; one that is not UTF-8 is a prep error of the cell.""" | ||
| try: | ||
| return (ws / rel).read_bytes().decode("utf-8") | ||
| except UnicodeDecodeError as exc: | ||
| raise PrepError(f"{arm}: {rel} is not UTF-8: {exc}") from exc | ||
|
|
||
|
|
||
| def _apply_as_skill(ws: Path, manifest: RepoManifest, matched: list[str]) -> tuple[list[str], dict[str, Any]]: | ||
| """The guidance files removed (as ``guidance:none`` removes them) and their text packaged as one skill, | ||
| ``.claude/skills/<repo>-guidance/SKILL.md``: each file under a ``## File: <path>`` line, its text verbatim.""" | ||
| if not matched: | ||
| raise PrepError("guidance:as-skill: the base has no guidance file to package") | ||
| name = guidance_skill_name(manifest.id) | ||
| skill_dir = ws / SKILLS_DIR / name | ||
| for part in (ws / ".claude", ws / SKILLS_DIR): # a repository link would write to the host | ||
| if part.is_symlink(): | ||
| raise PrepError(f"guidance:as-skill: {part.relative_to(ws)} is a symlink in the repository") | ||
| if skill_dir.exists() or skill_dir.is_symlink(): | ||
| raise PrepError(f"guidance:as-skill: the workspace already has {SKILLS_DIR}/{name}") | ||
| parts = [] | ||
| for rel in matched: | ||
| if (ws / rel).is_symlink(): # packaged once under its target's name; a link to anything else would be lost | ||
| _link_file(ws, rel, matched) | ||
| continue | ||
| text = _guidance_text(ws, rel, "guidance:as-skill") | ||
| parts.append(f"## File: {rel}\n\n" + (text if text.endswith("\n") else text + "\n")) | ||
| for rel in matched: | ||
| (ws / rel).unlink() | ||
| skill_dir.mkdir(parents=True) | ||
| frontmatter = f"---\nname: {name}\ndescription: {GUIDANCE_SKILL_DESCRIPTION}\n---\n\n" | ||
| (skill_dir / "SKILL.md").write_bytes((frontmatter + "\n".join(parts)).encode("utf-8")) # verbatim: no newline translation | ||
| return matched, {"guidance_skill": {"name": name, "path": f"{SKILLS_DIR}/{name}", | ||
| "sha256": tree_sha256(skill_dir)}} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.