diff --git a/dokimi/benchmarks/spec.v1.yaml b/dokimi/benchmarks/spec.v1.yaml index 25b413fdd..bf21281e0 100644 --- a/dokimi/benchmarks/spec.v1.yaml +++ b/dokimi/benchmarks/spec.v1.yaml @@ -76,7 +76,7 @@ future: # that matched almost nothing, max_docs one that escaped; raise it with each new doc. golden_manifest: min_docs: 10 - max_docs: 39 + max_docs: 40 include: - docs/**/*.md exclude: diff --git a/dokimi/changelog.d/20261031_120000_napetrov_ce-overlay.md b/dokimi/changelog.d/20261031_120000_napetrov_ce-overlay.md new file mode 100644 index 000000000..95bfad33e --- /dev/null +++ b/dokimi/changelog.d/20261031_120000_napetrov_ce-overlay.md @@ -0,0 +1,12 @@ + + +### Added + +- **`guidance:overlay` arm for per-agent AGENTS.md overlays** — `dokimi repo run --arm guidance:overlay` gives a cell the base `AGENTS.md` and the `AGENTS..md` of the harness under test, and removes the overlays of other agents (agentsmd/agents.md#185). A harness profile names its agent with the new `overlay` key (`claude` for Claude Code, `codex`, `copilot`). The arm refuses to run without that key, or when the base has no overlay for the agent. `dokimi repo reach` now probes every `AGENTS..md`, and reports a foreign overlay that the harness loaded as a `finding:` line and in the `overlays` block of `guidance_reach.v1`. Its `own_loaded` field says whether the harness loads its own overlay at all. + +### Changed + +- **`guidance:none` also removes the agents' overlays** — a `guidance:none` cell now deletes every `AGENTS.claude.md`, `AGENTS.codex.md` and `AGENTS.copilot.md` as well as the files the guidance globs match; a `guidance:as-skill` cell drops the same overlays and does not package them. The cell's execution id is unchanged, so on a repository with overlays a `guidance:none` or `guidance:as-skill` result is not comparable with one from an earlier version: start a new `--batch` (or re-run with `--force`) rather than resuming an old one. `repo tasks replay` now leaves overlays out of `change.patch` as it does guidance files; a replay task built earlier from a PR that edits an overlay fails to prepare under `guidance:none` and `guidance:as-skill`, and under `guidance:overlay` when the edited overlay is another agent's, until it is rebuilt with `--force`. diff --git a/dokimi/docs/benchmarking-and-comparison.md b/dokimi/docs/benchmarking-and-comparison.md index bf153c37c..ef12316fd 100644 --- a/dokimi/docs/benchmarking-and-comparison.md +++ b/dokimi/docs/benchmarking-and-comparison.md @@ -277,7 +277,7 @@ the answer model and judge fixed when comparing arms. ### Skill arms in repository mode `repo run` arms take the same `+` composition. Besides `guidance:none`, -`guidance:raw` and `guidance:bridge`, an arm can install Claude Code skills into +`guidance:raw`, `guidance:bridge` and `guidance:overlay`, an arm can install Claude Code skills into the task workspace at `.claude/skills//` before the agent starts: - `skill:`: one skill directory (holding `SKILL.md`); @@ -321,7 +321,7 @@ the cells the way the skill arms are compared above. components (`role`, `task`, `example`, ...) have no heading tag, so a POML arm needs a decision about which element a heading maps to. - `guidance:as-skill`: the guidance files are removed (as `guidance:none` removes - them) and their text goes into `.claude/skills/-guidance/SKILL.md`, each + them, the `AGENTS..md` overlays too, which are not packaged) and their text goes into `.claude/skills/-guidance/SKILL.md`, each file under a `## File: ` line with its text verbatim (a missing final newline is added); a symlink to another guidance file (`CLAUDE.md` to `AGENTS.md`) is packaged once under that file, and a symlink to any other file fails the cell. The frontmatter diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index 0b52048fe..ddffd08e3 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -7,7 +7,7 @@ SPDX-License-Identifier: Apache-2.0 > **Status: recommendations.** §1–§7 apply today. The measurement loop in §8 runs > today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`, -> `guidance:as-skill`, `guidance:format=xml`). +> `guidance:as-skill`, `guidance:format=xml`, `guidance:overlay`). > `guidance:bridge` measures the `CLAUDE.md` import (§7). The `skill:` and > `skill-catalog:` arms, which install skills, run today too > ([skill arms](benchmarking-and-comparison.md#skill-arms-in-repository-mode)). Arms @@ -256,7 +256,9 @@ Treat each item above as a hypothesis for your repository. 2. Run the baseline and the arm with `dokimi repo run`, on the current models you use. Today an arm can remove or keep the guidance (`guidance:none`, `guidance:raw`), or inject a file (`guidance:bridge`, which writes a `CLAUDE.md` - containing `@AGENTS.md` next to each `AGENTS.md` for profiles with a bridge key). + containing `@AGENTS.md` next to each `AGENTS.md` for profiles with a bridge key), or + keep only the base and the harness's own `AGENTS..md` (`guidance:overlay`, for + profiles with an `overlay` key). Testing an added tool, other than a skill (`skill:`, `skill-catalog:`), needs planned arms. Today, test one added guidance line with two task directories that differ only in a `setup_patches` entry (same prompt, same graders, distinct ids), or in their `base` commit (see diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 82b0121d2..f8a8ef06c 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -330,7 +330,7 @@ belongs to the repository's grader and is passed through unread, except `grader` 3. `git init` and commit (no history, so fixes cannot leak) 4. apply `setup_patches` 5. apply the arm: `guidance:none`, `guidance:raw`, `guidance:bridge`, - `guidance:rev:`, `patch:`, `guidance:format=xml` (the guidance + `guidance:overlay`, `guidance:rev:`, `patch:`, `guidance:format=xml` (the guidance files rendered as tagged sections, `dokimi/repos/guidance_format.py`) or `guidance:as-skill` (the guidance files removed, their text packaged as one skill, see [Guidance formats and packaging](benchmarking-and-comparison.md#guidance-formats-and-packaging)) @@ -346,13 +346,30 @@ Every `guidance:*` arm is made by the framework from `guidance.globs`, the harness profile and the source's own history; a repository supplies no arm content: -- `none` deletes every file the globs match at the base; `raw` keeps them. +- `none` deletes every file the globs match at the base and every overlay + `AGENTS..md`, which is matched by its name, not by the globs; `as-skill` + drops the same files, packaging only the matched ones; `raw` keeps them. The `rev:` arm keeps the overlays: a repository with overlays that wants a + revision without them lists `**/AGENTS.*.md` in `guidance.globs` too. - `bridge` exists only for a profile with a `bridge` key. It writes that file (for Claude Code, a `CLAUDE.md` containing `@AGENTS.md`) next to every matched `AGENTS.md`, not only at the root. Whether a harness needs it depends on its version: Claude Code 2.1.286 loads `AGENTS.md` itself, but only when no `CLAUDE.md` is in its start-up chain, so a repository-side bridge arm would mean different things on different versions. +- `overlay` exists only for a profile with an `overlay` key (the agent name, for + Claude Code `claude`). It is the per-agent overlay of agentsmd/agents.md#185: + `AGENTS..md` beside an `AGENTS.md`, at any depth. The workspace keeps + the base `AGENTS.md` and the `AGENTS..md` files and removes every + other `AGENTS..md` (not the harness's own names such as + `AGENTS.override.md`, nor files under `.claude`). The files are recognised by + name, not by `guidance.globs`. The name must be the `overlay` key of a builtin + profile (`claude`, `codex`, `copilot`): `AGENTS.ja.md` is an ordinary file. A + profile loaded from a path can reuse one of those names (a copy of `claude-code.yaml` + keeps `overlay: claude`), but one whose `overlay` key is not a builtin name has no + recognised overlay, so its arm is refused. A base with no `AGENTS.md` at its root, or + with no `AGENTS..md`, is a preparation error, so a cell cannot be the base alone + or an overlay with no base by accident. The arm + does not make a harness load its overlay: the reach probe reports that. - `rev:` keeps the base's code and replaces its guidance with that of another revision of the source: the base's matched files are deleted and the files the globs match at `` (outside `eval_dir`) are written. It answers @@ -909,8 +926,10 @@ that a commit on the first-parent path up to the reviewed revision answered (as **Guidance and `eval_dir` are left out.** Workspace preparation applies the arm before the review patch (§3, steps 5 and 7): `guidance:none` deletes the -files the guidance globs match and `guidance:rev:` replaces them, so a hunk on -one would not apply, and a guidance file the PR adds would reach every arm. +files the guidance globs match and every `AGENTS..md` overlay, +`guidance:overlay` deletes the overlays of other agents and `guidance:rev:` +replaces the matched files, so a hunk on one would not apply, and a guidance +file the PR adds would reach every arm. Overlays count as guidance here. `eval_dir` is never in a workspace. Files of either kind are left out of the patch and listed in `left_out_files`, and their threads are left out of `gold.json`. A pull request that changes nothing else gets no task. @@ -1025,6 +1044,7 @@ guidance: # DECLARED; verified by the reach probe, ne shadow_file: CLAUDE.local.md # written by `repo reach --scenarios shadow` (§7) claude_displaces_agents: true # repo budget: a root CLAUDE* turns off every AGENTS.md, a nested one its own directory's bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"} +overlay: claude # the AGENTS.claude.md the guidance:overlay arm keeps (agentsmd/agents.md#185) args_if_present: # argv added after the "{tools}" element when the workspace file exists and the tool mode is `all` .mcp.json: ["--mcp-config", ".mcp.json"] ``` @@ -1274,6 +1294,25 @@ per scenario). It is not a tree file, so it has no `files` entry, no records and no matrix row; the matrix prints one line per shadow scenario with its class. +- Overlays (`overlays` in `guidance_reach.v1`, the design note of §3): every + `AGENTS..md` is probed, even where the globs miss it, so a foreign overlay + is tested rather than assumed absent. The expectation is that foreign overlays are + `not_loaded`. `cross_loaded` lists each foreign overlay some scenario reported + as `loaded`, `truncated` or `fetched`; the text report prints it as a `finding:` + line. `own_loaded` lists the harness's own overlays that loaded in some scenario; + when none did, the text report says so, but only if some own-overlay scenario + read as `not_loaded` and every own-overlay scenario gave a usable answer (none + errored, none was a refusal); otherwise it says loading could not be + determined. `own` is the profile's `overlay`. + `ambiguous` lists the names of a load that could have been read under another + name: a file with aliases when any harness-native guidance file is present in + the probe (it could import the target), or when the alias group holds two + overlay names. Those loads are neither own nor cross loads. The rule does not + depend on the profile's `guidance.imports`: codex and copilot (`imports: null`) + have it too, so with their `AGENTS.md` in the probe every overlay link is + ambiguous and never a `finding:`. The text + report prints no own-overlay note for them, and a foreign target with an own + alias is listed there too, not reported as a cross load. - If payload capture is available, a `payload` field records whether each token was in the request bytes. This settles cells where the model under-reports, which is how a small model produces false `truncated` cells. diff --git a/dokimi/dokimi/commands/repo_run.py b/dokimi/dokimi/commands/repo_run.py index b39a7b733..4f1d38832 100644 --- a/dokimi/dokimi/commands/repo_run.py +++ b/dokimi/dokimi/commands/repo_run.py @@ -456,6 +456,7 @@ def register(repo_sub, eval_sub, positive_int, source_help: str, cache_help: str _common(run_p, source_help, cache_help, positive_int) run_p.add_argument("--arm", action="append", default=None, required=True, help="guidance:none | guidance:raw | guidance:bridge | guidance:as-skill | " + "guidance:overlay (the base with the AGENTS..md of the harness only) | " "guidance:format=xml | guidance:rev:REV (the base with the guidance files of revision REV " "of the source; the arm is named by REV's full SHA, resolved once per batch) | patch:FILE (a unified diff on this host, applied with git " "apply on top of guidance:raw; named patch:@, path and " diff --git a/dokimi/dokimi/repos/batch_ledger.py b/dokimi/dokimi/repos/batch_ledger.py index 1c7102d51..5eaeaabb9 100644 --- a/dokimi/dokimi/repos/batch_ledger.py +++ b/dokimi/dokimi/repos/batch_ledger.py @@ -70,9 +70,11 @@ def harness_sha256(path: Path) -> str: def execution_id(profile: HarnessProfile, tools: str, delegation: str = "native", harness_sha256: str | None = None, - harness_version: str | None = None, companion_sha256: dict[str, str | None] | None = None) -> str: + harness_version: str | None = None, companion_sha256: dict[str, str | None] | None = None, + overlay: str | None = None) -> str: """Short hash of how the agent is started: the profile's ``invoke`` template with its tool arguments, - environment, config, prompt suffix, egress, bridge, trace, answer format and companions. + environment, config, prompt suffix, egress, bridge, trace, answer format and companions, and for a cell of + the overlay arm the overlay it keeps (``overlay``; the other cells carry none, so their ids do not change). Two runs with the same id ran the same command line up to the per-cell values (prompt, model, caps); ``repo run`` reuses a finished cell only when its id matches. Includes the harness @@ -95,14 +97,17 @@ def execution_id(profile: HarnessProfile, tools: str, delegation: str = "native" if profile.bin_companions: key["bin_companions"] = (list(profile.bin_companions) if companion_sha256 is None else {n: companion_sha256.get(n) for n in profile.bin_companions}) + if overlay is not None: + key["overlay"] = overlay return hashlib.sha256(json.dumps(key, sort_keys=True).encode("utf-8")).hexdigest()[:12] def execution_record(profile: HarnessProfile, tools: str, delegation: str, harness_sha256: str, harness_version: str | None, - companion_sha256: dict[str, str | None] | None = None) -> dict[str, str]: + companion_sha256: dict[str, str | None] | None = None, + overlay: str | None = None) -> dict[str, str]: """How a cell's agent was started, as ``meta.json`` and ``result.json`` record it.""" - return {"id": execution_id(profile, tools, delegation, harness_sha256, harness_version, companion_sha256), + return {"id": execution_id(profile, tools, delegation, harness_sha256, harness_version, companion_sha256, overlay), "tools": tools, "delegation": profile.effective_delegation(tools, delegation)} diff --git a/dokimi/dokimi/repos/conformance_profiles/codex.yaml b/dokimi/dokimi/repos/conformance_profiles/codex.yaml index 0bde1d2ff..b11e06c54 100644 --- a/dokimi/dokimi/repos/conformance_profiles/codex.yaml +++ b/dokimi/dokimi/repos/conformance_profiles/codex.yaml @@ -82,3 +82,4 @@ skills: and .codex/skills in the project, $CODEX_HOME/skills and ~/.agents/skills, nested skills/group/ and symlinked skill dirs. Skills that share a name are all listed (identity is name + path). +overlay: codex # AGENTS.codex.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/guidance_arms.py b/dokimi/dokimi/repos/guidance_arms.py new file mode 100644 index 000000000..2cf49c0be --- /dev/null +++ b/dokimi/dokimi/repos/guidance_arms.py @@ -0,0 +1,91 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The ``guidance:format=`` and ``guidance:as-skill`` arms (workspace step 5, :mod:`.workspace`): the guidance +files rendered in place, or removed and packaged as one skill.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from .arms import SKILLS_DIR, guidance_skill_name, tree_sha256 +from .errors import PrepError +from .guidance_format import FormatError, to_format +from .manifest import RepoManifest + +#: The description of the skill ``guidance:as-skill`` packages the guidance into: the same for every repository. +GUIDANCE_SKILL_DESCRIPTION = ("Repository guidance for working in this codebase, its conventions, build and test " + "commands and rules. Load it before changing its code.") # no ": " (YAML plain scalar) + + +def _apply_format(ws: Path, matched: list[str], fmt: str) -> tuple[list[str], dict[str, Any]]: + """Each guidance file rewritten in ``fmt`` in place. A symlink is left alone: its target is rendered + once if it is a guidance file itself, and ``meta.json`` names the links that keep their raw bytes.""" + if not matched: + raise PrepError(f"guidance:format={fmt}: the base has no guidance file to render") + written, links = [], [] + for rel in matched: + if (ws / rel).is_symlink(): + _link_file(ws, rel, matched) # the link must reach a file that is rendered, not raw guidance + links.append(rel) + continue + try: + rendered = to_format(_guidance_text(ws, rel, f"guidance:format={fmt}"), fmt) + except FormatError as exc: + raise PrepError(f"guidance:format={fmt}: {rel}: {exc}") from exc + (ws / rel).write_bytes(rendered.encode("utf-8")) # no newline translation: the restore check holds on disk + written.append(rel) + return written, ({"guidance_links_kept": links} if links else {}) + + +def _link_file(ws: Path, rel: str, matched: list[str]) -> None: + """The regular file a guidance link of the workspace reaches, resolved as the filesystem resolves it (a linked + directory on the way counts). PrepError when the link dangles or cycles, or reaches a file outside the workspace + or outside the matched guidance files: that file would be dropped or left unrendered.""" + try: + target = (ws / rel).resolve(strict=True) + except (OSError, RuntimeError) as exc: # RuntimeError: a symlink loop, before Python 3.13 + raise PrepError(f"guidance link {rel} does not reach a file (a dangling link or a cycle): {exc}") from exc + root = ws.resolve() + if not target.is_relative_to(root) or not target.is_file(): + raise PrepError(f"guidance link {rel} reaches {target}, which is outside the workspace or not a regular file") + reached = target.relative_to(root).as_posix() + if reached not in matched: + raise PrepError(f"guidance link {rel} reaches {reached}, which is not a guidance file") + + +def _guidance_text(ws: Path, rel: str, arm: str) -> str: + """The text of a guidance file of the workspace; one that is not UTF-8 is a prep error of the cell.""" + try: + return (ws / rel).read_bytes().decode("utf-8") + except UnicodeDecodeError as exc: + raise PrepError(f"{arm}: {rel} is not UTF-8: {exc}") from exc + + +def _apply_as_skill(ws: Path, manifest: RepoManifest, matched: list[str]) -> tuple[list[str], dict[str, Any]]: + """The guidance files removed (as ``guidance:none`` removes them) and their text packaged as one skill, + ``.claude/skills/-guidance/SKILL.md``: each file under a ``## File: `` line, its text verbatim.""" + if not matched: + raise PrepError("guidance:as-skill: the base has no guidance file to package") + name = guidance_skill_name(manifest.id) + skill_dir = ws / SKILLS_DIR / name + for part in (ws / ".claude", ws / SKILLS_DIR): # a repository link would write to the host + if part.is_symlink(): + raise PrepError(f"guidance:as-skill: {part.relative_to(ws)} is a symlink in the repository") + if skill_dir.exists() or skill_dir.is_symlink(): + raise PrepError(f"guidance:as-skill: the workspace already has {SKILLS_DIR}/{name}") + parts = [] + for rel in matched: + if (ws / rel).is_symlink(): # packaged once under its target's name; a link to anything else would be lost + _link_file(ws, rel, matched) + continue + text = _guidance_text(ws, rel, "guidance:as-skill") + parts.append(f"## File: {rel}\n\n" + (text if text.endswith("\n") else text + "\n")) + for rel in matched: + (ws / rel).unlink() + skill_dir.mkdir(parents=True) + frontmatter = f"---\nname: {name}\ndescription: {GUIDANCE_SKILL_DESCRIPTION}\n---\n\n" + (skill_dir / "SKILL.md").write_bytes((frontmatter + "\n".join(parts)).encode("utf-8")) # verbatim: no newline translation + return matched, {"guidance_skill": {"name": name, "path": f"{SKILLS_DIR}/{name}", + "sha256": tree_sha256(skill_dir)}} diff --git a/dokimi/dokimi/repos/harness_profile.py b/dokimi/dokimi/repos/harness_profile.py index 547a7777c..432d3cd72 100644 --- a/dokimi/dokimi/repos/harness_profile.py +++ b/dokimi/dokimi/repos/harness_profile.py @@ -55,6 +55,7 @@ class HarnessProfile: guidance: dict[str, Any] = field(default_factory=dict) skills: dict[str, Any] = field(default_factory=dict) # declared skill dirs; verified only by `repo skill-reach` bridge: dict[str, str] | None = None + overlay: str | None = None # the agent name of its AGENTS..md (guidance:overlay, repos/overlays.py) prompt_suffix: str = "" config_files: dict[str, str] = field(default_factory=dict) bin_companions: tuple[str, ...] = () # files beside the executable that it needs; mounted next to it @@ -236,6 +237,12 @@ def _check_tools_element(raw: dict[str, Any], path: str) -> None: raise ProfileError(f"{path}: invoke must have exactly one {TOOLS_ELEMENT} element") +def _check_overlay(raw: dict[str, Any], path: str) -> None: + overlay = raw.get("overlay") + if overlay is not None and not re.fullmatch(r"[a-z0-9][a-z0-9-]*", str(overlay)): # fullmatch: `$` takes "\n" + raise ProfileError(f"{path}: overlay {overlay!r} is not an agent name (lower case, digits, dashes)") + + def parse_profile(raw: Any, path: str = "") -> HarnessProfile: errors = schema_errors("harness_profile.v1", raw) if errors: @@ -260,6 +267,7 @@ def parse_profile(raw: Any, path: str = "") -> HarnessProfile: raise ProfileError(f"{path}: invoke never passes {{prompt}} (set native_prompt for a harness that " "uses its own)") _check_tools_element(raw, path) + _check_overlay(raw, path) return HarnessProfile( name=raw["name"], bin=raw["bin"], @@ -275,6 +283,7 @@ def parse_profile(raw: Any, path: str = "") -> HarnessProfile: guidance=dict(raw.get("guidance") or {}), skills=dict(raw.get("skills") or {}), bridge=raw.get("bridge"), + overlay=raw.get("overlay"), prompt_suffix=raw.get("prompt_suffix") or "", config_files=dict(raw.get("config_files") or {}), bin_companions=tuple(raw.get("bin_companions") or ()), diff --git a/dokimi/dokimi/repos/overlay_paths.py b/dokimi/dokimi/repos/overlay_paths.py new file mode 100644 index 000000000..4389f185f --- /dev/null +++ b/dokimi/dokimi/repos/overlay_paths.py @@ -0,0 +1,85 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Link-aware path checks for the ``guidance:overlay`` arm (:mod:`.workspace`). + +The arm removes the overlays of other agents from a workspace. Before it does, it finds each one by its canonical +path (a name under a directory link is the file it reaches), and it checks whether a path of the base runs through +one of them, following symbolic links hop by hop as the workspace's own paths would be followed. +""" + +from __future__ import annotations + +import os +from pathlib import Path + +from .errors import PrepError + + +def _canon(path: Path) -> Path: + """``path`` with its directories resolved and its final name kept, so a link is compared by its own name.""" + return Path(os.path.realpath(path.parent)) / path.name + + +_MAX_HOPS = 64 # symbolic links followed by one walk; a cycle runs out + + +def _link_target(link: Path) -> Path: + """The target of a symbolic link as a path. On Windows ``os.readlink`` gives an absolute target with the ``\\\\?\\`` + long-path prefix, which the canonical paths of the workspace do not carry, so the prefix is dropped; a UNC share + keeps its server name as ``\\\\server\\share`` (``\\\\?\\UNC\\server\\share`` is that share, not a relative path).""" + target = os.readlink(link) + if target.startswith("\\\\?\\UNC\\"): + return Path("\\\\" + target[len("\\\\?\\UNC\\"):]) + return Path(target[len("\\\\?\\"):] if target.startswith("\\\\?\\") else target) + + +def _path_hits(ws: Path, rel: str, gone: set[Path]) -> bool: + """Whether the workspace path ``rel`` runs through one of the canonical paths ``gone``, a link or a plain name. + Walked from the workspace root: each name is checked before it is resolved, so a removed directory link named in + the middle of the path is a hit (the path would not exist once it is removed); a link is followed hop by hop, + and a cycle ends the walk.""" + resolved = Path(os.path.realpath(ws)) + todo = rel.split("/") + hops = 0 + while todo: + part = todo.pop(0) + if part in ("", "."): + continue + if part == "..": + resolved = resolved.parent + continue + cand = resolved / part + if cand in gone: + return True + if not cand.is_symlink(): + resolved = cand + continue + hops += 1 + if hops > _MAX_HOPS: + return False + target = _link_target(cand) # the native separators: a Windows target splits on backslashes + if target.is_absolute(): + resolved, parts = Path(target.anchor), target.parts[1:] + else: + parts = target.parts + todo[:0] = list(parts) + return False + + +def _overlay_targets(ws: Path, listed: list[str], arm: str = "guidance:overlay") -> dict[Path, str]: + """The overlays of other agents to remove, by canonical path (the directories resolved, so a name under a link + is the file it reaches), one name per file: a directory the index lists that a setup patch made a link to another + directory names the same file twice, and the first name in order stands for it.""" + names: dict[Path, str] = {} + for rel in listed: # a setup patch may have made a directory of the index a link out of the workspace + if not _canon(ws / rel).parent.is_relative_to(ws.resolve()): + raise PrepError(f"{arm}: {rel} is under a path that resolves outside the workspace") + names.setdefault(_canon(ws / rel), rel) + return names + + +def _supplies_a_file(ws: Path, rel: str) -> bool: + """Whether the workspace path ``rel`` resolves to a regular file inside the workspace.""" + real = Path(os.path.realpath(ws / rel)) + return real.is_file() and real.is_relative_to(ws.resolve()) diff --git a/dokimi/dokimi/repos/overlays.py b/dokimi/dokimi/repos/overlays.py new file mode 100644 index 000000000..6985165aa --- /dev/null +++ b/dokimi/dokimi/repos/overlays.py @@ -0,0 +1,56 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Per-agent overlays of AGENTS.md (agentsmd/agents.md#185): ``AGENTS..md`` beside an ``AGENTS.md``. + +An overlay is named by the agent it is for, in lower case (``AGENTS.claude.md``), and the name must be the +``overlay`` key of a builtin harness profile: ``AGENTS.ja.md`` or ``AGENTS.local.md`` is an ordinary file. The +``guidance:overlay`` arm keeps the overlay of that name and removes the others, and the reach probe reports any +foreign overlay a harness loaded anyway. +""" + +from __future__ import annotations + +import functools +import re +from pathlib import PurePosixPath + +from .harness_profile import PROFILE_DIR, builtin_profiles, load_profile + +OVERLAY_RE = re.compile(r"AGENTS\.([A-Za-z0-9][A-Za-z0-9-]*)\.md") + + +@functools.lru_cache(maxsize=1) +def native_names() -> frozenset[str]: + """Guidance names a builtin harness loads by its own convention (``AGENTS.override.md``): never an overlay.""" + return frozenset( + n + for name in builtin_profiles() + for n in load_profile(str(PROFILE_DIR / f"{name}.yaml")).guidance_names + ) + + +@functools.lru_cache(maxsize=1) +def overlay_names() -> frozenset[str]: + """The agent names a builtin harness profile has an overlay for (its ``overlay`` key): the only names a file + ``AGENTS..md`` is read as an overlay for. A profile loaded from outside the builtin set adds none.""" + names = set() + for name in builtin_profiles(): + overlay = load_profile(str(PROFILE_DIR / f"{name}.yaml")).overlay + if overlay is not None: + names.add(overlay) + return frozenset(names) + + +def overlay_of(path: str) -> str | None: + """The agent the guidance file ``path`` is an overlay for, ``None`` when it is not one. + + A file under ``.claude`` is skill content, not guidance, so it is never an overlay. A name that no builtin + profile has an overlay for (``AGENTS.ja.md``) is an ordinary file, not an overlay. + """ + parts = PurePosixPath(path).parts + if ".claude" in parts or parts[-1] in native_names(): + return None + match = OVERLAY_RE.fullmatch(parts[-1]) + name = match.group(1).lower() if match else None + return name if name in overlay_names() else None diff --git a/dokimi/dokimi/repos/profiles/claude-code.yaml b/dokimi/dokimi/repos/profiles/claude-code.yaml index b0e5ca1bf..97279f283 100644 --- a/dokimi/dokimi/repos/profiles/claude-code.yaml +++ b/dokimi/dokimi/repos/profiles/claude-code.yaml @@ -124,3 +124,4 @@ bridge: for: AGENTS.md write: CLAUDE.md content: "@AGENTS.md" +overlay: claude # AGENTS.claude.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/profiles/codex-review.yaml b/dokimi/dokimi/repos/profiles/codex-review.yaml index 05dc3c8f7..a6231f192 100644 --- a/dokimi/dokimi/repos/profiles/codex-review.yaml +++ b/dokimi/dokimi/repos/profiles/codex-review.yaml @@ -57,3 +57,4 @@ guidance: it opened AGENTS.md but reported only defects its P0-P3 rubric keeps; convention rules moved it only when the guidance called a violation merge-blocking (P1). It reports zero token usage in this mode. +overlay: codex # AGENTS.codex.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/profiles/codex.yaml b/dokimi/dokimi/repos/profiles/codex.yaml index dd450e2f9..467a18a0a 100644 --- a/dokimi/dokimi/repos/profiles/codex.yaml +++ b/dokimi/dokimi/repos/profiles/codex.yaml @@ -71,3 +71,4 @@ skills: and .codex/skills in the project, $CODEX_HOME/skills and ~/.agents/skills, nested skills/group/ and symlinked skill dirs. Skills that share a name are all listed (identity is name + path). +overlay: codex # AGENTS.codex.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/profiles/copilot-review.yaml b/dokimi/dokimi/repos/profiles/copilot-review.yaml index 0427aa917..8bb8da14d 100644 --- a/dokimi/dokimi/repos/profiles/copilot-review.yaml +++ b/dokimi/dokimi/repos/profiles/copilot-review.yaml @@ -76,3 +76,4 @@ guidance: REVIEW.md included. Measured with /review: the same files reached it, and the existing oneDAL guidance did not change its recall until the review rules sat in the root AGENTS.md. +overlay: copilot # AGENTS.copilot.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/profiles/copilot.yaml b/dokimi/dokimi/repos/profiles/copilot.yaml index b9bcce53c..121bd4a56 100644 --- a/dokimi/dokimi/repos/profiles/copilot.yaml +++ b/dokimi/dokimi/repos/profiles/copilot.yaml @@ -73,3 +73,4 @@ guidance: .github/instructions files are probed through the manifest's globs. The hosted Copilot code review (on a pull request) reads a different set, REVIEW.md included. +overlay: copilot # AGENTS.copilot.md, the guidance:overlay arm (agentsmd/agents.md#185) diff --git a/dokimi/dokimi/repos/reach.py b/dokimi/dokimi/repos/reach.py index d7f6c91b7..cf570098b 100644 --- a/dokimi/dokimi/repos/reach.py +++ b/dokimi/dokimi/repos/reach.py @@ -62,6 +62,8 @@ from .transcript import Transcript from .harness_profile import HarnessProfile from .manifest import RepoManifest, check_repo +from .overlays import overlay_of +from .reach_overlays import native_in_probe, overlay_report from .runner import SUPPORT_HOST from .source import RepoTree, SourceError @@ -208,7 +210,9 @@ def probe_files(tree: RepoTree, manifest: RepoManifest, profile: HarnessProfile) declared = set(check_repo(tree, manifest).guidance_files) names = set(profile.guidance_names) native = {p for p in tree.files() if PurePosixPath(p).name in names} - return sorted(p for p in declared | native if not manifest.in_eval_dir(p)) + # Every overlay, even where the globs miss it: whether a foreign one loads is the question (repos/overlays.py) + overlays = {p for p in tree.files() if overlay_of(p) is not None} + return sorted(p for p in declared | native | overlays if not manifest.in_eval_dir(p)) def check_files(files: list[str], profile: HarnessProfile) -> None: @@ -573,6 +577,8 @@ def run_reach( "unresolved": unresolved, "scenarios": scenario_rows, "records": records, + "overlays": overlay_report(records, profile.overlay, {s.path: s.aliases for s in seeded}, + native_in_probe(seeded, profile.guidance_names)), "summary": { "cost_usd": round(sum(r.get("cost_usd") or 0 for r in scenario_rows), 6), "by_class": by_class, @@ -663,6 +669,23 @@ def _matrix(records: list[dict[str, Any]]) -> dict[str, dict[str, str]]: return matrix +def _own_overlay_note(payload: dict[str, Any], overlays: dict[str, Any]) -> str: + """A "not loaded" note needs a usable negative record and no error: an errored scenario says nothing about loading.""" + group = {f["path"]: (f["path"], *f.get("aliases", ())) for f in payload["files"]} + own = set(overlays["own_files"]) + own_records = [r for r in payload["records"] if own & set(group.get(r["file"], (r["file"],)))] + seen = {r["class"] for r in own_records} + probed = {r["scenario"] for r in own_records} + # a scenario that errored, or answered with prose (a refusal), says nothing about what it loaded + usable = {s["id"] for s in payload["scenarios"] if not s.get("error") and (s.get("prose") or 0) <= PROSE_LIMIT} + negative = {r["class"] for r in own_records if r["scenario"] in usable} + files = ", ".join(overlays["own_files"]) + # a usable negative in one scenario does not clear a refusal in another: every probed scenario must be usable + if "not_loaded" in negative and "error" not in seen and probed <= usable: + return f"note: no {overlays['own']} overlay ({files}) was loaded in any scenario" + return f"note: could not determine whether the {overlays['own']} overlay ({files}) was loaded" + + def saved_gaps(payload: dict[str, Any], fmt: str) -> list[str]: """The fields a saved probe must carry to be rendered as ``fmt``: the schema leaves them optional.""" if fmt == "receipt": @@ -688,6 +711,19 @@ def saved_matrix_problem(payload: dict[str, Any]) -> str | None: return None +def _overlay_lines(payload: dict[str, Any]) -> list[str]: + """The overlay lines of the table: a ``finding:`` per foreign overlay that loaded, and the own-overlay note.""" + lines: list[str] = [] + overlays = payload.get("overlays") or {} + for path, seen in (overlays.get("cross_loaded") or {}).items(): + where = ", ".join(f"{sid} ({cls})" for sid, cls in seen.items()) + lines.append(f"finding: {path}, an overlay for another agent, was loaded by {payload['harness']['name']} in {where}") + own_ambiguous = set(overlays.get("own_files") or ()) & set(overlays.get("ambiguous") or ()) + if overlays.get("own_files") and not overlays.get("own_loaded") and not own_ambiguous: + lines.append(_own_overlay_note(payload, overlays)) + return lines + + def format_matrix(payload: dict[str, Any]) -> str: """A plain-text table: one row per guidance file, one column per scenario.""" short = _SHORT @@ -703,6 +739,7 @@ def format_matrix(payload: dict[str, Any]) -> str: row = payload["summary"]["matrix"].get(f["path"], {}) lines.append(f"{label[f['path']]:<{width}} " + " ".join(f"{short[row.get(s, 'error')]:>2}" for s in ids)) lines.append("L loaded T truncated F fetched (after a tool read) . not loaded E error") + lines += _overlay_lines(payload) if any(f.get("alias_kind") == "copy" for f in payload["files"]): lines.append("note: the paths after = are copies, not links (this OS refused a symlink); " "a harness that skips symlinks would still load them") diff --git a/dokimi/dokimi/repos/reach_overlays.py b/dokimi/dokimi/repos/reach_overlays.py new file mode 100644 index 000000000..464b2bf05 --- /dev/null +++ b/dokimi/dokimi/repos/reach_overlays.py @@ -0,0 +1,81 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Which guidance files of other agents, and which of the own overlay, a harness loaded: the overlay part of the +reach payload (``overlay_report``), kept apart from the probe itself so ``reach.py`` stays under its size cap.""" + +from __future__ import annotations + +from collections.abc import Iterable +from pathlib import PurePosixPath +from typing import Any + +from .overlays import overlay_of + +SEEN = ("loaded", "truncated", "fetched") # the classes that mean the harness had (some of) the file + + +def _pinned_names(links: dict[str, tuple[str, ...]], + native: set[str] | frozenset[str]) -> tuple[dict[str, tuple[str, ...]], set[str]]: + """Each probed file with the names a load of it can be pinned on, and the aliases that cannot be pinned.""" + names: dict[str, tuple[str, ...]] = {} + ambiguous: set[str] = set() + for target, aliases in links.items(): + group = (target, *aliases) + overlay_names = [n for n in group if overlay_of(n) is not None] + # a native file in the probe can import the target (CLAUDE.md with @docs/shared.md): the load is not the alias's; + # two overlay names name one file, so either could have been read + if aliases and (native or len(overlay_names) > 1): + ambiguous.update((*aliases, *overlay_names)) + names[target] = () + else: + names[target] = group + return names, ambiguous + + +def _loads(records: list[dict[str, Any]], names: dict[str, tuple[str, ...]], links: dict[str, tuple[str, ...]], + ambiguous: set[str], foreign: set[str], own_files: set[str]) -> tuple[dict[str, dict[str, str]], set[str], + set[str]]: + """The foreign overlays and own overlay files that loaded, by the names they can be pinned on, and the + aliases whose file loaded without a way to tell which of them was read.""" + cross: dict[str, dict[str, str]] = {} + own_loaded: set[str] = set() + unpinned: set[str] = set() + for r in records: + if r["class"] not in SEEN: + continue + for n in names.get(r["file"], ()): + if n in foreign: + cross.setdefault(n, {})[r["scenario"]] = r["class"] + if n in own_files: + own_loaded.add(n) + unpinned.update(n for n in (r["file"], *links.get(r["file"], ())) if n in ambiguous) + return cross, own_loaded, unpinned + + +def native_in_probe(seeded: list[Any], guidance_names: Iterable[str]) -> set[str]: + """The harness-native names (the profile's guidance names) that a probed file or link carries. A native file in + the probe can import a link's target; a configured name with no file in the probe cannot.""" + probed = {PurePosixPath(p).name for s in seeded for p in (s.path, *s.aliases)} + return probed & {PurePosixPath(n).name for n in guidance_names} + + +def overlay_report(records: list[dict[str, Any]], own: str | None, + links: dict[str, tuple[str, ...]], native: set[str] | frozenset[str] = frozenset()) -> dict[str, Any]: + """The overlay part of the payload: which of the ``own`` overlay files loaded, and which foreign overlays did + (``cross_loaded``, a finding: the harness read guidance written for another agent). ``links`` maps each probed + file to the links that name it: a file reached through an overlay-named link is an overlay under that name. + + A load is pinned on a link only when no other name of its file could have been read: the harness reads a + file of a ``native`` name by itself, and an own and a foreign overlay can name one file. Those links are + ``ambiguous`` when the file loaded, and are neither own nor cross loads.""" + every = {n for target, aliases in links.items() for n in (target, *aliases)} + own_files = {n for n in every if own and overlay_of(n) == own} + foreign = {n for n in every if overlay_of(n) not in (None, own)} + names, ambiguous = _pinned_names(links, native) + cross, own_loaded, unpinned = _loads(records, names, links, ambiguous, foreign, own_files) + report: dict[str, Any] = {"own": own, "own_files": sorted(own_files), "own_loaded": sorted(own_loaded), + "foreign_files": sorted(foreign), "cross_loaded": dict(sorted(cross.items()))} + if unpinned: + report["ambiguous"] = sorted(unpinned) + return report diff --git a/dokimi/dokimi/repos/replay/build.py b/dokimi/dokimi/repos/replay/build.py index 35fb5174f..ef7fce93c 100644 --- a/dokimi/dokimi/repos/replay/build.py +++ b/dokimi/dokimi/repos/replay/build.py @@ -29,6 +29,7 @@ from typing import Any from ..manifest import RepoManifest, match_glob +from ..overlays import overlay_of from .history import History, addressed, carry from .threads import PullRequest, ReplayError, Thread, gold_problem, review_bodies @@ -199,7 +200,8 @@ def _anchor_problem(h: History, manifest: RepoManifest, commit: str, path: str, def _left_out(manifest: RepoManifest, path: str) -> bool: - return any(match_glob(g, path) for g in manifest.guidance_globs) or manifest.in_eval_dir(path) + return (any(match_glob(g, path) for g in manifest.guidance_globs) or overlay_of(path) is not None + or manifest.in_eval_dir(path)) def _where(t: Thread) -> str: diff --git a/dokimi/dokimi/repos/runner.py b/dokimi/dokimi/repos/runner.py index 4fc174f6c..a5809f29a 100644 --- a/dokimi/dokimi/repos/runner.py +++ b/dokimi/dokimi/repos/runner.py @@ -77,7 +77,7 @@ from .host import probe_host, unmet, validate_tasks from .manifest import match_glob, scratch_name from .process_metrics import process_metrics -from .workspace import PrepError, Task, prepare_workspace +from .workspace import OVERLAY_ARM, PrepError, Task, parse_arm, prepare_workspace SUPPORT_HOST = Path(__file__).resolve().parent / "support" CONFIG_DIR = f"{RUN_DIR}/config" @@ -100,22 +100,26 @@ class AgentOptions: budget_usd: float = 2.0 -def execution_record(opts: AgentOptions) -> dict[str, str]: +def execution_record(opts: AgentOptions, arm: str | None = None) -> dict[str, str]: """How the cell's agent was started, as ``meta.json`` and ``result.json`` record it. The executable is digested for each cell, not once per batch: a cell records the bytes it actually ran, so a - binary replaced mid-batch shows up as cells of two executions, which the next run refuses.""" + binary replaced mid-batch shows up as cells of two executions, which the next run refuses. A cell of the + overlay arm also records the profile's overlay: the arm keeps that overlay's file, so another one is another run.""" beside = opts.harness_bin.resolve().parent # where harness_mounts takes the companions from companions = {n: harness_sha256(beside / n) if (beside / n).is_file() else None for n in opts.profile.bin_companions} + overlay = opts.profile.overlay if arm is not None and parse_arm(arm).base == OVERLAY_ARM else None return batch_ledger.execution_record(opts.profile, opts.tools, opts.delegation, harness_sha256(opts.harness_bin), - opts.harness_version, companions) + opts.harness_version, companions, overlay) def check_batch_executions(batch_dir: Path, options: list[AgentOptions], replacing: set[str] | None = None) -> None: """Refuse, before anything runs, a batch dir whose finished cells ran with another execution than ``options`` - (see :func:`batch_ledger.check_batch_executions`).""" - batch_ledger.check_batch_executions(batch_dir, {execution_record(o)["id"] for o in options}, replacing) + (see :func:`batch_ledger.check_batch_executions`). Each option's plain cells and its overlay cells are both + this execution, so a batch may hold either kind of cell of one profile.""" + wanted = {execution_record(o, a)["id"] for o in options for a in (None, OVERLAY_ARM)} + batch_ledger.check_batch_executions(batch_dir, wanted, replacing) def agent_network(opts: AgentOptions, ctx: EvalContext, task: Task) -> str: @@ -204,7 +208,7 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path, ``host`` (the probed facts of :func:`.host.probe_host`) is recorded in ``meta.json``.""" rd = batch_dir / cell_name(task.id, arm, opts.model, rep) - execution = execution_record(opts) + execution = execution_record(opts, arm) done = batch_ledger.reused_result(rd, execution, force) if done is not None: return {**done, "skipped": True} diff --git a/dokimi/dokimi/repos/workspace.py b/dokimi/dokimi/repos/workspace.py index e8183c36e..7cec53a9e 100644 --- a/dokimi/dokimi/repos/workspace.py +++ b/dokimi/dokimi/repos/workspace.py @@ -12,7 +12,7 @@ 3. ``git init`` and commit (no history, so an upstream fix cannot leak) 4. apply ``setup_patches`` 5. apply the arm (``guidance:none`` | ``guidance:raw`` | ``guidance:bridge`` | - ``guidance:as-skill`` | ``guidance:format=`` | ``guidance:rev:`` | + ``guidance:as-skill`` | ``guidance:overlay`` | ``guidance:format=`` | ``guidance:rev:`` | ``patch:@``, see :class:`Arm`), then its skills into ``.claude/skills/`` (:mod:`.arms`) 6. amend the commit @@ -50,20 +50,21 @@ from dokimi import process -from .arms import SKILLS_DIR, ArmError, Skill, guidance_skill_name, resolve_skills, tree_sha256 +from .arms import ArmError, Skill, resolve_skills, tree_sha256 from .errors import PrepError -from .guidance_format import FORMATS, FormatError, to_format +from .guidance_arms import GUIDANCE_SKILL_DESCRIPTION, _apply_as_skill, _apply_format # noqa: F401 +from .guidance_format import FORMATS from .guidance_history import arm_rev_history from .harness_profile import HarnessProfile from .manifest import RepoManifest, _is_within, match_glob +from .overlay_paths import _link_target, _overlay_targets, _path_hits, _supplies_a_file # noqa: F401 +from .overlays import overlay_names, overlay_of from .source import GitTree, RepoSource, RepoTree, SourceError, cache_path, default_cache_dir, isolated_git_env, resolve_rev from .uptake import MCP_CONFIG, base_servers, mcp_servers from .useful_spec import Useful, parse_useful -ARMS = ("guidance:none", "guidance:raw", "guidance:bridge", "guidance:as-skill") # the guidance parts -#: The description of the skill ``guidance:as-skill`` packages the guidance into: the same for every repository. -GUIDANCE_SKILL_DESCRIPTION = ("Repository guidance for working in this codebase, its conventions, build and test " - "commands and rules. Load it before changing its code.") # no ": " (YAML plain scalar) +ARMS = ("guidance:none", "guidance:raw", "guidance:bridge", "guidance:as-skill", "guidance:overlay") # the guidance parts +OVERLAY_ARM = "guidance:overlay" # base + the profile's AGENTS..md, other overlays removed (repos/overlays.py) REV_ARM, PATCH_ARM, FORMAT_ARM = "guidance:rev:", "patch:", "guidance:format=" #: Every form ``repo run --arm`` takes ARM_FORMS = (*ARMS, f"{REV_ARM}", f"{PATCH_ARM}", f"{FORMAT_ARM}<{'|'.join(FORMATS)}>", @@ -151,7 +152,7 @@ def _starts_part(text: str) -> bool: def _parse_guidance(spec: str) -> Arm: - """A guidance part: ``guidance:none|raw|bridge|as-skill``, ``guidance:format=``, + """A guidance part: ``guidance:none|raw|bridge|as-skill|overlay``, ``guidance:format=``, ``guidance:rev:`` or ``patch:``.""" if spec in ARMS: return Arm(spec) @@ -383,6 +384,13 @@ def prepare_workspace(git_dir: Path, manifest: RepoManifest, task: Task, arm: st if arm.base == "guidance:bridge" and not bridge: name = profile.name if profile is not None else "no harness" raise PrepError(f"guidance:bridge needs a harness profile with a `bridge` key; {name} has none") + overlay = profile.overlay if profile is not None else None + if arm.base == OVERLAY_ARM and not overlay: + name = profile.name if profile is not None else "no harness" + raise PrepError(f"guidance:overlay needs a harness profile with an `overlay` key; {name} has none") + if arm.base == OVERLAY_ARM and overlay not in overlay_names(): + names = ", ".join(sorted(overlay_names())) + raise PrepError(f"guidance:overlay: overlay {overlay!r} is not a builtin overlay name ({names})") try: base_sha = resolve_rev(git_dir, task.base) except SourceError as exc: @@ -401,7 +409,7 @@ def prepare_workspace(git_dir: Path, manifest: RepoManifest, task: Task, arm: st _git(ws, env, "commit", "-q", "--no-gpg-sign", "--allow-empty", "--no-verify", "-m", "snapshot") patches = _apply_setup_patches(ws, env, task) # 4 try: - arm_files, arm_meta = _apply_arm(ws, env, git_dir, manifest, arm, bridge) # 5 + arm_files, arm_meta = _apply_arm(ws, env, git_dir, manifest, arm, bridge, overlay) # 5 except PrepError as exc: raise PrepError(f"task {task.id}: {exc}") from exc installed = _install_skills(ws, task, skills) # the catalog's skills; a guidance:as-skill skill is in arm_meta @@ -465,99 +473,47 @@ def _forget(ws: Path, env: dict[str, str]) -> None: def _apply_arm(ws: Path, env: dict[str, str], git_dir: Path, manifest: RepoManifest, arm: Arm, - bridge: dict[str, str] | None) -> tuple[list[str], dict[str, Any]]: + bridge: dict[str, str] | None, overlay: str | None = None) -> tuple[list[str], dict[str, Any]]: """The files the arm removed (none), wrote (bridge, rev) or changed (patch), and its other - ``meta.json`` fields: bridge targets left alone, the revision, the patch file.""" + ``meta.json`` fields: bridge targets left alone, the revision, the patch file, the overlay kept.""" if arm.base == "guidance:raw": return [], {} if arm.patch is not None: return _apply_patch(ws, env, manifest, arm) + if arm.base == OVERLAY_ARM: + return _apply_overlay(ws, _workspace_files(ws, env), overlay or "") tracked = [p for p in _git(ws, env, "ls-files", "-z").split("\0") if p] matched = sorted(p for p in tracked if any(match_glob(g, p) for g in manifest.guidance_globs)) - if arm.base == "guidance:as-skill": - return _apply_as_skill(ws, manifest, matched) if arm.base.startswith(FORMAT_ARM): return _apply_format(ws, matched, arm.base[len(FORMAT_ARM):]) - if arm.base == "guidance:none" or arm.rev_sha is not None: - for rel in matched: - (ws / rel).unlink() - if arm.rev_sha is None: - return matched, {} - return _apply_rev(ws, git_dir, manifest, arm, matched) + if arm.base in ("guidance:none", "guidance:as-skill") or arm.rev_sha is not None: + return _apply_removal(ws, env, git_dir, manifest, arm, matched) assert bridge is not None return _apply_bridge(ws, matched, bridge) -def _apply_format(ws: Path, matched: list[str], fmt: str) -> tuple[list[str], dict[str, Any]]: - """Each guidance file rewritten in ``fmt`` in place. A symlink is left alone: its target is rendered - once if it is a guidance file itself, and ``meta.json`` names the links that keep their raw bytes.""" - if not matched: - raise PrepError(f"guidance:format={fmt}: the base has no guidance file to render") - written, links = [], [] - for rel in matched: - if (ws / rel).is_symlink(): - _link_file(ws, rel, matched) # the link must reach a file that is rendered, not raw guidance - links.append(rel) - continue - try: - rendered = to_format(_guidance_text(ws, rel, f"guidance:format={fmt}"), fmt) - except FormatError as exc: - raise PrepError(f"guidance:format={fmt}: {rel}: {exc}") from exc - (ws / rel).write_bytes(rendered.encode("utf-8")) # no newline translation: the restore check holds on disk - written.append(rel) - return written, ({"guidance_links_kept": links} if links else {}) - - -def _link_file(ws: Path, rel: str, matched: list[str]) -> None: - """The regular file a guidance link of the workspace reaches, resolved as the filesystem resolves it (a linked - directory on the way counts). PrepError when the link dangles or cycles, or reaches a file outside the workspace - or outside the matched guidance files: that file would be dropped or left unrendered.""" - try: - target = (ws / rel).resolve(strict=True) - except (OSError, RuntimeError) as exc: # RuntimeError: a symlink loop, before Python 3.13 - raise PrepError(f"guidance link {rel} does not reach a file (a dangling link or a cycle): {exc}") from exc - root = ws.resolve() - if not target.is_relative_to(root) or not target.is_file(): - raise PrepError(f"guidance link {rel} reaches {target}, which is outside the workspace or not a regular file") - reached = target.relative_to(root).as_posix() - if reached not in matched: - raise PrepError(f"guidance link {rel} reaches {reached}, which is not a guidance file") - - -def _guidance_text(ws: Path, rel: str, arm: str) -> str: - """The text of a guidance file of the workspace; one that is not UTF-8 is a prep error of the cell.""" - try: - return (ws / rel).read_bytes().decode("utf-8") - except UnicodeDecodeError as exc: - raise PrepError(f"{arm}: {rel} is not UTF-8: {exc}") from exc - - -def _apply_as_skill(ws: Path, manifest: RepoManifest, matched: list[str]) -> tuple[list[str], dict[str, Any]]: - """The guidance files removed (as ``guidance:none`` removes them) and their text packaged as one skill, - ``.claude/skills/-guidance/SKILL.md``: each file under a ``## File: `` line, its text verbatim.""" - if not matched: - raise PrepError("guidance:as-skill: the base has no guidance file to package") - name = guidance_skill_name(manifest.id) - skill_dir = ws / SKILLS_DIR / name - for part in (ws / ".claude", ws / SKILLS_DIR): # a repository link would write to the host - if part.is_symlink(): - raise PrepError(f"guidance:as-skill: {part.relative_to(ws)} is a symlink in the repository") - if skill_dir.exists() or skill_dir.is_symlink(): - raise PrepError(f"guidance:as-skill: the workspace already has {SKILLS_DIR}/{name}") - parts = [] - for rel in matched: - if (ws / rel).is_symlink(): # packaged once under its target's name; a link to anything else would be lost - _link_file(ws, rel, matched) - continue - text = _guidance_text(ws, rel, "guidance:as-skill") - parts.append(f"## File: {rel}\n\n" + (text if text.endswith("\n") else text + "\n")) - for rel in matched: - (ws / rel).unlink() - skill_dir.mkdir(parents=True) - frontmatter = f"---\nname: {name}\ndescription: {GUIDANCE_SKILL_DESCRIPTION}\n---\n\n" - (skill_dir / "SKILL.md").write_bytes((frontmatter + "\n".join(parts)).encode("utf-8")) # verbatim: no newline translation - return matched, {"guidance_skill": {"name": name, "path": f"{SKILLS_DIR}/{name}", - "sha256": tree_sha256(skill_dir)}} +def _dropped_overlays(ws: Path, env: dict[str, str], matched: list[str], arm: str) -> dict[Path, str]: + """The overlays the globs miss (a setup patch's too), by canonical path; refused before any unlink.""" + return _overlay_targets(ws, [p for p in _workspace_files(ws, env) + if overlay_of(p) is not None and p not in matched], arm) + + +def _apply_removal(ws: Path, env: dict[str, str], git_dir: Path, manifest: RepoManifest, arm: Arm, + matched: list[str]) -> tuple[list[str], dict[str, Any]]: + """``guidance:none`` and ``guidance:as-skill`` (the matched files and every overlay removed, as-skill packaging + the matched ones) or a pinned revision (the matched files replaced, the overlays kept).""" + dropped = _dropped_overlays(ws, env, matched, arm.base) if arm.rev_sha is None else {} + meta: dict[str, Any] = {} + if arm.base == "guidance:as-skill": + meta = _apply_as_skill(ws, manifest, matched)[1] # it removes the matched files + else: + for rel in matched: + (ws / rel).unlink() + for canon in dropped: + canon.unlink() + if arm.rev_sha is None: + return sorted([*matched, *dropped.values()]), meta + return _apply_rev(ws, git_dir, manifest, arm, matched) def _apply_rev(ws: Path, git_dir: Path, manifest: RepoManifest, arm: Arm, @@ -618,6 +574,60 @@ def _apply_bridge(ws: Path, matched: list[str], bridge: dict[str, str]) -> tuple return written, ({"bridge_skipped": skipped} if skipped else {}) +def _workspace_files(ws: Path, env: dict[str, str]) -> list[str]: + """The paths the workspace holds now: the indexed ones and the untracked ones (ignored ones too), since a setup + patch applied with ``git apply`` changes the tree and not the index. A path that is gone is no path.""" + listed = _git(ws, env, "ls-files", "-z", "--cached", "--others").split("\0") + return sorted({p for p in listed if p and os.path.lexists(ws / p)}) + + +def _kept_overlays(ws: Path, tracked: list[str], own: str) -> list[str]: + """The overlay ``own`` of the base, which the arm keeps; refused without the base ``AGENTS.md`` or without that + overlay. Each kept file must supply a file: a dangling, cyclic or directory link does not.""" + if "AGENTS.md" not in tracked: + raise PrepError("guidance:overlay: the base has no AGENTS.md to keep") + kept = sorted(p for p in tracked if overlay_of(p) == own) + if not kept: + raise PrepError(f"guidance:overlay: the base has no AGENTS.{own}.md to keep") + for rel in kept: # the arm keeps it, so it must supply a file: a dangling, cyclic or directory link does not + if not _supplies_a_file(ws, rel): + raise PrepError(f"guidance:overlay: {rel} does not resolve to a usable file in the workspace") + return kept + + +def _refuse_dangling(ws: Path, tracked: list[str], listed: set[str], gone: set[Path]) -> None: + """Refuse a path of the base that runs through an overlay the arm removes: the base would dangle.""" + for rel in tracked: + if rel not in listed and _path_hits(ws, rel, gone): + raise PrepError(f"guidance:overlay: {rel} resolves through an overlay the arm removes; " + "the base would dangle") + + +def _check_removable(ws: Path, targets: dict[Path, str]) -> None: + """Refuse a removal that leaves the base without a file, or that would remove a directory.""" + if not _supplies_a_file(ws, "AGENTS.md"): # the base is kept too, so it must supply a file as well + raise PrepError("guidance:overlay: AGENTS.md does not resolve to a usable file in the workspace") + for canon, rel in targets.items(): # all checks before the first unlink: a refusal leaves the workspace as it was + if canon.is_dir() and not canon.is_symlink(): + raise PrepError(f"guidance:overlay: {rel} is a directory, not an overlay file") + + +def _apply_overlay(ws: Path, tracked: list[str], own: str) -> tuple[list[str], dict[str, Any]]: + """Remove the overlays of other agents, keeping the base ``AGENTS.md`` and the ``own`` overlay; refused without + either (the arm's cell would be the base alone, or an overlay with no base to sit beside).""" + kept = _kept_overlays(ws, tracked, own) + listed = [p for p in tracked if overlay_of(p) not in (None, own)] + targets = _overlay_targets(ws, listed) + if targets: # the walk is only needed when some overlay is removed + _refuse_dangling(ws, tracked, set(listed), set(targets)) + _check_removable(ws, targets) + # Each canonical path has real directories only as its parents, so removing one leaves the others valid, whatever + # order the aliases sort in (a name under a link is unlinked by its canonical path, not through the link) + for canon in targets: + canon.unlink() + return sorted(targets.values()), {"arm_overlay": {"name": own, "kept": kept}} + + def _guidance_at(git_dir: Path, manifest: RepoManifest, sha: str) -> tuple[list[str], dict[str, str], dict[str, list[str]]]: """The files of commit ``sha`` the guidance globs match, outside ``eval_dir`` (step 2 deletes it), and for each diff --git a/dokimi/dokimi/schemas/guidance_reach.v1.json b/dokimi/dokimi/schemas/guidance_reach.v1.json index 54fa97944..9de5d1d0e 100644 --- a/dokimi/dokimi/schemas/guidance_reach.v1.json +++ b/dokimi/dokimi/schemas/guidance_reach.v1.json @@ -46,6 +46,19 @@ } } }, + "overlays": { + "type": "object", + "description": "agentsmd/agents.md#185 overlays. own: the harness's overlay name (null: none); own_files: AGENTS..md files probed; own_loaded: those loaded in some scenario; foreign_files: AGENTS..md files probed; cross_loaded: foreign file -> scenario -> loaded, truncated or fetched. Any cross_loaded entry is a finding. ambiguous (only when some): overlay aliases whose file loaded but could not be told apart from another name it could have been read under (any harness-native guidance file in the probe could import it, or the alias group holds two overlay names); they are neither own nor cross loads.", + "required": ["own", "own_files", "own_loaded", "foreign_files", "cross_loaded"], + "properties": { + "own": {"type": ["string", "null"]}, + "own_files": {"type": "array", "items": {"type": "string"}}, + "own_loaded": {"type": "array", "items": {"type": "string"}}, + "foreign_files": {"type": "array", "items": {"type": "string"}}, + "cross_loaded": {"type": "object", "additionalProperties": {"type": "object", "additionalProperties": {"enum": ["loaded", "truncated", "fetched"]}}}, + "ambiguous": {"type": "array", "items": {"type": "string"}} + } + }, "unresolved": { "type": "array", "description": "Probed paths that are symlinks the probe cannot follow inside the tree (escaping, dangling, looping or naming a directory). They are not seeded and have no records.", diff --git a/dokimi/dokimi/schemas/harness_profile.v1.json b/dokimi/dokimi/schemas/harness_profile.v1.json index ff04d7fd6..eb0f2dc6e 100644 --- a/dokimi/dokimi/schemas/harness_profile.v1.json +++ b/dokimi/dokimi/schemas/harness_profile.v1.json @@ -91,6 +91,11 @@ "write": {"type": "string"}, "content": {"type": "string"} } + }, + "overlay": { + "type": ["string", "null"], + "pattern": "^[a-z0-9][a-z0-9-]*$", + "description": "The agent name of the AGENTS..md the guidance:overlay arm keeps for this harness (agentsmd/agents.md#185). Absent or null: the arm cannot run with this profile." } } } diff --git a/dokimi/tests/test_repo_overlay.py b/dokimi/tests/test_repo_overlay.py new file mode 100644 index 000000000..73a2c44b4 --- /dev/null +++ b/dokimi/tests/test_repo_overlay.py @@ -0,0 +1,763 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The guidance:overlay arm and the overlay reach check (agentsmd/agents.md#185); offline, no LLM.""" + +from __future__ import annotations + +import dataclasses +import json +import os +import shutil +from pathlib import Path +from types import SimpleNamespace + +import pytest +import yaml + +from dokimi.repos import RepoSource, load_manifest, open_source +from dokimi.repos.harness_profile import PROFILE_DIR, ProfileError, load_profile, parse_profile +from dokimi.repos.overlays import native_names, overlay_of +from dokimi.repos.reach import format_matrix, native_in_probe, overlay_report, probe_files +from dokimi.repos.workspace import ( + ARM_FORMS, + OVERLAY_ARM, + SNAPSHOT_AUTHOR, + PrepError, + _apply_arm, + _apply_overlay, + _git_env, + _link_target, + _path_hits, + export_eval_dir, + load_tasks, + parse_arm, + prepare_workspace, + source_git_dir, +) +from repo_eval_fixture import git, make_repo, write + +OVERLAY_FILES = { + "AGENTS.md": "# guidance\n", + "AGENTS.claude.md": "# for claude\n", + "AGENTS.codex.md": "# for codex\n", + "sub/AGENTS.md": "# sub guidance\n", + "sub/AGENTS.copilot.md": "# sub, for copilot\n", +} + + +def _overlay_base(root: Path, base: str) -> str: + """A commit on ``base`` that adds the overlay files; the tasks move onto it, and HEAD holds the overlays too.""" + wt = root.parent / "overlay-wt" + git(root, "worktree", "add", "-q", "-b", "overlays", str(wt), base) + for rel, text in OVERLAY_FILES.items(): + write(wt, rel, text) + git(wt, "add", "-A") + git(wt, "commit", "-qm", "overlays") + sha = git(wt, "rev-parse", "HEAD") + git(root, "worktree", "remove", "--force", str(wt)) + for task in ("fix_add", "review_add"): + path = root / ".agent-eval" / "tasks" / task / "task.json" + data = json.loads(path.read_text(encoding="utf-8")) + data["base"] = sha + path.write_text(json.dumps(data), encoding="utf-8") + for rel, text in OVERLAY_FILES.items(): + write(root, rel, text) + git(root, "add", "-A") + git(root, "commit", "-qm", "tasks on the overlay base") + return sha + + +@pytest.fixture +def repo(tmp_path): + root, base = make_repo(tmp_path) + _overlay_base(root, base) + source = RepoSource.parse(f"{root}@HEAD") + tree = open_source(source) + manifest = load_manifest(tree, ".agent-eval/repo-eval.yaml") + content = export_eval_dir(tree, manifest, tmp_path / "content") + tasks = {t.id: t for t in load_tasks(content, manifest)} + return {"root": root, "tree": tree, "manifest": manifest, "tasks": tasks, + "git_dir": source_git_dir(source, tree), "tmp": tmp_path} + + +def _prep(repo, arm: str, profile, name: str = "rd"): + meta = prepare_workspace(repo["git_dir"], repo["manifest"], repo["tasks"]["fix_add"], arm, + repo["tmp"] / name, profile) + return meta, repo["tmp"] / name / "workspace" + + +@pytest.mark.parametrize("path, name", [ + ("AGENTS.claude.md", "claude"), + ("sub/AGENTS.CODEX.md", "codex"), + ("docs/AGENTS.copilot.md", "copilot"), + ("AGENTS.md", None), + ("AGENTS.override.md", None), # codex's own file, loaded by its own convention + ("CLAUDE.md", None), + ("AGENTS.a.b.md", None), + ("AGENTS.md.bak", None), + (".claude/skills/lint/AGENTS.codex.md", None), # skill content, not guidance + ("AGENTS.ja.md", None), # a word, but no builtin profile has an overlay for it: an ordinary file + ("AGENTS.local.md", None), + ("docs/AGENTS.template.md", None), + ("AGENTS.Override.md", None), # not the native name (lower case), and no profile's overlay +]) +def test_overlay_names_are_agent_files_beside_agents_md(path, name): + assert overlay_of(path) == name + + +def test_overlay_is_a_guidance_part_and_takes_skills(): + assert "guidance:overlay" in ARM_FORMS + assert parse_arm("guidance:overlay").base == "guidance:overlay" + arm = parse_arm("guidance:overlay+skill:skills/lint") + assert str(arm) == "guidance:overlay+skill:lint" and arm.base == "guidance:overlay" + with pytest.raises(PrepError, match="more than one guidance part"): + parse_arm("guidance:overlay+guidance:none+skill:skills/lint") + + +@pytest.mark.parametrize("profile, overlay", [("claude-code", "claude"), ("codex", "codex"), + ("codex-review", "codex"), ("copilot", "copilot"), + ("copilot-review", "copilot")]) +def test_builtin_profiles_name_the_overlay_of_their_agent(profile, overlay): + assert load_profile(profile).overlay == overlay + + +def test_external_copy_of_a_builtin_profile_keeps_its_builtin_overlay(tmp_path): + """A profile loaded from a path may reuse a builtin overlay name: a copy of claude-code.yaml keeps claude.""" + copy = tmp_path / "my-claude.yaml" + shutil.copy(PROFILE_DIR / "claude-code.yaml", copy) + assert load_profile(str(copy)).overlay == "claude" + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + removed, _ = _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md"], "claude") + assert removed == ["AGENTS.codex.md"] + + +def test_overlay_key_is_a_lowercase_agent_name(): + raw = yaml.safe_load((PROFILE_DIR / "codex.yaml").read_text(encoding="utf-8")) + assert parse_profile(raw).overlay == "codex" + with pytest.raises(ProfileError): + parse_profile({**raw, "overlay": "Codex/../x"}) + with pytest.raises(ProfileError): # the schema's `$` takes a trailing newline; the parser must not + parse_profile({**raw, "overlay": "codex\n"}) + + +def test_overlay_arm_keeps_base_and_the_harness_overlay(repo): + meta, ws = _prep(repo, "guidance:overlay", load_profile("claude-code")) + assert meta["arm"] == "guidance:overlay" + assert (ws / "AGENTS.md").read_text() == "# guidance\n" + assert (ws / "AGENTS.claude.md").read_text() == "# for claude\n" + assert (ws / "sub/AGENTS.md").is_file() + assert not (ws / "AGENTS.codex.md").exists() and not (ws / "sub/AGENTS.copilot.md").exists() + assert meta["arm_files"] == ["AGENTS.codex.md", "sub/AGENTS.copilot.md"] + assert meta["arm_overlay"] == {"name": "claude", "kept": ["AGENTS.claude.md"]} + + +def test_overlay_arm_follows_the_profile_not_the_arm_name(repo): + meta, ws = _prep(repo, "guidance:overlay", load_profile("codex"), name="rd-codex") + assert (ws / "AGENTS.codex.md").is_file() and not (ws / "AGENTS.claude.md").exists() + assert meta["arm_files"] == ["AGENTS.claude.md", "sub/AGENTS.copilot.md"] + assert meta["arm_overlay"] == {"name": "codex", "kept": ["AGENTS.codex.md"]} + + +def test_raw_arm_keeps_every_overlay(repo): + meta, ws = _prep(repo, "guidance:raw", load_profile("claude-code"), name="rd-raw") + assert (ws / "AGENTS.codex.md").is_file() and (ws / "AGENTS.claude.md").is_file() + assert "arm_overlay" not in meta + + +def test_overlay_arm_needs_a_profile_with_an_overlay_key(repo): + with pytest.raises(PrepError, match="needs a harness profile with an `overlay` key"): + _prep(repo, "guidance:overlay", None, name="rd-none") + with pytest.raises(PrepError, match="needs a harness profile with an `overlay` key"): + _prep(repo, "guidance:overlay", dataclasses.replace(load_profile("claude-code"), overlay=None), + name="rd-no-key") + + +def test_overlay_arm_refuses_an_overlay_name_no_builtin_profile_has(repo): + """``AGENTS.gemini.md`` is an ordinary file, not an overlay, so the arm could keep nothing: refused by its name.""" + with pytest.raises(PrepError, match="overlay 'gemini' is not a builtin overlay name \\(claude, codex, copilot\\)"): + _prep(repo, "guidance:overlay", dataclasses.replace(load_profile("claude-code"), overlay="gemini"), + name="rd-gemini") + + +def test_overlay_arm_refuses_a_base_with_no_overlay_for_its_agent(tmp_path): + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + with pytest.raises(PrepError, match="has no AGENTS.claude.md to keep"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md"], "claude") + + +def test_overlay_arm_refuses_a_base_with_no_agents_md(tmp_path): + """The workspace keeps the base AGENTS.md (the contract says so): overlays alone are refused, before any unlink.""" + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + with pytest.raises(PrepError, match="has no AGENTS.md to keep"): + _apply_overlay(tmp_path, ["AGENTS.claude.md", "AGENTS.codex.md"], "claude") + assert (tmp_path / "AGENTS.codex.md").is_file() + + +def test_probe_files_include_every_overlay_even_outside_the_globs(repo): + """The manifest's globs are ``**/AGENTS.md`` and the like: an overlay is probed for its own reach.""" + tree = repo["tree"] + files = probe_files(tree, load_manifest(tree, ".agent-eval/repo-eval.yaml"), load_profile("claude-code")) + assert {"AGENTS.claude.md", "AGENTS.codex.md", "sub/AGENTS.copilot.md"} <= set(files) + assert all(not f.startswith(".agent-eval/") for f in files) + + +def test_overlay_report_flags_a_foreign_overlay_that_loads(): + records = [ + {"file": "AGENTS.md", "scenario": "cwd=.", "class": "loaded"}, + {"file": "AGENTS.claude.md", "scenario": "cwd=.", "class": "loaded"}, + {"file": "AGENTS.codex.md", "scenario": "cwd=.", "class": "not_loaded"}, + {"file": "sub/AGENTS.copilot.md", "scenario": "cwd=sub", "class": "fetched"}, + {"file": "sub/AGENTS.copilot.md", "scenario": "cwd=.", "class": "error"}, + ] + paths = ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md", "sub/AGENTS.copilot.md"] + assert overlay_report(records, "claude", {p: () for p in paths}) == { + "own": "claude", "own_files": ["AGENTS.claude.md"], "own_loaded": ["AGENTS.claude.md"], + "foreign_files": ["AGENTS.codex.md", "sub/AGENTS.copilot.md"], + "cross_loaded": {"sub/AGENTS.copilot.md": {"cwd=sub": "fetched"}}, + } + + +def test_a_harness_without_an_overlay_has_only_foreign_overlays(): + records = [{"file": "AGENTS.codex.md", "scenario": "cwd=.", "class": "truncated"}] + report = overlay_report(records, None, {"AGENTS.md": (), "AGENTS.codex.md": ()}) + assert report["own_files"] == [] and report["cross_loaded"] == {"AGENTS.codex.md": {"cwd=.": "truncated"}} + + +def test_overlay_report_flags_a_foreign_overlay_reached_through_a_link(): + """The probe seeds the file the link names; the harness reads the link, so the report keys on the link name.""" + records = [{"file": "docs/codex.md", "scenario": "cwd=.", "class": "loaded"}] + report = overlay_report(records, "claude", {"docs/codex.md": ("AGENTS.codex.md",)}) + assert report["foreign_files"] == ["AGENTS.codex.md"] + assert report["cross_loaded"] == {"AGENTS.codex.md": {"cwd=.": "loaded"}} + + +def test_overlay_arm_refuses_a_base_link_into_an_overlay_it_removes(tmp_path): + (tmp_path / "AGENTS.md").symlink_to("AGENTS.codex.md") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md"], "claude") + assert (tmp_path / "AGENTS.codex.md").exists() # refused before anything is removed + + +def test_overlay_arm_keeps_a_base_link_whose_target_a_removed_link_also_names(tmp_path): + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "shared.md").write_text("shared\n", encoding="utf-8") + (tmp_path / "AGENTS.md").symlink_to(Path("docs", "shared.md")) + (tmp_path / "AGENTS.codex.md").symlink_to(Path("docs", "shared.md")) + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + removed, _ = _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md", "docs/shared.md"], + "claude") + assert removed == ["AGENTS.codex.md"] + assert (tmp_path / "AGENTS.md").read_text(encoding="utf-8") == "shared\n" + + +def test_a_link_target_with_the_windows_long_path_prefix_is_read_without_it(tmp_path, monkeypatch): + (tmp_path / "docs").mkdir() + target = Path(os.path.realpath(tmp_path / "docs" / "shared.md")) + target.write_text("shared\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").symlink_to(Path("docs", "shared.md")) + real_readlink = os.readlink + # os.readlink on Windows gives the absolute target with the \\?\ long-path prefix + monkeypatch.setattr(os, "readlink", lambda p: "\\\\?\\" + str(target) if Path(p).name == "AGENTS.claude.md" + else real_readlink(p)) + assert _path_hits(tmp_path, "AGENTS.claude.md", {target}) is True + + +def test_the_path_walk_runs_only_when_an_overlay_is_removed(tmp_path, monkeypatch): + for name in ("AGENTS.md", "AGENTS.claude.md"): + (tmp_path / name).write_text("x\n", encoding="utf-8") + walked = [] + monkeypatch.setattr("dokimi.repos.workspace._path_hits", lambda *args: walked.append(args) or False) + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.claude.md"], "claude") + assert walked == [] # no other agent's overlay to remove: nothing can dangle + (tmp_path / "AGENTS.codex.md").write_text("x\n", encoding="utf-8") + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md"], "claude") + assert walked and not (tmp_path / "AGENTS.codex.md").exists() + + +def test_a_link_target_on_a_windows_unc_share_keeps_the_server_name(tmp_path, monkeypatch): + """\\\\?\\UNC\\server\\share\\x is the share \\\\server\\share\\x; without the UNC rule it is the relative UNC\\server.""" + link = tmp_path / "AGENTS.codex.md" + monkeypatch.setattr(os, "readlink", lambda p: "\\\\?\\UNC\\server\\share\\ws\\AGENTS.codex.md") + assert str(_link_target(link)) == "\\\\server\\share\\ws\\AGENTS.codex.md" + + +@pytest.mark.skipif(os.name != "nt", reason="a backslash in a link target separates names on Windows only") +def test_overlay_arm_refuses_a_backslash_base_link_through_a_removed_directory_link(tmp_path): + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "base.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").symlink_to("docs", target_is_directory=True) + (tmp_path / "AGENTS.md").symlink_to("AGENTS.codex.md\\base.md") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md"], "claude") + + +def test_overlay_arm_refuses_a_base_link_through_a_directory_link(tmp_path): + (tmp_path / "real").mkdir() + (tmp_path / "real" / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "dir").symlink_to("real") + (tmp_path / "AGENTS.md").symlink_to("dir/AGENTS.codex.md") + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, ["AGENTS.md", "dir", "real/AGENTS.codex.md", "AGENTS.claude.md"], "claude") + assert (tmp_path / "real" / "AGENTS.codex.md").exists() + + +def test_overlay_arm_does_not_hang_on_a_symlink_loop_in_the_base(tmp_path): + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "a").symlink_to("b") + (tmp_path / "docs" / "b").symlink_to("a") + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + removed, _ = _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md", "docs/a", "docs/b"], + "claude") + assert removed == ["AGENTS.codex.md"] + + +def test_text_report_states_the_cross_load_as_a_finding(): + payload = { + "harness": {"name": "claude-code", "version": "2.1.0"}, "model": "m", + "summary": {"cost_usd": 0.0, "matrix": {}}, "files": [], "unresolved": [], + "scenarios": [{"id": "cwd=.", "error": None, "prose": 0, "unknown": []}], + "records": [{"file": "AGENTS.claude.md", "scenario": "cwd=.", "class": "not_loaded"}], + "overlays": {"own": "claude", "own_files": ["AGENTS.claude.md"], "own_loaded": [], + "foreign_files": ["AGENTS.codex.md"], "cross_loaded": {"AGENTS.codex.md": {"cwd=.": "loaded"}}}, + } + text = format_matrix(payload) + assert ("finding: AGENTS.codex.md, an overlay for another agent, was loaded by claude-code in cwd=. (loaded)" + in text) + assert "note: no claude overlay (AGENTS.claude.md) was loaded in any scenario" in text + + +def _payload(records, own, links, native=frozenset(), scenarios=("cwd=.",)): + """A text-report payload with the overlay part computed from ``records`` the way run_reach computes it.""" + files = [{"path": t, **({"aliases": list(a)} if a else {})} for t, a in links.items()] + return { + "harness": {"name": "claude-code", "version": "2.1.0"}, "model": "m", + "summary": {"cost_usd": 0.0, "matrix": {}}, "files": files, "unresolved": [], + "scenarios": [{"id": s, "error": None, "prose": 0, "unknown": []} for s in scenarios], + "records": records, "overlays": overlay_report(records, own, links, native), + } + + +@pytest.mark.parametrize("cls, note", [ + ("not_loaded", "note: no claude overlay (AGENTS.claude.md) was loaded in any scenario"), + ("error", "note: could not determine whether the claude overlay (AGENTS.claude.md) was loaded"), +]) +def test_text_report_says_not_loaded_only_for_a_usable_negative_record(cls, note): + records = [{"file": "AGENTS.claude.md", "scenario": "cwd=.", "class": cls}] + text = format_matrix(_payload(records, "claude", {"AGENTS.claude.md": ()})) + assert note in text + assert ("was loaded in any scenario" in text) == (cls == "not_loaded") + + +def test_text_report_does_not_claim_not_loaded_for_an_all_error_run(): + records = [{"file": "AGENTS.claude.md", "scenario": s, "class": "error"} for s in ("cwd=.", "read=sub")] + text = format_matrix(_payload(records, "claude", {"AGENTS.claude.md": ()}, scenarios=("cwd=.", "read=sub"))) + assert "was loaded in any scenario" not in text + assert "could not determine whether the claude overlay (AGENTS.claude.md) was loaded" in text + + +def test_text_report_keeps_the_ambiguous_load_out_of_the_own_overlay_note(): + """A loaded file that two own/foreign names could name is ambiguous: no own-overlay note is printed for it.""" + records = [{"file": "docs/x.md", "scenario": "cwd=.", "class": "loaded"}] + payload = _payload(records, "claude", {"docs/x.md": ("AGENTS.claude.md", "AGENTS.codex.md")}) + assert payload["overlays"]["ambiguous"] == ["AGENTS.claude.md", "AGENTS.codex.md"] + assert "note:" not in format_matrix(payload) + + +def test_text_report_does_not_claim_not_loaded_when_another_scenario_was_a_refusal(): + """A usable not_loaded in one scenario does not clear a refusal in another: the note cannot claim all scenarios.""" + records = [{"file": "AGENTS.claude.md", "scenario": s, "class": "not_loaded"} for s in ("cwd=.", "read=sub")] + payload = _payload(records, "claude", {"AGENTS.claude.md": ()}, scenarios=("cwd=.", "read=sub")) + payload["scenarios"][1]["prose"] = 40 + text = format_matrix(payload) + assert "was loaded in any scenario" not in text + assert "could not determine whether the claude overlay (AGENTS.claude.md) was loaded" in text + + +def test_overlay_of_reads_the_bundled_profiles_not_a_same_named_file_in_the_cwd(tmp_path, monkeypatch): + """A file named like a bundled profile, in the working directory, is not parsed as that profile.""" + (tmp_path / "codex").write_text("not: [a profile\n", encoding="utf-8") + monkeypatch.chdir(tmp_path) + native_names.cache_clear() + try: + assert overlay_of("AGENTS.claude.md") == "claude" + assert native_names() + finally: + native_names.cache_clear() + + +def test_overlay_report_leaves_a_link_load_unattributed_when_a_native_file_may_import_it(): + """CLAUDE.md can import docs/shared.md, so a load of that file is not a load of its AGENTS.codex.md alias.""" + records = [{"file": "docs/shared.md", "scenario": "cwd=.", "class": "loaded"}] + links = {"CLAUDE.md": (), "docs/shared.md": ("AGENTS.codex.md",)} + report = overlay_report(records, "claude", links, native={"CLAUDE.md"}) + assert report["cross_loaded"] == {} + assert report["ambiguous"] == ["AGENTS.codex.md"] + + +def test_text_report_still_says_not_loaded_for_an_own_overlay_next_to_an_unrelated_ambiguous_load(): + records = [{"file": "AGENTS.claude.md", "scenario": "cwd=.", "class": "not_loaded"}, + {"file": "AGENTS.codex.md", "scenario": "cwd=.", "class": "loaded"}] + payload = _payload(records, "claude", {"AGENTS.claude.md": (), "AGENTS.codex.md": ("CLAUDE.md",)}, + native={"CLAUDE.md"}) + assert payload["overlays"]["ambiguous"] == ["AGENTS.codex.md", "CLAUDE.md"] + assert "note: no claude overlay (AGENTS.claude.md) was loaded in any scenario" in format_matrix(payload) + + +def test_text_report_does_not_claim_not_loaded_from_a_refusal(): + """A scenario that answered with prose (a refusal) gives no usable negative record.""" + records = [{"file": "AGENTS.claude.md", "scenario": "read=sub", "class": "not_loaded"}] + payload = _payload(records, "claude", {"AGENTS.claude.md": ()}, scenarios=("read=sub",)) + payload["scenarios"][0]["prose"] = 40 + text = format_matrix(payload) + assert "was loaded in any scenario" not in text + assert "could not determine whether the claude overlay (AGENTS.claude.md) was loaded" in text + + +def test_overlay_report_leaves_a_load_of_two_foreign_aliases_unattributed(): + """A harness reading one of two foreign alias names of docs/shared.md reads the same file under both.""" + records = [{"file": "docs/shared.md", "scenario": "cwd=.", "class": "loaded"}] + links = {"docs/shared.md": ("AGENTS.codex.md", "AGENTS.copilot.md")} + report = overlay_report(records, "claude", links, native={"CLAUDE.md", "AGENTS.md"}) + assert report["cross_loaded"] == {} + assert report["ambiguous"] == ["AGENTS.codex.md", "AGENTS.copilot.md"] + + +def test_overlay_report_leaves_a_load_of_a_foreign_target_with_a_foreign_alias_unattributed(): + records = [{"file": "AGENTS.codex.md", "scenario": "cwd=.", "class": "loaded"}] + report = overlay_report(records, "claude", {"AGENTS.codex.md": ("AGENTS.copilot.md",)}) + assert report["cross_loaded"] == {} + assert report["ambiguous"] == ["AGENTS.codex.md", "AGENTS.copilot.md"] + + +@pytest.mark.parametrize("via", ["absolute", "dotdot", "dot-then-up", "chain"]) +def test_overlay_arm_refuses_a_base_link_to_a_removed_overlay_however_it_is_written(tmp_path, via): + (tmp_path / "sub").mkdir() + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + files = ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md"] + if via == "chain": + (tmp_path / "mid.md").symlink_to("AGENTS.codex.md") + (tmp_path / "AGENTS.md").symlink_to("mid.md") + files.append("mid.md") + else: + target = {"absolute": str(tmp_path / "AGENTS.codex.md"), + "dotdot": f"../{tmp_path.name}/AGENTS.codex.md", + "dot-then-up": "./sub/../AGENTS.codex.md"}[via] + (tmp_path / "AGENTS.md").symlink_to(target) + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, files, "claude") + assert (tmp_path / "AGENTS.codex.md").exists() + + +def test_overlay_arm_refuses_an_own_overlay_that_links_outside_the_workspace(tmp_path): + ws = tmp_path / "ws" + ws.mkdir() + (tmp_path / "guide.md").write_text("outside\n", encoding="utf-8") + (ws / "AGENTS.md").write_text("base\n", encoding="utf-8") + (ws / "AGENTS.claude.md").symlink_to("../guide.md") + (ws / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + with pytest.raises(PrepError, match="does not resolve to a usable file"): + _apply_overlay(ws, ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md"], "claude") + assert (ws / "AGENTS.codex.md").exists() + + +def test_overlay_arm_refuses_a_base_link_through_a_removed_directory_link(tmp_path): + """A removed overlay that is a directory link: the base link resolves through it, so it would dangle.""" + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "base.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").symlink_to("docs") + (tmp_path / "AGENTS.md").symlink_to("AGENTS.codex.md/base.md") + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md", "docs/base.md"], "claude") + assert (tmp_path / "AGENTS.codex.md").is_symlink() # refused before anything is removed + + +@pytest.mark.parametrize("own_target", ["missing.md", "AGENTS.claude.md", "docs"]) +def test_overlay_arm_refuses_an_own_overlay_that_supplies_no_file(tmp_path, own_target): + (tmp_path / "docs").mkdir() + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + (tmp_path / "AGENTS.claude.md").symlink_to(own_target) # dangling, a cycle, a directory + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + with pytest.raises(PrepError, match="does not resolve to a usable file"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md", "docs"], "claude") + assert (tmp_path / "AGENTS.codex.md").exists() + + +@pytest.mark.parametrize("base_target", ["missing.md", "AGENTS.md", "docs"]) +def test_overlay_arm_refuses_a_base_that_supplies_no_file(tmp_path, base_target): + (tmp_path / "docs").mkdir() + (tmp_path / "AGENTS.md").symlink_to(base_target) # dangling, a cycle, a directory + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + with pytest.raises(PrepError, match="AGENTS.md does not resolve"): + _apply_overlay(tmp_path, ["AGENTS.md", "AGENTS.claude.md", "AGENTS.codex.md", "docs"], "claude") + assert (tmp_path / "AGENTS.codex.md").exists() + + +def test_overlay_arm_reads_the_workspace_after_setup_patches_not_the_index(tmp_path): + """A setup patch applied with git apply changes the tree and not the index: the arm sees an added foreign + overlay (removes it) and a deleted one (skips it), instead of failing on a stale index entry.""" + ws = tmp_path / "ws" + ws.mkdir() + (ws / "AGENTS.md").write_text("base\n", encoding="utf-8") + (ws / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (ws / "AGENTS.copilot.md").write_text("copilot\n", encoding="utf-8") + git(ws, "init", "-q") + git(ws, "add", "-A") + git(ws, "commit", "-q", "-m", "base") + (ws / "AGENTS.copilot.md").unlink() # a setup patch deleted it + (ws / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") # and added this one, unstaged + env = _git_env(SNAPSHOT_AUTHOR) + removed, _ = _apply_arm(ws, env, tmp_path, None, parse_arm(OVERLAY_ARM), None, "claude") + assert removed == ["AGENTS.codex.md"] + assert not (ws / "AGENTS.codex.md").exists() and (ws / "AGENTS.claude.md").exists() + + +def test_overlay_arm_does_not_unlink_through_a_setup_patch_that_made_a_directory_a_link(tmp_path): + """A setup patch replaces the tracked directory sub/ with a link to a host directory. The index still lists + sub/AGENTS.codex.md and it now resolves outside the workspace: the arm must refuse, not delete the host file.""" + ws = tmp_path / "ws" + ws.mkdir() + host = tmp_path / "host" + host.mkdir() + (host / "AGENTS.codex.md").write_text("host\n", encoding="utf-8") + (ws / "AGENTS.md").write_text("base\n", encoding="utf-8") + (ws / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (ws / "sub").mkdir() + (ws / "sub" / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + git(ws, "init", "-q") + git(ws, "add", "-A") + git(ws, "commit", "-q", "-m", "base") + shutil.rmtree(ws / "sub") # the setup patch, applied without the index + (ws / "sub").symlink_to(host, target_is_directory=True) + env = _git_env(SNAPSHOT_AUTHOR) + with pytest.raises(PrepError, match="resolves outside the workspace"): + _apply_arm(ws, env, tmp_path, None, parse_arm(OVERLAY_ARM), None, "claude") + assert (host / "AGENTS.codex.md").read_text(encoding="utf-8") == "host\n" + + +def test_guidance_none_reads_the_workspace_after_setup_patches_not_the_index(tmp_path): + ws = tmp_path / "ws" + ws.mkdir() + for name in ("AGENTS.md", "AGENTS.claude.md", "AGENTS.copilot.md"): + (ws / name).write_text(f"{name}\n", encoding="utf-8") + git(ws, "init", "-q") + git(ws, "add", "-A") + git(ws, "commit", "-q", "-m", "base") + (ws / "AGENTS.copilot.md").unlink() # a setup patch deleted it + (ws / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") # and added this one, unstaged + manifest = SimpleNamespace(guidance_globs=["**/AGENTS.md"]) + removed, _ = _apply_arm(ws, _git_env(SNAPSHOT_AUTHOR), tmp_path, manifest, parse_arm("guidance:none"), None) + assert removed == ["AGENTS.claude.md", "AGENTS.codex.md", "AGENTS.md"] + assert not any(ws.glob("AGENTS*")) + + +def test_guidance_none_does_not_unlink_through_a_setup_patch_that_made_a_directory_a_link(tmp_path): + ws = tmp_path / "ws" + ws.mkdir() + host = tmp_path / "host" + host.mkdir() + (host / "AGENTS.codex.md").write_text("host\n", encoding="utf-8") + (ws / "AGENTS.md").write_text("base\n", encoding="utf-8") + (ws / "sub").mkdir() + (ws / "sub" / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + git(ws, "init", "-q") + git(ws, "add", "-A") + git(ws, "commit", "-q", "-m", "base") + shutil.rmtree(ws / "sub") # the setup patch, applied without the index + (ws / "sub").symlink_to(host, target_is_directory=True) + manifest = SimpleNamespace(guidance_globs=["**/AGENTS.md"]) + with pytest.raises(PrepError, match="guidance:none: sub/AGENTS.codex.md is under a path that resolves outside"): + _apply_arm(ws, _git_env(SNAPSHOT_AUTHOR), tmp_path, manifest, parse_arm("guidance:none"), None) + assert (host / "AGENTS.codex.md").read_text(encoding="utf-8") == "host\n" + assert (ws / "AGENTS.md").exists() # refused before the first unlink + + +def test_overlay_arm_refuses_a_kept_overlay_under_a_removed_directory_link(tmp_path): + """A setup patch replaces the indexed directory AGENTS.codex.md/ with a link to docs/. The stale index entry + AGENTS.codex.md/AGENTS.claude.md still resolves to a file, but once the removed link goes it no longer exists.""" + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").symlink_to("docs", target_is_directory=True) + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + tracked = ["AGENTS.md", "AGENTS.codex.md", "AGENTS.claude.md", "AGENTS.codex.md/AGENTS.claude.md", + "docs/AGENTS.claude.md"] + with pytest.raises(PrepError, match="the base would dangle"): + _apply_overlay(tmp_path, tracked, "claude") + assert (tmp_path / "AGENTS.codex.md").is_symlink() # refused before anything is removed + + +@pytest.mark.parametrize("alias", ["sub", "aa"]) # the alias sorts after the real name, or before it +def test_overlay_arm_removes_a_file_a_directory_alias_lists_twice_once(tmp_path, alias): + """A setup patch replaces the indexed directory `alias` with a link to real/: the index still lists + alias/AGENTS.codex.md next to real/AGENTS.codex.md, which is one file. It is removed once, not twice.""" + (tmp_path / "real").mkdir() + (tmp_path / "real" / "AGENTS.codex.md").write_text("codex\n", encoding="utf-8") + (tmp_path / alias).symlink_to("real", target_is_directory=True) + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + tracked = sorted(["AGENTS.md", "AGENTS.claude.md", "real/AGENTS.codex.md", f"{alias}/AGENTS.codex.md"]) + removed, _ = _apply_overlay(tmp_path, tracked, "claude") + assert len(removed) == 1 + assert not (tmp_path / "real" / "AGENTS.codex.md").exists() + + +@pytest.mark.parametrize("listed", [ + ["AGENTS.claude.md", "AGENTS.codex.md", "AGENTS.codex.md/AGENTS.copilot.md", "real/AGENTS.copilot.md"], + ["AGENTS.claude.md", "AGENTS.codex.md", "real/AGENTS.copilot.md"], +]) +def test_overlay_arm_removes_a_file_under_a_removed_directory_alias_before_the_alias(tmp_path, listed): + """AGENTS.codex.md is a link to real/ and the index also lists a file under it: the file goes before the link, + or its unlink finds no path. Either listing names the same file, so it is removed once.""" + (tmp_path / "real").mkdir() + (tmp_path / "real" / "AGENTS.copilot.md").write_text("copilot\n", encoding="utf-8") + (tmp_path / "AGENTS.codex.md").symlink_to("real", target_is_directory=True) + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + _apply_overlay(tmp_path, sorted([*listed, "AGENTS.md"]), "claude") + assert not os.path.lexists(tmp_path / "AGENTS.codex.md") + assert not (tmp_path / "real" / "AGENTS.copilot.md").exists() + assert (tmp_path / "AGENTS.claude.md").is_file() + + +def test_overlay_arm_removes_a_chained_directory_alias_listing_without_failing(tmp_path): + """AGENTS.codex.md links to AGENTS.copilot.md, which links to real/: the listed AGENTS.codex.md/AGENTS.copilot.md is + the same file as real/AGENTS.copilot.md. Removing AGENTS.copilot.md first would leave AGENTS.codex.md dangling before + the nested name is unlinked; each file is removed by its canonical path, so the order of the names does not matter.""" + (tmp_path / "real").mkdir() + (tmp_path / "real" / "AGENTS.copilot.md").write_text("copilot\n", encoding="utf-8") + (tmp_path / "AGENTS.copilot.md").symlink_to("real", target_is_directory=True) + (tmp_path / "AGENTS.codex.md").symlink_to("AGENTS.copilot.md", target_is_directory=True) + (tmp_path / "AGENTS.claude.md").write_text("claude\n", encoding="utf-8") + (tmp_path / "AGENTS.md").write_text("base\n", encoding="utf-8") + listed = ["AGENTS.md", "AGENTS.codex.md", "AGENTS.copilot.md", "AGENTS.codex.md/AGENTS.copilot.md", + "real/AGENTS.copilot.md", "AGENTS.claude.md"] + removed, _ = _apply_overlay(tmp_path, sorted(listed), "claude") + assert removed == ["AGENTS.codex.md", "AGENTS.codex.md/AGENTS.copilot.md", "AGENTS.copilot.md"] + assert not os.path.lexists(tmp_path / "AGENTS.codex.md") + assert not os.path.lexists(tmp_path / "AGENTS.copilot.md") + assert not (tmp_path / "real" / "AGENTS.copilot.md").exists() + assert (tmp_path / "AGENTS.claude.md").is_file() + + +def test_native_in_probe_counts_only_the_native_names_a_probed_file_or_link_carries(): + """The profile names CLAUDE.md, but a probe with no CLAUDE.md file or link has no native name to import a target: + the configured name alone does not make an alias ambiguous.""" + shared = SimpleNamespace(path="docs/shared.md", aliases=("AGENTS.codex.md",)) + assert native_in_probe([shared], ["CLAUDE.md"]) == set() + assert native_in_probe([shared, SimpleNamespace(path="sub/CLAUDE.md", aliases=())], ["CLAUDE.md"]) == {"CLAUDE.md"} + assert native_in_probe([SimpleNamespace(path="CLAUDE.md", aliases=("AGENTS.codex.md",))], ["CLAUDE.md"]) == {"CLAUDE.md"} + + +def test_overlay_report_leaves_a_load_of_a_shared_target_unattributed(): + """A harness that loads AGENTS.md reads the file its AGENTS.claude.md and AGENTS.codex.md aliases name too: + neither alias is pinned as loaded, and the load is reported as ambiguous.""" + records = [{"file": "AGENTS.md", "scenario": "cwd=.", "class": "loaded"}] + links = {"AGENTS.md": ("AGENTS.claude.md", "AGENTS.codex.md")} + report = overlay_report(records, "claude", links, native={"AGENTS.md"}) + assert report["own_loaded"] == [] and report["cross_loaded"] == {} + assert report["ambiguous"] == ["AGENTS.claude.md", "AGENTS.codex.md"] + assert report["own_files"] == ["AGENTS.claude.md"] and report["foreign_files"] == ["AGENTS.codex.md"] + + +def test_overlay_report_does_not_pin_a_shared_target_read_through_a_native_alias(): + """CLAUDE.md and AGENTS.codex.md both link to docs/shared.md: a Claude load of it is CLAUDE.md, not the foreign alias.""" + records = [{"file": "docs/shared.md", "scenario": "cwd=.", "class": "loaded"}] + links = {"docs/shared.md": ("CLAUDE.md", "AGENTS.codex.md")} + report = overlay_report(records, "claude", links, native={"CLAUDE.md"}) + assert report["cross_loaded"] == {} + assert "AGENTS.codex.md" in report["ambiguous"] + + +def test_overlay_report_leaves_a_load_of_an_own_and_a_foreign_alias_unattributed(): + records = [{"file": "docs/x.md", "scenario": "cwd=.", "class": "loaded"}] + report = overlay_report(records, "claude", {"docs/x.md": ("AGENTS.claude.md", "AGENTS.codex.md")}) + assert report["own_loaded"] == [] and report["cross_loaded"] == {} + assert report["ambiguous"] == ["AGENTS.claude.md", "AGENTS.codex.md"] + + +def test_an_overlay_cell_is_identified_by_the_overlay_it_keeps(tmp_path): + """A resumed batch must not reuse an overlay cell that kept another agent's overlay; other cells keep their ids.""" + from dokimi.repos.runner import AgentOptions, ExecutionMismatch, check_batch_executions, execution_record + + harness = tmp_path / "harness" + harness.write_bytes(b"harness\n") + claude = load_profile("claude-code") + codex = dataclasses.replace(claude, overlay="codex") + + def options(profile): + return AgentOptions(profile=profile, model="m", harness_bin=harness) + + def cell_id(profile, arm): + return execution_record(options(profile), arm)["id"] + + assert cell_id(claude, "guidance:overlay") != cell_id(codex, "guidance:overlay") + assert cell_id(claude, "guidance:none") == cell_id(codex, "guidance:none") + assert cell_id(claude, "guidance:overlay") == execution_record(options(claude), "guidance:overlay")["id"] + + batch = tmp_path / "batch" + cell = batch / "task__guidance-overlay__m__r1" + cell.mkdir(parents=True) + (cell / "result.json").write_text( + json.dumps({"execution": {"id": cell_id(claude, "guidance:overlay"), "tools": "all", "delegation": "native"}}), + encoding="utf-8") + check_batch_executions(batch, [options(claude)]) # the same execution: no refusal + with pytest.raises(ExecutionMismatch, match="another execution"): + check_batch_executions(batch, [options(codex)]) + plain = batch / "task__guidance-none__m__r1" + plain.mkdir() + (plain / "result.json").write_text( + json.dumps({"execution": {"id": cell_id(claude, "guidance:none"), "tools": "all", "delegation": "native"}}), + encoding="utf-8") + check_batch_executions(batch, [options(claude)]) # a plain cell beside an overlay cell of one profile + + +def test_overlay_report_leaves_a_load_of_a_foreign_target_with_an_own_alias_unattributed(): + records = [{"file": "AGENTS.codex.md", "scenario": "cwd=.", "class": "loaded"}] + report = overlay_report(records, "claude", {"AGENTS.codex.md": ("AGENTS.claude.md",)}) + assert report["cross_loaded"] == {} and report["own_loaded"] == [] + assert report["ambiguous"] == ["AGENTS.claude.md", "AGENTS.codex.md"] + + +def test_guidance_none_drops_every_overlay_as_well_as_the_base(repo): + meta, ws = _prep(repo, "guidance:none", load_profile("claude-code"), name="rd-none") + assert not (ws / "AGENTS.md").exists() and not (ws / "sub/AGENTS.md").exists() + assert not (ws / "AGENTS.claude.md").exists() and not (ws / "AGENTS.codex.md").exists() + assert not (ws / "sub/AGENTS.copilot.md").exists() + assert {"AGENTS.claude.md", "AGENTS.codex.md", "sub/AGENTS.copilot.md"} <= set(meta["arm_files"]) + + +def test_guidance_none_removes_an_overlay_the_globs_also_match_once(repo): + """A repository whose globs list **/AGENTS.*.md (so rev: drops its overlays) still runs guidance:none.""" + repo = {**repo, "manifest": dataclasses.replace(repo["manifest"], + guidance_globs=("**/AGENTS.md", "**/AGENTS.*.md"))} + meta, ws = _prep(repo, "guidance:none", load_profile("claude-code"), name="rd-none-globs") + assert meta["arm_files"] == sorted(OVERLAY_FILES) + assert not any((ws / rel).exists() for rel in OVERLAY_FILES) + + +def test_guidance_as_skill_drops_the_overlays_as_guidance_none_does(repo): + meta, ws = _prep(repo, "guidance:as-skill", load_profile("claude-code"), name="rd-as-skill") + assert not any((ws / rel).exists() for rel in OVERLAY_FILES) + assert set(OVERLAY_FILES) <= set(meta["arm_files"]) + skill = (ws / meta["guidance_skill"]["path"] / "SKILL.md").read_text(encoding="utf-8") + assert "## File: AGENTS.md" in skill and "# for claude" not in skill # dropped, not packaged + + +def test_guidance_rev_keeps_the_overlays(repo): + meta, ws = _prep(repo, "guidance:rev:overlays", load_profile("claude-code"), name="rd-rev") + assert meta["arm_files"] == ["AGENTS.md", "sub/AGENTS.md"] + for rel in ("AGENTS.claude.md", "AGENTS.codex.md", "sub/AGENTS.copilot.md"): + assert (ws / rel).read_text(encoding="utf-8") == OVERLAY_FILES[rel] diff --git a/dokimi/tests/test_repo_reach_links.py b/dokimi/tests/test_repo_reach_links.py new file mode 100644 index 000000000..2a149f447 --- /dev/null +++ b/dokimi/tests/test_repo_reach_links.py @@ -0,0 +1,29 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Overlay links in the L1 reach probe: which names a harness is read as, and what that pins (no LLM).""" + +from __future__ import annotations + +from dokimi.artifacts import validate_artifact +from dokimi.repos import load_manifest +from dokimi.repos.harness_profile import load_profile +from dokimi.repos.reach import run_reach +from test_repo_reach import _commit_links, _profile, _repo, _write + + +def test_reach_payload_leaves_a_native_alias_of_a_foreign_overlay_unpinned(tmp_path, monkeypatch): + """run_reach gives overlay_report the profile's native names: CLAUDE.md -> AGENTS.codex.md is read as the + native file, so a load of it is not a load of the foreign overlay.""" + monkeypatch.setenv("FAKE_MODE", "ok") + root = _repo(tmp_path) + _write(root, "AGENTS.codex.md", "# codex overlay\n") + (root / "CLAUDE.md").unlink() + (root / "sub/CLAUDE.md").unlink() + tree = _commit_links(root, {"CLAUDE.md": "AGENTS.codex.md"}) + payload = run_reach(tree=tree, manifest=load_manifest(tree, "evals/repo-eval.yaml"), + profile=load_profile(_profile(tmp_path, overlay="claude")), model="m", + out_dir=tmp_path / "out", work_dir=tmp_path, salt="s", kinds=("root",)) + validate_artifact("guidance_reach", {**payload, "schema_version": "guidance_reach.v1"}) + assert payload["overlays"]["cross_loaded"] == {} + assert payload["overlays"]["ambiguous"] == ["AGENTS.codex.md", "CLAUDE.md"] diff --git a/dokimi/tests/test_repo_replay.py b/dokimi/tests/test_repo_replay.py index db5d32ac4..24001c0af 100644 --- a/dokimi/tests/test_repo_replay.py +++ b/dokimi/tests/test_repo_replay.py @@ -333,6 +333,30 @@ def test_the_tasks_pass_the_self_check_and_apply_under_guidance_none(hist, tmp_p assert arm == "guidance:none" or _read(ws, "AGENTS.md") == ["# guidance"] # the base's, not the PR's +def test_a_pr_that_edits_an_agents_overlay_leaves_it_out_and_applies_under_guidance_none(hist, tmp_path): + root = hist["root"] + main = git(root, "rev-parse", "--abbrev-ref", "HEAD") + _put(root, "AGENTS.codex.md", ["# for codex"]) + _commit(root, "an overlay") + git(root, "checkout", "-qb", "pr20") + _put(root, "AGENTS.codex.md", ["# for codex", "pr20"]) + _set(root, "src/r.py", 3, "r3 pr") + rev = _commit(root, "overlay and code") + merge = _squash(root, main, "pr20") + git(root, "branch", "-q", "-D", "pr20") + _replay(root, _pages(_pr(20, rev, merge, [_t("src/r.py", 3, rev)])), root / ".agent-eval/tasks") + task, _, patch = _load(root / ".agent-eval/tasks", 20) + assert task["left_out_files"] == ["AGENTS.codex.md"] and "AGENTS.codex.md" not in patch + _commit(root, "replay task") + source = RepoSource.parse(f"{root}@HEAD") + tree = open_source(source) + manifest = load_manifest(tree, ".agent-eval/repo-eval.yaml") + task = load_tasks(export_eval_dir(tree, manifest, tmp_path / "content"), manifest, ["review_replay_pr20"])[0] + prepare_workspace(source_git_dir(source, tree), manifest, task, "guidance:none", tmp_path / "none") + ws = tmp_path / "none" / "workspace" + assert _read(ws, "src/r.py")[2] == "r3 pr" and not (ws / "AGENTS.codex.md").exists() + + # ------------------------------------------------------------------ the other pull requests