zizi-al/muse-agent-example
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

实现: 建立正文三臂冻结回放与可信判定

Browse Source
This commit is contained in:
zizi 2026-07-20 21:00:00 +08:00
parent 8727eeaa46
commit 3e269e582d
12 changed files with 4478 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.claude/agents/judge.md
View File

@ -5,7 +5,7 @@ tools: Read, Grep, Glob, Write
model: opus
---
你是质量评委,保护节点角色(不可被装配替换)。**功能合同=`quality-gate` skill**——正文任务使用 `quality_gate` profile；细纲回放任务使用 `fine_outline_replay` profile。维度、证据要求和稳定性口径全在那里,你只执行不自造维度。报告按当前运行合同落临时评测目录或 `works/<书>/评审/第NNN章-评分.md`,除此不写正式创作文件。
你是质量评委,保护节点角色(不可被装配替换)。**功能合同=`quality-gate` skill**——在线正文质量详情使用 `quality_gate` profile；正文离线回放使用独立 `writer_replay` profile；细纲回放使用 `fine_outline_replay` profile。维度、证据要求和稳定性口径全在那里,你只执行不自造维度。报告按当前运行合同落临时评测目录或 `works/<书>/评审/第NNN章-评分.md`,除此不写正式创作文件。
## 元数据纪律(怎么用元数据)
@ -13,11 +13,13 @@ model: opus
- 上下文=writer 基线包(**同证独立**):评的是"在写手所知条件下写得好不好",不索取额外资料,不拿包外信息扣分。
- 细纲回放只使用冻结快照和结构化 `reference scaffold proxy`;不得读取目标章全文或完整目标章细纲。
- 细纲回放每个分数必须有 `evidence` 字段；不得使用正文文风、文笔或可读性作为评分维度。
- 正文回放只接收盲化候选 ID，不得在报告中输出 A/B/C、真实臂名、文件名或提示词差异。每维证据必须标为 `fine_outline`、`historical_prose` 或 `judge_inference`；盲评输入不含卡，禁止伪造 `card_index` 归因。卡效用只由去盲后的 B-A/C-A 差分判断。
## 打分纪律
- 每维给分必附一句引文证据(好在哪/差在哪,引原句);没有证据的分数无效。
- 同一维度复评同一章分差应 ≤0.5;严格度不因收敛压力改变,不放水不加戏。
- 正文回放使用 0-10 分、0.5 步长。双评同维差异大于 0.5 时只请求一次第三评委；第三评后仍无任意稳定配对时标记 `invalid_unstable`，不强行给臂输赢。
- 末尾「最值得改的三点」按提升空间排序:问题→根因层猜测(prompt/上下文/设定卡)→具体改法。
- 细纲回放时，把末尾建议替换为“最值得补齐的三项结构缺口”，并标注它属于候选结构、公共大纲、卡注入、原文检索还是标准事实不确定；若两次同维分差大于 0.5，只写稳定性警告，不强行裁决。

6
.claude/skills/eval/SKILL.md
View File

@ -26,3 +26,9 @@ description: 质量收敛环——judge 打分、锁最低维、单变量改进(
## 红线
评分必须有引文证据;禁止为了分数让 judge 放水(rubric 与严格度不因收敛压力改变);同一 rubric 复评分差 >0.5 时先查评委稳定性再查正文。
## 正文离线回放
正文 A/B/C 回放使用独立 `writer_replay` profile，不复用本 skill 的在线 1-5 分收敛口径，也不覆盖 `fine_outline_replay`。五维均为 0-10 分、0.5 步长：`setting_entity_fidelity`、`fine_outline_fidelity`、`style_consistency`、`narrative_tension`、`prose_readability`。
两名独立评委只看盲化候选和所有臂一致的共同参考，第二名严格反序。同维分差大于 0.5 才增加一次第三评；三评分至少一对差值不大于 0.5 时取中位数，否则样本为 `invalid_unstable`，不进入方向结论。每个维度都必须把证据归因为细纲、历史原文或评委自行推断。盲评输入不含卡，卡效用由去盲后的 B-A/C-A 差分识别。

16
.claude/skills/quality-gate/SKILL.md
View File

@ -1,6 +1,6 @@
---
name: quality-gate
description: 质量评分的功能合同(scenario: quality_gate/fine_outline_replay,保护节点)。正文质量维度和细纲回放 rubric 分开管理;rubric 与达标口径的单一来源。
description: 质量评分的功能合同(scenario: quality_gate/writer_replay/fine_outline_replay,保护节点)。在线正文、正文回放和细纲回放 rubric 分开管理;rubric 与达标口径的单一来源。
disable-model-invocation: true
---
@ -58,3 +58,17 @@ scores:
```
两次独立评审的同维差异大于 `0.5` 时只输出稳定性警告，不据此下卡效用结论。三臂顺序、arm 名称和卡 manifest 对评委隐藏；汇总报告分别给出每维分数、结构门结果、`B-A` 和 `C-A`，不压成没有统计意义的单一“卡质量总分”。
## 正文回放 profile（`writer_replay`）
正文回放与在线 `quality_gate`、细纲 `fine_outline_replay` 完全独立。每维 0-10 分、0.5 步长；双评委独立无会话，第二评委严格反序。报告在去盲前只保留随机候选 ID，禁止出现 A/B/C 或证据策略名称。
| 维度 ID | 评分问题 |
|---|---|
| `setting_entity_fidelity` | 设定、实体、关系、能力边界和冻结时点状态是否保真 |
| `fine_outline_fidelity` | 细纲硬事件、结果方向、伏笔动作、出场实体和章末钩子是否忠实 |
| `style_consistency` | 与冻结历史原文的人物声音、动作习惯、句式和叙事质感是否一致 |
| `narrative_tension` | 场景推进、冲突升级、不可逆变化和章末驱动力是否成立 |
| `prose_readability` | 语言是否准确、自然、清晰且没有明显阅读阻力 |
每维证据数组中的 `sourceType` 只能是 `fine_outline`、`historical_prose` 或 `judge_inference`；评委自行推断必须显式标注。盲评输入不含卡，禁止评委声称证据来自 `card_index`，卡效用只由去盲后的 B-A/C-A 差分归因。双评同维分差大于 0.5 时启动且只启动一次第三评委。第三评后，任一评分对差值不大于 0.5 则取三者中位数；否则整章为 `invalid_unstable`，不得强行计算胜负或进入 Gate 增益统计。

40
.claude/skills/replay-eval/SKILL.md
View File

@ -4,9 +4,9 @@ description: 回放评测的冻结与结果边界合同。把参考作品冻结
disable-model-invocation: true
---
# 回放评测（`next_fine_outline_replay_v0`）
# 回放评测
本 skill 只负责确定性的评测编排边界，不调用模型、不替代统一读取器，也不写正式规划或知识。评测目的、三臂定义和细纲评分合同见 `docs/2026-07-19-回放评测-细纲首跑设计与计划.md`。
本 skill 只负责确定性的评测编排边界，不调用模型、不替代统一读取器，也不写正式规划或知识。细纲回放见 `docs/2026-07-19-回放评测-细纲首跑设计与计划.md`；正文回放的唯一任务 SoT 是 `docs/2026-07-20-正文智能体正式优化设计与计划.md`。
真实参考作品配置由 `scripts/load_reference_work.py` 从实验库只读组装；它只取作品元数据、窗级大纲、章级细纲摘要和预注册卡 ID 对应的候选卡历史。候选卡必须标记为 `eval_draft`，不能当作生产知识检索结果。
@ -50,3 +50,39 @@ disable-model-invocation: true
- `scripts/write_report.py`：从 `run_result.json` 生成安全摘要；它不会读取候选正文，也不会把候选路径以外的原始响应写入报告。
真实作品运行前必须先从权威来源取得不可变授权快照。数据库没有该字段时，使用 dry-run 证明机制并保持 `blocked_authorization`，不得用本地配置或口头许可伪造放行。
## 正文 A/B/C 回放
正文 Gate A 固定配置为 `configs/writer-gate-a-deep-space-v1.json`，预注册深空之影五章：489 战斗、321 人物对话、544 转折、199 信息揭示、523 老角色回归。不得根据候选结果换章、换场景分类或修改冻结点；每个冻结点必须等于 `targetChapter-1`。
每个样本必须提供 `writerContextInput`。仓内配置只允许 `contentMode=sanitized_contract_fixture`，`recentChapters` 只能放连续四章的脱敏合成短文本，用于证明 WriterContext 合同、章号冻结和 A/B/C 差异策略；它不是历史原文，不证明真实正文完整性或文学质量。配置可以记录细纲硬约束、主要实体、预期篇幅、新角色比例和泄漏哨兵，但不得包含原书全文、完整目标细纲或标准答案。
三臂都必须构造并校验 `WriterContext v1`，且固定 `mode=diagnostic_only`、`purpose=evaluation`、`acceptanceEligible=false`：
- A：`evidenceStrategy=historical_prose_only`，保留连续四章脱敏基线，不放 `indexHints`。
- B：`evidenceStrategy=card_index_only`，只放 `retrievalResult.indexHints`，`proseEvidence` 为空；这是合同内可审计的诊断基线例外，不是生产旁路。
- C：`evidenceStrategy=card_index_plus_prose`，保留连续四章脱敏基线，并放冻结 `indexHints` 与配置中的补充原文证据。
`indexHints` 每项只能包含 `cardId/name/type/content/sourceId/sourceVersion/asOf`，只能用于 `diagnostic_only` 的 `evaluation/diagnostic`，不得进入生产上下文；`claimLedger` 不得引用它。所有 `asOf` 和来源章号不得超过样本冻结点。
每个样本必须记录 `frozenRecentHanCounts` 和 `targetLengthBasis`。篇幅只能使用冻结点前连续四章汉字数，经 `calculate_target_chars` 复算；当前 Gate A 固定 `hardEventCount=1`、`foreshadowingActionCount=0`、`requiredSceneCount=0`，禁止读取目标章实际长度。`expectedLength` 与 WriterContext 的输出合同必须等于机械目标的正负 10%，再受 2000-10000 边界限制。
`newCharacterRatio` 定义为：具名 `requiredCharacters` 中，在 `asOfChapter` 前无记录的角色比例。泛称角色不参与猜测；例如第 544 章的“内应”无法确定具体身份，必须记录 `newCharacterRatio=null` 和 `newCharacterRatioStatus=unresolved_generic_role`，不得默认成 0。
盲评使用独立的 `writer-blind-input-v1` 内容边界。judge 只能看到 `blind-1/2/3` 候选正文/哈希，以及所有臂完全相同的 `sharedEvaluationReference`：目标章细纲硬约束、实体、必需角色、伏笔、章末钩子和冻结点前连续四章历史原文基准。共同参考可以在真实运行时临时传给 judge，但不能进入安全摘要。它不得包含 `indexHints`、各臂补充原文、卡 manifest、`evidenceStrategy`、真实 A/B/C 映射、各臂 WriterContext、`raw` 目录或任何包含 `candidate-A/B/C` 的路径；卡的正确性只能由共同历史原文基准验证，不能把被测卡本身当裁判。映射只保留在编排器内存中，评分完成后才去盲。
正文 dry-run 命令：
```bash
.venv/bin/python .claude/skills/replay-eval/scripts/run_writer_replay.py \
--config .claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json \
--dry-run
```
dry-run 只输出计划、manifest 和上下文摘要，不调用 writer、semantic detector 或 judge，不生成候选正文，也不得声称真实回放完成。当前 CLI 的 `--execute` 因真实 semantic detector 和 judge adapter 尚未接线而失败关闭；只有实现并验证这两个受控 adapter 后，才可在预算确认后启用真实模型回放。
## 正文 Gate 输入
`writer_gate.py` 只信任 `gate-input.json.samples[]` 的逐样本脱敏结果。有效样本数、作品数、场景覆盖、C 臂硬门统计、C-A 五维增量、退化比例和五类混淆项全部由判定器内部计算；顶层传入的同名聚合值和聚合 `confounds` 不参与裁决。五类混淆项是：假阴、假阳、泄露、评委不稳定、新角色无卡。
真实回放未接线期间不能生成合法 Gate 输入，也不能运行 Gate A/B 得出通过结论。Gate A/B 的终态只能由 `writer_gate.py` 按任务 SoT 的唯一顺序产生，人工不得改写。

1196
.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json Normal file
View File

File diff suppressed because it is too large Load Diff

1101
.claude/skills/replay-eval/scripts/run_writer_replay.py Normal file
View File

File diff suppressed because it is too large Load Diff

736
.claude/skills/replay-eval/scripts/test_run_writer_replay.py Normal file
View File

@ -0,0 +1,736 @@
#!/usr/bin/env python3
"""正文 A/B/C 冻结回放编排的无网络、无真实模型测试。"""
from __future__ import annotations
import copy
import hashlib
import json
import pathlib
import subprocess
import sys
import tempfile
import unittest
SCRIPT_DIR = pathlib.Path(__file__).resolve().parent
SKILLS_DIR = SCRIPT_DIR.parents[1]
READ_CONTEXT_DIR = SKILLS_DIR / "read-context" / "scripts"
for import_path in (SCRIPT_DIR, READ_CONTEXT_DIR):
sys.path.insert(0, str(import_path))
from run_writer_replay import ( # noqa: E402
WriterReplayError,
WriterReplayTestAdapters,
run_writer_replay,
)
from writer_contract import calculate_target_chars, han_count, validate_writer_context # noqa: E402
from writer_rubric import DIMENSIONS, RUBRIC_PROFILE # noqa: E402
AUTHORIZATION = {
"sourceStatus": "authorized",
"copyrightStatus": "owned",
"sourceVersion": "raw-file-v1:sha256:" + "a" * 64,
"allowedPurpose": ["offline_evaluation"],
"authorizationSnapshot": {
"id": "auth-work-8",
"version": "auth-work-8-v1",
"immutable": True,
"sourceVersion": "raw-file-v1:sha256:" + "a" * 64,
"sourceStatus": "authorized",
"allowedPurpose": ["offline_evaluation"],
"checkedAt": "2026-07-20T00:00:00Z",
"revalidationAt": "2026-08-19T00:00:00Z",
},
}
def _prose(chapter: int, block_id: int, text: str) -> dict[str, object]:
"""构造只用于合同证明的脱敏合成原文片段。"""
return {
"chapter": chapter,
"sourceRef": {
"sourceId": f"fixture:chapter:{chapter}:block:{block_id}",
"sourceVersion": f"sanitized-fixture-{chapter}-v1",
"chapter": chapter,
"blockId": block_id,
"startCodePoint": 0,
"endCodePoint": len(text),
},
"text": text,
}
def _writer_context_input() -> dict[str, object]:
"""构造可让 A/B/C 都通过 WriterContext v1 的脱敏夹具。"""
recent = [
_prose(chapter, 1000 + chapter, f"脱敏合同夹具第{chapter}章，人物保持冻结状态。")
for chapter in range(485, 489)
]
supplemental = _prose(120, 1120, "脱敏补充片段，旧徽章曾经出现。")
return {
"contentMode": "sanitized_contract_fixture",
"sourceVersion": AUTHORIZATION["sourceVersion"],
"authorizationSnapshot": {
"snapshotId": "auth-work-8",
"allowedPurpose": "offline_evaluation",
"verifiedAt": "2026-07-20T00:00:00Z",
},
"sourceStatus": "authorized",
"cardIndexVersion": "cards-frozen-488-v1",
"proseIndexVersion": "prose-frozen-488-v1",
"retrievalResult": {
"cards": [
{
"name": "林澈",
"type": "character",
"sourceRefs": [copy.deepcopy(supplemental["sourceRef"])],
}
],
"factEvidence": [],
"proseEvidence": [supplemental],
"indexHints": [
{
"cardId": "card-1",
"name": "林澈",
"type": "character",
"content": "林澈在冻结点仍位于圣蒂曼",
"sourceId": "fixture:card:1",
"sourceVersion": "cards-frozen-488-v1",
"asOf": 488,
}
],
"manifest": {"omittedSources": []},
},
"fineOutline": {
"sourceRef": {
"sourceId": "fixture:fine-outline:489",
"sourceVersion": "fine-outline-fixture-v1",
"chapter": 489,
},
"hardConstraints": ["必须完成围攻突围"],
"adjustableBeats": [],
"declaredNewFacts": [],
"entities": [{"id": "character:林澈", "type": "character", "name": "林澈"}],
"relations": [],
"items": [{"id": "item:旧徽章", "type": "item", "name": "旧徽章"}],
"locations": [{"id": "location:圣蒂曼", "type": "location", "name": "圣蒂曼"}],
"powerSystems": [],
},
"narrativeState": {
"time": "围攻当日",
"location": "圣蒂曼",
"characterPositions": {"林澈": "城内"},
"immediateSituation": "围攻持续",
},
"recentChapters": recent,
"outputContract": {
"targetChars": 4000,
"minChars": 3600,
"maxChars": 4400,
"frontmatterRequired": False,
"newSettingDeclarationRequired": True,
},
"tokenBudget": {"maxContextChars": 50000},
"generatedAt": "2026-07-20T00:00:00Z",
"requirements": {
"requiredEvents": [
{"requirementId": "event-1", "anchors": ["完成围攻突围"]}
],
"requiredCharacters": ["林澈"],
"foreshadowingActions": [
{"requirementId": "foreshadow-1", "anchors": ["旧徽章"]}
],
"chapterEndHook": {
"requirementId": "hook-1",
"anchors": ["城门忽然打开"],
"maxDistanceFromEnd": 20,
},
"detectedNewSettings": [],
"frozenConflicts": [],
},
}
def config() -> dict[str, object]:
"""构造不含原文全文的单样本预注册配置。"""
common = {
"workId": 8,
"asOfChapter": 488,
"targetChapter": 489,
"outlineSource": "outline:481-488",
"fineOutlineSource": "scaffold:489",
"targetChars": 4000,
"modelVersion": "opus",
"sampling": {"temperature": 0, "topP": 1, "seed": 489},
"detectorProfile": "writer-detector-report-v1",
}
return {
"profile": "writer_replay",
"evaluationSetVersion": "writer-gate-a-test-v1",
"strategyVersion": "writer-abc-v1",
"referenceWork": {
"id": 8,
"title": "深空之影",
"version": AUTHORIZATION["sourceVersion"],
},
"authorization": copy.deepcopy(AUTHORIZATION),
"commonControls": common,
"samples": [
{
"sampleId": "deep-space-489",
"scenario": "combat",
"asOfChapter": 488,
"targetChapter": 489,
"snapshotVersion": "writer-deep-space-489-v1",
"frozenRecentHanCounts": [4600, 4600, 4800, 4800],
"targetLengthBasis": {
"algorithm": "calculate_target_chars",
"sourceChapters": [485, 486, 487, 488],
"hardEventCount": 1,
"foreshadowingActionCount": 0,
"requiredSceneCount": 0,
"minChars": 2000,
"maxChars": 10000,
"usesTargetChapterLength": False,
},
"targetChars": 4000,
"expectedLength": {
"targetChars": 4000,
"minChars": 3600,
"maxChars": 4400,
},
"newCharacterRatio": 0.0,
"newCharacterRatioStatus": "resolved",
"newCharacterBasis": {
"definition": "named_required_characters_absent_before_as_of_ratio",
"asOfChapter": 488,
"requiredCharacters": ["林澈"],
"knownBeforeAsOf": ["林澈"],
"absentBeforeAsOf": [],
"genericRoles": [],
},
"sources": [
{
"sourceId": "fixture:chapter:485-488",
"sourceVersion": AUTHORIZATION["sourceVersion"],
"chapterRange": "485-488",
},
{
"sourceId": "fixture:card-index:488",
"sourceVersion": "cards-frozen-488-v1",
"chapterRange": "1-488",
},
],
"snapshotData": {
"chapters": [
{
"chapter": 488,
"sourceId": "fixture:chapter:488",
"contentSha256": "sha256:" + "1" * 64,
}
],
"cards": [],
},
"writerContextInput": _writer_context_input(),
"leakageAudit": {
"method": "target-fact-hash-and-chapter-bound-audit",
"targetFacts": {
"targetChapter": 489,
"forbiddenFacts": [
{
"id": "target-489-1",
"firstChapter": 489,
"text": "目标章专属秘密",
}
],
},
},
}
],
}
def _candidate_body(marker: str) -> str:
"""构造满足动态篇幅和全部机械锚点的三臂候选。"""
prefix = f"林澈{marker}完成围攻突围，又把旧徽章压回掌心。"
hook = "城门忽然打开"
filler_count = 4000 - han_count(prefix) - han_count(hook)
return prefix + "文" * filler_count + hook
def _writer_output(context: dict[str, object]) -> dict[str, object]:
"""从 subprocess stdin 上下文构造严格绑定的 WriterOutput。"""
marker = {
"historical_prose_only": "甲",
"card_index_only": "乙",
"card_index_plus_prose": "丙",
}[context["evidenceStrategy"]]
body = _candidate_body(marker)
candidate_hash = "sha256:" + hashlib.sha256(body.encode("utf-8")).hexdigest()
return {
"schemaVersion": "writer-output-v1",
"runId": context["runId"],
"attempt": context["attempt"],
"mode": context["mode"],
"qualityPolicyVersion": context["qualityPolicyVersion"],
"contextSnapshotId": context["contextSnapshot"]["manifestId"],
"contextSnapshotSha256": context["contextSnapshot"]["contextSha256"],
"candidateVersion": context["attempt"],
"candidateSha256": candidate_hash,
"acceptanceEligible": False,
"candidateBody": body,
"claimLedger": [],
"evidenceRequests": [],
"newSettingDeclarations": [],
"selfCheck": {"hardConstraintsCovered": True, "notes": []},
}
class FakeSubprocessRunner:
"""只替换 subprocess runner，候选仍由 run_writer() 解析和校验。"""
def __init__(self, *, illegal_index_claim: bool = False):
self.calls: list[tuple[list[str], dict[str, object]]] = []
self.contexts: list[dict[str, object]] = []
self.illegal_index_claim = illegal_index_claim
def __call__(self, command: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]:
context = json.loads(str(kwargs["input"]))
validate_writer_context(context)
self.calls.append((command, kwargs))
self.contexts.append(context)
output = _writer_output(context)
if self.illegal_index_claim and context["evidenceStrategy"] == "card_index_only":
output["claimLedger"] = [
{
"claimId": "claim-index-1",
"candidateSha256": output["candidateSha256"],
"startCodePoint": 0,
"endCodePoint": 2,
"factType": "character_state",
"factEvidenceId": "card-1",
"coverageState": "supported",
}
]
stdout = json.dumps(
{"type": "result", "result": json.dumps(output, ensure_ascii=False)},
ensure_ascii=False,
)
return subprocess.CompletedProcess(command, 0, stdout=stdout, stderr="")
class FakeSemanticDetector:
"""记录机械门输出，并返回与候选绑定的语义通过报告。"""
def __init__(self) -> None:
self.calls: list[tuple[str, bool]] = []
def __call__(self, context, candidate, mechanical_report):
self.calls.append((context["evidenceStrategy"], mechanical_report["passed"]))
return {
"status": "passed",
"candidateVersion": candidate["candidateVersion"],
"candidateSha256": candidate["candidateSha256"],
"blockingFailures": [],
"suggestions": [],
}
def judge_report(
reviewer_id: str,
sample_id: str,
blind_id: str,
order: list[str],
score: float = 8.0,
) -> dict[str, object]:
"""构造测试盲评报告。"""
return {
"profile": RUBRIC_PROFILE,
"reviewerId": reviewer_id,
"sampleId": sample_id,
"blindCandidateId": blind_id,
"candidateOrder": order,
"scores": {
dimension: {
"score": score,
"evidence": [
{
"sourceType": "judge_inference",
"sourceRef": f"candidate:{blind_id}",
"excerpt": f"证据-{dimension}",
}
],
}
for dimension in DIMENSIONS
},
}
class FakeJudge:
"""模拟双评差异和必要第三评，不读取网络或真实模型。"""
def __init__(self, *, unstable: bool = False):
self.calls: list[tuple[str, str]] = []
self.unstable = unstable
def __call__(self, blind_input, reviewer_id):
blind_id = blind_input["blindCandidateId"]
candidate_order = blind_input["candidateOrder"]
self.calls.append((reviewer_id, blind_id))
score = 8.0
if self.unstable and blind_id == "blind-1" and reviewer_id == "judge-2":
score = 7.0
if self.unstable and blind_id == "blind-1" and reviewer_id == "judge-3":
score = 7.5
return judge_report(
reviewer_id,
blind_input["sample"]["sampleId"],
blind_id,
candidate_order,
score,
)
class MaliciousJudge(FakeJudge):
"""主动扫描全部可见输入，证明 judge 无法发现真实臂或原始目录。"""
def __init__(self) -> None:
super().__init__()
self.visible_inputs: list[dict[str, object]] = []
def __call__(self, blind_input, reviewer_id):
rendered = json.dumps(blind_input, ensure_ascii=False, sort_keys=True)
forbidden = (
"candidate-A",
"candidate-B",
"candidate-C",
"pipeline-A",
"pipeline-B",
"pipeline-C",
"evidenceStrategy",
"writerContextInput",
"indexHints",
"retrievalManifest",
"evidenceCoverage",
"raw_dir",
"rawDir",
"_contexts",
)
for marker in forbidden:
if marker in rendered:
raise AssertionError(f"judge 可见输入泄露: {marker}")
self.visible_inputs.append(copy.deepcopy(blind_input))
return super().__call__(blind_input, reviewer_id)
def _test_adapters(
*, unstable: bool = False, illegal_index_claim: bool = False
) -> tuple[WriterReplayTestAdapters, FakeSubprocessRunner, FakeSemanticDetector, FakeJudge]:
"""集中构造测试注入对象，避免它们被误认为生产 runtime。"""
runner = FakeSubprocessRunner(illegal_index_claim=illegal_index_claim)
detector = FakeSemanticDetector()
judge = FakeJudge(unstable=unstable)
return (
WriterReplayTestAdapters(runner, detector, judge),
runner,
detector,
judge,
)
class WriterReplayDryRunTest(unittest.TestCase):
"""验证 dry-run 的合同、冻结、安全和控制变量。"""
def test_three_arms_validate_full_context_and_only_change_evidence_strategy(self):
result = run_writer_replay(config(), run_id="dry-1")
self.assertEqual(result["status"], "ready")
sample = result["samples"][0]
arms = sample["arms"]
self.assertEqual(set(arms), {"A", "B", "C"})
self.assertEqual(arms["A"]["evidenceStrategy"], "historical_prose_only")
self.assertEqual(arms["B"]["evidenceStrategy"], "card_index_only")
self.assertEqual(arms["C"]["evidenceStrategy"], "card_index_plus_prose")
self.assertEqual(len({arm["commonControlsSha256"] for arm in arms.values()}), 1)
self.assertTrue(all(not arm["acceptanceEligible"] for arm in arms.values()))
self.assertEqual(arms["A"]["contextSummary"]["recentBaselineChapters"], [485, 486, 487, 488])
self.assertEqual(arms["B"]["contextSummary"]["proseEvidenceCount"], 0)
self.assertEqual(arms["B"]["contextSummary"]["indexHintCount"], 1)
self.assertEqual(arms["C"]["contextSummary"]["recentBaselineChapters"], [485, 486, 487, 488])
self.assertEqual(arms["C"]["contextSummary"]["indexHintCount"], 1)
self.assertTrue(
all(
arm["contextSummary"]["contentMode"] == "sanitized_contract_fixture"
for arm in arms.values()
)
)
rendered = json.dumps(result, ensure_ascii=False)
self.assertNotIn("脱敏合同夹具", rendered)
self.assertNotIn("林澈在冻结点仍位于圣蒂曼", rendered)
def test_future_index_hint_invalidates_whole_sample(self):
leaked = config()
leaked["samples"][0]["writerContextInput"]["retrievalResult"]["indexHints"][0]["asOf"] = 489
result = run_writer_replay(leaked, run_id="dry-leak")
self.assertFalse(result["ok"])
self.assertEqual(result["status"], "invalid_leakage")
def test_invalid_b_arm_contract_cannot_be_reported_ready(self):
invalid = config()
invalid["samples"][0]["writerContextInput"]["retrievalResult"]["indexHints"] = []
result = run_writer_replay(invalid, run_id="dry-invalid-context")
self.assertFalse(result["ok"])
self.assertEqual(result["status"], "invalid_writer_context")
def test_target_length_is_mechanically_recomputed_and_target_chapter_length_is_forbidden(self):
invalid = config()
invalid["samples"][0]["targetChars"] = 4100
with self.assertRaisesRegex(WriterReplayError, "机械复算"):
run_writer_replay(invalid, run_id="dry-length-tampered")
leaked = config()
leaked["samples"][0]["targetLengthBasis"]["usesTargetChapterLength"] = True
with self.assertRaisesRegex(WriterReplayError, "禁止读取目标章"):
run_writer_replay(leaked, run_id="dry-length-leaked")
def test_unresolved_generic_role_must_remain_null_instead_of_defaulting_to_zero(self):
unresolved = config()
sample = unresolved["samples"][0]
sample["writerContextInput"]["requirements"]["requiredCharacters"] = ["内应"]
sample["newCharacterRatio"] = None
sample["newCharacterRatioStatus"] = "unresolved_generic_role"
sample["newCharacterBasis"] = {
"definition": "named_required_characters_absent_before_as_of_ratio",
"asOfChapter": 488,
"requiredCharacters": ["内应"],
"knownBeforeAsOf": [],
"absentBeforeAsOf": [],
"genericRoles": ["内应"],
}
self.assertTrue(run_writer_replay(unresolved, run_id="dry-generic-role")["ok"])
unresolved["samples"][0]["newCharacterRatio"] = 0
with self.assertRaisesRegex(WriterReplayError, "必须为 null"):
run_writer_replay(unresolved, run_id="dry-generic-role-zero")
def test_character_declared_absent_before_freeze_cannot_have_card_index_hint(self):
inconsistent = config()
sample = inconsistent["samples"][0]
sample["newCharacterRatio"] = 1.0
sample["newCharacterBasis"]["knownBeforeAsOf"] = []
sample["newCharacterBasis"]["absentBeforeAsOf"] = ["林澈"]
with self.assertRaisesRegex(WriterReplayError, "不得同时出现在卡索引"):
run_writer_replay(inconsistent, run_id="dry-absent-card-conflict")
def test_gate_a_preregistration_mechanically_recomputes_all_lengths_and_ratios(self):
gate_config_path = SCRIPT_DIR.parent / "configs" / "writer-gate-a-deep-space-v1.json"
gate_config = json.loads(gate_config_path.read_text(encoding="utf-8"))
expected_targets = {489: 7500, 321: 7600, 544: 6700, 199: 2000, 523: 6100}
expected_ratios = {489: 1.0, 321: 0.0, 544: None, 199: 0.0, 523: 0.0}
for sample in gate_config["samples"]:
basis = sample["targetLengthBasis"]
recalculated = calculate_target_chars(
recent_chapter_han_counts=sample["frozenRecentHanCounts"],
hard_event_count=basis["hardEventCount"],
foreshadowing_action_count=basis["foreshadowingActionCount"],
required_scene_count=basis["requiredSceneCount"],
min_chars=basis["minChars"],
max_chars=basis["maxChars"],
)
target_chapter = sample["targetChapter"]
self.assertEqual(recalculated, expected_targets[target_chapter])
self.assertEqual(sample["targetChars"], recalculated)
self.assertEqual(sample["newCharacterRatio"], expected_ratios[target_chapter])
if target_chapter == 544:
self.assertEqual(sample["newCharacterRatioStatus"], "unresolved_generic_role")
else:
self.assertEqual(sample["newCharacterRatioStatus"], "resolved")
result = run_writer_replay(gate_config, run_id="gate-a-preregistered-dry-run")
self.assertTrue(result["ok"])
self.assertEqual(len(result["samples"]), 5)
class WriterReplayExecuteBoundaryTest(unittest.TestCase):
"""验证真实执行失败关闭和测试注入仍经过正式管线。"""
def test_execute_requires_private_tmp_output(self):
with tempfile.TemporaryDirectory() as directory:
with self.assertRaisesRegex(WriterReplayError, "/private/tmp"):
run_writer_replay(
config(),
run_id="real-bad-path",
output_dir=pathlib.Path(directory),
execute=True,
)
def test_execute_without_production_adapters_fails_closed(self):
with tempfile.TemporaryDirectory(dir="/private/tmp") as directory:
with self.assertRaisesRegex(WriterReplayError, "真实 execute 未接线"):
run_writer_replay(
config(),
run_id="real-not-wired",
output_dir=pathlib.Path(directory),
execute=True,
)
def test_context_authorization_cannot_swap_snapshot_or_source_before_runner(self):
"""上下文版本、快照、用途、时间和状态都必须绑定顶层授权。"""
mutations = {
"source_version": lambda value: value["samples"][0]["writerContextInput"].update(
{"sourceVersion": "swapped-source-v2"}
),
"snapshot_id": lambda value: value["samples"][0]["writerContextInput"][
"authorizationSnapshot"
].update({"snapshotId": "auth-swapped"}),
"purpose": lambda value: value["samples"][0]["writerContextInput"][
"authorizationSnapshot"
].update({"allowedPurpose": "diagnostic"}),
"verified_at": lambda value: value["samples"][0]["writerContextInput"][
"authorizationSnapshot"
].update({"verifiedAt": "2026-07-21T00:00:00Z"}),
"source_status": lambda value: value["samples"][0]["writerContextInput"].update(
{"sourceStatus": "revoked"}
),
}
for name, mutate in mutations.items():
invalid = config()
mutate(invalid)
adapters, runner, _detector, _judge = _test_adapters()
with self.subTest(name=name), tempfile.TemporaryDirectory(
dir="/private/tmp"
) as directory:
result = run_writer_replay(
invalid,
run_id=f"auth-binding-{name}",
output_dir=pathlib.Path(directory) / "run",
execute=True,
test_adapters=adapters,
)
self.assertFalse(result["ok"])
self.assertEqual(result["status"], "blocked_authorization")
self.assertEqual(runner.calls, [])
def test_test_injection_runs_writer_pipeline_mechanical_and_semantic_detector(self):
adapters, runner, detector, judge = _test_adapters(unstable=True)
with tempfile.TemporaryDirectory(dir="/private/tmp") as directory:
output_dir = pathlib.Path(directory) / "run"
result = run_writer_replay(
config(),
run_id="test-pipeline",
output_dir=output_dir,
execute=True,
test_adapters=adapters,
)
raw_candidates = sorted((output_dir / "raw" / "deep-space-489").glob("candidate-*.json"))
self.assertEqual(len(raw_candidates), 3)
self.assertIn("candidateBody", raw_candidates[0].read_text(encoding="utf-8"))
public_manifest = (output_dir / "manifest.json").read_text(encoding="utf-8")
self.assertNotIn("candidateBody", public_manifest)
self.assertNotIn("脱敏合同夹具", public_manifest)
self.assertEqual(result["status"], "completed_test_pipeline")
self.assertEqual([item["evidenceStrategy"] for item in runner.contexts], [
"historical_prose_only",
"card_index_only",
"card_index_plus_prose",
])
self.assertTrue(all("--agent" in command for command, _kwargs in runner.calls))
self.assertEqual(detector.calls, [
("historical_prose_only", True),
("card_index_only", True),
("card_index_plus_prose", True),
])
self.assertEqual(len([call for call in judge.calls if call[0] == "judge-3"]), 1)
candidates = result["samples"][0]["candidates"]
self.assertTrue(all(not candidate["acceptanceEligible"] for candidate in candidates.values()))
def test_index_hint_cannot_be_claim_ledger_evidence(self):
adapters, _runner, detector, _judge = _test_adapters(illegal_index_claim=True)
with tempfile.TemporaryDirectory(dir="/private/tmp") as directory:
with self.assertRaisesRegex(WriterReplayError, "claim_evidence_reference_invalid"):
run_writer_replay(
config(),
run_id="test-index-claim",
output_dir=pathlib.Path(directory) / "run",
execute=True,
test_adapters=adapters,
)
# B 臂在 writer adapter 已失败，不能进入 semantic detector。
self.assertEqual(detector.calls, [("historical_prose_only", True)])
def test_malicious_judge_only_sees_blind_candidates_without_raw_paths_or_arm_contexts(self):
runner = FakeSubprocessRunner()
detector = FakeSemanticDetector()
judge = MaliciousJudge()
adapters = WriterReplayTestAdapters(runner, detector, judge)
with tempfile.TemporaryDirectory(dir="/private/tmp") as directory:
output_dir = pathlib.Path(directory) / "run"
result = run_writer_replay(
config(),
run_id="blind-isolation",
output_dir=output_dir,
execute=True,
test_adapters=adapters,
)
self.assertTrue(result["ok"])
self.assertTrue(judge.visible_inputs)
shared_references = []
for visible in judge.visible_inputs:
self.assertEqual(
set(visible),
{
"schemaVersion",
"sample",
"blindCandidateId",
"candidateOrder",
"sharedEvaluationReference",
"candidates",
},
)
self.assertEqual(set(visible["candidateOrder"]), {"blind-1", "blind-2", "blind-3"})
shared = visible["sharedEvaluationReference"]
shared_references.append(copy.deepcopy(shared))
self.assertEqual(shared["fineOutline"]["hardConstraints"], ["必须完成围攻突围"])
self.assertEqual(shared["requirements"]["requiredCharacters"], ["林澈"])
self.assertEqual(
shared["requirements"]["chapterEndHook"]["anchors"],
["城门忽然打开"],
)
self.assertEqual(
[item["chapter"] for item in shared["historicalProseBaseline"]],
[485, 486, 487, 488],
)
self.assertTrue(
all(
set(candidate)
== {"blindCandidateId", "candidateSha256", "candidateBody"}
and candidate["blindCandidateId"].startswith("blind-")
for candidate in visible["candidates"]
)
)
self.assertTrue(
all(reference == shared_references[0] for reference in shared_references[1:]),
"评委顺序变化不得改变共同评测参考",
)
if __name__ == "__main__":
unittest.main()

326
.claude/skills/replay-eval/scripts/test_writer_gate.py Normal file
View File

@ -0,0 +1,326 @@
#!/usr/bin/env python3
"""正文 Gate A/B 逐样本唯一判定器的表驱动测试。"""
from __future__ import annotations
import copy
import pathlib
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from writer_gate import ( # noqa: E402
CONFOUND_LABELS,
REQUIRED_SCENARIOS,
decide_gate_a,
decide_gate_b,
)
from writer_rubric import DIMENSIONS # noqa: E402
SCENARIOS = tuple(sorted(REQUIRED_SCENARIOS))
def scores(
*,
b_fidelity_delta: float = 0.0,
c_fidelity_delta: float = 0.5,
) -> dict[str, dict[str, float]]:
"""构造三臂五维终分，分别控制 B-A 与 C-A 的保真增量。"""
a_scores = {dimension: 7.0 for dimension in DIMENSIONS}
b_scores = dict(a_scores)
c_scores = dict(a_scores)
b_scores["setting_entity_fidelity"] += b_fidelity_delta
c_scores["setting_entity_fidelity"] += c_fidelity_delta
return {"A": a_scores, "B": b_scores, "C": c_scores}
def sample(
number: int,
*,
work_id: int = 8,
scenario: str | None = None,
**overrides,
) -> dict[str, object]:
"""构造合法的逐样本脱敏结果。"""
result: dict[str, object] = {
"sampleId": f"{work_id}-{number}",
"workId": work_id,
"scenario": scenario or SCENARIOS[(number - 1) % len(SCENARIOS)],
"schemaValid": True,
"futureLeakage": False,
"systemFailure": False,
"reviewStatus": "stable_report",
"cArm": {
"hardConstraintCoverage": 1.0,
"highSeverityResidualCount": 0,
},
"scores": scores(),
"confounders": {category: [] for category in CONFOUND_LABELS},
}
result.update(overrides)
return result
def gate_a_input(samples: list[dict[str, object]], **forged) -> dict[str, object]:
"""构造 Gate A 输入；forged 用于证明顶层聚合值无效。"""
return {"gate": "A", "samples": samples, **forged}
def gate_b_input(samples: list[dict[str, object]], **overrides) -> dict[str, object]:
"""构造 Gate B 输入。"""
return {"gate": "B", "gateAStatus": "passed", "samples": samples, **overrides}
def ten_samples() -> list[dict[str, object]]:
"""构造两书各五章并覆盖五类场景的 Gate B 基线。"""
return [
sample(index + 1, work_id=work_id, scenario=SCENARIOS[index])
for work_id in (8, 4)
for index in range(5)
]
class GateATest(unittest.TestCase):
"""验证 Gate A 的逐样本统计与唯一短路顺序。"""
def test_table_driven_terminal_priority(self):
base = [sample(index) for index in range(1, 6)]
cases = []
insufficient = copy.deepcopy(base[:4])
insufficient[0]["futureLeakage"] = True
cases.append(("不足优先于泄露", insufficient, "insufficient_evidence"))
for label, field, value in (
("schema 非法", "schemaValid", False),
("未来泄露", "futureLeakage", True),
("系统失败", "systemFailure", True),
):
samples = copy.deepcopy(base)
samples.append(sample(6, **{field: value}))
cases.append((label, samples, "failed"))
high_severity = copy.deepcopy(base)
high_severity[0]["cArm"]["highSeverityResidualCount"] = 1
cases.append(("C 高严重度", high_severity, "failed"))
low_coverage = copy.deepcopy(base)
low_coverage[0]["cArm"]["hardConstraintCoverage"] = 0.99
cases.append(("覆盖不足", low_coverage, "failed"))
cases.append(("全部通过", base, "passed"))
for label, samples, expected in cases:
with self.subTest(label=label):
self.assertEqual(decide_gate_a(gate_a_input(samples))["status"], expected)
def test_pre_c_arm_failures_do_not_require_c_arm_results(self):
"""前置机械或系统失败没有 C 臂结果时仍按唯一顺序裁决。"""
base = [sample(index) for index in range(1, 6)]
cases = []
for label, field, value in (
("schema 非法", "schemaValid", False),
("未来泄露", "futureLeakage", True),
("系统失败", "systemFailure", True),
):
failed_sample = sample(6, **{field: value})
failed_sample.pop("cArm")
cases.append((f"五有效加{label}", [*copy.deepcopy(base), failed_sample], "failed"))
cases.append((f"四有效加{label}", [*copy.deepcopy(base[:4]), failed_sample], "insufficient_evidence"))
for label, samples, expected in cases:
with self.subTest(label=label):
self.assertEqual(decide_gate_a(gate_a_input(samples))["status"], expected)
def test_forged_upstream_aggregates_cannot_change_result(self):
forged = gate_a_input(
[sample(index) for index in range(1, 5)],
validSampleCount=999,
invalidSchemaCount=0,
futureLeakageCount=0,
cArmHardConstraintCoverage=1.0,
)
report = decide_gate_a(forged)
self.assertEqual(report["status"], "insufficient_evidence")
self.assertEqual(report["metrics"]["validSampleCount"], 4)
def test_valid_sample_boundary_counts_each_sample_once(self):
"""四个有效样本仍不足，补到五个后才进入硬门判断。"""
four_valid = [sample(index) for index in range(1, 5)]
multi_failed = sample(
5,
schemaValid=False,
futureLeakage=True,
systemFailure=True,
reviewStatus="invalid_unstable",
)
multi_failed.pop("cArm")
report = decide_gate_a(gate_a_input([*four_valid, multi_failed]))
self.assertEqual(report["status"], "insufficient_evidence")
self.assertEqual(report["metrics"]["sampleCount"], 5)
self.assertEqual(report["metrics"]["validSampleCount"], 4)
report = decide_gate_a(
gate_a_input([*four_valid, sample(6), multi_failed])
)
self.assertEqual(report["status"], "failed")
self.assertEqual(report["metrics"]["validSampleCount"], 5)
def test_five_confounder_categories_are_aggregated_per_sample(self):
samples = [sample(index) for index in range(1, 7)]
categories = list(CONFOUND_LABELS)
for index, category in enumerate(categories):
samples[index]["confounders"][category] = [{"code": f"case-{index}"}]
samples[5]["futureLeakage"] = True
samples[4]["reviewStatus"] = "invalid_unstable"
report = decide_gate_a(
gate_a_input(samples, confounds={"falseNegatives": ["伪造聚合"]})
)
self.assertEqual(set(report["confounds"]), set(CONFOUND_LABELS))
self.assertNotIn("伪造聚合", str(report["confounds"]))
for index, category in enumerate(categories):
self.assertTrue(
any(item["sampleId"] == f"8-{index + 1}" for item in report["confounds"][category])
)
self.assertTrue(
any(item["sampleId"] == "8-6" for item in report["confounds"]["leakage"])
)
class GateBTest(unittest.TestCase):
"""验证 Gate B 的充分性、退化与增益短路顺序。"""
def test_gate_a_terminal_has_highest_priority(self):
insufficient = [sample(index) for index in range(1, 5)]
report = decide_gate_b(gate_b_input(insufficient, gateAStatus="passed"))
self.assertEqual(report["status"], "insufficient_evidence")
self.assertEqual(report["metrics"]["recomputedGateAStatus"], "insufficient_evidence")
failed = [sample(index) for index in range(1, 6)]
failed.append(sample(6, futureLeakage=True))
report = decide_gate_b(gate_b_input(failed, gateAStatus="passed"))
self.assertEqual(report["status"], "failed")
self.assertEqual(report["metrics"]["recomputedGateAStatus"], "failed")
def test_forged_gate_a_status_cannot_override_recomputed_result(self):
"""顶层 Gate A 终态只是非可信输入，不能放行或阻断同批样本。"""
passed = decide_gate_b(gate_b_input(ten_samples(), gateAStatus="failed"))
self.assertEqual(passed["status"], "passed")
self.assertEqual(passed["metrics"]["recomputedGateAStatus"], "passed")
insufficient = decide_gate_b(
gate_b_input([sample(index) for index in range(1, 5)], gateAStatus="passed")
)
self.assertEqual(insufficient["status"], "insufficient_evidence")
def test_gate_a_short_circuit_keeps_sample_confounds_without_changing_status(self):
"""Gate A 终态优先，但合法逐样本混淆项仍应进入 Gate B 报告。"""
valid = [sample(index) for index in range(1, 6)]
leaked = sample(6, futureLeakage=True)
leaked.pop("cArm")
leaked["confounders"]["falsePositives"] = ["detector 误报候选"]
report = decide_gate_b(
{
"gateAStatus": "failed",
"samples": [*valid, leaked],
"confounds": {"falsePositives": ["伪造聚合"]},
}
)
self.assertEqual(report["status"], "failed")
self.assertIn("detector 误报候选", str(report["confounds"]["falsePositives"]))
self.assertTrue(report["confounds"]["leakage"])
self.assertNotIn("伪造聚合", str(report["confounds"]))
def test_sample_sufficiency_precedes_quality(self):
samples = ten_samples()[:-1]
for item in samples[:3]:
item["scores"]["C"]["style_consistency"] = 6.0
self.assertEqual(decide_gate_b(gate_b_input(samples))["status"], "insufficient_evidence")
unstable = ten_samples()
for index in range(3):
unstable[index]["reviewStatus"] = "invalid_unstable"
unstable[index].pop("scores")
self.assertEqual(
decide_gate_b(gate_b_input(unstable))["status"], "insufficient_evidence"
)
def test_missing_scene_or_single_work_is_insufficient(self):
missing_scene = ten_samples()
for item in missing_scene:
if item["scenario"] == "returning_character":
item["scenario"] = "battle"
self.assertEqual(
decide_gate_b(gate_b_input(missing_scene))["status"],
"insufficient_evidence",
)
one_work = [sample(index) for index in range(1, 11)]
self.assertEqual(
decide_gate_b(gate_b_input(one_work))["status"], "insufficient_evidence"
)
def test_hard_failure_and_quality_regression_fail(self):
low_coverage = ten_samples()
low_coverage[0]["cArm"]["hardConstraintCoverage"] = 0.9
self.assertEqual(decide_gate_b(gate_b_input(low_coverage))["status"], "failed")
decline = ten_samples()
for item in decline[:3]:
item["scores"]["C"]["style_consistency"] = 6.0
report = decide_gate_b(
gate_b_input(
decline,
anyDimensionDeclineOverHalfRatio=0.0,
averageDeltas={"setting_entity_fidelity": 999},
)
)
self.assertEqual(report["status"], "failed")
self.assertEqual(report["metrics"]["anyDimensionDeclineOverHalfRatio"], 0.3)
def test_gain_threshold_distinguishes_passed_and_no_gain(self):
self.assertEqual(decide_gate_b(gate_b_input(ten_samples()))["status"], "passed")
no_gain = ten_samples()
for item in no_gain:
item["scores"] = scores(c_fidelity_delta=0.0)
self.assertEqual(decide_gate_b(gate_b_input(no_gain))["status"], "no_gain")
def test_stable_sample_requires_complete_a_b_c_scores(self):
missing_b = ten_samples()
missing_b[0]["scores"].pop("B")
with self.assertRaisesRegex(ValueError, "精确包含 A/B/C"):
decide_gate_b(gate_b_input(missing_b))
def test_b_minus_a_is_diagnostic_and_c_minus_a_drives_gate(self):
"""即使 B-A 明显退化，只要 C-A 达标，正式 Gate B 仍按 C-A 通过。"""
samples = ten_samples()
for item in samples:
item["scores"] = scores(
b_fidelity_delta=-2.0,
c_fidelity_delta=0.5,
)
report = decide_gate_b(gate_b_input(samples))
self.assertEqual(report["status"], "passed")
self.assertEqual(
report["metrics"]["averageDeltas"]["B-A"]["setting_entity_fidelity"],
-2.0,
)
self.assertEqual(
report["metrics"]["averageDeltas"]["C-A"]["setting_entity_fidelity"],
0.5,
)
if __name__ == "__main__":
unittest.main()

182
.claude/skills/replay-eval/scripts/test_writer_rubric.py Normal file
View File

@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""正文五维盲评量表的离线回归测试。"""
from __future__ import annotations
import pathlib
import sys
import unittest
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from writer_rubric import ( # noqa: E402
DIMENSIONS,
RUBRIC_PROFILE,
adjudicate_reviews,
deblind_reports,
validate_blind_pair,
validate_report,
)
def scorecard(score: float = 8.0) -> dict[str, dict[str, object]]:
"""构造包含四类证据归因的合法五维评分。"""
source_types = ("fine_outline", "historical_prose", "judge_inference")
return {
dimension: {
"score": score,
"evidence": [
{
"sourceType": source_types[index % len(source_types)],
"sourceRef": f"source:{dimension}",
"excerpt": f"证据-{dimension}",
}
],
}
for index, dimension in enumerate(DIMENSIONS)
}
def report(
reviewer_id: str,
scores: dict[str, dict[str, object]] | None = None,
*,
order: tuple[str, ...] = ("blind-1", "blind-2", "blind-3"),
) -> dict[str, object]:
"""构造不携带真实臂名的单评委报告。"""
return {
"profile": RUBRIC_PROFILE,
"reviewerId": reviewer_id,
"sampleId": "deep-space-489",
"blindCandidateId": "blind-1",
"candidateOrder": list(order),
"scores": scores or scorecard(),
}
class WriterRubricValidationTest(unittest.TestCase):
"""验证量表结构与盲评纪律。"""
def test_five_dimensions_use_zero_to_ten_half_steps(self):
self.assertEqual(len(DIMENSIONS), 5)
self.assertEqual(validate_report(report("judge-1", scorecard(0.0))), [])
self.assertEqual(validate_report(report("judge-1", scorecard(10.0))), [])
bad = scorecard(8.0)
bad[DIMENSIONS[0]]["score"] = 8.25
self.assertTrue(any("0.5" in error for error in validate_report(report("judge-1", bad))))
def test_each_dimension_requires_typed_evidence(self):
bad = scorecard()
bad[DIMENSIONS[2]]["evidence"] = []
self.assertTrue(any("证据" in error for error in validate_report(report("judge-1", bad))))
bad = scorecard()
bad[DIMENSIONS[3]]["evidence"][0]["sourceType"] = "target_answer"
self.assertTrue(any("来源类型" in error for error in validate_report(report("judge-1", bad))))
hidden_card = scorecard()
hidden_card[DIMENSIONS[0]]["evidence"][0]["sourceType"] = "card_index"
self.assertTrue(
any("来源类型" in error for error in validate_report(report("judge-1", hidden_card)))
)
def test_pair_must_be_independent_blind_and_reverse_ordered(self):
first = report("judge-1")
second = report("judge-2", order=("blind-3", "blind-2", "blind-1"))
self.assertEqual(validate_blind_pair(first, second), [])
same_order = report("judge-2")
self.assertTrue(any("反序" in error for error in validate_blind_pair(first, same_order)))
different_candidate = report(
"judge-2", order=("blind-3", "blind-2", "blind-1")
)
different_candidate["blindCandidateId"] = "blind-2"
self.assertTrue(
any("同一盲化候选" in error for error in validate_blind_pair(first, different_candidate))
)
leaked = report("judge-2", order=("blind-3", "blind-2", "blind-1"))
leaked["arm"] = "C"
self.assertTrue(any("去盲" in error for error in validate_blind_pair(first, leaked)))
def test_deblind_uses_preregistered_mapping_not_position(self):
reports = [
report("judge-1"),
{**report("judge-1"), "blindCandidateId": "blind-3"},
]
mapping = {"blind-1": "C", "blind-2": "A", "blind-3": "B"}
deblinded = deblind_reports(reports, mapping)
self.assertEqual([item["arm"] for item in deblinded], ["C", "B"])
class WriterRubricAdjudicationTest(unittest.TestCase):
"""验证双评稳定性与最多一次第三评委仲裁。"""
def test_gap_above_half_requires_third_reviewer_once(self):
first = report("judge-1", scorecard(8.0))
second_scores = scorecard(8.0)
second_scores[DIMENSIONS[0]]["score"] = 7.0
second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1"))
result = adjudicate_reviews(first, second)
self.assertEqual(result["status"], "needs_third_reviewer")
self.assertEqual(result["unstableDimensions"], [DIMENSIONS[0]])
def test_third_reviewer_uses_median_when_any_stable_pair_exists(self):
first = report("judge-1", scorecard(8.0))
second_scores = scorecard(8.0)
second_scores[DIMENSIONS[1]]["score"] = 6.5
second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1"))
third_scores = scorecard(8.0)
third_scores[DIMENSIONS[1]]["score"] = 7.5
third = report("judge-3", third_scores)
result = adjudicate_reviews(first, second, third)
self.assertEqual(result["status"], "adjudicated_report")
self.assertEqual(result["scores"][DIMENSIONS[1]], 7.5)
def test_third_reviewer_must_bind_same_candidate_and_candidate_set(self):
"""第三评不得用另一候选或漂移后的候选集合参与仲裁。"""
first = report("judge-1", scorecard(8.0))
second_scores = scorecard(8.0)
second_scores[DIMENSIONS[1]]["score"] = 6.5
second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1"))
different_candidate = report("judge-3", scorecard(7.5))
different_candidate["blindCandidateId"] = "blind-2"
candidate_result = adjudicate_reviews(first, second, different_candidate)
self.assertEqual(candidate_result["status"], "invalid_report")
self.assertTrue(any("同一盲化候选" in error for error in candidate_result["errors"]))
different_set = report(
"judge-3",
scorecard(7.5),
order=("blind-1", "blind-2", "blind-4"),
)
set_result = adjudicate_reviews(first, second, different_set)
self.assertEqual(set_result["status"], "invalid_report")
self.assertTrue(any("相同的盲化候选集合" in error for error in set_result["errors"]))
def test_no_stable_pair_marks_sample_invalid(self):
first = report("judge-1", scorecard(8.0))
second_scores = scorecard(8.0)
second_scores[DIMENSIONS[4]]["score"] = 6.5
second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1"))
third_scores = scorecard(8.0)
third_scores[DIMENSIONS[4]]["score"] = 9.5
third = report("judge-3", third_scores)
result = adjudicate_reviews(first, second, third)
self.assertEqual(result["status"], "invalid_unstable")
self.assertEqual(result["unstableDimensions"], [DIMENSIONS[4]])
self.assertNotIn("winner", result)
if __name__ == "__main__":
unittest.main()

586
.claude/skills/replay-eval/scripts/writer_gate.py Normal file
View File

@ -0,0 +1,586 @@
#!/usr/bin/env python3
"""正文智能体 Gate A/B 的确定性唯一判定器。
判定器只消费逐样本脱敏结果，并在可信边界内计算全部统计值。顶层传入的
样本数、平均分、覆盖率或混淆项等聚合字段一律不参与裁决，避免上游通过
伪造汇总改变终态。命中较高优先级终态后不再使用较低优先级条件改写结果。
"""
from __future__ import annotations
import argparse
import json
import math
import sys
from pathlib import Path
from typing import Any, Mapping, Sequence
from writer_rubric import DIMENSIONS
REQUIRED_SCENARIOS = frozenset(
{
"battle",
"character_dialogue",
"turning_point",
"information_reveal",
"returning_character",
}
)
CONFOUND_LABELS = {
"falseNegatives": "假阴",
"falsePositives": "假阳",
"leakage": "泄露",
"reviewerInstability": "评委不稳定",
"newCharactersWithoutCards": "新角色无卡",
}
STABLE_REVIEW_STATUSES = frozenset({"stable_report", "adjudicated_report"})
REVIEW_STATUSES = frozenset({*STABLE_REVIEW_STATUSES, "invalid_unstable"})
def _require_mapping(value: object, field: str) -> Mapping[str, Any]:
"""取得必需对象字段，避免缺失数据被静默当成空对象。"""
if not isinstance(value, Mapping):
raise ValueError(f"{field} 必须是对象")
return value
def _require_samples(value: object) -> list[Mapping[str, Any]]:
"""取得非空逐样本数组，Gate 不接受只有聚合值的输入。"""
if not isinstance(value, list) or not value:
raise ValueError("samples 必须是非空逐样本数组")
if any(not isinstance(item, Mapping) for item in value):
raise ValueError("samples 每项必须是对象")
return list(value)
def _require_integer(value: object, field: str) -> int:
"""取得非负整数计数。"""
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise ValueError(f"{field} 必须是非负整数")
return value
def _require_number(value: object, field: str) -> float:
"""取得有限数值，显式拒绝布尔值、NaN 和无穷值。"""
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(f"{field} 必须是有限数值")
result = float(value)
if not math.isfinite(result):
raise ValueError(f"{field} 必须是有限数值")
return result
def _require_ratio(value: object, field: str) -> float:
"""取得闭区间 0-1 内的比例。"""
result = _require_number(value, field)
if not 0.0 <= result <= 1.0:
raise ValueError(f"{field} 必须在 0-1 之间")
return result
def _require_boolean(value: object, field: str) -> bool:
"""取得严格布尔值，拒绝 0/1 等隐式真假值。"""
if not isinstance(value, bool):
raise ValueError(f"{field} 必须是布尔值")
return value
def _sample_identity(sample: Mapping[str, Any], index: int) -> tuple[str, str, str]:
"""校验并返回样本 ID、作品 ID 和场景分类。"""
sample_id = sample.get("sampleId")
if not isinstance(sample_id, str) or not sample_id.strip():
raise ValueError(f"samples[{index}].sampleId 必须是非空字符串")
work_id = sample.get("workId")
if isinstance(work_id, bool) or not isinstance(work_id, (int, str)) or not str(work_id).strip():
raise ValueError(f"samples[{index}].workId 必须是非空整数或字符串")
scenario = sample.get("scenario")
if scenario not in REQUIRED_SCENARIOS:
raise ValueError(f"samples[{index}].scenario 未登记")
return sample_id, str(work_id), str(scenario)
def _sample_status(sample: Mapping[str, Any], index: int) -> dict[str, Any]:
"""从单样本机械结果推导有效性，不接受上游 valid 标记。"""
schema_valid = _require_boolean(sample.get("schemaValid"), f"samples[{index}].schemaValid")
future_leakage = _require_boolean(
sample.get("futureLeakage"), f"samples[{index}].futureLeakage"
)
system_failure = _require_boolean(
sample.get("systemFailure"), f"samples[{index}].systemFailure"
)
review_status = sample.get("reviewStatus")
if review_status not in REVIEW_STATUSES:
raise ValueError(
f"samples[{index}].reviewStatus 必须是 stable_report、"
"adjudicated_report 或 invalid_unstable"
)
return {
"schemaValid": schema_valid,
"futureLeakage": future_leakage,
"systemFailure": system_failure,
"reviewStatus": review_status,
"valid": (
schema_valid
and not future_leakage
and not system_failure
and review_status in STABLE_REVIEW_STATUSES
),
}
def _sample_c_arm(sample: Mapping[str, Any], index: int) -> dict[str, Any]:
"""读取单样本 C 臂机械门结果，聚合时不允许平均掩盖硬错误。"""
c_arm = _require_mapping(sample.get("cArm"), f"samples[{index}].cArm")
return {
"hardConstraintCoverage": _require_ratio(
c_arm.get("hardConstraintCoverage"),
f"samples[{index}].cArm.hardConstraintCoverage",
),
"highSeverityResidualCount": _require_integer(
c_arm.get("highSeverityResidualCount"),
f"samples[{index}].cArm.highSeverityResidualCount",
),
}
def _sample_scores(sample: Mapping[str, Any], index: int) -> dict[str, dict[str, float]]:
"""读取稳定样本的 A/B/C 三臂五维脱敏终分。"""
scores = _require_mapping(sample.get("scores"), f"samples[{index}].scores")
if set(scores) != {"A", "B", "C"}:
raise ValueError(f"samples[{index}].scores 必须精确包含 A/B/C")
result: dict[str, dict[str, float]] = {}
for arm in ("A", "B", "C"):
arm_scores = _require_mapping(scores[arm], f"samples[{index}].scores.{arm}")
if set(arm_scores) != set(DIMENSIONS):
raise ValueError(f"samples[{index}].scores.{arm} 必须精确包含正文五维")
result[arm] = {
dimension: _require_number(
arm_scores[dimension], f"samples[{index}].scores.{arm}.{dimension}"
)
for dimension in DIMENSIONS
}
return result
def _sample_confounds(sample: Mapping[str, Any], index: int) -> dict[str, list[Any]]:
"""校验单样本五类混淆项；顶层聚合混淆项不在信任边界内。"""
source = sample.get("confounders", {})
source = _require_mapping(source, f"samples[{index}].confounders")
unknown = sorted(set(source) - set(CONFOUND_LABELS))
if unknown:
raise ValueError(
f"samples[{index}].confounders 存在未知分类: {','.join(unknown)}"
)
result: dict[str, list[Any]] = {}
for category in CONFOUND_LABELS:
items = source.get(category, [])
if not isinstance(items, list):
raise ValueError(f"samples[{index}].confounders.{category} 必须是数组")
if any(not isinstance(item, (str, Mapping)) for item in items):
raise ValueError(
f"samples[{index}].confounders.{category} 只能包含字符串或对象"
)
result[category] = list(items)
return result
def _aggregate_confounds(samples: Sequence[Mapping[str, Any]]) -> dict[str, list[Any]]:
"""逐样本汇总五类混淆项，并补入可机械推导的泄露与不稳定记录。"""
aggregated = {category: [] for category in CONFOUND_LABELS}
for index, sample in enumerate(samples):
sample_id, _, _ = _sample_identity(sample, index)
status = _sample_status(sample, index)
for category, items in _sample_confounds(sample, index).items():
for item in items:
if isinstance(item, str):
aggregated[category].append({"sampleId": sample_id, "detail": item})
else:
aggregated[category].append({**dict(item), "sampleId": sample_id})
if status["futureLeakage"] and not any(
item.get("sampleId") == sample_id for item in aggregated["leakage"]
):
aggregated["leakage"].append(
{"sampleId": sample_id, "code": "future_leakage_detected"}
)
if status["reviewStatus"] == "invalid_unstable" and not any(
item.get("sampleId") == sample_id
for item in aggregated["reviewerInstability"]
):
aggregated["reviewerInstability"].append(
{"sampleId": sample_id, "code": "invalid_unstable"}
)
return aggregated
def _report(
gate: str,
status: str,
reasons: list[str],
confounds: dict[str, list[Any]],
metrics: Mapping[str, Any],
) -> dict[str, Any]:
"""组装稳定报告结构；reasons 顺序同时表达当前层内优先级。"""
return {
"schemaVersion": "writer-gate-report-v1",
"gate": gate,
"status": status,
"primaryReason": reasons[0],
"reasons": reasons,
"metrics": dict(metrics),
"confounds": confounds,
}
def _gate_a_metrics(samples: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
"""完全从逐样本结果计算 Gate A 统计。"""
statuses: list[dict[str, Any]] = []
c_arms: list[dict[str, Any]] = []
seen_ids: set[str] = set()
for index, sample in enumerate(samples):
sample_id, _, _ = _sample_identity(sample, index)
if sample_id in seen_ids:
raise ValueError(f"sampleId 重复: {sample_id}")
seen_ids.add(sample_id)
status = _sample_status(sample, index)
statuses.append(status)
# schema、泄漏或系统失败可能发生在 C 臂生成前，此时没有 cArm 是合法的
# 逐样本失败形态，不能让缺失的下游结果覆盖 Gate 的既定裁决顺序。
if (
status["schemaValid"]
and not status["futureLeakage"]
and not status["systemFailure"]
):
c_arms.append(_sample_c_arm(sample, index))
return {
"sampleCount": len(samples),
"validSampleCount": sum(item["valid"] for item in statuses),
"invalidSchemaCount": sum(not item["schemaValid"] for item in statuses),
"futureLeakageCount": sum(item["futureLeakage"] for item in statuses),
"systemFailureCount": sum(item["systemFailure"] for item in statuses),
"unstableSampleCount": sum(
item["reviewStatus"] == "invalid_unstable" for item in statuses
),
"cArmHighSeverityResidualCount": sum(
item["highSeverityResidualCount"] for item in c_arms
),
"cArmHardConstraintCoverage": min(
(item["hardConstraintCoverage"] for item in c_arms), default=1.0
),
}
def _short_circuit_confounds(gate_input: Mapping[str, Any]) -> dict[str, list[Any]]:
"""Gate A 已决定 Gate B 终态时，尽量保留合法逐样本混淆项。"""
if "samples" not in gate_input:
return {category: [] for category in CONFOUND_LABELS}
try:
return _aggregate_confounds(_require_samples(gate_input.get("samples")))
except ValueError:
# 混淆项属于报告信息，不能反向推翻更高优先级的 Gate A 终态。
return {category: [] for category in CONFOUND_LABELS}
def decide_gate_a(gate_input: Mapping[str, Any]) -> dict[str, Any]:
"""按“证据不足 -> 硬失败 -> 通过”唯一顺序裁决 Gate A。"""
gate_input = _require_mapping(gate_input, "Gate A 输入")
samples = _require_samples(gate_input.get("samples"))
normalized = _gate_a_metrics(samples)
confounds = _aggregate_confounds(samples)
if normalized["validSampleCount"] < 5:
return _report(
"A",
"insufficient_evidence",
["valid_sample_count_below_5"],
confounds,
normalized,
)
failure_reasons: list[str] = []
if normalized["invalidSchemaCount"] > 0:
failure_reasons.append("schema_invalid")
if normalized["futureLeakageCount"] > 0:
failure_reasons.append("future_leakage")
if normalized["systemFailureCount"] > 0:
failure_reasons.append("system_failure")
if normalized["cArmHighSeverityResidualCount"] > 0:
failure_reasons.append("c_arm_high_severity_residual")
if normalized["cArmHardConstraintCoverage"] < 1.0:
failure_reasons.append("c_arm_hard_constraint_coverage_below_100_percent")
if failure_reasons:
return _report("A", "failed", failure_reasons, confounds, normalized)
return _report(
"A", "passed", ["all_gate_a_conditions_met"], confounds, normalized
)
def _gate_b_metrics(samples: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
"""完全从逐样本结果计算 Gate B 充分性、退化和增益指标。"""
work_counts: dict[str, int] = {}
scenarios: set[str] = set()
statuses: list[dict[str, Any]] = []
c_arms: list[dict[str, Any]] = []
deltas: dict[str, list[dict[str, float]]] = {"B-A": [], "C-A": []}
seen_ids: set[str] = set()
for index, sample in enumerate(samples):
sample_id, work_id, scenario = _sample_identity(sample, index)
if sample_id in seen_ids:
raise ValueError(f"sampleId 重复: {sample_id}")
seen_ids.add(sample_id)
work_counts[work_id] = work_counts.get(work_id, 0) + 1
scenarios.add(scenario)
status = _sample_status(sample, index)
statuses.append(status)
c_arms.append(_sample_c_arm(sample, index))
if status["reviewStatus"] in STABLE_REVIEW_STATUSES:
scores = _sample_scores(sample, index)
for comparison, arm in (("B-A", "B"), ("C-A", "C")):
deltas[comparison].append(
{
dimension: scores[arm][dimension] - scores["A"][dimension]
for dimension in DIMENSIONS
}
)
total_count = len(samples)
stable_count = len(deltas["C-A"])
averages = {
comparison: {
dimension: (
sum(item[dimension] for item in comparison_deltas) / stable_count
if stable_count
else 0.0
)
for dimension in DIMENSIONS
}
for comparison, comparison_deltas in deltas.items()
}
# Gate B 的正式退化与增益阈值只消费 C-A；B-A 仅作为卡索引单线的诊断信息。
c_minus_a_deltas = deltas["C-A"]
decline_ratios = {
dimension: (
sum(item[dimension] < -0.5 for item in c_minus_a_deltas) / stable_count
if stable_count
else 0.0
)
for dimension in DIMENSIONS
}
any_decline_ratio = (
sum(
any(item[dimension] < -0.5 for dimension in DIMENSIONS)
for item in c_minus_a_deltas
)
/ stable_count
if stable_count
else 0.0
)
fidelity_positive_ratio = (
sum(
item["setting_entity_fidelity"] > 0 for item in c_minus_a_deltas
)
/ stable_count
if stable_count
else 0.0
)
return {
"workSampleCounts": dict(sorted(work_counts.items())),
"workCount": len(work_counts),
"totalSampleCount": total_count,
"stableSampleCount": stable_count,
"coveredScenarios": sorted(scenarios),
"unstableSampleRatio": sum(
item["reviewStatus"] == "invalid_unstable" for item in statuses
)
/ total_count,
"cArmHardConstraintCoverage": min(
(item["hardConstraintCoverage"] for item in c_arms), default=0.0
),
"cArmHighSeverityResidualCount": sum(
item["highSeverityResidualCount"] for item in c_arms
),
"averageDeltas": averages,
"bMinusAAverageDeltas": averages["B-A"],
"cMinusAAverageDeltas": averages["C-A"],
"dimensionDeclineOverHalfRatios": decline_ratios,
"anyDimensionDeclineOverHalfRatio": any_decline_ratio,
"fidelityPositiveSampleRatio": fidelity_positive_ratio,
}
def decide_gate_b(gate_input: Mapping[str, Any]) -> dict[str, Any]:
"""按 Gate A、证据、退化、增益四层唯一顺序裁决 Gate B。"""
gate_input = _require_mapping(gate_input, "Gate B 输入")
samples = _require_samples(gate_input.get("samples"))
# 顶层 gateAStatus 不在可信边界内；必须用 Gate B 的同批逐样本结果重算。
gate_a_report = decide_gate_a({"gate": "A", "samples": samples})
gate_a_status = gate_a_report["status"]
gate_a_metrics = {
"recomputedGateAStatus": gate_a_status,
"recomputedGateAMetrics": gate_a_report["metrics"],
}
if gate_a_status == "insufficient_evidence":
return _report(
"B",
"insufficient_evidence",
["gate_a_insufficient_evidence"],
gate_a_report["confounds"],
gate_a_metrics,
)
if gate_a_status == "failed":
return _report(
"B",
"failed",
["gate_a_failed"],
gate_a_report["confounds"],
gate_a_metrics,
)
normalized = _gate_b_metrics(samples)
normalized.update(gate_a_metrics)
confounds = _aggregate_confounds(samples)
evidence_reasons: list[str] = []
if normalized["workCount"] < 2:
evidence_reasons.append("work_count_below_2")
if any(count < 5 for count in normalized["workSampleCounts"].values()):
evidence_reasons.append("per_work_sample_count_below_5")
if normalized["totalSampleCount"] < 10:
evidence_reasons.append("total_sample_count_below_10")
if not REQUIRED_SCENARIOS.issubset(normalized["coveredScenarios"]):
evidence_reasons.append("scenario_coverage_incomplete")
if normalized["unstableSampleRatio"] > 0.20:
evidence_reasons.append("unstable_sample_ratio_above_20_percent")
if evidence_reasons:
return _report(
"B", "insufficient_evidence", evidence_reasons, confounds, normalized
)
quality_reasons: list[str] = []
if normalized["cArmHardConstraintCoverage"] < 1.0:
quality_reasons.append("c_arm_hard_constraint_coverage_below_100_percent")
if normalized["cArmHighSeverityResidualCount"] > 0:
quality_reasons.append("c_arm_high_severity_residual")
averages = normalized["averageDeltas"]["C-A"]
if averages["style_consistency"] < -0.25:
quality_reasons.append("style_consistency_average_delta_below_minus_0_25")
if averages["narrative_tension"] < -0.25:
quality_reasons.append("narrative_tension_average_delta_below_minus_0_25")
if normalized["anyDimensionDeclineOverHalfRatio"] > 0.20:
quality_reasons.append("dimension_decline_over_half_ratio_above_20_percent")
if quality_reasons:
return _report("B", "failed", quality_reasons, confounds, normalized)
if (
averages["setting_entity_fidelity"] >= 0.25
and normalized["fidelityPositiveSampleRatio"] >= 0.60
):
return _report(
"B", "passed", ["fidelity_gain_threshold_met"], confounds, normalized
)
return _report(
"B", "no_gain", ["fidelity_gain_threshold_not_met"], confounds, normalized
)
def decide_gate(gate_input: Mapping[str, Any]) -> dict[str, Any]:
"""从带 gate 字段的统一输入分派 Gate A 或 Gate B。"""
gate_input = _require_mapping(gate_input, "gate-input.json")
gate = gate_input.get("gate")
if gate == "A":
return decide_gate_a(gate_input)
if gate == "B":
return decide_gate_b(gate_input)
raise ValueError("gate 必须是 A 或 B")
def render_summary(report: Mapping[str, Any]) -> str:
"""渲染只含内部统计、终态和混淆项的 Markdown 摘要。"""
lines = [
f"# Writer Gate {report['gate']} 裁决摘要",
"",
f"- 终态：`{report['status']}`",
f"- 主原因：`{report['primaryReason']}`",
f"- 原因码：`{', '.join(report['reasons'])}`",
"",
"## 混淆项",
"",
]
confounds = _require_mapping(report.get("confounds"), "report.confounds")
for category, label in CONFOUND_LABELS.items():
lines.append(f"### {label}")
lines.append("")
items = confounds.get(category, [])
if not items:
lines.append("- 未观察到或未报告")
else:
for item in items:
rendered = (
item
if isinstance(item, str)
else json.dumps(item, ensure_ascii=False, sort_keys=True)
)
lines.append(f"- {rendered}")
lines.append("")
return "\n".join(lines)
def _parse_args() -> argparse.Namespace:
"""解析任务计划约定的运行目录与摘要输出参数。"""
parser = argparse.ArgumentParser(description="裁决正文智能体 Gate A/B")
parser.add_argument(
"--run-dir", type=Path, required=True, help="包含 gate-input.json 的回放运行目录"
)
parser.add_argument("--summary-output", type=Path, help="可选的脱敏 Markdown 摘要输出路径")
return parser.parse_args()
def main() -> int:
"""执行离线判定，并写出唯一 JSON 终态和可选摘要。"""
args = _parse_args()
input_path = args.run_dir / "gate-input.json"
output_path = args.run_dir / "gate-report.json"
try:
gate_input = json.loads(input_path.read_text(encoding="utf-8"))
report = decide_gate(gate_input)
output_path.write_text(
json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
if args.summary_output is not None:
args.summary_output.parent.mkdir(parents=True, exist_ok=True)
args.summary_output.write_text(render_summary(report), encoding="utf-8")
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"writer_gate: {error}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())

254
.claude/skills/replay-eval/scripts/writer_rubric.py Normal file
View File

@ -0,0 +1,254 @@
#!/usr/bin/env python3
"""正文回放五维量表与确定性稳定性仲裁。"""
from __future__ import annotations
from statistics import median
from typing import Any, Mapping, Sequence
RUBRIC_PROFILE = "writer_replay"
DIMENSIONS = (
"setting_entity_fidelity",
"fine_outline_fidelity",
"style_consistency",
"narrative_tension",
"prose_readability",
)
EVIDENCE_SOURCE_TYPES = frozenset(
{"fine_outline", "historical_prose", "judge_inference"}
)
REPORT_FIELDS = frozenset(
{"profile", "reviewerId", "sampleId", "blindCandidateId", "candidateOrder", "scores"}
)
def _is_half_step(value: Any) -> bool:
"""只接受 0 到 10 的数字和 0.5 步长，布尔值不算数字。"""
return (
not isinstance(value, bool)
and isinstance(value, (int, float))
and 0 <= float(value) <= 10
and float(value) * 2 == int(float(value) * 2)
)
def validate_scores(scores: Any) -> list[str]:
"""校验五维分数、步长和逐项证据归因。"""
if not isinstance(scores, Mapping):
return ["scores 必须是对象"]
errors: list[str] = []
missing = [dimension for dimension in DIMENSIONS if dimension not in scores]
unexpected = sorted(set(scores) - set(DIMENSIONS))
if missing:
errors.append(f"缺少正文 rubric 维度: {','.join(missing)}")
if unexpected:
errors.append(f"存在未登记正文 rubric 维度: {','.join(unexpected)}")
for dimension in DIMENSIONS:
item = scores.get(dimension)
if not isinstance(item, Mapping):
errors.append(f"维度必须包含 score/evidence 对象: {dimension}")
continue
if set(item) != {"score", "evidence"}:
errors.append(f"维度字段必须精确为 score/evidence: {dimension}")
if not _is_half_step(item.get("score")):
errors.append(f"分数必须在 0-10 且使用 0.5 步长: {dimension}")
evidence = item.get("evidence")
if not isinstance(evidence, list) or not evidence:
errors.append(f"分数缺少证据: {dimension}")
continue
for index, raw in enumerate(evidence):
if not isinstance(raw, Mapping):
errors.append(f"证据必须是对象: {dimension}[{index}]")
continue
if set(raw) != {"sourceType", "sourceRef", "excerpt"}:
errors.append(f"证据字段必须精确为 sourceType/sourceRef/excerpt: {dimension}[{index}]")
continue
if raw.get("sourceType") not in EVIDENCE_SOURCE_TYPES:
errors.append(f"证据来源类型未登记: {dimension}[{index}]")
for field in ("sourceRef", "excerpt"):
if not isinstance(raw.get(field), str) or not raw[field].strip():
errors.append(f"证据 {field} 不能为空: {dimension}[{index}]")
return errors
def validate_report(report: Any) -> list[str]:
"""校验单个盲评报告，真实臂名或额外字段一律失败关闭。"""
if not isinstance(report, Mapping):
return ["评委报告必须是对象"]
errors: list[str] = []
if set(report) != REPORT_FIELDS:
errors.append("评委报告字段非法，去盲前不得包含真实臂名或额外信息")
if report.get("profile") != RUBRIC_PROFILE:
errors.append(f"profile 必须是 {RUBRIC_PROFILE}")
for field in ("reviewerId", "sampleId", "blindCandidateId"):
if not isinstance(report.get(field), str) or not report[field].strip():
errors.append(f"{field} 必须是非空字符串")
order = report.get("candidateOrder")
if (
not isinstance(order, list)
or len(order) < 2
or any(not isinstance(item, str) or not item for item in order)
or len(set(order)) != len(order)
):
errors.append("candidateOrder 必须是无重复盲化候选 ID 数组")
elif report.get("blindCandidateId") not in order:
errors.append("blindCandidateId 不在本轮 candidateOrder 中")
errors.extend(validate_scores(report.get("scores")))
return errors
def validate_blind_pair(first: Any, second: Any) -> list[str]:
"""确认双评独立、同样本、同候选集合且第二评委严格反序。"""
errors = [*validate_report(first), *validate_report(second)]
if errors or not isinstance(first, Mapping) or not isinstance(second, Mapping):
return errors
if first["reviewerId"] == second["reviewerId"]:
errors.append("双评委必须使用独立 reviewerId")
if first["sampleId"] != second["sampleId"]:
errors.append("双评委必须评审同一样本")
if first["blindCandidateId"] != second["blindCandidateId"]:
errors.append("双评委必须评审同一盲化候选")
if set(first["candidateOrder"]) != set(second["candidateOrder"]):
errors.append("双评委的 candidateOrder 必须包含同一盲化候选集合")
if second["candidateOrder"] != list(reversed(first["candidateOrder"])):
errors.append("第二评委必须对相同盲化候选严格反序")
return errors
def deblind_reports(
reports: Sequence[Mapping[str, Any]], mapping: Mapping[str, str]
) -> list[dict[str, Any]]:
"""评分结束后按预注册映射去盲，绝不依赖展示位置推断真实臂。"""
if set(mapping.values()) != {"A", "B", "C"} or len(mapping) != 3:
raise ValueError("去盲映射必须将三个盲 ID 一一映射到 A/B/C")
result: list[dict[str, Any]] = []
for index, report in enumerate(reports):
errors = validate_report(report)
if errors:
raise ValueError(f"reports[{index}] 非法: {'; '.join(errors)}")
blind_id = str(report["blindCandidateId"])
if blind_id not in mapping:
raise ValueError(f"reports[{index}] 的 blindCandidateId 未预注册")
result.append({**dict(report), "arm": mapping[blind_id]})
return result
def _numeric_scores(report: Mapping[str, Any]) -> dict[str, float]:
"""从已校验报告提取确定性浮点分数。"""
return {
dimension: float(report["scores"][dimension]["score"])
for dimension in DIMENSIONS
}
def _stable_pair_exists(values: Sequence[float], threshold: float) -> bool:
"""判断三个评分中是否至少存在一对落在稳定阈值内。"""
return any(
abs(values[left] - values[right]) <= threshold
for left in range(len(values))
for right in range(left + 1, len(values))
)
def adjudicate_reviews(
first: Mapping[str, Any],
second: Mapping[str, Any],
third: Mapping[str, Any] | None = None,
*,
threshold: float = 0.5,
) -> dict[str, Any]:
"""按双评差异触发最多一次第三评委，并输出稳定终态。"""
pair_errors = validate_blind_pair(first, second)
if pair_errors:
return {"status": "invalid_report", "errors": pair_errors}
first_scores = _numeric_scores(first)
second_scores = _numeric_scores(second)
unstable = [
dimension
for dimension in DIMENSIONS
if abs(first_scores[dimension] - second_scores[dimension]) > threshold
]
if not unstable:
return {
"status": "stable_report",
"scores": {
dimension: (first_scores[dimension] + second_scores[dimension]) / 2
for dimension in DIMENSIONS
},
"unstableDimensions": [],
"reviewCount": 2,
}
if third is None:
return {
"status": "needs_third_reviewer",
"unstableDimensions": unstable,
"reviewCount": 2,
}
third_errors = validate_report(third)
if third_errors:
return {"status": "invalid_report", "errors": third_errors}
if third["sampleId"] != first["sampleId"]:
return {"status": "invalid_report", "errors": ["第三评委必须评审同一样本"]}
if third["blindCandidateId"] != first["blindCandidateId"]:
return {"status": "invalid_report", "errors": ["第三评委必须评审同一盲化候选"]}
allowed_third_orders = (
first["candidateOrder"],
list(reversed(first["candidateOrder"])),
)
if third["candidateOrder"] not in allowed_third_orders:
return {
"status": "invalid_report",
"errors": ["第三评委必须使用与双评相同的盲化候选集合及第一评或反序顺序"],
}
if third["reviewerId"] in {first["reviewerId"], second["reviewerId"]}:
return {"status": "invalid_report", "errors": ["第三评委必须使用独立 reviewerId"]}
third_scores = _numeric_scores(third)
unresolved = [
dimension
for dimension in unstable
if not _stable_pair_exists(
[first_scores[dimension], second_scores[dimension], third_scores[dimension]],
threshold,
)
]
if unresolved:
return {
"status": "invalid_unstable",
"unstableDimensions": unresolved,
"reviewCount": 3,
}
return {
"status": "adjudicated_report",
"scores": {
dimension: float(
median(
[first_scores[dimension], second_scores[dimension], third_scores[dimension]]
)
)
for dimension in DIMENSIONS
},
"unstableDimensions": unstable,
"reviewCount": 3,
}
__all__ = [
"RUBRIC_PROFILE",
"DIMENSIONS",
"EVIDENCE_SOURCE_TYPES",
"validate_scores",
"validate_report",
"validate_blind_pair",
"deblind_reports",
"adjudicate_reviews",
]

35
docs/2026-07-20-正文智能体正式优化设计与计划.md
View File

@ -197,6 +197,8 @@ writer 可以创造非 Canonical 的环境细节、动作、过渡、无名配
| `narrativeState` | object | 是 | 时间、地点、角色位置、即时局面 |
| `factEvidence[]` | array | 是 | 事实 ID、来源类型、sourceRef、hash |
| `proseEvidence[]` | array | 是 | 章号、offset、hash、用途、临时片段 |
| `indexHints[]` | array | 否 | 仅诊断上下文可用的冻结卡提示；严格字段为 `cardId/name/type/content/sourceId/sourceVersion/asOf` |
| `evidenceStrategy` | enum | 否 | 生产缺省视为 `production_dual_evidence`；正文 A/B/C 回放必须显式记录各臂证据策略 |
| `patternReferences[]` | array | 否 | 已授权范式卡，只作结构方法参考 |
| `evidenceCoverage[]` | array | 是 | 细纲要素到证据的覆盖状态 |
| `outputContract` | object | 是 | 篇幅、场景、frontmatter、申报规则 |
@ -204,6 +206,8 @@ writer 可以创造非 Canonical 的环境细节、动作、过渡、无名配
所有对象使用严格 schema，额外字段失败；引用的 ID、版本和 hash 必须存在且一致。
`evidenceStrategy` 是兼容字段：生产上下文未填写时按 `production_dual_evidence` 校验；正文 A/B/C 回放不得使用缺省值，必须分别显式填写 `historical_prose_only`、`card_index_only`、`card_index_plus_prose`。`indexHints` 只能在 `mode=diagnostic_only`、`purpose=evaluation/diagnostic` 且 `acceptanceEligible=false` 时出现；生产上下文硬拒绝。其 `asOf` 不得超过上下文冻结点，`claimLedger` 不得引用 `indexHints`。只有显式填写 `evidenceStrategy=card_index_only` 的 B 臂允许在合同内跳过连续四章基线，并且必须保持 `proseEvidence=[]`；该例外不能用于生产。
### 7.2 `WriterOutput v1`
```text
@ -269,6 +273,7 @@ detector 只阻断可定位、可验证的问题：细纲硬约束漏项、事
- 离线优化允许最多 5 轮，且每轮只改一个变量；与生产两轮返修是两套状态机。
- 三臂使用相同细纲、冻结点、模型、篇幅算法和最大生成预算。
- 候选臂名映射为随机化 ID；顺序种子由预注册 `evaluationSetVersion + sampleId` 派生并固定。
- judge 的可信输入边界是独立 `writer-blind-input-v1` 内容，包含 `blind-1/2/3` 候选正文/哈希，以及所有臂完全相同的 `sharedEvaluationReference`。共同参考只取目标章细纲硬约束、实体、必需角色、伏笔、章末钩子和冻结点前连续四章历史原文基准，用于评价细纲忠实、设定、文风和卡索引正确性；它可在真实运行时临时传递，但不进入安全摘要。共同参考严禁包含 `indexHints`、各臂补充原文、卡 manifest、`evidenceStrategy`、真实映射或臂名，不能用被测卡本身给被测候选背书。真实 A/B/C 映射仅在编排器内存中存在，judge 不得访问各臂 WriterContext、原始运行目录或含 `candidate-A/B/C` 的路径。
- 两个评委使用独立无会话实例，第二评委反转顺序；评分步长为 0.5。
- 同维分差 > 0.5 时判不稳定，最多增加一次第三评委。三评分中若至少一对差值 <=0.5，则该维最终分取三者中位数；若不存在稳定配对，则样本进入 `invalid_unstable`，不参与方向结论。
- 五维：设定与实体保真、情节与细纲忠实、叙事完整与张力、文风一致、文笔质量。
@ -288,6 +293,36 @@ detector 只阻断可定位、可验证的问题：细纲硬约束漏项、事
只有 Gate B=`passed` 才解锁细纲智能体真实能力验收；`no_gain` 只形成“卡未证明增益”结论，不视为通过。
### 10.2 Task10 Gate A 预注册与当前可验证边界
唯一配置为 `.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json`。样本固定如下，不得根据生成结果替换章节或场景分类：
| 目标章 | 冻结点 | 场景 | 预注册目标 | 大纲/细纲来源 | 预期长度 | 主要实体 | 新角色比例 | 泄漏检查 |
|---:|---:|---|---|---|---:|---|---:|---|
| 489 | 488 | 战斗 | 圣蒂曼围攻/加特朗战 | `outline-window:deep-space:481-488` / `fine-outline:deep-space:489` | 7500 | 圣蒂曼、加特朗 | 1.0 | 禁读 489 及以后；目标/未来读取数必须为 0 |
| 321 | 320 | 人物对话 | 莫妮卡道别与朋友确认 | `outline-window:deep-space:313-320` / `fine-outline:deep-space:321` | 7600 | 莫妮卡、朋友关系 | 0 | 禁读 321 及以后；目标/未来读取数必须为 0 |
| 544 | 543 | 转折 | 迷途之地内应反水 | `outline-window:deep-space:536-543` / `fine-outline:deep-space:544` | 6700 | 迷途之地、内应 | 未知 | 禁读 544 及以后；目标/未来读取数必须为 0 |
| 199 | 198 | 信息揭示 | 赛莉丝身份与联赛锁死真相 | `outline-window:deep-space:191-198` / `fine-outline:deep-space:199` | 2000 | 赛莉丝、联赛 | 0 | 禁读 199 及以后；目标/未来读取数必须为 0 |
| 523 | 522 | 老角色回归 | 伊蕾莉雅与旧部重逢 | `outline-window:deep-space:515-522` / `fine-outline:deep-space:523` | 6100 | 伊蕾莉雅、旧部 | 0 | 禁读 523 及以后；目标/未来读取数必须为 0 |
预期长度不是人工填写：配置记录目标章之前连续四章的 `frozenRecentHanCounts`，统一以 `hardEventCount=1`、`foreshadowingActionCount=0`、`requiredSceneCount=0` 调用 `calculate_target_chars`，且 `usesTargetChapterLength=false`。五章机械结果固定为 489=7500、321=7600、544=6700、199=2000、523=6100；上下限取正负 10% 后再受 2000-10000 限幅。
新角色比例只统计具名 `requiredCharacters` 在冻结点前无记录的比例。489 的加特朗在 488 前无记录，因此为 1.0；321/199/523 的具名角色已有记录，因此为 0；544 的“内应”是泛称、无法机械判定具体身份，必须为 `null + unresolved_generic_role`，不允许默认成 0。
仓内配置不含原文全文。`writerContextInput.contentMode=sanitized_contract_fixture`，连续四章只放脱敏合成短文本，用于 dry-run 机械证明 WriterContext 合同、冻结边界、三臂差异、manifest 可复现和候选不可接受；不能据此声称历史原文完整、生成质量通过或 real-run 完成。
dry-run 命令：
```bash
.venv/bin/python .claude/skills/replay-eval/scripts/run_writer_replay.py \
--config .claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json \
--dry-run
```
dry-run 只生成计划、manifest 和上下文摘要，不调用模型。测试注入路径虽可验证 writer、detector 与盲评编排，但 judge 只接收独立盲化内容和共同评测参考，不能取得 raw 目录、真实臂映射或任一臂专属证据；评委顺序变化不得改变共同参考。当前真实 semantic detector 与 judge adapter 尚未实现，CLI `--execute` 必须明确失败关闭；在 adapter 接线、离线测试和预算确认完成前，不得声称真实回放或 Gate A 已完成。
Gate 判定只消费逐样本脱敏输入。`writer_gate.py` 在可信边界内自行计算 Gate A/B 的有效样本、作品/场景覆盖、C 臂硬门、C-A 五维增量、退化比例与五类混淆项；不信任上游聚合数字。五类混淆项为假阴、假阳、泄露、评委不稳定和新角色无卡，必须逐样本记录后汇总。
## 11. 用户闭环与下游交接
Shadow 候选展示后提供三决策：