diff --git a/.claude/agents/judge.md b/.claude/agents/judge.md index 19eebe2..0a87e15 100644 --- a/.claude/agents/judge.md +++ b/.claude/agents/judge.md @@ -5,7 +5,7 @@ tools: Read, Grep, Glob, Write model: opus --- -你是质量评委,保护节点角色(不可被装配替换)。**功能合同=`quality-gate` skill**——正文任务使用 `quality_gate` profile；细纲回放任务使用 `fine_outline_replay` profile。维度、证据要求和稳定性口径全在那里,你只执行不自造维度。报告按当前运行合同落临时评测目录或 `works/<书>/评审/第NNN章-评分.md`,除此不写正式创作文件。 +你是质量评委,保护节点角色(不可被装配替换)。**功能合同=`quality-gate` skill**——在线正文质量详情使用 `quality_gate` profile；正文离线回放使用独立 `writer_replay` profile；细纲回放使用 `fine_outline_replay` profile。维度、证据要求和稳定性口径全在那里,你只执行不自造维度。报告按当前运行合同落临时评测目录或 `works/<书>/评审/第NNN章-评分.md`,除此不写正式创作文件。 ## 元数据纪律(怎么用元数据) @@ -13,11 +13,13 @@ model: opus - 上下文=writer 基线包(**同证独立**):评的是"在写手所知条件下写得好不好",不索取额外资料,不拿包外信息扣分。 - 细纲回放只使用冻结快照和结构化 `reference scaffold proxy`;不得读取目标章全文或完整目标章细纲。 - 细纲回放每个分数必须有 `evidence` 字段；不得使用正文文风、文笔或可读性作为评分维度。 +- 正文回放只接收盲化候选 ID，不得在报告中输出 A/B/C、真实臂名、文件名或提示词差异。每维证据必须标为 `fine_outline`、`historical_prose` 或 `judge_inference`；盲评输入不含卡，禁止伪造 `card_index` 归因。卡效用只由去盲后的 B-A/C-A 差分判断。 ## 打分纪律 - 每维给分必附一句引文证据(好在哪/差在哪,引原句);没有证据的分数无效。 - 同一维度复评同一章分差应 ≤0.5;严格度不因收敛压力改变,不放水不加戏。 +- 正文回放使用 0-10 分、0.5 步长。双评同维差异大于 0.5 时只请求一次第三评委；第三评后仍无任意稳定配对时标记 `invalid_unstable`，不强行给臂输赢。 - 末尾「最值得改的三点」按提升空间排序:问题→根因层猜测(prompt/上下文/设定卡)→具体改法。 - 细纲回放时，把末尾建议替换为“最值得补齐的三项结构缺口”，并标注它属于候选结构、公共大纲、卡注入、原文检索还是标准事实不确定；若两次同维分差大于 0.5，只写稳定性警告，不强行裁决。 diff --git a/.claude/skills/eval/SKILL.md b/.claude/skills/eval/SKILL.md index 829cdba..0e99785 100644 --- a/.claude/skills/eval/SKILL.md +++ b/.claude/skills/eval/SKILL.md @@ -26,3 +26,9 @@ description: 质量收敛环——judge 打分、锁最低维、单变量改进( ## 红线 评分必须有引文证据;禁止为了分数让 judge 放水(rubric 与严格度不因收敛压力改变);同一 rubric 复评分差 >0.5 时先查评委稳定性再查正文。 + +## 正文离线回放 + +正文 A/B/C 回放使用独立 `writer_replay` profile，不复用本 skill 的在线 1-5 分收敛口径，也不覆盖 `fine_outline_replay`。五维均为 0-10 分、0.5 步长：`setting_entity_fidelity`、`fine_outline_fidelity`、`style_consistency`、`narrative_tension`、`prose_readability`。 + +两名独立评委只看盲化候选和所有臂一致的共同参考，第二名严格反序。同维分差大于 0.5 才增加一次第三评；三评分至少一对差值不大于 0.5 时取中位数，否则样本为 `invalid_unstable`，不进入方向结论。每个维度都必须把证据归因为细纲、历史原文或评委自行推断。盲评输入不含卡，卡效用由去盲后的 B-A/C-A 差分识别。 diff --git a/.claude/skills/quality-gate/SKILL.md b/.claude/skills/quality-gate/SKILL.md index a979f68..61906fb 100644 --- a/.claude/skills/quality-gate/SKILL.md +++ b/.claude/skills/quality-gate/SKILL.md @@ -1,6 +1,6 @@ --- name: quality-gate -description: 质量评分的功能合同(scenario: quality_gate/fine_outline_replay,保护节点)。正文质量维度和细纲回放 rubric 分开管理;rubric 与达标口径的单一来源。 +description: 质量评分的功能合同(scenario: quality_gate/writer_replay/fine_outline_replay,保护节点)。在线正文、正文回放和细纲回放 rubric 分开管理;rubric 与达标口径的单一来源。 disable-model-invocation: true --- @@ -58,3 +58,17 @@ scores: ``` 两次独立评审的同维差异大于 `0.5` 时只输出稳定性警告，不据此下卡效用结论。三臂顺序、arm 名称和卡 manifest 对评委隐藏；汇总报告分别给出每维分数、结构门结果、`B-A` 和 `C-A`，不压成没有统计意义的单一“卡质量总分”。 + +## 正文回放 profile（`writer_replay`） + +正文回放与在线 `quality_gate`、细纲 `fine_outline_replay` 完全独立。每维 0-10 分、0.5 步长；双评委独立无会话，第二评委严格反序。报告在去盲前只保留随机候选 ID，禁止出现 A/B/C 或证据策略名称。 + +| 维度 ID | 评分问题 | +|---|---| +| `setting_entity_fidelity` | 设定、实体、关系、能力边界和冻结时点状态是否保真 | +| `fine_outline_fidelity` | 细纲硬事件、结果方向、伏笔动作、出场实体和章末钩子是否忠实 | +| `style_consistency` | 与冻结历史原文的人物声音、动作习惯、句式和叙事质感是否一致 | +| `narrative_tension` | 场景推进、冲突升级、不可逆变化和章末驱动力是否成立 | +| `prose_readability` | 语言是否准确、自然、清晰且没有明显阅读阻力 | + +每维证据数组中的 `sourceType` 只能是 `fine_outline`、`historical_prose` 或 `judge_inference`；评委自行推断必须显式标注。盲评输入不含卡，禁止评委声称证据来自 `card_index`，卡效用只由去盲后的 B-A/C-A 差分归因。双评同维分差大于 0.5 时启动且只启动一次第三评委。第三评后，任一评分对差值不大于 0.5 则取三者中位数；否则整章为 `invalid_unstable`，不得强行计算胜负或进入 Gate 增益统计。 diff --git a/.claude/skills/replay-eval/SKILL.md b/.claude/skills/replay-eval/SKILL.md index 707e634..5a4ae37 100644 --- a/.claude/skills/replay-eval/SKILL.md +++ b/.claude/skills/replay-eval/SKILL.md @@ -4,9 +4,9 @@ description: 回放评测的冻结与结果边界合同。把参考作品冻结 disable-model-invocation: true --- -# 回放评测（`next_fine_outline_replay_v0`） +# 回放评测 -本 skill 只负责确定性的评测编排边界，不调用模型、不替代统一读取器，也不写正式规划或知识。评测目的、三臂定义和细纲评分合同见 `docs/2026-07-19-回放评测-细纲首跑设计与计划.md`。 +本 skill 只负责确定性的评测编排边界，不调用模型、不替代统一读取器，也不写正式规划或知识。细纲回放见 `docs/2026-07-19-回放评测-细纲首跑设计与计划.md`；正文回放的唯一任务 SoT 是 `docs/2026-07-20-正文智能体正式优化设计与计划.md`。 真实参考作品配置由 `scripts/load_reference_work.py` 从实验库只读组装；它只取作品元数据、窗级大纲、章级细纲摘要和预注册卡 ID 对应的候选卡历史。候选卡必须标记为 `eval_draft`，不能当作生产知识检索结果。 @@ -50,3 +50,39 @@ disable-model-invocation: true - `scripts/write_report.py`：从 `run_result.json` 生成安全摘要；它不会读取候选正文，也不会把候选路径以外的原始响应写入报告。 真实作品运行前必须先从权威来源取得不可变授权快照。数据库没有该字段时，使用 dry-run 证明机制并保持 `blocked_authorization`，不得用本地配置或口头许可伪造放行。 + +## 正文 A/B/C 回放 + +正文 Gate A 固定配置为 `configs/writer-gate-a-deep-space-v1.json`，预注册深空之影五章：489 战斗、321 人物对话、544 转折、199 信息揭示、523 老角色回归。不得根据候选结果换章、换场景分类或修改冻结点；每个冻结点必须等于 `targetChapter-1`。 + +每个样本必须提供 `writerContextInput`。仓内配置只允许 `contentMode=sanitized_contract_fixture`，`recentChapters` 只能放连续四章的脱敏合成短文本，用于证明 WriterContext 合同、章号冻结和 A/B/C 差异策略；它不是历史原文，不证明真实正文完整性或文学质量。配置可以记录细纲硬约束、主要实体、预期篇幅、新角色比例和泄漏哨兵，但不得包含原书全文、完整目标细纲或标准答案。 + +三臂都必须构造并校验 `WriterContext v1`，且固定 `mode=diagnostic_only`、`purpose=evaluation`、`acceptanceEligible=false`： + +- A：`evidenceStrategy=historical_prose_only`，保留连续四章脱敏基线，不放 `indexHints`。 +- B：`evidenceStrategy=card_index_only`，只放 `retrievalResult.indexHints`，`proseEvidence` 为空；这是合同内可审计的诊断基线例外，不是生产旁路。 +- C：`evidenceStrategy=card_index_plus_prose`，保留连续四章脱敏基线，并放冻结 `indexHints` 与配置中的补充原文证据。 + +`indexHints` 每项只能包含 `cardId/name/type/content/sourceId/sourceVersion/asOf`，只能用于 `diagnostic_only` 的 `evaluation/diagnostic`，不得进入生产上下文；`claimLedger` 不得引用它。所有 `asOf` 和来源章号不得超过样本冻结点。 + +每个样本必须记录 `frozenRecentHanCounts` 和 `targetLengthBasis`。篇幅只能使用冻结点前连续四章汉字数，经 `calculate_target_chars` 复算；当前 Gate A 固定 `hardEventCount=1`、`foreshadowingActionCount=0`、`requiredSceneCount=0`，禁止读取目标章实际长度。`expectedLength` 与 WriterContext 的输出合同必须等于机械目标的正负 10%，再受 2000-10000 边界限制。 + +`newCharacterRatio` 定义为：具名 `requiredCharacters` 中，在 `asOfChapter` 前无记录的角色比例。泛称角色不参与猜测；例如第 544 章的“内应”无法确定具体身份，必须记录 `newCharacterRatio=null` 和 `newCharacterRatioStatus=unresolved_generic_role`，不得默认成 0。 + +盲评使用独立的 `writer-blind-input-v1` 内容边界。judge 只能看到 `blind-1/2/3` 候选正文/哈希，以及所有臂完全相同的 `sharedEvaluationReference`：目标章细纲硬约束、实体、必需角色、伏笔、章末钩子和冻结点前连续四章历史原文基准。共同参考可以在真实运行时临时传给 judge，但不能进入安全摘要。它不得包含 `indexHints`、各臂补充原文、卡 manifest、`evidenceStrategy`、真实 A/B/C 映射、各臂 WriterContext、`raw` 目录或任何包含 `candidate-A/B/C` 的路径；卡的正确性只能由共同历史原文基准验证，不能把被测卡本身当裁判。映射只保留在编排器内存中，评分完成后才去盲。 + +正文 dry-run 命令： + +```bash +.venv/bin/python .claude/skills/replay-eval/scripts/run_writer_replay.py \ + --config .claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json \ + --dry-run +``` + +dry-run 只输出计划、manifest 和上下文摘要，不调用 writer、semantic detector 或 judge，不生成候选正文，也不得声称真实回放完成。当前 CLI 的 `--execute` 因真实 semantic detector 和 judge adapter 尚未接线而失败关闭；只有实现并验证这两个受控 adapter 后，才可在预算确认后启用真实模型回放。 + +## 正文 Gate 输入 + +`writer_gate.py` 只信任 `gate-input.json.samples[]` 的逐样本脱敏结果。有效样本数、作品数、场景覆盖、C 臂硬门统计、C-A 五维增量、退化比例和五类混淆项全部由判定器内部计算；顶层传入的同名聚合值和聚合 `confounds` 不参与裁决。五类混淆项是：假阴、假阳、泄露、评委不稳定、新角色无卡。 + +真实回放未接线期间不能生成合法 Gate 输入，也不能运行 Gate A/B 得出通过结论。Gate A/B 的终态只能由 `writer_gate.py` 按任务 SoT 的唯一顺序产生，人工不得改写。 diff --git a/.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json b/.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json new file mode 100644 index 0000000..7f225a1 --- /dev/null +++ b/.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json @@ -0,0 +1,1196 @@ +{ + "profile": "writer_replay", + "evaluationSetVersion": "writer-gate-a-deep-space-v1", + "strategyVersion": "writer-abc-context-v1", + "referenceWork": { + "id": 8, + "title": "深空之影", + "version": "db-work-8-gate-a-preregistered-v1" + }, + "authorization": { + "sourceStatus": "authorized", + "copyrightStatus": "owned", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "allowedPurpose": [ + "offline_evaluation" + ], + "authorizationSnapshot": { + "id": "auth-work-8-gate-a-v1", + "version": "auth-work-8-gate-a-v1", + "immutable": true, + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "sourceStatus": "authorized", + "allowedPurpose": [ + "offline_evaluation" + ], + "checkedAt": "2026-07-20T00:00:00Z", + "revalidationAt": "2026-08-19T00:00:00Z" + } + }, + "commonControls": { + "workId": 8, + "modelVersion": "writer-model-unwired", + "sampling": { + "temperature": 0, + "topP": 1, + "seed": 20260720 + }, + "detectorProfile": "writer-detector-report-v1" + }, + "samples": [ + { + "sampleId": "deep-space-489-battle", + "targetTitle": "圣蒂曼围攻/加特朗战", + "targetDescription": "第489章固定为圣蒂曼围攻中的加特朗战斗场景。", + "workId": 8, + "scenario": "battle", + "asOfChapter": 488, + "targetChapter": 489, + "snapshotVersion": "writer-deep-space-489-v1", + "outlineSource": "outline-window:deep-space:481-488", + "fineOutlineSource": "fine-outline:deep-space:489", + "frozenRecentHanCounts": [8915, 8582, 8716, 9114], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [485, 486, 487, 488], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": false + }, + "targetChars": 7500, + "expectedLength": { + "targetChars": 7500, + "minChars": 6750, + "maxChars": 8250 + }, + "mainEntities": [ + "圣蒂曼", + "加特朗" + ], + "newCharacterRatio": 1.0, + "newCharacterRatioStatus": "resolved", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 488, + "requiredCharacters": ["加特朗"], + "knownBeforeAsOf": [], + "absentBeforeAsOf": ["加特朗"], + "genericRoles": [] + }, + "leakageCheck": { + "method": "chapter-bound-and-target-fact-sentinel", + "forbiddenReadFromChapter": 489, + "expectedTargetOrFutureReadCount": 0 + }, + "sources": [ + { + "sourceId": "fixture-chapters:485-488", + "sourceVersion": "sanitized-fixture-v1", + "chapterRange": "485-488" + }, + { + "sourceId": "fixture-card-index:488", + "sourceVersion": "fixture-card-v1", + "chapterRange": "1-488" + } + ], + "snapshotData": { + "chapters": [ + { + "chapter": 488, + "sourceId": "fixture-chapter:488", + "contentSha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + } + ], + "cards": [] + }, + "leakageAudit": { + "method": "chapter-bound-and-target-fact-sentinel", + "targetFacts": { + "targetChapter": 489, + "forbiddenFacts": [ + { + "id": "target-489-sentinel", + "firstChapter": 489, + "text": "TARGET_SENTINEL_489" + } + ] + } + }, + "writerContextInput": { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "authorizationSnapshot": { + "snapshotId": "auth-work-8-gate-a-v1", + "allowedPurpose": "evaluation", + "verifiedAt": "2026-07-20T00:00:00Z" + }, + "sourceStatus": "active", + "retrievalResult": { + "cards": [], + "factEvidence": [], + "proseEvidence": [], + "indexHints": [ + { + "cardId": "fixture-card-489-1", + "name": "圣蒂曼", + "type": "location", + "content": "截至第488章的脱敏索引提示", + "sourceId": "fixture-card-source:489-1", + "sourceVersion": "fixture-card-v1", + "asOf": 488 + } + ], + "manifest": { + "omittedSources": [] + } + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fine-outline:deep-space:489", + "sourceVersion": "fine-outline-preregistered-v1", + "chapter": 489 + }, + "hardConstraints": [ + "完成圣蒂曼围攻中的加特朗战斗主事件" + ], + "adjustableBeats": [ + "战斗动作与局部节奏允许调整" + ], + "declaredNewFacts": [], + "entities": [ + { + "id": "entity-489-location", + "type": "location", + "name": "圣蒂曼" + }, + { + "id": "entity-489-character", + "type": "character", + "name": "加特朗" + } + ] + }, + "narrativeState": { + "time": "第488章结束后", + "location": "圣蒂曼", + "characterPositions": {}, + "immediateSituation": "围攻战斗待续" + }, + "recentChapters": [ + { + "chapter": 485, + "sourceRef": { + "sourceId": "fixture-chapter:485", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第485章冻结位置。" + }, + { + "chapter": 486, + "sourceRef": { + "sourceId": "fixture-chapter:486", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第486章冻结位置。" + }, + { + "chapter": 487, + "sourceRef": { + "sourceId": "fixture-chapter:487", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第487章冻结位置。" + }, + { + "chapter": 488, + "sourceRef": { + "sourceId": "fixture-chapter:488", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第488章冻结位置。" + } + ], + "outputContract": { + "targetChars": 7500, + "minChars": 6750, + "maxChars": 8250, + "frontmatterRequired": false, + "newSettingDeclarationRequired": true + }, + "tokenBudget": { + "maxContextChars": 50000 + }, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + { + "requirementId": "event-489-1", + "anchors": [ + "圣蒂曼", + "加特朗" + ] + } + ], + "requiredCharacters": [ + "加特朗" + ], + "foreshadowingActions": [], + "chapterEndHook": { + "requirementId": "hook-489", + "anchors": [ + "战局" + ], + "maxDistanceFromEnd": 120 + }, + "detectedNewSettings": [], + "frozenConflicts": [] + } + } + }, + { + "sampleId": "deep-space-321-character-dialogue", + "targetTitle": "莫妮卡道别与朋友确认", + "targetDescription": "第321章固定为莫妮卡道别并确认朋友关系的人物对话场景。", + "workId": 8, + "scenario": "character_dialogue", + "asOfChapter": 320, + "targetChapter": 321, + "snapshotVersion": "writer-deep-space-321-v1", + "outlineSource": "outline-window:deep-space:313-320", + "fineOutlineSource": "fine-outline:deep-space:321", + "frozenRecentHanCounts": [9013, 8950, 9475, 8731], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [317, 318, 319, 320], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": false + }, + "targetChars": 7600, + "expectedLength": { + "targetChars": 7600, + "minChars": 6840, + "maxChars": 8360 + }, + "mainEntities": [ + "莫妮卡", + "朋友关系" + ], + "newCharacterRatio": 0.0, + "newCharacterRatioStatus": "resolved", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 320, + "requiredCharacters": ["莫妮卡"], + "knownBeforeAsOf": ["莫妮卡"], + "absentBeforeAsOf": [], + "genericRoles": [] + }, + "leakageCheck": { + "method": "chapter-bound-and-target-fact-sentinel", + "forbiddenReadFromChapter": 321, + "expectedTargetOrFutureReadCount": 0 + }, + "sources": [ + { + "sourceId": "fixture-chapters:317-320", + "sourceVersion": "sanitized-fixture-v1", + "chapterRange": "317-320" + }, + { + "sourceId": "fixture-card-index:320", + "sourceVersion": "fixture-card-v1", + "chapterRange": "1-320" + } + ], + "snapshotData": { + "chapters": [ + { + "chapter": 320, + "sourceId": "fixture-chapter:320", + "contentSha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + } + ], + "cards": [] + }, + "leakageAudit": { + "method": "chapter-bound-and-target-fact-sentinel", + "targetFacts": { + "targetChapter": 321, + "forbiddenFacts": [ + { + "id": "target-321-sentinel", + "firstChapter": 321, + "text": "TARGET_SENTINEL_321" + } + ] + } + }, + "writerContextInput": { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "authorizationSnapshot": { + "snapshotId": "auth-work-8-gate-a-v1", + "allowedPurpose": "evaluation", + "verifiedAt": "2026-07-20T00:00:00Z" + }, + "sourceStatus": "active", + "retrievalResult": { + "cards": [], + "factEvidence": [], + "proseEvidence": [], + "indexHints": [ + { + "cardId": "fixture-card-321-1", + "name": "莫妮卡", + "type": "character", + "content": "截至第320章的脱敏索引提示", + "sourceId": "fixture-card-source:321-1", + "sourceVersion": "fixture-card-v1", + "asOf": 320 + } + ], + "manifest": { + "omittedSources": [] + } + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fine-outline:deep-space:321", + "sourceVersion": "fine-outline-preregistered-v1", + "chapter": 321 + }, + "hardConstraints": [ + "完成莫妮卡道别与朋友关系确认" + ], + "adjustableBeats": [ + "对白顺序与停顿节奏允许调整" + ], + "declaredNewFacts": [], + "entities": [ + { + "id": "entity-321-character", + "type": "character", + "name": "莫妮卡" + } + ], + "relations": [ + { + "id": "relation-321-friend", + "type": "character_relation", + "name": "朋友关系" + } + ] + }, + "narrativeState": { + "time": "第320章结束后", + "location": "道别场景", + "characterPositions": {}, + "immediateSituation": "关系确认待发生" + }, + "recentChapters": [ + { + "chapter": 317, + "sourceRef": { + "sourceId": "fixture-chapter:317", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第317章冻结位置。" + }, + { + "chapter": 318, + "sourceRef": { + "sourceId": "fixture-chapter:318", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第318章冻结位置。" + }, + { + "chapter": 319, + "sourceRef": { + "sourceId": "fixture-chapter:319", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第319章冻结位置。" + }, + { + "chapter": 320, + "sourceRef": { + "sourceId": "fixture-chapter:320", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第320章冻结位置。" + } + ], + "outputContract": { + "targetChars": 7600, + "minChars": 6840, + "maxChars": 8360, + "frontmatterRequired": false, + "newSettingDeclarationRequired": true + }, + "tokenBudget": { + "maxContextChars": 50000 + }, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + { + "requirementId": "event-321-1", + "anchors": [ + "道别", + "朋友" + ] + } + ], + "requiredCharacters": [ + "莫妮卡" + ], + "foreshadowingActions": [], + "chapterEndHook": { + "requirementId": "hook-321", + "anchors": [ + "离开" + ], + "maxDistanceFromEnd": 120 + }, + "detectedNewSettings": [], + "frozenConflicts": [] + } + } + }, + { + "sampleId": "deep-space-544-turning-point", + "targetTitle": "迷途之地内应反水", + "targetDescription": "第544章固定为迷途之地内应反水的情节转折场景。", + "workId": 8, + "scenario": "turning_point", + "asOfChapter": 543, + "targetChapter": 544, + "snapshotVersion": "writer-deep-space-544-v1", + "outlineSource": "outline-window:deep-space:536-543", + "fineOutlineSource": "fine-outline:deep-space:544", + "frozenRecentHanCounts": [6931, 8011, 7785, 8827], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [540, 541, 542, 543], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": false + }, + "targetChars": 6700, + "expectedLength": { + "targetChars": 6700, + "minChars": 6030, + "maxChars": 7370 + }, + "mainEntities": [ + "迷途之地", + "内应" + ], + "newCharacterRatio": null, + "newCharacterRatioStatus": "unresolved_generic_role", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 543, + "requiredCharacters": ["内应"], + "knownBeforeAsOf": [], + "absentBeforeAsOf": [], + "genericRoles": ["内应"] + }, + "leakageCheck": { + "method": "chapter-bound-and-target-fact-sentinel", + "forbiddenReadFromChapter": 544, + "expectedTargetOrFutureReadCount": 0 + }, + "sources": [ + { + "sourceId": "fixture-chapters:540-543", + "sourceVersion": "sanitized-fixture-v1", + "chapterRange": "540-543" + }, + { + "sourceId": "fixture-card-index:543", + "sourceVersion": "fixture-card-v1", + "chapterRange": "1-543" + } + ], + "snapshotData": { + "chapters": [ + { + "chapter": 543, + "sourceId": "fixture-chapter:543", + "contentSha256": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" + } + ], + "cards": [] + }, + "leakageAudit": { + "method": "chapter-bound-and-target-fact-sentinel", + "targetFacts": { + "targetChapter": 544, + "forbiddenFacts": [ + { + "id": "target-544-sentinel", + "firstChapter": 544, + "text": "TARGET_SENTINEL_544" + } + ] + } + }, + "writerContextInput": { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "authorizationSnapshot": { + "snapshotId": "auth-work-8-gate-a-v1", + "allowedPurpose": "evaluation", + "verifiedAt": "2026-07-20T00:00:00Z" + }, + "sourceStatus": "active", + "retrievalResult": { + "cards": [], + "factEvidence": [], + "proseEvidence": [], + "indexHints": [ + { + "cardId": "fixture-card-544-1", + "name": "迷途之地", + "type": "location", + "content": "截至第543章的脱敏索引提示", + "sourceId": "fixture-card-source:544-1", + "sourceVersion": "fixture-card-v1", + "asOf": 543 + }, + { + "cardId": "fixture-card-544-2", + "name": "内应", + "type": "character_role", + "content": "截至第543章的脱敏索引提示", + "sourceId": "fixture-card-source:544-2", + "sourceVersion": "fixture-card-v1", + "asOf": 543 + } + ], + "manifest": { + "omittedSources": [] + } + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fine-outline:deep-space:544", + "sourceVersion": "fine-outline-preregistered-v1", + "chapter": 544 + }, + "hardConstraints": [ + "完成迷途之地内应反水的转折" + ], + "adjustableBeats": [ + "转折铺垫与揭示顺序允许微调" + ], + "declaredNewFacts": [], + "entities": [ + { + "id": "entity-544-location", + "type": "location", + "name": "迷途之地" + }, + { + "id": "entity-544-role", + "type": "character_role", + "name": "内应" + } + ] + }, + "narrativeState": { + "time": "第543章结束后", + "location": "迷途之地", + "characterPositions": {}, + "immediateSituation": "阵营转折待发生" + }, + "recentChapters": [ + { + "chapter": 540, + "sourceRef": { + "sourceId": "fixture-chapter:540", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第540章冻结位置。" + }, + { + "chapter": 541, + "sourceRef": { + "sourceId": "fixture-chapter:541", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第541章冻结位置。" + }, + { + "chapter": 542, + "sourceRef": { + "sourceId": "fixture-chapter:542", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第542章冻结位置。" + }, + { + "chapter": 543, + "sourceRef": { + "sourceId": "fixture-chapter:543", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第543章冻结位置。" + } + ], + "outputContract": { + "targetChars": 6700, + "minChars": 6030, + "maxChars": 7370, + "frontmatterRequired": false, + "newSettingDeclarationRequired": true + }, + "tokenBudget": { + "maxContextChars": 50000 + }, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + { + "requirementId": "event-544-1", + "anchors": [ + "内应", + "反水" + ] + } + ], + "requiredCharacters": [ + "内应" + ], + "foreshadowingActions": [], + "chapterEndHook": { + "requirementId": "hook-544", + "anchors": [ + "立场" + ], + "maxDistanceFromEnd": 120 + }, + "detectedNewSettings": [], + "frozenConflicts": [] + } + } + }, + { + "sampleId": "deep-space-199-information-reveal", + "targetTitle": "赛莉丝身份与联赛锁死真相", + "targetDescription": "第199章固定为揭示赛莉丝身份与联赛锁死真相的信息揭示场景。", + "workId": 8, + "scenario": "information_reveal", + "asOfChapter": 198, + "targetChapter": 199, + "snapshotVersion": "writer-deep-space-199-v1", + "outlineSource": "outline-window:deep-space:191-198", + "fineOutlineSource": "fine-outline:deep-space:199", + "frozenRecentHanCounts": [1820, 1760, 1808, 1797], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [195, 196, 197, 198], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": false + }, + "targetChars": 2000, + "expectedLength": { + "targetChars": 2000, + "minChars": 2000, + "maxChars": 2200 + }, + "mainEntities": [ + "赛莉丝", + "联赛" + ], + "newCharacterRatio": 0.0, + "newCharacterRatioStatus": "resolved", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 198, + "requiredCharacters": ["赛莉丝"], + "knownBeforeAsOf": ["赛莉丝"], + "absentBeforeAsOf": [], + "genericRoles": [] + }, + "leakageCheck": { + "method": "chapter-bound-and-target-fact-sentinel", + "forbiddenReadFromChapter": 199, + "expectedTargetOrFutureReadCount": 0 + }, + "sources": [ + { + "sourceId": "fixture-chapters:195-198", + "sourceVersion": "sanitized-fixture-v1", + "chapterRange": "195-198" + }, + { + "sourceId": "fixture-card-index:198", + "sourceVersion": "fixture-card-v1", + "chapterRange": "1-198" + } + ], + "snapshotData": { + "chapters": [ + { + "chapter": 198, + "sourceId": "fixture-chapter:198", + "contentSha256": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" + } + ], + "cards": [] + }, + "leakageAudit": { + "method": "chapter-bound-and-target-fact-sentinel", + "targetFacts": { + "targetChapter": 199, + "forbiddenFacts": [ + { + "id": "target-199-sentinel", + "firstChapter": 199, + "text": "TARGET_SENTINEL_199" + } + ] + } + }, + "writerContextInput": { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "authorizationSnapshot": { + "snapshotId": "auth-work-8-gate-a-v1", + "allowedPurpose": "evaluation", + "verifiedAt": "2026-07-20T00:00:00Z" + }, + "sourceStatus": "active", + "retrievalResult": { + "cards": [], + "factEvidence": [], + "proseEvidence": [], + "indexHints": [ + { + "cardId": "fixture-card-199-1", + "name": "赛莉丝", + "type": "character", + "content": "截至第198章的脱敏索引提示", + "sourceId": "fixture-card-source:199-1", + "sourceVersion": "fixture-card-v1", + "asOf": 198 + }, + { + "cardId": "fixture-card-199-2", + "name": "联赛", + "type": "event", + "content": "截至第198章的脱敏索引提示", + "sourceId": "fixture-card-source:199-2", + "sourceVersion": "fixture-card-v1", + "asOf": 198 + } + ], + "manifest": { + "omittedSources": [] + } + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fine-outline:deep-space:199", + "sourceVersion": "fine-outline-preregistered-v1", + "chapter": 199 + }, + "hardConstraints": [ + "揭示赛莉丝身份与联赛锁死真相" + ], + "adjustableBeats": [ + "信息揭示的铺陈顺序允许调整" + ], + "declaredNewFacts": [], + "entities": [ + { + "id": "entity-199-character", + "type": "character", + "name": "赛莉丝" + }, + { + "id": "entity-199-event", + "type": "event", + "name": "联赛" + } + ] + }, + "narrativeState": { + "time": "第198章结束后", + "location": "联赛相关场景", + "characterPositions": {}, + "immediateSituation": "身份与真相待揭示" + }, + "recentChapters": [ + { + "chapter": 195, + "sourceRef": { + "sourceId": "fixture-chapter:195", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第195章冻结位置。" + }, + { + "chapter": 196, + "sourceRef": { + "sourceId": "fixture-chapter:196", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第196章冻结位置。" + }, + { + "chapter": 197, + "sourceRef": { + "sourceId": "fixture-chapter:197", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第197章冻结位置。" + }, + { + "chapter": 198, + "sourceRef": { + "sourceId": "fixture-chapter:198", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第198章冻结位置。" + } + ], + "outputContract": { + "targetChars": 2000, + "minChars": 2000, + "maxChars": 2200, + "frontmatterRequired": false, + "newSettingDeclarationRequired": true + }, + "tokenBudget": { + "maxContextChars": 50000 + }, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + { + "requirementId": "event-199-1", + "anchors": [ + "赛莉丝", + "联赛" + ] + } + ], + "requiredCharacters": [ + "赛莉丝" + ], + "foreshadowingActions": [], + "chapterEndHook": { + "requirementId": "hook-199", + "anchors": [ + "真相" + ], + "maxDistanceFromEnd": 120 + }, + "detectedNewSettings": [], + "frozenConflicts": [] + } + } + }, + { + "sampleId": "deep-space-523-returning-character", + "targetTitle": "伊蕾莉雅与旧部重逢", + "targetDescription": "第523章固定为伊蕾莉雅与旧部重逢的老角色回归场景。", + "workId": 8, + "scenario": "returning_character", + "asOfChapter": 522, + "targetChapter": 523, + "snapshotVersion": "writer-deep-space-523-v1", + "outlineSource": "outline-window:deep-space:515-522", + "fineOutlineSource": "fine-outline:deep-space:523", + "frozenRecentHanCounts": [9312, 6815, 7466, 6952], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [519, 520, 521, 522], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": false + }, + "targetChars": 6100, + "expectedLength": { + "targetChars": 6100, + "minChars": 5490, + "maxChars": 6710 + }, + "mainEntities": [ + "伊蕾莉雅", + "旧部" + ], + "newCharacterRatio": 0.0, + "newCharacterRatioStatus": "resolved", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 522, + "requiredCharacters": ["伊蕾莉雅"], + "knownBeforeAsOf": ["伊蕾莉雅"], + "absentBeforeAsOf": [], + "genericRoles": [] + }, + "leakageCheck": { + "method": "chapter-bound-and-target-fact-sentinel", + "forbiddenReadFromChapter": 523, + "expectedTargetOrFutureReadCount": 0 + }, + "sources": [ + { + "sourceId": "fixture-chapters:519-522", + "sourceVersion": "sanitized-fixture-v1", + "chapterRange": "519-522" + }, + { + "sourceId": "fixture-card-index:522", + "sourceVersion": "fixture-card-v1", + "chapterRange": "1-522" + } + ], + "snapshotData": { + "chapters": [ + { + "chapter": 522, + "sourceId": "fixture-chapter:522", + "contentSha256": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + } + ], + "cards": [] + }, + "leakageAudit": { + "method": "chapter-bound-and-target-fact-sentinel", + "targetFacts": { + "targetChapter": 523, + "forbiddenFacts": [ + { + "id": "target-523-sentinel", + "firstChapter": 523, + "text": "TARGET_SENTINEL_523" + } + ] + } + }, + "writerContextInput": { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": "db-work-8-gate-a-preregistered-v1", + "authorizationSnapshot": { + "snapshotId": "auth-work-8-gate-a-v1", + "allowedPurpose": "evaluation", + "verifiedAt": "2026-07-20T00:00:00Z" + }, + "sourceStatus": "active", + "retrievalResult": { + "cards": [], + "factEvidence": [], + "proseEvidence": [], + "indexHints": [ + { + "cardId": "fixture-card-523-1", + "name": "伊蕾莉雅", + "type": "character", + "content": "截至第522章的脱敏索引提示", + "sourceId": "fixture-card-source:523-1", + "sourceVersion": "fixture-card-v1", + "asOf": 522 + }, + { + "cardId": "fixture-card-523-2", + "name": "旧部", + "type": "character_group", + "content": "截至第522章的脱敏索引提示", + "sourceId": "fixture-card-source:523-2", + "sourceVersion": "fixture-card-v1", + "asOf": 522 + } + ], + "manifest": { + "omittedSources": [] + } + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fine-outline:deep-space:523", + "sourceVersion": "fine-outline-preregistered-v1", + "chapter": 523 + }, + "hardConstraints": [ + "完成伊蕾莉雅与旧部重逢" + ], + "adjustableBeats": [ + "重逢动作与对白次序允许调整" + ], + "declaredNewFacts": [], + "entities": [ + { + "id": "entity-523-character", + "type": "character", + "name": "伊蕾莉雅" + }, + { + "id": "entity-523-group", + "type": "character_group", + "name": "旧部" + } + ] + }, + "narrativeState": { + "time": "第522章结束后", + "location": "重逢场景", + "characterPositions": {}, + "immediateSituation": "旧角色回归待发生" + }, + "recentChapters": [ + { + "chapter": 519, + "sourceRef": { + "sourceId": "fixture-chapter:519", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第519章冻结位置。" + }, + { + "chapter": 520, + "sourceRef": { + "sourceId": "fixture-chapter:520", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第520章冻结位置。" + }, + { + "chapter": 521, + "sourceRef": { + "sourceId": "fixture-chapter:521", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第521章冻结位置。" + }, + { + "chapter": 522, + "sourceRef": { + "sourceId": "fixture-chapter:522", + "sourceVersion": "sanitized-fixture-v1", + "blockId": 1, + "startCodePoint": 0, + "endCodePoint": 17 + }, + "text": "脱敏合同夹具，第522章冻结位置。" + } + ], + "outputContract": { + "targetChars": 6100, + "minChars": 5490, + "maxChars": 6710, + "frontmatterRequired": false, + "newSettingDeclarationRequired": true + }, + "tokenBudget": { + "maxContextChars": 50000 + }, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + { + "requirementId": "event-523-1", + "anchors": [ + "伊蕾莉雅", + "旧部" + ] + } + ], + "requiredCharacters": [ + "伊蕾莉雅" + ], + "foreshadowingActions": [], + "chapterEndHook": { + "requirementId": "hook-523", + "anchors": [ + "重逢" + ], + "maxDistanceFromEnd": 120 + }, + "detectedNewSettings": [], + "frozenConflicts": [] + } + } + } + ] +} diff --git a/.claude/skills/replay-eval/scripts/run_writer_replay.py b/.claude/skills/replay-eval/scripts/run_writer_replay.py new file mode 100644 index 0000000..b44b8ee --- /dev/null +++ b/.claude/skills/replay-eval/scripts/run_writer_replay.py @@ -0,0 +1,1101 @@ +#!/usr/bin/env python3 +"""正文 A/B/C 同条件冻结回放编排器。 + +默认 dry-run 只组装并校验 WriterContext，公开计划、清单和脱敏摘要， +不调用模型。真实 semantic detector 与盲评 adapter 尚未接线，因此 CLI +的 ``--execute`` 明确失败关闭；测试只能通过专用注入接口替换 subprocess +runner，并且候选仍必须经过正式 writer adapter 和完整审查管线。 +""" + +from __future__ import annotations + +import argparse +import copy +import json +import random +import re +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable, Mapping, Protocol + +SCRIPT_DIR = Path(__file__).resolve().parent +SKILLS_DIR = SCRIPT_DIR.parents[1] +CONTINUATION_DIR = SKILLS_DIR / "continuation" / "scripts" +READ_CONTEXT_DIR = SKILLS_DIR / "read-context" / "scripts" +for import_path in (CONTINUATION_DIR, READ_CONTEXT_DIR): + if str(import_path) not in sys.path: + sys.path.insert(0, str(import_path)) + +from assemble_writer_context import AssemblyError, assemble_context # noqa: E402 +from audit_leakage import audit_snapshot # noqa: E402 +from build_snapshot import SnapshotError, build_snapshot, normalize_chapter, sha256_value # noqa: E402 +from check_snapshot import ( # noqa: E402 + STATUS_READY, + check_arm_manifests, + check_authorization, + check_target_sources, +) +from run_writer import WriterAdapterError, run_writer # noqa: E402 +from run_writer_pipeline import ( # noqa: E402 + InMemoryCasStateStore, + PipelineError, + atomic_write_json, + run_writer_pipeline, +) +from writer_contract import ( # noqa: E402 + ContractError, + calculate_target_chars, + retrieval_identity, + validate_writer_context, +) +from writer_rubric import adjudicate_reviews, deblind_reports # noqa: E402 + + +PROFILE = "writer_replay" +REQUIRED_ARMS = ("A", "B", "C") +EVIDENCE_STRATEGIES = { + "A": "historical_prose_only", + "B": "card_index_only", + "C": "card_index_plus_prose", +} +PRIVATE_TMP = Path("/private/tmp").resolve() +SNAPSHOT_COMPAT_ARMS = ( + "outline_only", + "outline_plus_cards", + "outline_plus_placebo_cards", +) +CONTENT_MODES = frozenset({"sanitized_contract_fixture", "canonical_frozen_prose"}) + + +class WriterReplayError(ValueError): + """正文回放配置或可信边界不合法。""" + + +class AuthorizationBindingError(WriterReplayError): + """WriterContext 输入试图脱离顶层不可变授权快照。""" + + +class ReplayJudge(Protocol): + """测试盲评边界；只接收盲化内容，生产 judge adapter 未接线。""" + + def __call__( + self, + blind_input: Mapping[str, Any], + reviewer_id: str, + ) -> Mapping[str, Any]: ... + + +@dataclass(frozen=True) +class WriterReplayTestAdapters: + """仅供无模型测试注入的三类边界，不能由 CLI 构造。""" + + writer_runner: Callable[..., subprocess.CompletedProcess[str]] + semantic_detector: Callable[ + [Mapping[str, Any], Mapping[str, Any], Mapping[str, Any]], Mapping[str, Any] + ] + judge: ReplayJudge + + +def _safe_json(value: Any) -> str: + """生成稳定 JSON，便于哈希与机械比较。""" + + return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + + +def _mapping(value: Any, field: str) -> Mapping[str, Any]: + """读取必填对象字段并失败关闭。""" + + if not isinstance(value, Mapping): + raise WriterReplayError(f"{field} 必须是对象") + return value + + +def _sequence(value: Any, field: str) -> list[Any]: + """读取必填数组字段，不猜测字符串或单对象。""" + + if not isinstance(value, list): + raise WriterReplayError(f"{field} 必须是数组") + return list(value) + + +def _run_id(value: str) -> str: + """把外部 run/sample 标识收敛到 WriterContext 允许的稳定字符集。""" + + normalized = re.sub(r"[^A-Za-z0-9._:-]+", "-", value).strip("-._:") + if not normalized: + raise WriterReplayError("runId/sampleId 不能规范化为空") + return normalized[:128] + + +def _length_bounds(target_chars: int) -> tuple[int, int]: + """按预注册目标的正负 10% 计算边界，并执行正文合同的总限幅。""" + + return max(2000, target_chars * 9 // 10), min(10000, target_chars * 11 // 10) + + +def _validate_preregistered_length(sample: Mapping[str, Any], as_of: int) -> int: + """只用冻结点前四章汉字数机械复算篇幅，禁止读取目标章实际长度。""" + + counts = _sequence(sample.get("frozenRecentHanCounts"), "frozenRecentHanCounts") + if ( + len(counts) != 4 + or any(isinstance(value, bool) or not isinstance(value, int) or value < 500 for value in counts) + ): + raise WriterReplayError("frozenRecentHanCounts 必须是冻结点前四章的有效汉字数") + basis = _mapping(sample.get("targetLengthBasis"), "targetLengthBasis") + expected_fields = { + "algorithm", + "sourceChapters", + "hardEventCount", + "foreshadowingActionCount", + "requiredSceneCount", + "minChars", + "maxChars", + "usesTargetChapterLength", + } + if set(basis) != expected_fields: + raise WriterReplayError("targetLengthBasis 字段必须严格匹配预注册篇幅合同") + if basis.get("algorithm") != "calculate_target_chars": + raise WriterReplayError("targetLengthBasis.algorithm 非法") + source_chapters = _sequence(basis.get("sourceChapters"), "targetLengthBasis.sourceChapters") + if source_chapters != list(range(max(1, as_of - 3), as_of + 1)): + raise WriterReplayError("篇幅基线必须严格来自冻结点前连续四章") + if basis.get("usesTargetChapterLength") is not False: + raise WriterReplayError("篇幅预注册禁止读取目标章实际长度") + integer_fields = ( + "hardEventCount", + "foreshadowingActionCount", + "requiredSceneCount", + "minChars", + "maxChars", + ) + if any(isinstance(basis.get(field), bool) or not isinstance(basis.get(field), int) for field in integer_fields): + raise WriterReplayError("targetLengthBasis 计数与边界必须是整数") + try: + target = calculate_target_chars( + recent_chapter_han_counts=counts, + hard_event_count=int(basis["hardEventCount"]), + foreshadowing_action_count=int(basis["foreshadowingActionCount"]), + required_scene_count=int(basis["requiredSceneCount"]), + min_chars=int(basis["minChars"]), + max_chars=int(basis["maxChars"]), + ) + except ContractError as error: + raise WriterReplayError(f"篇幅预注册参数非法: {error}") from error + if sample.get("targetChars") != target: + raise WriterReplayError("targetChars 与冻结历史机械复算结果不一致") + expected_min, expected_max = _length_bounds(target) + expected_length = _mapping(sample.get("expectedLength"), "expectedLength") + if dict(expected_length) != { + "targetChars": target, + "minChars": expected_min, + "maxChars": expected_max, + }: + raise WriterReplayError("expectedLength 必须等于机械目标的正负 10% 限幅") + context_input = _mapping(sample.get("writerContextInput"), "writerContextInput") + output_contract = _mapping(context_input.get("outputContract"), "writerContextInput.outputContract") + for field, expected in ( + ("targetChars", target), + ("minChars", expected_min), + ("maxChars", expected_max), + ): + if output_contract.get(field) != expected: + raise WriterReplayError(f"writerContextInput.outputContract.{field} 未绑定预注册篇幅") + return target + + +def _validate_new_character_ratio(sample: Mapping[str, Any], as_of: int) -> None: + """校验具名必需角色在冻结点前无记录的比例；泛称角色必须保持未知。""" + + context_input = _mapping(sample.get("writerContextInput"), "writerContextInput") + requirements = _mapping(context_input.get("requirements"), "writerContextInput.requirements") + required = _sequence( + requirements.get("requiredCharacters"), + "writerContextInput.requirements.requiredCharacters", + ) + if not required or any(not isinstance(item, str) or not item.strip() for item in required): + raise WriterReplayError("requiredCharacters 必须是非空字符串数组") + status = sample.get("newCharacterRatioStatus") + basis = _mapping(sample.get("newCharacterBasis"), "newCharacterBasis") + expected_fields = { + "definition", + "asOfChapter", + "requiredCharacters", + "knownBeforeAsOf", + "absentBeforeAsOf", + "genericRoles", + } + if set(basis) != expected_fields: + raise WriterReplayError("newCharacterBasis 字段必须严格匹配预注册合同") + if basis.get("definition") != "named_required_characters_absent_before_as_of_ratio": + raise WriterReplayError("newCharacterBasis.definition 非法") + if basis.get("asOfChapter") != as_of: + raise WriterReplayError("newCharacterBasis.asOfChapter 未绑定冻结点") + if _sequence(basis.get("requiredCharacters"), "newCharacterBasis.requiredCharacters") != required: + raise WriterReplayError("newCharacterBasis.requiredCharacters 与细纲要求不一致") + known = _sequence(basis.get("knownBeforeAsOf"), "newCharacterBasis.knownBeforeAsOf") + absent = _sequence(basis.get("absentBeforeAsOf"), "newCharacterBasis.absentBeforeAsOf") + generic = _sequence(basis.get("genericRoles"), "newCharacterBasis.genericRoles") + for field, values in (("knownBeforeAsOf", known), ("absentBeforeAsOf", absent), ("genericRoles", generic)): + if any(not isinstance(item, str) or not item.strip() for item in values) or len(set(values)) != len(values): + raise WriterReplayError(f"newCharacterBasis.{field} 必须是无重复非空字符串数组") + if status == "unresolved_generic_role": + if sample.get("newCharacterRatio") is not None or not generic: + raise WriterReplayError("泛称角色无法判定时 newCharacterRatio 必须为 null") + if set(generic) != set(required) or known or absent: + raise WriterReplayError("泛称角色不得被默认归为已知或新角色") + return + if status != "resolved" or generic: + raise WriterReplayError("newCharacterRatioStatus 非法") + if set(known).intersection(absent) or set(known).union(absent) != set(required): + raise WriterReplayError("具名 requiredCharacters 必须被已知/无记录集合完整且互斥地覆盖") + retrieval_result = _mapping(context_input.get("retrievalResult"), "writerContextInput.retrievalResult") + raw_hints = retrieval_result.get("indexHints") + if raw_hints is None: + raw_hints = retrieval_result.get("unverifiedIndexHints", []) + hints = _sequence(raw_hints, "writerContextInput.retrievalResult.indexHints") + hinted_names = { + str(hint.get("name") or "") + for hint in hints + if isinstance(hint, Mapping) + } + if set(absent).intersection(hinted_names): + raise WriterReplayError("冻结点前无记录角色不得同时出现在卡索引提示中") + expected_ratio = len(absent) / len(required) + ratio = sample.get("newCharacterRatio") + if isinstance(ratio, bool) or not isinstance(ratio, (int, float)) or float(ratio) != expected_ratio: + raise WriterReplayError("newCharacterRatio 与冻结点前记录分区不一致") + + +def _common_controls( + config: Mapping[str, Any], sample: Mapping[str, Any] +) -> dict[str, Any]: + """冻结三臂共享的作品、输入、模型、采样和 detector。""" + + common = dict(_mapping(config.get("commonControls"), "commonControls")) + as_of = normalize_chapter(sample.get("asOfChapter")) + target = normalize_chapter(sample.get("targetChapter")) + if as_of is None or target != as_of + 1: + raise WriterReplayError("样本 targetChapter 必须等于 asOfChapter+1") + _validate_preregistered_length(sample, as_of) + _validate_new_character_ratio(sample, as_of) + for field in ( + "outlineSource", + "fineOutlineSource", + "targetChars", + "modelVersion", + "sampling", + "detectorProfile", + ): + value = sample.get(field, common.get(field)) + if value is None: + raise WriterReplayError(f"样本缺少公共控制字段: {field}") + common[field] = value + reference = _mapping(config.get("referenceWork"), "referenceWork") + common.update( + { + "workId": reference.get("id"), + "referenceWorkVersion": reference.get("version"), + "asOfChapter": as_of, + "targetChapter": target, + } + ) + if isinstance(common["workId"], bool) or not isinstance(common["workId"], int): + raise WriterReplayError("referenceWork.id 必须是正整数") + if common["workId"] <= 0: + raise WriterReplayError("referenceWork.id 必须是正整数") + if sample.get("workId", common["workId"]) != common["workId"]: + raise WriterReplayError("样本 workId 与 referenceWork 不一致") + return common + + +def _retrieval_plan( + *, run_id: str, common: Mapping[str, Any], context_input: Mapping[str, Any] +) -> dict[str, Any]: + """为三臂生成同一份确定性检索计划；证据差异只进入 manifest。""" + + token_budget = _mapping(context_input.get("tokenBudget"), "writerContextInput.tokenBudget") + plan = { + "planVersion": "writer-retrieval-plan-v1", + "runId": run_id, + "asOf": common["asOfChapter"], + "queries": copy.deepcopy(_sequence(context_input.get("retrievalQueries", []), "retrievalQueries")), + "cardIndexVersion": str(context_input.get("cardIndexVersion") or "writer-replay-cards-v1"), + "proseIndexVersion": str(context_input.get("proseIndexVersion") or "writer-replay-prose-v1"), + "filters": { + "workId": common["workId"], + "asOfChapter": common["asOfChapter"], + "sourceStatus": str(context_input.get("sourceStatus") or ""), + "authorizationRequired": True, + }, + "tieBreak": "score DESC, sourceVersion ASC, sourceId ASC, sourceOffset ASC", + "tokenBudget": {"maxContextChars": token_budget.get("maxContextChars")}, + } + plan["planId"] = retrieval_identity(plan) + return plan + + +def _requirements(context_input: Mapping[str, Any]) -> dict[str, Any]: + """读取机械 detector 的完整输入，dry-run 也先校验结构。""" + + raw = dict(_mapping(context_input.get("requirements"), "writerContextInput.requirements")) + for field in ( + "requiredEvents", + "requiredCharacters", + "foreshadowingActions", + "detectedNewSettings", + "frozenConflicts", + ): + _sequence(raw.get(field), f"writerContextInput.requirements.{field}") + _mapping(raw.get("chapterEndHook"), "writerContextInput.requirements.chapterEndHook") + return raw + + +def _validate_context_authorization_binding( + config: Mapping[str, Any], context_input: Mapping[str, Any] +) -> None: + """把 WriterContext 来源、版本、用途和校验时间绑定到顶层授权快照。 + + 顶层 ``offline_evaluation`` 在 WriterContext 授权快照中允许保留原值， + 或规范化为合同目的 ``evaluation``。顶层来源状态是授权登记状态，WriterContext 的 + ``active`` 是其运行期投影；只有下表明确列出的投影允许通过。 + """ + + reference = _mapping(config.get("referenceWork"), "referenceWork") + authorization = _mapping(config.get("authorization"), "authorization") + top_snapshot = _mapping( + authorization.get("authorizationSnapshot"), + "authorization.authorizationSnapshot", + ) + context_snapshot = _mapping( + context_input.get("authorizationSnapshot"), + "writerContextInput.authorizationSnapshot", + ) + + versions = { + "referenceWork.version": str(reference.get("version") or ""), + "authorization.sourceVersion": str(authorization.get("sourceVersion") or ""), + "authorization.authorizationSnapshot.sourceVersion": str( + top_snapshot.get("sourceVersion") or "" + ), + "writerContextInput.sourceVersion": str(context_input.get("sourceVersion") or ""), + } + if any(not value for value in versions.values()) or len(set(versions.values())) != 1: + raise AuthorizationBindingError( + "WriterContext sourceVersion 未绑定 referenceWork 与不可变授权快照" + ) + + if str(context_snapshot.get("snapshotId") or "") != str(top_snapshot.get("id") or ""): + raise AuthorizationBindingError("WriterContext authorizationSnapshot.snapshotId 不匹配") + context_allowed_purpose = str(context_snapshot.get("allowedPurpose") or "") + if context_allowed_purpose not in {"evaluation", "offline_evaluation"}: + raise AuthorizationBindingError( + "WriterContext 授权用途必须对应 evaluation/offline_evaluation" + ) + top_allowed = authorization.get("allowedPurpose") + snapshot_allowed = top_snapshot.get("allowedPurpose") + if ( + not isinstance(top_allowed, list) + or "offline_evaluation" not in top_allowed + or not isinstance(snapshot_allowed, list) + or "offline_evaluation" not in snapshot_allowed + ): + raise AuthorizationBindingError("顶层授权快照未授权 offline_evaluation") + if str(context_snapshot.get("verifiedAt") or "") != str(top_snapshot.get("checkedAt") or ""): + raise AuthorizationBindingError( + "WriterContext authorizationSnapshot.verifiedAt 未绑定顶层 checkedAt" + ) + context_snapshot_source = context_snapshot.get("sourceVersion") + if context_snapshot_source is not None and str(context_snapshot_source) != versions[ + "referenceWork.version" + ]: + raise AuthorizationBindingError( + "WriterContext authorizationSnapshot.sourceVersion 不匹配" + ) + + top_status = str(authorization.get("sourceStatus") or "").lower() + context_status = str(context_input.get("sourceStatus") or "").lower() + allowed_context_statuses = { + "active": frozenset({"active"}), + "approved": frozenset({"active"}), + "authorized": frozenset({"active", "authorized", "frozen_authorized"}), + "licensed": frozenset({"active"}), + } + if context_status not in allowed_context_statuses.get(top_status, frozenset()): + raise AuthorizationBindingError( + "WriterContext sourceStatus 与顶层授权状态语义不一致" + ) + + +def _build_arm_contexts( + *, + config: Mapping[str, Any], + sample: Mapping[str, Any], + common: Mapping[str, Any], + replay_run_id: str, +) -> tuple[dict[str, dict[str, Any]], dict[str, Any], str]: + """经唯一组装入口构造 A/B/C，并再次执行 WriterContext 合同校验。""" + + context_input = _mapping(sample.get("writerContextInput"), "writerContextInput") + _validate_context_authorization_binding(config, context_input) + content_mode = str(context_input.get("contentMode") or "") + if content_mode not in CONTENT_MODES: + raise WriterReplayError( + "writerContextInput.contentMode 必须明确为 sanitized_contract_fixture 或 canonical_frozen_prose" + ) + requirements = _requirements(context_input) + sample_run_id = _run_id(f"{replay_run_id}:{sample['sampleId']}") + plan = _retrieval_plan(run_id=sample_run_id, common=common, context_input=context_input) + retrieval_result = _mapping( + context_input.get("retrievalResult"), "writerContextInput.retrievalResult" + ) + contexts: dict[str, dict[str, Any]] = {} + for arm in REQUIRED_ARMS: + try: + assembled = assemble_context( + run_id=sample_run_id, + attempt=1, + mode="diagnostic_only", + purpose="evaluation", + quality_policy_version="writer-eval-v1", + work_id=int(common["workId"]), + target_chapter=int(common["targetChapter"]), + as_of=int(common["asOfChapter"]), + source_version=str(context_input.get("sourceVersion") or ""), + authorization_snapshot=_mapping( + context_input.get("authorizationSnapshot"), + "writerContextInput.authorizationSnapshot", + ), + source_status=str(context_input.get("sourceStatus") or ""), + retrieval_plan=plan, + retrieval_result=retrieval_result, + fine_outline=_mapping( + context_input.get("fineOutline"), "writerContextInput.fineOutline" + ), + narrative_state=_mapping( + context_input.get("narrativeState"), "writerContextInput.narrativeState" + ), + recent_chapters=_sequence( + context_input.get("recentChapters"), "writerContextInput.recentChapters" + ), + output_contract=_mapping( + context_input.get("outputContract"), "writerContextInput.outputContract" + ), + token_budget=_mapping( + context_input.get("tokenBudget"), "writerContextInput.tokenBudget" + ), + pattern_references=_sequence( + context_input.get("patternReferences", []), + "writerContextInput.patternReferences", + ), + generated_at=str(context_input.get("generatedAt") or ""), + evidence_strategy=EVIDENCE_STRATEGIES[arm], + ) + # 组装器内部已校验；这里保留显式二次校验，证明每一臂都经过合同入口。 + contexts[arm] = validate_writer_context(assembled["context"]) + except (AssemblyError, ContractError, TypeError, ValueError) as error: + raise WriterReplayError(f"{sample['sampleId']} {arm} 臂 WriterContext 非法: {error}") from error + return contexts, requirements, content_mode + + +def _context_summary(context: Mapping[str, Any], *, content_mode: str) -> dict[str, Any]: + """只公开合同、冻结和来源元数据，不泄露证据文本或提示内容。""" + + prose = context["proseEvidence"] + hints = context.get("indexHints", []) + return { + "schemaVersion": context["schemaVersion"], + "mode": context["mode"], + "purpose": context["purpose"], + "evidenceStrategy": context["evidenceStrategy"], + "acceptanceEligible": context["acceptanceEligible"], + "asOf": context["asOf"], + "targetChapter": context["targetChapter"], + "contentMode": content_mode, + "contextSnapshotId": context["contextSnapshot"]["manifestId"], + "contextSnapshotSha256": context["contextSnapshot"]["contextSha256"], + "retrievalPlanId": context["retrievalPlan"]["planId"], + "retrievalManifestId": context["retrievalManifest"]["manifestId"], + "factEvidenceCount": len(context["factEvidence"]), + "proseEvidenceCount": len(prose), + "proseChapters": [item["chapter"] for item in prose], + "recentBaselineChapters": [ + item["chapter"] for item in prose if item["isRecentBaseline"] is True + ], + "indexHintCount": len(hints), + "indexHintAsOf": sorted({item["asOf"] for item in hints}), + "sourceCount": len(context["retrievalManifest"]["sources"]), + "omittedSourceCount": len(context["retrievalManifest"]["omittedSources"]), + } + + +def _arm_manifests( + *, + common: Mapping[str, Any], + contexts: Mapping[str, Mapping[str, Any]], + snapshot_sha256: str, + content_mode: str, +) -> dict[str, dict[str, Any]]: + """从已校验上下文生成三臂清单，避免配置宣称与实际输入漂移。""" + + common_hash = sha256_value(common) + manifests: dict[str, dict[str, Any]] = {} + for arm in REQUIRED_ARMS: + context = contexts[arm] + prose = context["proseEvidence"] + manifests[arm] = { + "arm": arm, + "profile": PROFILE, + "evidenceStrategy": EVIDENCE_STRATEGIES[arm], + "commonControlsSha256": common_hash, + "snapshotManifestSha256": snapshot_sha256, + "acceptanceEligible": False, + "retrievalPlan": copy.deepcopy(context["retrievalPlan"]), + "retrievalManifest": copy.deepcopy(context["retrievalManifest"]), + "contextSummary": _context_summary(context, content_mode=content_mode), + "diagnosticContext": { + "proseSourceIds": [item["sourceRef"]["sourceId"] for item in prose], + "indexHintSourceIds": [item["sourceId"] for item in context.get("indexHints", [])], + "cardExpandedProseSourceIds": [ + item["sourceRef"]["sourceId"] + for item in prose + if item["isRecentBaseline"] is False + ], + }, + } + control_manifests = { + arm: { + "arm": arm, + "asOfChapter": common["asOfChapter"], + "targetChapter": common["targetChapter"], + "commonControls": dict(common), + "snapshotManifestSha256": snapshot_sha256, + } + for arm in REQUIRED_ARMS + } + control_check = check_arm_manifests( + control_manifests, + required_arms=REQUIRED_ARMS, + as_of_chapter=int(common["asOfChapter"]), + target_chapter=int(common["targetChapter"]), + smoke=True, + ) + if not control_check["ok"]: + raise WriterReplayError("三臂公共控制不一致: " + "; ".join(control_check["errors"])) + return manifests + + +def _prepare_sample( + config: Mapping[str, Any], sample: Mapping[str, Any], *, replay_run_id: str +) -> dict[str, Any]: + """复用授权、章界、冻结和泄露逻辑，再组装三臂 WriterContext。""" + + sample_id = str(sample.get("sampleId") or "") + if not sample_id: + raise WriterReplayError("样本缺少 sampleId") + common = _common_controls(config, sample) + sources = sample.get("sources") + if not isinstance(sources, list): + raise WriterReplayError(f"{sample_id}.sources 必须是数组") + source_check = check_target_sources(int(common["targetChapter"]), sources) + if not source_check["ok"]: + return { + "sampleId": sample_id, + "ok": False, + "status": source_check["status"], + "errors": source_check["errors"], + } + + reference = _mapping(config.get("referenceWork"), "referenceWork") + authorization = _mapping(config.get("authorization"), "authorization") + try: + frozen = build_snapshot( + _mapping(sample.get("snapshotData"), f"{sample_id}.snapshotData"), + int(common["asOfChapter"]), + str(sample.get("snapshotVersion") or ""), + target_chapter=int(common["targetChapter"]), + manifest_metadata={ + "referenceWork": { + "id": str(reference.get("id")), + "version": str(reference.get("version")), + }, + "evaluationSetVersion": str(config.get("evaluationSetVersion") or ""), + "strategyVersion": str(config.get("strategyVersion") or ""), + "authorizationSnapshot": authorization.get("authorizationSnapshot"), + "runPermissions": {"purpose": "offline_evaluation", "mode": "diagnostic_only"}, + "armConfig": {"arms": list(SNAPSHOT_COMPAT_ARMS)}, + }, + ) + except SnapshotError as error: + return { + "sampleId": sample_id, + "ok": False, + "status": "invalid_snapshot", + "errors": [str(error)], + } + future_omissions = [ + item + for item in frozen["manifest"].get("omittedSources", []) + if isinstance(item, Mapping) and item.get("reason") == "future_or_crosses_as_of" + ] + if future_omissions: + return { + "sampleId": sample_id, + "ok": False, + "status": "invalid_leakage", + "errors": ["冻结输入包含目标章或未来章记录"], + "leakageAudit": { + "status": "invalid_snapshot", + "findingCount": len(future_omissions), + }, + } + + leakage = _mapping(sample.get("leakageAudit"), f"{sample_id}.leakageAudit") + context_input = _mapping(sample.get("writerContextInput"), f"{sample_id}.writerContextInput") + # 目标章细纲本来就带 targetChapter/sourceRef，不能交给“历史来源章界”扫描器； + # 泄露审计只扫描真正注入写手的冻结历史证据和卡索引提示。 + audit = audit_snapshot( + { + "snapshot": frozen["snapshot"], + "writerEvidence": { + "recentChapters": context_input.get("recentChapters", []), + "retrievalResult": context_input.get("retrievalResult", {}), + "narrativeState": context_input.get("narrativeState", {}), + }, + }, + leakage.get("targetFacts"), + as_of=int(common["asOfChapter"]), + target=int(common["targetChapter"]), + ) + if not audit["ok"]: + return { + "sampleId": sample_id, + "ok": False, + "status": audit["status"], + "errors": audit["errors"], + "leakageAudit": audit, + } + try: + contexts, requirements, content_mode = _build_arm_contexts( + config=config, + sample=sample, + common=common, + replay_run_id=replay_run_id, + ) + arms = _arm_manifests( + common=common, + contexts=contexts, + snapshot_sha256=frozen["manifest"]["manifestSha256"], + content_mode=content_mode, + ) + except WriterReplayError as error: + error_text = str(error) + if isinstance(error, AuthorizationBindingError): + status = "blocked_authorization" + elif "超出冻结线" in error_text or "目标章或未来章" in error_text: + status = "invalid_leakage" + else: + status = "invalid_writer_context" + failure = { + "sampleId": sample_id, + "ok": False, + "status": status, + "errors": [error_text], + } + if status == "invalid_leakage": + failure["leakageAudit"] = { + "status": "invalid_snapshot", + "findingCount": 1, + } + return failure + return { + "sampleId": sample_id, + "scenario": str(sample.get("scenario") or ""), + "ok": True, + "status": STATUS_READY, + "asOfChapter": common["asOfChapter"], + "targetChapter": common["targetChapter"], + "snapshotManifestSha256": frozen["manifest"]["manifestSha256"], + "leakageAudit": { + "status": audit["status"], + "findingCount": audit["findingCount"], + }, + "requirementsSha256": sha256_value(requirements), + "arms": arms, + "commonControls": common, + "_contexts": contexts, + "_requirements": requirements, + } + + +def _public_sample(prepared: Mapping[str, Any]) -> dict[str, Any]: + """移除仅供执行使用的完整上下文，避免 dry-run 回显脱敏夹具文本。""" + + return {key: copy.deepcopy(value) for key, value in prepared.items() if not key.startswith("_")} + + +def _blind_mapping(evaluation_set_version: str, sample_id: str) -> tuple[dict[str, str], list[str]]: + """从预注册评测集和样本 ID 派生稳定盲化顺序。""" + + seed = int(sha256_value(f"{evaluation_set_version}:{sample_id}")[:16], 16) + shuffled = list(REQUIRED_ARMS) + random.Random(seed).shuffle(shuffled) + order = [f"blind-{index}" for index in range(1, 4)] + return dict(zip(order, shuffled, strict=True)), order + + +def _candidate_summary(candidate: Mapping[str, Any], arm: str) -> dict[str, Any]: + """只保留安全候选元数据，正文和完整响应不得进入结构化结果。""" + + if candidate.get("acceptanceEligible") is not False: + raise WriterReplayError(f"{arm} 臂候选必须 acceptanceEligible=false") + candidate_hash = candidate.get("candidateSha256") + candidate_version = candidate.get("candidateVersion") + if not isinstance(candidate_hash, str) or not candidate_hash.startswith("sha256:"): + raise WriterReplayError(f"{arm} 臂候选缺少 candidateSha256") + if isinstance(candidate_version, bool) or not isinstance(candidate_version, int): + raise WriterReplayError(f"{arm} 臂候选缺少 candidateVersion") + return { + "candidateId": f"{candidate['runId']}:{arm}:v{candidate_version}", + "candidateVersion": candidate_version, + "candidateSha256": candidate_hash, + "acceptanceEligible": False, + } + + +def _build_blind_input( + *, + raw_sample: Mapping[str, Any], + candidates: Mapping[str, Mapping[str, Any]], + mapping: Mapping[str, str], + blind_id: str, + candidate_order: list[str], +) -> dict[str, Any]: + """构造 judge 唯一可见输入；只含共同参考与盲化候选。""" + + if blind_id not in mapping or set(mapping.values()) != set(REQUIRED_ARMS): + raise WriterReplayError("盲评映射非法") + if set(candidate_order) != set(mapping): + raise WriterReplayError("盲评候选顺序与预注册盲 ID 不一致") + blind_candidates: list[dict[str, Any]] = [] + for current_blind_id in candidate_order: + arm = mapping[current_blind_id] + candidate = candidates[arm] + body = candidate.get("candidateBody") + candidate_hash = candidate.get("candidateSha256") + if not isinstance(body, str) or not body: + raise WriterReplayError("盲评候选缺少正文内容") + if not isinstance(candidate_hash, str) or not candidate_hash.startswith("sha256:"): + raise WriterReplayError("盲评候选缺少正文哈希") + blind_candidates.append( + { + "blindCandidateId": current_blind_id, + "candidateSha256": candidate_hash, + "candidateBody": body, + } + ) + context_input = _mapping(raw_sample.get("writerContextInput"), "writerContextInput") + requirements = _mapping(context_input.get("requirements"), "writerContextInput.requirements") + shared_reference = { + "fineOutline": copy.deepcopy( + _mapping(context_input.get("fineOutline"), "writerContextInput.fineOutline") + ), + "requirements": { + "requiredEvents": copy.deepcopy( + _sequence(requirements.get("requiredEvents"), "requirements.requiredEvents") + ), + "requiredCharacters": copy.deepcopy( + _sequence(requirements.get("requiredCharacters"), "requirements.requiredCharacters") + ), + "foreshadowingActions": copy.deepcopy( + _sequence( + requirements.get("foreshadowingActions"), + "requirements.foreshadowingActions", + ) + ), + "chapterEndHook": copy.deepcopy( + _mapping(requirements.get("chapterEndHook"), "requirements.chapterEndHook") + ), + }, + "historicalProseBaseline": copy.deepcopy( + _sequence(context_input.get("recentChapters"), "writerContextInput.recentChapters") + ), + } + return { + "schemaVersion": "writer-blind-input-v1", + "sample": { + "sampleId": str(raw_sample.get("sampleId") or ""), + "scenario": str(raw_sample.get("scenario") or ""), + "targetChapter": raw_sample.get("targetChapter"), + "targetTitle": str(raw_sample.get("targetTitle") or ""), + "targetDescription": str(raw_sample.get("targetDescription") or ""), + }, + "blindCandidateId": blind_id, + "candidateOrder": list(candidate_order), + "sharedEvaluationReference": shared_reference, + "candidates": blind_candidates, + } + + +def _execute_sample_for_test( + *, + raw_sample: Mapping[str, Any], + prepared: Mapping[str, Any], + evaluation_set_version: str, + adapters: WriterReplayTestAdapters, + raw_dir: Path, +) -> dict[str, Any]: + """仅供测试验证正式 writer/pipeline/detector 编排，不代表生产已接线。""" + + raw_candidates: dict[str, dict[str, Any]] = {} + candidate_summaries: dict[str, dict[str, Any]] = {} + detector: dict[str, dict[str, Any]] = {} + for arm in REQUIRED_ARMS: + context = prepared["_contexts"][arm] + captured: dict[str, Mapping[str, Any]] = {} + + def writer( + current_context: Mapping[str, Any], + candidate_version: int, + _repair_failures: list[dict[str, Any]], + ) -> Mapping[str, Any]: + """通过正式 adapter 调 subprocess runner，禁止直接返回候选。""" + + try: + candidate = run_writer(current_context, runner=adapters.writer_runner) + except WriterAdapterError as error: + raise PipelineError(error.code, str(error), details=error.details) from error + if candidate["candidateVersion"] != candidate_version: + raise PipelineError("WRITER_OUTPUT_STALE", "候选版本未绑定本次管线状态") + captured["candidate"] = candidate + return candidate + + def no_evidence_provider(*_args: Any, **_kwargs: Any) -> Mapping[str, Any]: + """固定三臂实验不允许生成中途改变证据集合。""" + + raise PipelineError("REPLAY_EVIDENCE_REQUEST_FORBIDDEN", "固定回放禁止补证") + + try: + pipeline_result = run_writer_pipeline( + context=context, + requirements=prepared["_requirements"], + writer=writer, + evidence_provider=no_evidence_provider, + semantic_detector=adapters.semantic_detector, + state_store=InMemoryCasStateStore(), + result_path=raw_dir / f"pipeline-{arm}.json", + ) + except PipelineError as error: + raise WriterReplayError(f"{prepared['sampleId']} {arm} 臂管线失败: {error.code}") from error + candidate = captured.get("candidate") + if candidate is None: + raise WriterReplayError(f"{arm} 臂管线未留下已校验候选") + atomic_write_json(raw_dir / f"candidate-{arm}.json", candidate) + raw_candidates[arm] = candidate + candidate_summaries[arm] = _candidate_summary(candidate, arm) + last_trace = pipeline_result["trace"][-1] + detector[arm] = { + "passed": pipeline_result["status"] == "PASSED", + "highSeverityCount": 0, + "hardConstraintCoverage": 1.0, + "mechanicalPassed": last_trace.get("mechanicalPassed") is True, + "semanticStatus": last_trace.get("semanticStatus"), + } + + mapping, blind_order = _blind_mapping(evaluation_set_version, str(prepared["sampleId"])) + reviews: dict[str, Any] = {} + first_reports: list[Mapping[str, Any]] = [] + for blind_id in blind_order: + first_input = _build_blind_input( + raw_sample=raw_sample, + candidates=raw_candidates, + mapping=mapping, + blind_id=blind_id, + candidate_order=blind_order, + ) + first = adapters.judge(first_input, "judge-1") + first_reports.append(first) + second_input = _build_blind_input( + raw_sample=raw_sample, + candidates=raw_candidates, + mapping=mapping, + blind_id=blind_id, + candidate_order=list(reversed(blind_order)), + ) + second = adapters.judge(second_input, "judge-2") + adjudication = adjudicate_reviews(first, second) + if adjudication.get("status") == "needs_third_reviewer": + third_input = _build_blind_input( + raw_sample=raw_sample, + candidates=raw_candidates, + mapping=mapping, + blind_id=blind_id, + candidate_order=blind_order, + ) + third = adapters.judge(third_input, "judge-3") + adjudication = adjudicate_reviews(first, second, third) + reviews[blind_id] = adjudication + deblind_reports(first_reports, mapping) + reviews_by_arm = {mapping[blind_id]: reviews[blind_id] for blind_id in blind_order} + return { + **_public_sample(prepared), + "status": "completed_test_pipeline", + "candidates": candidate_summaries, + "detector": detector, + "blindOrderSha256": sha256_value(blind_order), + "reviews": reviews_by_arm, + } + + +def run_writer_replay( + config: Mapping[str, Any], + *, + run_id: str | None = None, + output_dir: Path | None = None, + execute: bool = False, + test_adapters: WriterReplayTestAdapters | None = None, +) -> dict[str, Any]: + """准备正文回放；测试可验证执行管线，生产 execute 当前失败关闭。""" + + if not isinstance(config, Mapping) or config.get("profile") != PROFILE: + raise WriterReplayError(f"profile 必须是 {PROFILE}") + evaluation_set_version = str(config.get("evaluationSetVersion") or "") + strategy_version = str(config.get("strategyVersion") or "") + replay_run_id = _run_id(str(run_id or "dry-run")) + if not evaluation_set_version or not strategy_version: + raise WriterReplayError("evaluationSetVersion/strategyVersion 不能为空") + authorization = check_authorization(config.get("authorization")) + if not authorization["ok"]: + return { + "runId": replay_run_id, + "mode": "execute" if execute else "dry_run", + "status": authorization["status"], + "ok": False, + "errors": authorization["errors"], + "samples": [], + } + samples = config.get("samples") + if not isinstance(samples, list) or not samples or any( + not isinstance(item, Mapping) for item in samples + ): + raise WriterReplayError("samples 必须是非空对象数组") + sample_ids = [str(item.get("sampleId") or "") for item in samples] + if any(not item for item in sample_ids) or len(set(sample_ids)) != len(sample_ids): + raise WriterReplayError("sampleId 必须非空且不能重复") + + prepared = [ + _prepare_sample(config, sample, replay_run_id=replay_run_id) for sample in samples + ] + invalid = [item for item in prepared if not item["ok"]] + result: dict[str, Any] = { + "runId": replay_run_id, + "mode": "execute" if execute else "dry_run", + "profile": PROFILE, + "evaluationSetVersion": evaluation_set_version, + "strategyVersion": strategy_version, + "status": "ready", + "ok": not invalid, + "executeAdapterStatus": "semantic_and_judge_not_wired", + "samples": [_public_sample(item) for item in prepared], + } + if invalid: + statuses = {item["status"] for item in invalid} + result["status"] = sorted(statuses)[0] + if any(item.get("leakageAudit") for item in invalid): + result["status"] = "invalid_leakage" + result["ok"] = False + return result + if not execute: + return result + + if output_dir is None: + raise WriterReplayError("execute 模式必须显式提供 output_dir") + resolved_output = output_dir.resolve() + if resolved_output == PRIVATE_TMP or not resolved_output.is_relative_to(PRIVATE_TMP): + raise WriterReplayError("execute 输出目录必须位于 /private/tmp 的独立子目录") + if test_adapters is None: + raise WriterReplayError( + "真实 execute 未接线：缺少生产 semantic detector 与 judge adapter，已失败关闭" + ) + + # 这一路径仅证明边界可测试；CLI 无法注入 test_adapters，不能宣称 real-run 完成。 + raw_dir = resolved_output / "raw" + raw_dir.mkdir(parents=True, exist_ok=True) + executed: list[dict[str, Any]] = [] + for raw_sample, prepared_sample in zip(samples, prepared, strict=True): + sample_raw_dir = raw_dir / prepared_sample["sampleId"] + sample_raw_dir.mkdir(parents=True, exist_ok=True) + executed.append( + _execute_sample_for_test( + raw_sample=raw_sample, + prepared=prepared_sample, + evaluation_set_version=evaluation_set_version, + adapters=test_adapters, + raw_dir=sample_raw_dir, + ) + ) + result.update( + { + "status": "completed_test_pipeline", + "ok": True, + "executeAdapterStatus": "test_injection_only", + "samples": executed, + } + ) + resolved_output.mkdir(parents=True, exist_ok=True) + atomic_write_json(resolved_output / "manifest.json", result) + return result + + +def _parse_args() -> argparse.Namespace: + """解析固定 CLI；默认 dry-run，显式 execute 仍受未接线门阻断。""" + + parser = argparse.ArgumentParser(description="运行 writer_replay A/B/C 回放") + parser.add_argument("--config", type=Path, required=True) + parser.add_argument("--run-id") + parser.add_argument("--output-dir", type=Path) + mode = parser.add_mutually_exclusive_group() + mode.add_argument("--dry-run", action="store_true") + mode.add_argument("--execute", action="store_true") + return parser.parse_args() + + +def main() -> int: + """CLI 只真正开放零模型 dry-run；execute 未接线时返回稳定阻塞。""" + + args = _parse_args() + config = json.loads(args.config.read_text(encoding="utf-8")) + result = run_writer_replay( + config, + run_id=args.run_id, + output_dir=args.output_dir, + execute=args.execute, + ) + print(json.dumps(result, ensure_ascii=False, sort_keys=True, indent=2)) + return 0 if result["ok"] else 2 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except (WriterReplayError, OSError, json.JSONDecodeError) as error: + print( + json.dumps( + {"status": "blocked_writer_replay", "error": str(error)}, + ensure_ascii=False, + ) + ) + raise SystemExit(2) + + +__all__ = [ + "CONTENT_MODES", + "EVIDENCE_STRATEGIES", + "PROFILE", + "REQUIRED_ARMS", + "WriterReplayError", + "WriterReplayTestAdapters", + "run_writer_replay", +] diff --git a/.claude/skills/replay-eval/scripts/test_run_writer_replay.py b/.claude/skills/replay-eval/scripts/test_run_writer_replay.py new file mode 100644 index 0000000..862c9a1 --- /dev/null +++ b/.claude/skills/replay-eval/scripts/test_run_writer_replay.py @@ -0,0 +1,736 @@ +#!/usr/bin/env python3 +"""正文 A/B/C 冻结回放编排的无网络、无真实模型测试。""" + +from __future__ import annotations + +import copy +import hashlib +import json +import pathlib +import subprocess +import sys +import tempfile +import unittest + +SCRIPT_DIR = pathlib.Path(__file__).resolve().parent +SKILLS_DIR = SCRIPT_DIR.parents[1] +READ_CONTEXT_DIR = SKILLS_DIR / "read-context" / "scripts" +for import_path in (SCRIPT_DIR, READ_CONTEXT_DIR): + sys.path.insert(0, str(import_path)) + +from run_writer_replay import ( # noqa: E402 + WriterReplayError, + WriterReplayTestAdapters, + run_writer_replay, +) +from writer_contract import calculate_target_chars, han_count, validate_writer_context # noqa: E402 +from writer_rubric import DIMENSIONS, RUBRIC_PROFILE # noqa: E402 + + +AUTHORIZATION = { + "sourceStatus": "authorized", + "copyrightStatus": "owned", + "sourceVersion": "raw-file-v1:sha256:" + "a" * 64, + "allowedPurpose": ["offline_evaluation"], + "authorizationSnapshot": { + "id": "auth-work-8", + "version": "auth-work-8-v1", + "immutable": True, + "sourceVersion": "raw-file-v1:sha256:" + "a" * 64, + "sourceStatus": "authorized", + "allowedPurpose": ["offline_evaluation"], + "checkedAt": "2026-07-20T00:00:00Z", + "revalidationAt": "2026-08-19T00:00:00Z", + }, +} + + +def _prose(chapter: int, block_id: int, text: str) -> dict[str, object]: + """构造只用于合同证明的脱敏合成原文片段。""" + + return { + "chapter": chapter, + "sourceRef": { + "sourceId": f"fixture:chapter:{chapter}:block:{block_id}", + "sourceVersion": f"sanitized-fixture-{chapter}-v1", + "chapter": chapter, + "blockId": block_id, + "startCodePoint": 0, + "endCodePoint": len(text), + }, + "text": text, + } + + +def _writer_context_input() -> dict[str, object]: + """构造可让 A/B/C 都通过 WriterContext v1 的脱敏夹具。""" + + recent = [ + _prose(chapter, 1000 + chapter, f"脱敏合同夹具第{chapter}章，人物保持冻结状态。") + for chapter in range(485, 489) + ] + supplemental = _prose(120, 1120, "脱敏补充片段，旧徽章曾经出现。") + return { + "contentMode": "sanitized_contract_fixture", + "sourceVersion": AUTHORIZATION["sourceVersion"], + "authorizationSnapshot": { + "snapshotId": "auth-work-8", + "allowedPurpose": "offline_evaluation", + "verifiedAt": "2026-07-20T00:00:00Z", + }, + "sourceStatus": "authorized", + "cardIndexVersion": "cards-frozen-488-v1", + "proseIndexVersion": "prose-frozen-488-v1", + "retrievalResult": { + "cards": [ + { + "name": "林澈", + "type": "character", + "sourceRefs": [copy.deepcopy(supplemental["sourceRef"])], + } + ], + "factEvidence": [], + "proseEvidence": [supplemental], + "indexHints": [ + { + "cardId": "card-1", + "name": "林澈", + "type": "character", + "content": "林澈在冻结点仍位于圣蒂曼", + "sourceId": "fixture:card:1", + "sourceVersion": "cards-frozen-488-v1", + "asOf": 488, + } + ], + "manifest": {"omittedSources": []}, + }, + "fineOutline": { + "sourceRef": { + "sourceId": "fixture:fine-outline:489", + "sourceVersion": "fine-outline-fixture-v1", + "chapter": 489, + }, + "hardConstraints": ["必须完成围攻突围"], + "adjustableBeats": [], + "declaredNewFacts": [], + "entities": [{"id": "character:林澈", "type": "character", "name": "林澈"}], + "relations": [], + "items": [{"id": "item:旧徽章", "type": "item", "name": "旧徽章"}], + "locations": [{"id": "location:圣蒂曼", "type": "location", "name": "圣蒂曼"}], + "powerSystems": [], + }, + "narrativeState": { + "time": "围攻当日", + "location": "圣蒂曼", + "characterPositions": {"林澈": "城内"}, + "immediateSituation": "围攻持续", + }, + "recentChapters": recent, + "outputContract": { + "targetChars": 4000, + "minChars": 3600, + "maxChars": 4400, + "frontmatterRequired": False, + "newSettingDeclarationRequired": True, + }, + "tokenBudget": {"maxContextChars": 50000}, + "generatedAt": "2026-07-20T00:00:00Z", + "requirements": { + "requiredEvents": [ + {"requirementId": "event-1", "anchors": ["完成围攻突围"]} + ], + "requiredCharacters": ["林澈"], + "foreshadowingActions": [ + {"requirementId": "foreshadow-1", "anchors": ["旧徽章"]} + ], + "chapterEndHook": { + "requirementId": "hook-1", + "anchors": ["城门忽然打开"], + "maxDistanceFromEnd": 20, + }, + "detectedNewSettings": [], + "frozenConflicts": [], + }, + } + + +def config() -> dict[str, object]: + """构造不含原文全文的单样本预注册配置。""" + + common = { + "workId": 8, + "asOfChapter": 488, + "targetChapter": 489, + "outlineSource": "outline:481-488", + "fineOutlineSource": "scaffold:489", + "targetChars": 4000, + "modelVersion": "opus", + "sampling": {"temperature": 0, "topP": 1, "seed": 489}, + "detectorProfile": "writer-detector-report-v1", + } + return { + "profile": "writer_replay", + "evaluationSetVersion": "writer-gate-a-test-v1", + "strategyVersion": "writer-abc-v1", + "referenceWork": { + "id": 8, + "title": "深空之影", + "version": AUTHORIZATION["sourceVersion"], + }, + "authorization": copy.deepcopy(AUTHORIZATION), + "commonControls": common, + "samples": [ + { + "sampleId": "deep-space-489", + "scenario": "combat", + "asOfChapter": 488, + "targetChapter": 489, + "snapshotVersion": "writer-deep-space-489-v1", + "frozenRecentHanCounts": [4600, 4600, 4800, 4800], + "targetLengthBasis": { + "algorithm": "calculate_target_chars", + "sourceChapters": [485, 486, 487, 488], + "hardEventCount": 1, + "foreshadowingActionCount": 0, + "requiredSceneCount": 0, + "minChars": 2000, + "maxChars": 10000, + "usesTargetChapterLength": False, + }, + "targetChars": 4000, + "expectedLength": { + "targetChars": 4000, + "minChars": 3600, + "maxChars": 4400, + }, + "newCharacterRatio": 0.0, + "newCharacterRatioStatus": "resolved", + "newCharacterBasis": { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 488, + "requiredCharacters": ["林澈"], + "knownBeforeAsOf": ["林澈"], + "absentBeforeAsOf": [], + "genericRoles": [], + }, + "sources": [ + { + "sourceId": "fixture:chapter:485-488", + "sourceVersion": AUTHORIZATION["sourceVersion"], + "chapterRange": "485-488", + }, + { + "sourceId": "fixture:card-index:488", + "sourceVersion": "cards-frozen-488-v1", + "chapterRange": "1-488", + }, + ], + "snapshotData": { + "chapters": [ + { + "chapter": 488, + "sourceId": "fixture:chapter:488", + "contentSha256": "sha256:" + "1" * 64, + } + ], + "cards": [], + }, + "writerContextInput": _writer_context_input(), + "leakageAudit": { + "method": "target-fact-hash-and-chapter-bound-audit", + "targetFacts": { + "targetChapter": 489, + "forbiddenFacts": [ + { + "id": "target-489-1", + "firstChapter": 489, + "text": "目标章专属秘密", + } + ], + }, + }, + } + ], + } + + +def _candidate_body(marker: str) -> str: + """构造满足动态篇幅和全部机械锚点的三臂候选。""" + + prefix = f"林澈{marker}完成围攻突围，又把旧徽章压回掌心。" + hook = "城门忽然打开" + filler_count = 4000 - han_count(prefix) - han_count(hook) + return prefix + "文" * filler_count + hook + + +def _writer_output(context: dict[str, object]) -> dict[str, object]: + """从 subprocess stdin 上下文构造严格绑定的 WriterOutput。""" + + marker = { + "historical_prose_only": "甲", + "card_index_only": "乙", + "card_index_plus_prose": "丙", + }[context["evidenceStrategy"]] + body = _candidate_body(marker) + candidate_hash = "sha256:" + hashlib.sha256(body.encode("utf-8")).hexdigest() + return { + "schemaVersion": "writer-output-v1", + "runId": context["runId"], + "attempt": context["attempt"], + "mode": context["mode"], + "qualityPolicyVersion": context["qualityPolicyVersion"], + "contextSnapshotId": context["contextSnapshot"]["manifestId"], + "contextSnapshotSha256": context["contextSnapshot"]["contextSha256"], + "candidateVersion": context["attempt"], + "candidateSha256": candidate_hash, + "acceptanceEligible": False, + "candidateBody": body, + "claimLedger": [], + "evidenceRequests": [], + "newSettingDeclarations": [], + "selfCheck": {"hardConstraintsCovered": True, "notes": []}, + } + + +class FakeSubprocessRunner: + """只替换 subprocess runner，候选仍由 run_writer() 解析和校验。""" + + def __init__(self, *, illegal_index_claim: bool = False): + self.calls: list[tuple[list[str], dict[str, object]]] = [] + self.contexts: list[dict[str, object]] = [] + self.illegal_index_claim = illegal_index_claim + + def __call__(self, command: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]: + context = json.loads(str(kwargs["input"])) + validate_writer_context(context) + self.calls.append((command, kwargs)) + self.contexts.append(context) + output = _writer_output(context) + if self.illegal_index_claim and context["evidenceStrategy"] == "card_index_only": + output["claimLedger"] = [ + { + "claimId": "claim-index-1", + "candidateSha256": output["candidateSha256"], + "startCodePoint": 0, + "endCodePoint": 2, + "factType": "character_state", + "factEvidenceId": "card-1", + "coverageState": "supported", + } + ] + stdout = json.dumps( + {"type": "result", "result": json.dumps(output, ensure_ascii=False)}, + ensure_ascii=False, + ) + return subprocess.CompletedProcess(command, 0, stdout=stdout, stderr="") + + +class FakeSemanticDetector: + """记录机械门输出，并返回与候选绑定的语义通过报告。""" + + def __init__(self) -> None: + self.calls: list[tuple[str, bool]] = [] + + def __call__(self, context, candidate, mechanical_report): + self.calls.append((context["evidenceStrategy"], mechanical_report["passed"])) + return { + "status": "passed", + "candidateVersion": candidate["candidateVersion"], + "candidateSha256": candidate["candidateSha256"], + "blockingFailures": [], + "suggestions": [], + } + + +def judge_report( + reviewer_id: str, + sample_id: str, + blind_id: str, + order: list[str], + score: float = 8.0, +) -> dict[str, object]: + """构造测试盲评报告。""" + + return { + "profile": RUBRIC_PROFILE, + "reviewerId": reviewer_id, + "sampleId": sample_id, + "blindCandidateId": blind_id, + "candidateOrder": order, + "scores": { + dimension: { + "score": score, + "evidence": [ + { + "sourceType": "judge_inference", + "sourceRef": f"candidate:{blind_id}", + "excerpt": f"证据-{dimension}", + } + ], + } + for dimension in DIMENSIONS + }, + } + + +class FakeJudge: + """模拟双评差异和必要第三评，不读取网络或真实模型。""" + + def __init__(self, *, unstable: bool = False): + self.calls: list[tuple[str, str]] = [] + self.unstable = unstable + + def __call__(self, blind_input, reviewer_id): + blind_id = blind_input["blindCandidateId"] + candidate_order = blind_input["candidateOrder"] + self.calls.append((reviewer_id, blind_id)) + score = 8.0 + if self.unstable and blind_id == "blind-1" and reviewer_id == "judge-2": + score = 7.0 + if self.unstable and blind_id == "blind-1" and reviewer_id == "judge-3": + score = 7.5 + return judge_report( + reviewer_id, + blind_input["sample"]["sampleId"], + blind_id, + candidate_order, + score, + ) + + +class MaliciousJudge(FakeJudge): + """主动扫描全部可见输入，证明 judge 无法发现真实臂或原始目录。""" + + def __init__(self) -> None: + super().__init__() + self.visible_inputs: list[dict[str, object]] = [] + + def __call__(self, blind_input, reviewer_id): + rendered = json.dumps(blind_input, ensure_ascii=False, sort_keys=True) + forbidden = ( + "candidate-A", + "candidate-B", + "candidate-C", + "pipeline-A", + "pipeline-B", + "pipeline-C", + "evidenceStrategy", + "writerContextInput", + "indexHints", + "retrievalManifest", + "evidenceCoverage", + "raw_dir", + "rawDir", + "_contexts", + ) + for marker in forbidden: + if marker in rendered: + raise AssertionError(f"judge 可见输入泄露: {marker}") + self.visible_inputs.append(copy.deepcopy(blind_input)) + return super().__call__(blind_input, reviewer_id) + + +def _test_adapters( + *, unstable: bool = False, illegal_index_claim: bool = False +) -> tuple[WriterReplayTestAdapters, FakeSubprocessRunner, FakeSemanticDetector, FakeJudge]: + """集中构造测试注入对象，避免它们被误认为生产 runtime。""" + + runner = FakeSubprocessRunner(illegal_index_claim=illegal_index_claim) + detector = FakeSemanticDetector() + judge = FakeJudge(unstable=unstable) + return ( + WriterReplayTestAdapters(runner, detector, judge), + runner, + detector, + judge, + ) + + +class WriterReplayDryRunTest(unittest.TestCase): + """验证 dry-run 的合同、冻结、安全和控制变量。""" + + def test_three_arms_validate_full_context_and_only_change_evidence_strategy(self): + result = run_writer_replay(config(), run_id="dry-1") + + self.assertEqual(result["status"], "ready") + sample = result["samples"][0] + arms = sample["arms"] + self.assertEqual(set(arms), {"A", "B", "C"}) + self.assertEqual(arms["A"]["evidenceStrategy"], "historical_prose_only") + self.assertEqual(arms["B"]["evidenceStrategy"], "card_index_only") + self.assertEqual(arms["C"]["evidenceStrategy"], "card_index_plus_prose") + self.assertEqual(len({arm["commonControlsSha256"] for arm in arms.values()}), 1) + self.assertTrue(all(not arm["acceptanceEligible"] for arm in arms.values())) + self.assertEqual(arms["A"]["contextSummary"]["recentBaselineChapters"], [485, 486, 487, 488]) + self.assertEqual(arms["B"]["contextSummary"]["proseEvidenceCount"], 0) + self.assertEqual(arms["B"]["contextSummary"]["indexHintCount"], 1) + self.assertEqual(arms["C"]["contextSummary"]["recentBaselineChapters"], [485, 486, 487, 488]) + self.assertEqual(arms["C"]["contextSummary"]["indexHintCount"], 1) + self.assertTrue( + all( + arm["contextSummary"]["contentMode"] == "sanitized_contract_fixture" + for arm in arms.values() + ) + ) + rendered = json.dumps(result, ensure_ascii=False) + self.assertNotIn("脱敏合同夹具", rendered) + self.assertNotIn("林澈在冻结点仍位于圣蒂曼", rendered) + + def test_future_index_hint_invalidates_whole_sample(self): + leaked = config() + leaked["samples"][0]["writerContextInput"]["retrievalResult"]["indexHints"][0]["asOf"] = 489 + + result = run_writer_replay(leaked, run_id="dry-leak") + + self.assertFalse(result["ok"]) + self.assertEqual(result["status"], "invalid_leakage") + + def test_invalid_b_arm_contract_cannot_be_reported_ready(self): + invalid = config() + invalid["samples"][0]["writerContextInput"]["retrievalResult"]["indexHints"] = [] + + result = run_writer_replay(invalid, run_id="dry-invalid-context") + + self.assertFalse(result["ok"]) + self.assertEqual(result["status"], "invalid_writer_context") + + def test_target_length_is_mechanically_recomputed_and_target_chapter_length_is_forbidden(self): + invalid = config() + invalid["samples"][0]["targetChars"] = 4100 + + with self.assertRaisesRegex(WriterReplayError, "机械复算"): + run_writer_replay(invalid, run_id="dry-length-tampered") + + leaked = config() + leaked["samples"][0]["targetLengthBasis"]["usesTargetChapterLength"] = True + with self.assertRaisesRegex(WriterReplayError, "禁止读取目标章"): + run_writer_replay(leaked, run_id="dry-length-leaked") + + def test_unresolved_generic_role_must_remain_null_instead_of_defaulting_to_zero(self): + unresolved = config() + sample = unresolved["samples"][0] + sample["writerContextInput"]["requirements"]["requiredCharacters"] = ["内应"] + sample["newCharacterRatio"] = None + sample["newCharacterRatioStatus"] = "unresolved_generic_role" + sample["newCharacterBasis"] = { + "definition": "named_required_characters_absent_before_as_of_ratio", + "asOfChapter": 488, + "requiredCharacters": ["内应"], + "knownBeforeAsOf": [], + "absentBeforeAsOf": [], + "genericRoles": ["内应"], + } + self.assertTrue(run_writer_replay(unresolved, run_id="dry-generic-role")["ok"]) + + unresolved["samples"][0]["newCharacterRatio"] = 0 + with self.assertRaisesRegex(WriterReplayError, "必须为 null"): + run_writer_replay(unresolved, run_id="dry-generic-role-zero") + + def test_character_declared_absent_before_freeze_cannot_have_card_index_hint(self): + inconsistent = config() + sample = inconsistent["samples"][0] + sample["newCharacterRatio"] = 1.0 + sample["newCharacterBasis"]["knownBeforeAsOf"] = [] + sample["newCharacterBasis"]["absentBeforeAsOf"] = ["林澈"] + + with self.assertRaisesRegex(WriterReplayError, "不得同时出现在卡索引"): + run_writer_replay(inconsistent, run_id="dry-absent-card-conflict") + + def test_gate_a_preregistration_mechanically_recomputes_all_lengths_and_ratios(self): + gate_config_path = SCRIPT_DIR.parent / "configs" / "writer-gate-a-deep-space-v1.json" + gate_config = json.loads(gate_config_path.read_text(encoding="utf-8")) + expected_targets = {489: 7500, 321: 7600, 544: 6700, 199: 2000, 523: 6100} + expected_ratios = {489: 1.0, 321: 0.0, 544: None, 199: 0.0, 523: 0.0} + + for sample in gate_config["samples"]: + basis = sample["targetLengthBasis"] + recalculated = calculate_target_chars( + recent_chapter_han_counts=sample["frozenRecentHanCounts"], + hard_event_count=basis["hardEventCount"], + foreshadowing_action_count=basis["foreshadowingActionCount"], + required_scene_count=basis["requiredSceneCount"], + min_chars=basis["minChars"], + max_chars=basis["maxChars"], + ) + target_chapter = sample["targetChapter"] + self.assertEqual(recalculated, expected_targets[target_chapter]) + self.assertEqual(sample["targetChars"], recalculated) + self.assertEqual(sample["newCharacterRatio"], expected_ratios[target_chapter]) + if target_chapter == 544: + self.assertEqual(sample["newCharacterRatioStatus"], "unresolved_generic_role") + else: + self.assertEqual(sample["newCharacterRatioStatus"], "resolved") + + result = run_writer_replay(gate_config, run_id="gate-a-preregistered-dry-run") + self.assertTrue(result["ok"]) + self.assertEqual(len(result["samples"]), 5) + + +class WriterReplayExecuteBoundaryTest(unittest.TestCase): + """验证真实执行失败关闭和测试注入仍经过正式管线。""" + + def test_execute_requires_private_tmp_output(self): + with tempfile.TemporaryDirectory() as directory: + with self.assertRaisesRegex(WriterReplayError, "/private/tmp"): + run_writer_replay( + config(), + run_id="real-bad-path", + output_dir=pathlib.Path(directory), + execute=True, + ) + + def test_execute_without_production_adapters_fails_closed(self): + with tempfile.TemporaryDirectory(dir="/private/tmp") as directory: + with self.assertRaisesRegex(WriterReplayError, "真实 execute 未接线"): + run_writer_replay( + config(), + run_id="real-not-wired", + output_dir=pathlib.Path(directory), + execute=True, + ) + + def test_context_authorization_cannot_swap_snapshot_or_source_before_runner(self): + """上下文版本、快照、用途、时间和状态都必须绑定顶层授权。""" + + mutations = { + "source_version": lambda value: value["samples"][0]["writerContextInput"].update( + {"sourceVersion": "swapped-source-v2"} + ), + "snapshot_id": lambda value: value["samples"][0]["writerContextInput"][ + "authorizationSnapshot" + ].update({"snapshotId": "auth-swapped"}), + "purpose": lambda value: value["samples"][0]["writerContextInput"][ + "authorizationSnapshot" + ].update({"allowedPurpose": "diagnostic"}), + "verified_at": lambda value: value["samples"][0]["writerContextInput"][ + "authorizationSnapshot" + ].update({"verifiedAt": "2026-07-21T00:00:00Z"}), + "source_status": lambda value: value["samples"][0]["writerContextInput"].update( + {"sourceStatus": "revoked"} + ), + } + for name, mutate in mutations.items(): + invalid = config() + mutate(invalid) + adapters, runner, _detector, _judge = _test_adapters() + with self.subTest(name=name), tempfile.TemporaryDirectory( + dir="/private/tmp" + ) as directory: + result = run_writer_replay( + invalid, + run_id=f"auth-binding-{name}", + output_dir=pathlib.Path(directory) / "run", + execute=True, + test_adapters=adapters, + ) + self.assertFalse(result["ok"]) + self.assertEqual(result["status"], "blocked_authorization") + self.assertEqual(runner.calls, []) + + def test_test_injection_runs_writer_pipeline_mechanical_and_semantic_detector(self): + adapters, runner, detector, judge = _test_adapters(unstable=True) + with tempfile.TemporaryDirectory(dir="/private/tmp") as directory: + output_dir = pathlib.Path(directory) / "run" + result = run_writer_replay( + config(), + run_id="test-pipeline", + output_dir=output_dir, + execute=True, + test_adapters=adapters, + ) + raw_candidates = sorted((output_dir / "raw" / "deep-space-489").glob("candidate-*.json")) + self.assertEqual(len(raw_candidates), 3) + self.assertIn("candidateBody", raw_candidates[0].read_text(encoding="utf-8")) + public_manifest = (output_dir / "manifest.json").read_text(encoding="utf-8") + self.assertNotIn("candidateBody", public_manifest) + self.assertNotIn("脱敏合同夹具", public_manifest) + + self.assertEqual(result["status"], "completed_test_pipeline") + self.assertEqual([item["evidenceStrategy"] for item in runner.contexts], [ + "historical_prose_only", + "card_index_only", + "card_index_plus_prose", + ]) + self.assertTrue(all("--agent" in command for command, _kwargs in runner.calls)) + self.assertEqual(detector.calls, [ + ("historical_prose_only", True), + ("card_index_only", True), + ("card_index_plus_prose", True), + ]) + self.assertEqual(len([call for call in judge.calls if call[0] == "judge-3"]), 1) + candidates = result["samples"][0]["candidates"] + self.assertTrue(all(not candidate["acceptanceEligible"] for candidate in candidates.values())) + + def test_index_hint_cannot_be_claim_ledger_evidence(self): + adapters, _runner, detector, _judge = _test_adapters(illegal_index_claim=True) + with tempfile.TemporaryDirectory(dir="/private/tmp") as directory: + with self.assertRaisesRegex(WriterReplayError, "claim_evidence_reference_invalid"): + run_writer_replay( + config(), + run_id="test-index-claim", + output_dir=pathlib.Path(directory) / "run", + execute=True, + test_adapters=adapters, + ) + # B 臂在 writer adapter 已失败，不能进入 semantic detector。 + self.assertEqual(detector.calls, [("historical_prose_only", True)]) + + def test_malicious_judge_only_sees_blind_candidates_without_raw_paths_or_arm_contexts(self): + runner = FakeSubprocessRunner() + detector = FakeSemanticDetector() + judge = MaliciousJudge() + adapters = WriterReplayTestAdapters(runner, detector, judge) + with tempfile.TemporaryDirectory(dir="/private/tmp") as directory: + output_dir = pathlib.Path(directory) / "run" + result = run_writer_replay( + config(), + run_id="blind-isolation", + output_dir=output_dir, + execute=True, + test_adapters=adapters, + ) + + self.assertTrue(result["ok"]) + self.assertTrue(judge.visible_inputs) + shared_references = [] + for visible in judge.visible_inputs: + self.assertEqual( + set(visible), + { + "schemaVersion", + "sample", + "blindCandidateId", + "candidateOrder", + "sharedEvaluationReference", + "candidates", + }, + ) + self.assertEqual(set(visible["candidateOrder"]), {"blind-1", "blind-2", "blind-3"}) + shared = visible["sharedEvaluationReference"] + shared_references.append(copy.deepcopy(shared)) + self.assertEqual(shared["fineOutline"]["hardConstraints"], ["必须完成围攻突围"]) + self.assertEqual(shared["requirements"]["requiredCharacters"], ["林澈"]) + self.assertEqual( + shared["requirements"]["chapterEndHook"]["anchors"], + ["城门忽然打开"], + ) + self.assertEqual( + [item["chapter"] for item in shared["historicalProseBaseline"]], + [485, 486, 487, 488], + ) + self.assertTrue( + all( + set(candidate) + == {"blindCandidateId", "candidateSha256", "candidateBody"} + and candidate["blindCandidateId"].startswith("blind-") + for candidate in visible["candidates"] + ) + ) + self.assertTrue( + all(reference == shared_references[0] for reference in shared_references[1:]), + "评委顺序变化不得改变共同评测参考", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/replay-eval/scripts/test_writer_gate.py b/.claude/skills/replay-eval/scripts/test_writer_gate.py new file mode 100644 index 0000000..9337564 --- /dev/null +++ b/.claude/skills/replay-eval/scripts/test_writer_gate.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +"""正文 Gate A/B 逐样本唯一判定器的表驱动测试。""" + +from __future__ import annotations + +import copy +import pathlib +import sys +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +from writer_gate import ( # noqa: E402 + CONFOUND_LABELS, + REQUIRED_SCENARIOS, + decide_gate_a, + decide_gate_b, +) +from writer_rubric import DIMENSIONS # noqa: E402 + + +SCENARIOS = tuple(sorted(REQUIRED_SCENARIOS)) + + +def scores( + *, + b_fidelity_delta: float = 0.0, + c_fidelity_delta: float = 0.5, +) -> dict[str, dict[str, float]]: + """构造三臂五维终分，分别控制 B-A 与 C-A 的保真增量。""" + + a_scores = {dimension: 7.0 for dimension in DIMENSIONS} + b_scores = dict(a_scores) + c_scores = dict(a_scores) + b_scores["setting_entity_fidelity"] += b_fidelity_delta + c_scores["setting_entity_fidelity"] += c_fidelity_delta + return {"A": a_scores, "B": b_scores, "C": c_scores} + + +def sample( + number: int, + *, + work_id: int = 8, + scenario: str | None = None, + **overrides, +) -> dict[str, object]: + """构造合法的逐样本脱敏结果。""" + + result: dict[str, object] = { + "sampleId": f"{work_id}-{number}", + "workId": work_id, + "scenario": scenario or SCENARIOS[(number - 1) % len(SCENARIOS)], + "schemaValid": True, + "futureLeakage": False, + "systemFailure": False, + "reviewStatus": "stable_report", + "cArm": { + "hardConstraintCoverage": 1.0, + "highSeverityResidualCount": 0, + }, + "scores": scores(), + "confounders": {category: [] for category in CONFOUND_LABELS}, + } + result.update(overrides) + return result + + +def gate_a_input(samples: list[dict[str, object]], **forged) -> dict[str, object]: + """构造 Gate A 输入；forged 用于证明顶层聚合值无效。""" + + return {"gate": "A", "samples": samples, **forged} + + +def gate_b_input(samples: list[dict[str, object]], **overrides) -> dict[str, object]: + """构造 Gate B 输入。""" + + return {"gate": "B", "gateAStatus": "passed", "samples": samples, **overrides} + + +def ten_samples() -> list[dict[str, object]]: + """构造两书各五章并覆盖五类场景的 Gate B 基线。""" + + return [ + sample(index + 1, work_id=work_id, scenario=SCENARIOS[index]) + for work_id in (8, 4) + for index in range(5) + ] + + +class GateATest(unittest.TestCase): + """验证 Gate A 的逐样本统计与唯一短路顺序。""" + + def test_table_driven_terminal_priority(self): + base = [sample(index) for index in range(1, 6)] + cases = [] + + insufficient = copy.deepcopy(base[:4]) + insufficient[0]["futureLeakage"] = True + cases.append(("不足优先于泄露", insufficient, "insufficient_evidence")) + + for label, field, value in ( + ("schema 非法", "schemaValid", False), + ("未来泄露", "futureLeakage", True), + ("系统失败", "systemFailure", True), + ): + samples = copy.deepcopy(base) + samples.append(sample(6, **{field: value})) + cases.append((label, samples, "failed")) + + high_severity = copy.deepcopy(base) + high_severity[0]["cArm"]["highSeverityResidualCount"] = 1 + cases.append(("C 高严重度", high_severity, "failed")) + + low_coverage = copy.deepcopy(base) + low_coverage[0]["cArm"]["hardConstraintCoverage"] = 0.99 + cases.append(("覆盖不足", low_coverage, "failed")) + cases.append(("全部通过", base, "passed")) + + for label, samples, expected in cases: + with self.subTest(label=label): + self.assertEqual(decide_gate_a(gate_a_input(samples))["status"], expected) + + def test_pre_c_arm_failures_do_not_require_c_arm_results(self): + """前置机械或系统失败没有 C 臂结果时仍按唯一顺序裁决。""" + + base = [sample(index) for index in range(1, 6)] + cases = [] + for label, field, value in ( + ("schema 非法", "schemaValid", False), + ("未来泄露", "futureLeakage", True), + ("系统失败", "systemFailure", True), + ): + failed_sample = sample(6, **{field: value}) + failed_sample.pop("cArm") + cases.append((f"五有效加{label}", [*copy.deepcopy(base), failed_sample], "failed")) + cases.append((f"四有效加{label}", [*copy.deepcopy(base[:4]), failed_sample], "insufficient_evidence")) + + for label, samples, expected in cases: + with self.subTest(label=label): + self.assertEqual(decide_gate_a(gate_a_input(samples))["status"], expected) + + def test_forged_upstream_aggregates_cannot_change_result(self): + forged = gate_a_input( + [sample(index) for index in range(1, 5)], + validSampleCount=999, + invalidSchemaCount=0, + futureLeakageCount=0, + cArmHardConstraintCoverage=1.0, + ) + report = decide_gate_a(forged) + self.assertEqual(report["status"], "insufficient_evidence") + self.assertEqual(report["metrics"]["validSampleCount"], 4) + + def test_valid_sample_boundary_counts_each_sample_once(self): + """四个有效样本仍不足，补到五个后才进入硬门判断。""" + + four_valid = [sample(index) for index in range(1, 5)] + multi_failed = sample( + 5, + schemaValid=False, + futureLeakage=True, + systemFailure=True, + reviewStatus="invalid_unstable", + ) + multi_failed.pop("cArm") + report = decide_gate_a(gate_a_input([*four_valid, multi_failed])) + self.assertEqual(report["status"], "insufficient_evidence") + self.assertEqual(report["metrics"]["sampleCount"], 5) + self.assertEqual(report["metrics"]["validSampleCount"], 4) + + report = decide_gate_a( + gate_a_input([*four_valid, sample(6), multi_failed]) + ) + self.assertEqual(report["status"], "failed") + self.assertEqual(report["metrics"]["validSampleCount"], 5) + + def test_five_confounder_categories_are_aggregated_per_sample(self): + samples = [sample(index) for index in range(1, 7)] + categories = list(CONFOUND_LABELS) + for index, category in enumerate(categories): + samples[index]["confounders"][category] = [{"code": f"case-{index}"}] + samples[5]["futureLeakage"] = True + samples[4]["reviewStatus"] = "invalid_unstable" + + report = decide_gate_a( + gate_a_input(samples, confounds={"falseNegatives": ["伪造聚合"]}) + ) + self.assertEqual(set(report["confounds"]), set(CONFOUND_LABELS)) + self.assertNotIn("伪造聚合", str(report["confounds"])) + for index, category in enumerate(categories): + self.assertTrue( + any(item["sampleId"] == f"8-{index + 1}" for item in report["confounds"][category]) + ) + self.assertTrue( + any(item["sampleId"] == "8-6" for item in report["confounds"]["leakage"]) + ) + + +class GateBTest(unittest.TestCase): + """验证 Gate B 的充分性、退化与增益短路顺序。""" + + def test_gate_a_terminal_has_highest_priority(self): + insufficient = [sample(index) for index in range(1, 5)] + report = decide_gate_b(gate_b_input(insufficient, gateAStatus="passed")) + self.assertEqual(report["status"], "insufficient_evidence") + self.assertEqual(report["metrics"]["recomputedGateAStatus"], "insufficient_evidence") + + failed = [sample(index) for index in range(1, 6)] + failed.append(sample(6, futureLeakage=True)) + report = decide_gate_b(gate_b_input(failed, gateAStatus="passed")) + self.assertEqual(report["status"], "failed") + self.assertEqual(report["metrics"]["recomputedGateAStatus"], "failed") + + def test_forged_gate_a_status_cannot_override_recomputed_result(self): + """顶层 Gate A 终态只是非可信输入，不能放行或阻断同批样本。""" + + passed = decide_gate_b(gate_b_input(ten_samples(), gateAStatus="failed")) + self.assertEqual(passed["status"], "passed") + self.assertEqual(passed["metrics"]["recomputedGateAStatus"], "passed") + + insufficient = decide_gate_b( + gate_b_input([sample(index) for index in range(1, 5)], gateAStatus="passed") + ) + self.assertEqual(insufficient["status"], "insufficient_evidence") + + def test_gate_a_short_circuit_keeps_sample_confounds_without_changing_status(self): + """Gate A 终态优先，但合法逐样本混淆项仍应进入 Gate B 报告。""" + + valid = [sample(index) for index in range(1, 6)] + leaked = sample(6, futureLeakage=True) + leaked.pop("cArm") + leaked["confounders"]["falsePositives"] = ["detector 误报候选"] + report = decide_gate_b( + { + "gateAStatus": "failed", + "samples": [*valid, leaked], + "confounds": {"falsePositives": ["伪造聚合"]}, + } + ) + self.assertEqual(report["status"], "failed") + self.assertIn("detector 误报候选", str(report["confounds"]["falsePositives"])) + self.assertTrue(report["confounds"]["leakage"]) + self.assertNotIn("伪造聚合", str(report["confounds"])) + + def test_sample_sufficiency_precedes_quality(self): + samples = ten_samples()[:-1] + for item in samples[:3]: + item["scores"]["C"]["style_consistency"] = 6.0 + self.assertEqual(decide_gate_b(gate_b_input(samples))["status"], "insufficient_evidence") + + unstable = ten_samples() + for index in range(3): + unstable[index]["reviewStatus"] = "invalid_unstable" + unstable[index].pop("scores") + self.assertEqual( + decide_gate_b(gate_b_input(unstable))["status"], "insufficient_evidence" + ) + + def test_missing_scene_or_single_work_is_insufficient(self): + missing_scene = ten_samples() + for item in missing_scene: + if item["scenario"] == "returning_character": + item["scenario"] = "battle" + self.assertEqual( + decide_gate_b(gate_b_input(missing_scene))["status"], + "insufficient_evidence", + ) + one_work = [sample(index) for index in range(1, 11)] + self.assertEqual( + decide_gate_b(gate_b_input(one_work))["status"], "insufficient_evidence" + ) + + def test_hard_failure_and_quality_regression_fail(self): + low_coverage = ten_samples() + low_coverage[0]["cArm"]["hardConstraintCoverage"] = 0.9 + self.assertEqual(decide_gate_b(gate_b_input(low_coverage))["status"], "failed") + + decline = ten_samples() + for item in decline[:3]: + item["scores"]["C"]["style_consistency"] = 6.0 + report = decide_gate_b( + gate_b_input( + decline, + anyDimensionDeclineOverHalfRatio=0.0, + averageDeltas={"setting_entity_fidelity": 999}, + ) + ) + self.assertEqual(report["status"], "failed") + self.assertEqual(report["metrics"]["anyDimensionDeclineOverHalfRatio"], 0.3) + + def test_gain_threshold_distinguishes_passed_and_no_gain(self): + self.assertEqual(decide_gate_b(gate_b_input(ten_samples()))["status"], "passed") + no_gain = ten_samples() + for item in no_gain: + item["scores"] = scores(c_fidelity_delta=0.0) + self.assertEqual(decide_gate_b(gate_b_input(no_gain))["status"], "no_gain") + + def test_stable_sample_requires_complete_a_b_c_scores(self): + missing_b = ten_samples() + missing_b[0]["scores"].pop("B") + with self.assertRaisesRegex(ValueError, "精确包含 A/B/C"): + decide_gate_b(gate_b_input(missing_b)) + + def test_b_minus_a_is_diagnostic_and_c_minus_a_drives_gate(self): + """即使 B-A 明显退化，只要 C-A 达标，正式 Gate B 仍按 C-A 通过。""" + + samples = ten_samples() + for item in samples: + item["scores"] = scores( + b_fidelity_delta=-2.0, + c_fidelity_delta=0.5, + ) + report = decide_gate_b(gate_b_input(samples)) + self.assertEqual(report["status"], "passed") + self.assertEqual( + report["metrics"]["averageDeltas"]["B-A"]["setting_entity_fidelity"], + -2.0, + ) + self.assertEqual( + report["metrics"]["averageDeltas"]["C-A"]["setting_entity_fidelity"], + 0.5, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/replay-eval/scripts/test_writer_rubric.py b/.claude/skills/replay-eval/scripts/test_writer_rubric.py new file mode 100644 index 0000000..6d472c7 --- /dev/null +++ b/.claude/skills/replay-eval/scripts/test_writer_rubric.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""正文五维盲评量表的离线回归测试。""" + +from __future__ import annotations + +import pathlib +import sys +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +from writer_rubric import ( # noqa: E402 + DIMENSIONS, + RUBRIC_PROFILE, + adjudicate_reviews, + deblind_reports, + validate_blind_pair, + validate_report, +) + + +def scorecard(score: float = 8.0) -> dict[str, dict[str, object]]: + """构造包含四类证据归因的合法五维评分。""" + + source_types = ("fine_outline", "historical_prose", "judge_inference") + return { + dimension: { + "score": score, + "evidence": [ + { + "sourceType": source_types[index % len(source_types)], + "sourceRef": f"source:{dimension}", + "excerpt": f"证据-{dimension}", + } + ], + } + for index, dimension in enumerate(DIMENSIONS) + } + + +def report( + reviewer_id: str, + scores: dict[str, dict[str, object]] | None = None, + *, + order: tuple[str, ...] = ("blind-1", "blind-2", "blind-3"), +) -> dict[str, object]: + """构造不携带真实臂名的单评委报告。""" + + return { + "profile": RUBRIC_PROFILE, + "reviewerId": reviewer_id, + "sampleId": "deep-space-489", + "blindCandidateId": "blind-1", + "candidateOrder": list(order), + "scores": scores or scorecard(), + } + + +class WriterRubricValidationTest(unittest.TestCase): + """验证量表结构与盲评纪律。""" + + def test_five_dimensions_use_zero_to_ten_half_steps(self): + self.assertEqual(len(DIMENSIONS), 5) + self.assertEqual(validate_report(report("judge-1", scorecard(0.0))), []) + self.assertEqual(validate_report(report("judge-1", scorecard(10.0))), []) + + bad = scorecard(8.0) + bad[DIMENSIONS[0]]["score"] = 8.25 + self.assertTrue(any("0.5" in error for error in validate_report(report("judge-1", bad)))) + + def test_each_dimension_requires_typed_evidence(self): + bad = scorecard() + bad[DIMENSIONS[2]]["evidence"] = [] + self.assertTrue(any("证据" in error for error in validate_report(report("judge-1", bad)))) + + bad = scorecard() + bad[DIMENSIONS[3]]["evidence"][0]["sourceType"] = "target_answer" + self.assertTrue(any("来源类型" in error for error in validate_report(report("judge-1", bad)))) + + hidden_card = scorecard() + hidden_card[DIMENSIONS[0]]["evidence"][0]["sourceType"] = "card_index" + self.assertTrue( + any("来源类型" in error for error in validate_report(report("judge-1", hidden_card))) + ) + + def test_pair_must_be_independent_blind_and_reverse_ordered(self): + first = report("judge-1") + second = report("judge-2", order=("blind-3", "blind-2", "blind-1")) + self.assertEqual(validate_blind_pair(first, second), []) + + same_order = report("judge-2") + self.assertTrue(any("反序" in error for error in validate_blind_pair(first, same_order))) + + different_candidate = report( + "judge-2", order=("blind-3", "blind-2", "blind-1") + ) + different_candidate["blindCandidateId"] = "blind-2" + self.assertTrue( + any("同一盲化候选" in error for error in validate_blind_pair(first, different_candidate)) + ) + + leaked = report("judge-2", order=("blind-3", "blind-2", "blind-1")) + leaked["arm"] = "C" + self.assertTrue(any("去盲" in error for error in validate_blind_pair(first, leaked))) + + def test_deblind_uses_preregistered_mapping_not_position(self): + reports = [ + report("judge-1"), + {**report("judge-1"), "blindCandidateId": "blind-3"}, + ] + mapping = {"blind-1": "C", "blind-2": "A", "blind-3": "B"} + + deblinded = deblind_reports(reports, mapping) + self.assertEqual([item["arm"] for item in deblinded], ["C", "B"]) + + +class WriterRubricAdjudicationTest(unittest.TestCase): + """验证双评稳定性与最多一次第三评委仲裁。""" + + def test_gap_above_half_requires_third_reviewer_once(self): + first = report("judge-1", scorecard(8.0)) + second_scores = scorecard(8.0) + second_scores[DIMENSIONS[0]]["score"] = 7.0 + second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1")) + + result = adjudicate_reviews(first, second) + self.assertEqual(result["status"], "needs_third_reviewer") + self.assertEqual(result["unstableDimensions"], [DIMENSIONS[0]]) + + def test_third_reviewer_uses_median_when_any_stable_pair_exists(self): + first = report("judge-1", scorecard(8.0)) + second_scores = scorecard(8.0) + second_scores[DIMENSIONS[1]]["score"] = 6.5 + second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1")) + third_scores = scorecard(8.0) + third_scores[DIMENSIONS[1]]["score"] = 7.5 + third = report("judge-3", third_scores) + + result = adjudicate_reviews(first, second, third) + self.assertEqual(result["status"], "adjudicated_report") + self.assertEqual(result["scores"][DIMENSIONS[1]], 7.5) + + def test_third_reviewer_must_bind_same_candidate_and_candidate_set(self): + """第三评不得用另一候选或漂移后的候选集合参与仲裁。""" + + first = report("judge-1", scorecard(8.0)) + second_scores = scorecard(8.0) + second_scores[DIMENSIONS[1]]["score"] = 6.5 + second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1")) + + different_candidate = report("judge-3", scorecard(7.5)) + different_candidate["blindCandidateId"] = "blind-2" + candidate_result = adjudicate_reviews(first, second, different_candidate) + self.assertEqual(candidate_result["status"], "invalid_report") + self.assertTrue(any("同一盲化候选" in error for error in candidate_result["errors"])) + + different_set = report( + "judge-3", + scorecard(7.5), + order=("blind-1", "blind-2", "blind-4"), + ) + set_result = adjudicate_reviews(first, second, different_set) + self.assertEqual(set_result["status"], "invalid_report") + self.assertTrue(any("相同的盲化候选集合" in error for error in set_result["errors"])) + + def test_no_stable_pair_marks_sample_invalid(self): + first = report("judge-1", scorecard(8.0)) + second_scores = scorecard(8.0) + second_scores[DIMENSIONS[4]]["score"] = 6.5 + second = report("judge-2", second_scores, order=("blind-3", "blind-2", "blind-1")) + third_scores = scorecard(8.0) + third_scores[DIMENSIONS[4]]["score"] = 9.5 + third = report("judge-3", third_scores) + + result = adjudicate_reviews(first, second, third) + self.assertEqual(result["status"], "invalid_unstable") + self.assertEqual(result["unstableDimensions"], [DIMENSIONS[4]]) + self.assertNotIn("winner", result) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/replay-eval/scripts/writer_gate.py b/.claude/skills/replay-eval/scripts/writer_gate.py new file mode 100644 index 0000000..b0c2a78 --- /dev/null +++ b/.claude/skills/replay-eval/scripts/writer_gate.py @@ -0,0 +1,586 @@ +#!/usr/bin/env python3 +"""正文智能体 Gate A/B 的确定性唯一判定器。 + +判定器只消费逐样本脱敏结果，并在可信边界内计算全部统计值。顶层传入的 +样本数、平均分、覆盖率或混淆项等聚合字段一律不参与裁决，避免上游通过 +伪造汇总改变终态。命中较高优先级终态后不再使用较低优先级条件改写结果。 +""" + +from __future__ import annotations + +import argparse +import json +import math +import sys +from pathlib import Path +from typing import Any, Mapping, Sequence + +from writer_rubric import DIMENSIONS + +REQUIRED_SCENARIOS = frozenset( + { + "battle", + "character_dialogue", + "turning_point", + "information_reveal", + "returning_character", + } +) + +CONFOUND_LABELS = { + "falseNegatives": "假阴", + "falsePositives": "假阳", + "leakage": "泄露", + "reviewerInstability": "评委不稳定", + "newCharactersWithoutCards": "新角色无卡", +} + +STABLE_REVIEW_STATUSES = frozenset({"stable_report", "adjudicated_report"}) +REVIEW_STATUSES = frozenset({*STABLE_REVIEW_STATUSES, "invalid_unstable"}) + + +def _require_mapping(value: object, field: str) -> Mapping[str, Any]: + """取得必需对象字段，避免缺失数据被静默当成空对象。""" + + if not isinstance(value, Mapping): + raise ValueError(f"{field} 必须是对象") + return value + + +def _require_samples(value: object) -> list[Mapping[str, Any]]: + """取得非空逐样本数组，Gate 不接受只有聚合值的输入。""" + + if not isinstance(value, list) or not value: + raise ValueError("samples 必须是非空逐样本数组") + if any(not isinstance(item, Mapping) for item in value): + raise ValueError("samples 每项必须是对象") + return list(value) + + +def _require_integer(value: object, field: str) -> int: + """取得非负整数计数。""" + + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError(f"{field} 必须是非负整数") + return value + + +def _require_number(value: object, field: str) -> float: + """取得有限数值，显式拒绝布尔值、NaN 和无穷值。""" + + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ValueError(f"{field} 必须是有限数值") + result = float(value) + if not math.isfinite(result): + raise ValueError(f"{field} 必须是有限数值") + return result + + +def _require_ratio(value: object, field: str) -> float: + """取得闭区间 0-1 内的比例。""" + + result = _require_number(value, field) + if not 0.0 <= result <= 1.0: + raise ValueError(f"{field} 必须在 0-1 之间") + return result + + +def _require_boolean(value: object, field: str) -> bool: + """取得严格布尔值，拒绝 0/1 等隐式真假值。""" + + if not isinstance(value, bool): + raise ValueError(f"{field} 必须是布尔值") + return value + + +def _sample_identity(sample: Mapping[str, Any], index: int) -> tuple[str, str, str]: + """校验并返回样本 ID、作品 ID 和场景分类。""" + + sample_id = sample.get("sampleId") + if not isinstance(sample_id, str) or not sample_id.strip(): + raise ValueError(f"samples[{index}].sampleId 必须是非空字符串") + work_id = sample.get("workId") + if isinstance(work_id, bool) or not isinstance(work_id, (int, str)) or not str(work_id).strip(): + raise ValueError(f"samples[{index}].workId 必须是非空整数或字符串") + scenario = sample.get("scenario") + if scenario not in REQUIRED_SCENARIOS: + raise ValueError(f"samples[{index}].scenario 未登记") + return sample_id, str(work_id), str(scenario) + + +def _sample_status(sample: Mapping[str, Any], index: int) -> dict[str, Any]: + """从单样本机械结果推导有效性，不接受上游 valid 标记。""" + + schema_valid = _require_boolean(sample.get("schemaValid"), f"samples[{index}].schemaValid") + future_leakage = _require_boolean( + sample.get("futureLeakage"), f"samples[{index}].futureLeakage" + ) + system_failure = _require_boolean( + sample.get("systemFailure"), f"samples[{index}].systemFailure" + ) + review_status = sample.get("reviewStatus") + if review_status not in REVIEW_STATUSES: + raise ValueError( + f"samples[{index}].reviewStatus 必须是 stable_report、" + "adjudicated_report 或 invalid_unstable" + ) + return { + "schemaValid": schema_valid, + "futureLeakage": future_leakage, + "systemFailure": system_failure, + "reviewStatus": review_status, + "valid": ( + schema_valid + and not future_leakage + and not system_failure + and review_status in STABLE_REVIEW_STATUSES + ), + } + + +def _sample_c_arm(sample: Mapping[str, Any], index: int) -> dict[str, Any]: + """读取单样本 C 臂机械门结果，聚合时不允许平均掩盖硬错误。""" + + c_arm = _require_mapping(sample.get("cArm"), f"samples[{index}].cArm") + return { + "hardConstraintCoverage": _require_ratio( + c_arm.get("hardConstraintCoverage"), + f"samples[{index}].cArm.hardConstraintCoverage", + ), + "highSeverityResidualCount": _require_integer( + c_arm.get("highSeverityResidualCount"), + f"samples[{index}].cArm.highSeverityResidualCount", + ), + } + + +def _sample_scores(sample: Mapping[str, Any], index: int) -> dict[str, dict[str, float]]: + """读取稳定样本的 A/B/C 三臂五维脱敏终分。""" + + scores = _require_mapping(sample.get("scores"), f"samples[{index}].scores") + if set(scores) != {"A", "B", "C"}: + raise ValueError(f"samples[{index}].scores 必须精确包含 A/B/C") + result: dict[str, dict[str, float]] = {} + for arm in ("A", "B", "C"): + arm_scores = _require_mapping(scores[arm], f"samples[{index}].scores.{arm}") + if set(arm_scores) != set(DIMENSIONS): + raise ValueError(f"samples[{index}].scores.{arm} 必须精确包含正文五维") + result[arm] = { + dimension: _require_number( + arm_scores[dimension], f"samples[{index}].scores.{arm}.{dimension}" + ) + for dimension in DIMENSIONS + } + return result + + +def _sample_confounds(sample: Mapping[str, Any], index: int) -> dict[str, list[Any]]: + """校验单样本五类混淆项；顶层聚合混淆项不在信任边界内。""" + + source = sample.get("confounders", {}) + source = _require_mapping(source, f"samples[{index}].confounders") + unknown = sorted(set(source) - set(CONFOUND_LABELS)) + if unknown: + raise ValueError( + f"samples[{index}].confounders 存在未知分类: {','.join(unknown)}" + ) + result: dict[str, list[Any]] = {} + for category in CONFOUND_LABELS: + items = source.get(category, []) + if not isinstance(items, list): + raise ValueError(f"samples[{index}].confounders.{category} 必须是数组") + if any(not isinstance(item, (str, Mapping)) for item in items): + raise ValueError( + f"samples[{index}].confounders.{category} 只能包含字符串或对象" + ) + result[category] = list(items) + return result + + +def _aggregate_confounds(samples: Sequence[Mapping[str, Any]]) -> dict[str, list[Any]]: + """逐样本汇总五类混淆项，并补入可机械推导的泄露与不稳定记录。""" + + aggregated = {category: [] for category in CONFOUND_LABELS} + for index, sample in enumerate(samples): + sample_id, _, _ = _sample_identity(sample, index) + status = _sample_status(sample, index) + for category, items in _sample_confounds(sample, index).items(): + for item in items: + if isinstance(item, str): + aggregated[category].append({"sampleId": sample_id, "detail": item}) + else: + aggregated[category].append({**dict(item), "sampleId": sample_id}) + if status["futureLeakage"] and not any( + item.get("sampleId") == sample_id for item in aggregated["leakage"] + ): + aggregated["leakage"].append( + {"sampleId": sample_id, "code": "future_leakage_detected"} + ) + if status["reviewStatus"] == "invalid_unstable" and not any( + item.get("sampleId") == sample_id + for item in aggregated["reviewerInstability"] + ): + aggregated["reviewerInstability"].append( + {"sampleId": sample_id, "code": "invalid_unstable"} + ) + return aggregated + + +def _report( + gate: str, + status: str, + reasons: list[str], + confounds: dict[str, list[Any]], + metrics: Mapping[str, Any], +) -> dict[str, Any]: + """组装稳定报告结构；reasons 顺序同时表达当前层内优先级。""" + + return { + "schemaVersion": "writer-gate-report-v1", + "gate": gate, + "status": status, + "primaryReason": reasons[0], + "reasons": reasons, + "metrics": dict(metrics), + "confounds": confounds, + } + + +def _gate_a_metrics(samples: Sequence[Mapping[str, Any]]) -> dict[str, Any]: + """完全从逐样本结果计算 Gate A 统计。""" + + statuses: list[dict[str, Any]] = [] + c_arms: list[dict[str, Any]] = [] + seen_ids: set[str] = set() + for index, sample in enumerate(samples): + sample_id, _, _ = _sample_identity(sample, index) + if sample_id in seen_ids: + raise ValueError(f"sampleId 重复: {sample_id}") + seen_ids.add(sample_id) + status = _sample_status(sample, index) + statuses.append(status) + # schema、泄漏或系统失败可能发生在 C 臂生成前，此时没有 cArm 是合法的 + # 逐样本失败形态，不能让缺失的下游结果覆盖 Gate 的既定裁决顺序。 + if ( + status["schemaValid"] + and not status["futureLeakage"] + and not status["systemFailure"] + ): + c_arms.append(_sample_c_arm(sample, index)) + return { + "sampleCount": len(samples), + "validSampleCount": sum(item["valid"] for item in statuses), + "invalidSchemaCount": sum(not item["schemaValid"] for item in statuses), + "futureLeakageCount": sum(item["futureLeakage"] for item in statuses), + "systemFailureCount": sum(item["systemFailure"] for item in statuses), + "unstableSampleCount": sum( + item["reviewStatus"] == "invalid_unstable" for item in statuses + ), + "cArmHighSeverityResidualCount": sum( + item["highSeverityResidualCount"] for item in c_arms + ), + "cArmHardConstraintCoverage": min( + (item["hardConstraintCoverage"] for item in c_arms), default=1.0 + ), + } + + +def _short_circuit_confounds(gate_input: Mapping[str, Any]) -> dict[str, list[Any]]: + """Gate A 已决定 Gate B 终态时，尽量保留合法逐样本混淆项。""" + + if "samples" not in gate_input: + return {category: [] for category in CONFOUND_LABELS} + try: + return _aggregate_confounds(_require_samples(gate_input.get("samples"))) + except ValueError: + # 混淆项属于报告信息，不能反向推翻更高优先级的 Gate A 终态。 + return {category: [] for category in CONFOUND_LABELS} + + +def decide_gate_a(gate_input: Mapping[str, Any]) -> dict[str, Any]: + """按“证据不足 -> 硬失败 -> 通过”唯一顺序裁决 Gate A。""" + + gate_input = _require_mapping(gate_input, "Gate A 输入") + samples = _require_samples(gate_input.get("samples")) + normalized = _gate_a_metrics(samples) + confounds = _aggregate_confounds(samples) + + if normalized["validSampleCount"] < 5: + return _report( + "A", + "insufficient_evidence", + ["valid_sample_count_below_5"], + confounds, + normalized, + ) + + failure_reasons: list[str] = [] + if normalized["invalidSchemaCount"] > 0: + failure_reasons.append("schema_invalid") + if normalized["futureLeakageCount"] > 0: + failure_reasons.append("future_leakage") + if normalized["systemFailureCount"] > 0: + failure_reasons.append("system_failure") + if normalized["cArmHighSeverityResidualCount"] > 0: + failure_reasons.append("c_arm_high_severity_residual") + if normalized["cArmHardConstraintCoverage"] < 1.0: + failure_reasons.append("c_arm_hard_constraint_coverage_below_100_percent") + if failure_reasons: + return _report("A", "failed", failure_reasons, confounds, normalized) + + return _report( + "A", "passed", ["all_gate_a_conditions_met"], confounds, normalized + ) + + +def _gate_b_metrics(samples: Sequence[Mapping[str, Any]]) -> dict[str, Any]: + """完全从逐样本结果计算 Gate B 充分性、退化和增益指标。""" + + work_counts: dict[str, int] = {} + scenarios: set[str] = set() + statuses: list[dict[str, Any]] = [] + c_arms: list[dict[str, Any]] = [] + deltas: dict[str, list[dict[str, float]]] = {"B-A": [], "C-A": []} + seen_ids: set[str] = set() + + for index, sample in enumerate(samples): + sample_id, work_id, scenario = _sample_identity(sample, index) + if sample_id in seen_ids: + raise ValueError(f"sampleId 重复: {sample_id}") + seen_ids.add(sample_id) + work_counts[work_id] = work_counts.get(work_id, 0) + 1 + scenarios.add(scenario) + status = _sample_status(sample, index) + statuses.append(status) + c_arms.append(_sample_c_arm(sample, index)) + if status["reviewStatus"] in STABLE_REVIEW_STATUSES: + scores = _sample_scores(sample, index) + for comparison, arm in (("B-A", "B"), ("C-A", "C")): + deltas[comparison].append( + { + dimension: scores[arm][dimension] - scores["A"][dimension] + for dimension in DIMENSIONS + } + ) + + total_count = len(samples) + stable_count = len(deltas["C-A"]) + averages = { + comparison: { + dimension: ( + sum(item[dimension] for item in comparison_deltas) / stable_count + if stable_count + else 0.0 + ) + for dimension in DIMENSIONS + } + for comparison, comparison_deltas in deltas.items() + } + # Gate B 的正式退化与增益阈值只消费 C-A；B-A 仅作为卡索引单线的诊断信息。 + c_minus_a_deltas = deltas["C-A"] + decline_ratios = { + dimension: ( + sum(item[dimension] < -0.5 for item in c_minus_a_deltas) / stable_count + if stable_count + else 0.0 + ) + for dimension in DIMENSIONS + } + any_decline_ratio = ( + sum( + any(item[dimension] < -0.5 for dimension in DIMENSIONS) + for item in c_minus_a_deltas + ) + / stable_count + if stable_count + else 0.0 + ) + fidelity_positive_ratio = ( + sum( + item["setting_entity_fidelity"] > 0 for item in c_minus_a_deltas + ) + / stable_count + if stable_count + else 0.0 + ) + return { + "workSampleCounts": dict(sorted(work_counts.items())), + "workCount": len(work_counts), + "totalSampleCount": total_count, + "stableSampleCount": stable_count, + "coveredScenarios": sorted(scenarios), + "unstableSampleRatio": sum( + item["reviewStatus"] == "invalid_unstable" for item in statuses + ) + / total_count, + "cArmHardConstraintCoverage": min( + (item["hardConstraintCoverage"] for item in c_arms), default=0.0 + ), + "cArmHighSeverityResidualCount": sum( + item["highSeverityResidualCount"] for item in c_arms + ), + "averageDeltas": averages, + "bMinusAAverageDeltas": averages["B-A"], + "cMinusAAverageDeltas": averages["C-A"], + "dimensionDeclineOverHalfRatios": decline_ratios, + "anyDimensionDeclineOverHalfRatio": any_decline_ratio, + "fidelityPositiveSampleRatio": fidelity_positive_ratio, + } + + +def decide_gate_b(gate_input: Mapping[str, Any]) -> dict[str, Any]: + """按 Gate A、证据、退化、增益四层唯一顺序裁决 Gate B。""" + + gate_input = _require_mapping(gate_input, "Gate B 输入") + samples = _require_samples(gate_input.get("samples")) + # 顶层 gateAStatus 不在可信边界内；必须用 Gate B 的同批逐样本结果重算。 + gate_a_report = decide_gate_a({"gate": "A", "samples": samples}) + gate_a_status = gate_a_report["status"] + gate_a_metrics = { + "recomputedGateAStatus": gate_a_status, + "recomputedGateAMetrics": gate_a_report["metrics"], + } + if gate_a_status == "insufficient_evidence": + return _report( + "B", + "insufficient_evidence", + ["gate_a_insufficient_evidence"], + gate_a_report["confounds"], + gate_a_metrics, + ) + if gate_a_status == "failed": + return _report( + "B", + "failed", + ["gate_a_failed"], + gate_a_report["confounds"], + gate_a_metrics, + ) + + normalized = _gate_b_metrics(samples) + normalized.update(gate_a_metrics) + confounds = _aggregate_confounds(samples) + + evidence_reasons: list[str] = [] + if normalized["workCount"] < 2: + evidence_reasons.append("work_count_below_2") + if any(count < 5 for count in normalized["workSampleCounts"].values()): + evidence_reasons.append("per_work_sample_count_below_5") + if normalized["totalSampleCount"] < 10: + evidence_reasons.append("total_sample_count_below_10") + if not REQUIRED_SCENARIOS.issubset(normalized["coveredScenarios"]): + evidence_reasons.append("scenario_coverage_incomplete") + if normalized["unstableSampleRatio"] > 0.20: + evidence_reasons.append("unstable_sample_ratio_above_20_percent") + if evidence_reasons: + return _report( + "B", "insufficient_evidence", evidence_reasons, confounds, normalized + ) + + quality_reasons: list[str] = [] + if normalized["cArmHardConstraintCoverage"] < 1.0: + quality_reasons.append("c_arm_hard_constraint_coverage_below_100_percent") + if normalized["cArmHighSeverityResidualCount"] > 0: + quality_reasons.append("c_arm_high_severity_residual") + averages = normalized["averageDeltas"]["C-A"] + if averages["style_consistency"] < -0.25: + quality_reasons.append("style_consistency_average_delta_below_minus_0_25") + if averages["narrative_tension"] < -0.25: + quality_reasons.append("narrative_tension_average_delta_below_minus_0_25") + if normalized["anyDimensionDeclineOverHalfRatio"] > 0.20: + quality_reasons.append("dimension_decline_over_half_ratio_above_20_percent") + if quality_reasons: + return _report("B", "failed", quality_reasons, confounds, normalized) + + if ( + averages["setting_entity_fidelity"] >= 0.25 + and normalized["fidelityPositiveSampleRatio"] >= 0.60 + ): + return _report( + "B", "passed", ["fidelity_gain_threshold_met"], confounds, normalized + ) + + return _report( + "B", "no_gain", ["fidelity_gain_threshold_not_met"], confounds, normalized + ) + + +def decide_gate(gate_input: Mapping[str, Any]) -> dict[str, Any]: + """从带 gate 字段的统一输入分派 Gate A 或 Gate B。""" + + gate_input = _require_mapping(gate_input, "gate-input.json") + gate = gate_input.get("gate") + if gate == "A": + return decide_gate_a(gate_input) + if gate == "B": + return decide_gate_b(gate_input) + raise ValueError("gate 必须是 A 或 B") + + +def render_summary(report: Mapping[str, Any]) -> str: + """渲染只含内部统计、终态和混淆项的 Markdown 摘要。""" + + lines = [ + f"# Writer Gate {report['gate']} 裁决摘要", + "", + f"- 终态：`{report['status']}`", + f"- 主原因：`{report['primaryReason']}`", + f"- 原因码：`{', '.join(report['reasons'])}`", + "", + "## 混淆项", + "", + ] + confounds = _require_mapping(report.get("confounds"), "report.confounds") + for category, label in CONFOUND_LABELS.items(): + lines.append(f"### {label}") + lines.append("") + items = confounds.get(category, []) + if not items: + lines.append("- 未观察到或未报告") + else: + for item in items: + rendered = ( + item + if isinstance(item, str) + else json.dumps(item, ensure_ascii=False, sort_keys=True) + ) + lines.append(f"- {rendered}") + lines.append("") + return "\n".join(lines) + + +def _parse_args() -> argparse.Namespace: + """解析任务计划约定的运行目录与摘要输出参数。""" + + parser = argparse.ArgumentParser(description="裁决正文智能体 Gate A/B") + parser.add_argument( + "--run-dir", type=Path, required=True, help="包含 gate-input.json 的回放运行目录" + ) + parser.add_argument("--summary-output", type=Path, help="可选的脱敏 Markdown 摘要输出路径") + return parser.parse_args() + + +def main() -> int: + """执行离线判定，并写出唯一 JSON 终态和可选摘要。""" + + args = _parse_args() + input_path = args.run_dir / "gate-input.json" + output_path = args.run_dir / "gate-report.json" + try: + gate_input = json.loads(input_path.read_text(encoding="utf-8")) + report = decide_gate(gate_input) + output_path.write_text( + json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + if args.summary_output is not None: + args.summary_output.parent.mkdir(parents=True, exist_ok=True) + args.summary_output.write_text(render_summary(report), encoding="utf-8") + except (OSError, ValueError, json.JSONDecodeError) as error: + print(f"writer_gate: {error}", file=sys.stderr) + return 2 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/replay-eval/scripts/writer_rubric.py b/.claude/skills/replay-eval/scripts/writer_rubric.py new file mode 100644 index 0000000..b224493 --- /dev/null +++ b/.claude/skills/replay-eval/scripts/writer_rubric.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +"""正文回放五维量表与确定性稳定性仲裁。""" + +from __future__ import annotations + +from statistics import median +from typing import Any, Mapping, Sequence + + +RUBRIC_PROFILE = "writer_replay" +DIMENSIONS = ( + "setting_entity_fidelity", + "fine_outline_fidelity", + "style_consistency", + "narrative_tension", + "prose_readability", +) +EVIDENCE_SOURCE_TYPES = frozenset( + {"fine_outline", "historical_prose", "judge_inference"} +) +REPORT_FIELDS = frozenset( + {"profile", "reviewerId", "sampleId", "blindCandidateId", "candidateOrder", "scores"} +) + + +def _is_half_step(value: Any) -> bool: + """只接受 0 到 10 的数字和 0.5 步长，布尔值不算数字。""" + + return ( + not isinstance(value, bool) + and isinstance(value, (int, float)) + and 0 <= float(value) <= 10 + and float(value) * 2 == int(float(value) * 2) + ) + + +def validate_scores(scores: Any) -> list[str]: + """校验五维分数、步长和逐项证据归因。""" + + if not isinstance(scores, Mapping): + return ["scores 必须是对象"] + errors: list[str] = [] + missing = [dimension for dimension in DIMENSIONS if dimension not in scores] + unexpected = sorted(set(scores) - set(DIMENSIONS)) + if missing: + errors.append(f"缺少正文 rubric 维度: {','.join(missing)}") + if unexpected: + errors.append(f"存在未登记正文 rubric 维度: {','.join(unexpected)}") + for dimension in DIMENSIONS: + item = scores.get(dimension) + if not isinstance(item, Mapping): + errors.append(f"维度必须包含 score/evidence 对象: {dimension}") + continue + if set(item) != {"score", "evidence"}: + errors.append(f"维度字段必须精确为 score/evidence: {dimension}") + if not _is_half_step(item.get("score")): + errors.append(f"分数必须在 0-10 且使用 0.5 步长: {dimension}") + evidence = item.get("evidence") + if not isinstance(evidence, list) or not evidence: + errors.append(f"分数缺少证据: {dimension}") + continue + for index, raw in enumerate(evidence): + if not isinstance(raw, Mapping): + errors.append(f"证据必须是对象: {dimension}[{index}]") + continue + if set(raw) != {"sourceType", "sourceRef", "excerpt"}: + errors.append(f"证据字段必须精确为 sourceType/sourceRef/excerpt: {dimension}[{index}]") + continue + if raw.get("sourceType") not in EVIDENCE_SOURCE_TYPES: + errors.append(f"证据来源类型未登记: {dimension}[{index}]") + for field in ("sourceRef", "excerpt"): + if not isinstance(raw.get(field), str) or not raw[field].strip(): + errors.append(f"证据 {field} 不能为空: {dimension}[{index}]") + return errors + + +def validate_report(report: Any) -> list[str]: + """校验单个盲评报告，真实臂名或额外字段一律失败关闭。""" + + if not isinstance(report, Mapping): + return ["评委报告必须是对象"] + errors: list[str] = [] + if set(report) != REPORT_FIELDS: + errors.append("评委报告字段非法，去盲前不得包含真实臂名或额外信息") + if report.get("profile") != RUBRIC_PROFILE: + errors.append(f"profile 必须是 {RUBRIC_PROFILE}") + for field in ("reviewerId", "sampleId", "blindCandidateId"): + if not isinstance(report.get(field), str) or not report[field].strip(): + errors.append(f"{field} 必须是非空字符串") + order = report.get("candidateOrder") + if ( + not isinstance(order, list) + or len(order) < 2 + or any(not isinstance(item, str) or not item for item in order) + or len(set(order)) != len(order) + ): + errors.append("candidateOrder 必须是无重复盲化候选 ID 数组") + elif report.get("blindCandidateId") not in order: + errors.append("blindCandidateId 不在本轮 candidateOrder 中") + errors.extend(validate_scores(report.get("scores"))) + return errors + + +def validate_blind_pair(first: Any, second: Any) -> list[str]: + """确认双评独立、同样本、同候选集合且第二评委严格反序。""" + + errors = [*validate_report(first), *validate_report(second)] + if errors or not isinstance(first, Mapping) or not isinstance(second, Mapping): + return errors + if first["reviewerId"] == second["reviewerId"]: + errors.append("双评委必须使用独立 reviewerId") + if first["sampleId"] != second["sampleId"]: + errors.append("双评委必须评审同一样本") + if first["blindCandidateId"] != second["blindCandidateId"]: + errors.append("双评委必须评审同一盲化候选") + if set(first["candidateOrder"]) != set(second["candidateOrder"]): + errors.append("双评委的 candidateOrder 必须包含同一盲化候选集合") + if second["candidateOrder"] != list(reversed(first["candidateOrder"])): + errors.append("第二评委必须对相同盲化候选严格反序") + return errors + + +def deblind_reports( + reports: Sequence[Mapping[str, Any]], mapping: Mapping[str, str] +) -> list[dict[str, Any]]: + """评分结束后按预注册映射去盲，绝不依赖展示位置推断真实臂。""" + + if set(mapping.values()) != {"A", "B", "C"} or len(mapping) != 3: + raise ValueError("去盲映射必须将三个盲 ID 一一映射到 A/B/C") + result: list[dict[str, Any]] = [] + for index, report in enumerate(reports): + errors = validate_report(report) + if errors: + raise ValueError(f"reports[{index}] 非法: {'; '.join(errors)}") + blind_id = str(report["blindCandidateId"]) + if blind_id not in mapping: + raise ValueError(f"reports[{index}] 的 blindCandidateId 未预注册") + result.append({**dict(report), "arm": mapping[blind_id]}) + return result + + +def _numeric_scores(report: Mapping[str, Any]) -> dict[str, float]: + """从已校验报告提取确定性浮点分数。""" + + return { + dimension: float(report["scores"][dimension]["score"]) + for dimension in DIMENSIONS + } + + +def _stable_pair_exists(values: Sequence[float], threshold: float) -> bool: + """判断三个评分中是否至少存在一对落在稳定阈值内。""" + + return any( + abs(values[left] - values[right]) <= threshold + for left in range(len(values)) + for right in range(left + 1, len(values)) + ) + + +def adjudicate_reviews( + first: Mapping[str, Any], + second: Mapping[str, Any], + third: Mapping[str, Any] | None = None, + *, + threshold: float = 0.5, +) -> dict[str, Any]: + """按双评差异触发最多一次第三评委，并输出稳定终态。""" + + pair_errors = validate_blind_pair(first, second) + if pair_errors: + return {"status": "invalid_report", "errors": pair_errors} + first_scores = _numeric_scores(first) + second_scores = _numeric_scores(second) + unstable = [ + dimension + for dimension in DIMENSIONS + if abs(first_scores[dimension] - second_scores[dimension]) > threshold + ] + if not unstable: + return { + "status": "stable_report", + "scores": { + dimension: (first_scores[dimension] + second_scores[dimension]) / 2 + for dimension in DIMENSIONS + }, + "unstableDimensions": [], + "reviewCount": 2, + } + if third is None: + return { + "status": "needs_third_reviewer", + "unstableDimensions": unstable, + "reviewCount": 2, + } + third_errors = validate_report(third) + if third_errors: + return {"status": "invalid_report", "errors": third_errors} + if third["sampleId"] != first["sampleId"]: + return {"status": "invalid_report", "errors": ["第三评委必须评审同一样本"]} + if third["blindCandidateId"] != first["blindCandidateId"]: + return {"status": "invalid_report", "errors": ["第三评委必须评审同一盲化候选"]} + allowed_third_orders = ( + first["candidateOrder"], + list(reversed(first["candidateOrder"])), + ) + if third["candidateOrder"] not in allowed_third_orders: + return { + "status": "invalid_report", + "errors": ["第三评委必须使用与双评相同的盲化候选集合及第一评或反序顺序"], + } + if third["reviewerId"] in {first["reviewerId"], second["reviewerId"]}: + return {"status": "invalid_report", "errors": ["第三评委必须使用独立 reviewerId"]} + + third_scores = _numeric_scores(third) + unresolved = [ + dimension + for dimension in unstable + if not _stable_pair_exists( + [first_scores[dimension], second_scores[dimension], third_scores[dimension]], + threshold, + ) + ] + if unresolved: + return { + "status": "invalid_unstable", + "unstableDimensions": unresolved, + "reviewCount": 3, + } + return { + "status": "adjudicated_report", + "scores": { + dimension: float( + median( + [first_scores[dimension], second_scores[dimension], third_scores[dimension]] + ) + ) + for dimension in DIMENSIONS + }, + "unstableDimensions": unstable, + "reviewCount": 3, + } + + +__all__ = [ + "RUBRIC_PROFILE", + "DIMENSIONS", + "EVIDENCE_SOURCE_TYPES", + "validate_scores", + "validate_report", + "validate_blind_pair", + "deblind_reports", + "adjudicate_reviews", +] diff --git a/docs/2026-07-20-正文智能体正式优化设计与计划.md b/docs/2026-07-20-正文智能体正式优化设计与计划.md index b59e5b8..0195068 100644 --- a/docs/2026-07-20-正文智能体正式优化设计与计划.md +++ b/docs/2026-07-20-正文智能体正式优化设计与计划.md @@ -197,6 +197,8 @@ writer 可以创造非 Canonical 的环境细节、动作、过渡、无名配 | `narrativeState` | object | 是 | 时间、地点、角色位置、即时局面 | | `factEvidence[]` | array | 是 | 事实 ID、来源类型、sourceRef、hash | | `proseEvidence[]` | array | 是 | 章号、offset、hash、用途、临时片段 | +| `indexHints[]` | array | 否 | 仅诊断上下文可用的冻结卡提示；严格字段为 `cardId/name/type/content/sourceId/sourceVersion/asOf` | +| `evidenceStrategy` | enum | 否 | 生产缺省视为 `production_dual_evidence`；正文 A/B/C 回放必须显式记录各臂证据策略 | | `patternReferences[]` | array | 否 | 已授权范式卡，只作结构方法参考 | | `evidenceCoverage[]` | array | 是 | 细纲要素到证据的覆盖状态 | | `outputContract` | object | 是 | 篇幅、场景、frontmatter、申报规则 | @@ -204,6 +206,8 @@ writer 可以创造非 Canonical 的环境细节、动作、过渡、无名配 所有对象使用严格 schema，额外字段失败；引用的 ID、版本和 hash 必须存在且一致。 +`evidenceStrategy` 是兼容字段：生产上下文未填写时按 `production_dual_evidence` 校验；正文 A/B/C 回放不得使用缺省值，必须分别显式填写 `historical_prose_only`、`card_index_only`、`card_index_plus_prose`。`indexHints` 只能在 `mode=diagnostic_only`、`purpose=evaluation/diagnostic` 且 `acceptanceEligible=false` 时出现；生产上下文硬拒绝。其 `asOf` 不得超过上下文冻结点，`claimLedger` 不得引用 `indexHints`。只有显式填写 `evidenceStrategy=card_index_only` 的 B 臂允许在合同内跳过连续四章基线，并且必须保持 `proseEvidence=[]`；该例外不能用于生产。 + ### 7.2 `WriterOutput v1` ```text @@ -269,6 +273,7 @@ detector 只阻断可定位、可验证的问题：细纲硬约束漏项、事 - 离线优化允许最多 5 轮，且每轮只改一个变量；与生产两轮返修是两套状态机。 - 三臂使用相同细纲、冻结点、模型、篇幅算法和最大生成预算。 - 候选臂名映射为随机化 ID；顺序种子由预注册 `evaluationSetVersion + sampleId` 派生并固定。 +- judge 的可信输入边界是独立 `writer-blind-input-v1` 内容，包含 `blind-1/2/3` 候选正文/哈希，以及所有臂完全相同的 `sharedEvaluationReference`。共同参考只取目标章细纲硬约束、实体、必需角色、伏笔、章末钩子和冻结点前连续四章历史原文基准，用于评价细纲忠实、设定、文风和卡索引正确性；它可在真实运行时临时传递，但不进入安全摘要。共同参考严禁包含 `indexHints`、各臂补充原文、卡 manifest、`evidenceStrategy`、真实映射或臂名，不能用被测卡本身给被测候选背书。真实 A/B/C 映射仅在编排器内存中存在，judge 不得访问各臂 WriterContext、原始运行目录或含 `candidate-A/B/C` 的路径。 - 两个评委使用独立无会话实例，第二评委反转顺序；评分步长为 0.5。 - 同维分差 > 0.5 时判不稳定，最多增加一次第三评委。三评分中若至少一对差值 <=0.5，则该维最终分取三者中位数；若不存在稳定配对，则样本进入 `invalid_unstable`，不参与方向结论。 - 五维：设定与实体保真、情节与细纲忠实、叙事完整与张力、文风一致、文笔质量。 @@ -288,6 +293,36 @@ detector 只阻断可定位、可验证的问题：细纲硬约束漏项、事 只有 Gate B=`passed` 才解锁细纲智能体真实能力验收；`no_gain` 只形成“卡未证明增益”结论，不视为通过。 +### 10.2 Task10 Gate A 预注册与当前可验证边界 + +唯一配置为 `.claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json`。样本固定如下，不得根据生成结果替换章节或场景分类： + +| 目标章 | 冻结点 | 场景 | 预注册目标 | 大纲/细纲来源 | 预期长度 | 主要实体 | 新角色比例 | 泄漏检查 | +|---:|---:|---|---|---|---:|---|---:|---| +| 489 | 488 | 战斗 | 圣蒂曼围攻/加特朗战 | `outline-window:deep-space:481-488` / `fine-outline:deep-space:489` | 7500 | 圣蒂曼、加特朗 | 1.0 | 禁读 489 及以后；目标/未来读取数必须为 0 | +| 321 | 320 | 人物对话 | 莫妮卡道别与朋友确认 | `outline-window:deep-space:313-320` / `fine-outline:deep-space:321` | 7600 | 莫妮卡、朋友关系 | 0 | 禁读 321 及以后；目标/未来读取数必须为 0 | +| 544 | 543 | 转折 | 迷途之地内应反水 | `outline-window:deep-space:536-543` / `fine-outline:deep-space:544` | 6700 | 迷途之地、内应 | 未知 | 禁读 544 及以后；目标/未来读取数必须为 0 | +| 199 | 198 | 信息揭示 | 赛莉丝身份与联赛锁死真相 | `outline-window:deep-space:191-198` / `fine-outline:deep-space:199` | 2000 | 赛莉丝、联赛 | 0 | 禁读 199 及以后；目标/未来读取数必须为 0 | +| 523 | 522 | 老角色回归 | 伊蕾莉雅与旧部重逢 | `outline-window:deep-space:515-522` / `fine-outline:deep-space:523` | 6100 | 伊蕾莉雅、旧部 | 0 | 禁读 523 及以后；目标/未来读取数必须为 0 | + +预期长度不是人工填写：配置记录目标章之前连续四章的 `frozenRecentHanCounts`，统一以 `hardEventCount=1`、`foreshadowingActionCount=0`、`requiredSceneCount=0` 调用 `calculate_target_chars`，且 `usesTargetChapterLength=false`。五章机械结果固定为 489=7500、321=7600、544=6700、199=2000、523=6100；上下限取正负 10% 后再受 2000-10000 限幅。 + +新角色比例只统计具名 `requiredCharacters` 在冻结点前无记录的比例。489 的加特朗在 488 前无记录，因此为 1.0；321/199/523 的具名角色已有记录，因此为 0；544 的“内应”是泛称、无法机械判定具体身份，必须为 `null + unresolved_generic_role`，不允许默认成 0。 + +仓内配置不含原文全文。`writerContextInput.contentMode=sanitized_contract_fixture`，连续四章只放脱敏合成短文本，用于 dry-run 机械证明 WriterContext 合同、冻结边界、三臂差异、manifest 可复现和候选不可接受；不能据此声称历史原文完整、生成质量通过或 real-run 完成。 + +dry-run 命令： + +```bash +.venv/bin/python .claude/skills/replay-eval/scripts/run_writer_replay.py \ + --config .claude/skills/replay-eval/configs/writer-gate-a-deep-space-v1.json \ + --dry-run +``` + +dry-run 只生成计划、manifest 和上下文摘要，不调用模型。测试注入路径虽可验证 writer、detector 与盲评编排，但 judge 只接收独立盲化内容和共同评测参考，不能取得 raw 目录、真实臂映射或任一臂专属证据；评委顺序变化不得改变共同参考。当前真实 semantic detector 与 judge adapter 尚未实现，CLI `--execute` 必须明确失败关闭；在 adapter 接线、离线测试和预算确认完成前，不得声称真实回放或 Gate A 已完成。 + +Gate 判定只消费逐样本脱敏输入。`writer_gate.py` 在可信边界内自行计算 Gate A/B 的有效样本、作品/场景覆盖、C 臂硬门、C-A 五维增量、退化比例与五类混淆项；不信任上游聚合数字。五类混淆项为假阴、假阳、泄露、评委不稳定和新角色无卡，必须逐样本记录后汇总。 + ## 11. 用户闭环与下游交接 Shadow 候选展示后提供三决策：