修复: 收紧回放评测失败关闭与盲化边界
复用目录先原子落盘非成功态,并为三类外部 runner 增加可配置超时。detector 改为真正盲检,安全报告与嵌套响应统一执行严格闭集校验。
This commit is contained in:
parent
ca7d0ada1a
commit
1d32a96399
@ -47,9 +47,10 @@ disable-model-invocation: true
|
||||
- `scripts/run_replay.py --mode dry_run`:只执行授权、来源、冻结和三臂 manifest 预检,不调用模型;这是首个机制 smoke 入口。
|
||||
- `scripts/load_reference_work.py`:从 PostgreSQL 只读事务组装仓库外临时配置;缺授权字段仍会生成可审计配置,但送入 `run_replay` 后必须保持 `blocked_authorization`。
|
||||
- `scripts/run_replay.py --mode execute`:在全部前置门通过后,依次执行三臂 planner、整组 schema、逐臂盲 detector、两个独立盲 judge、rubric 校验、稳定性门和去盲汇总;`--output-dir` 必须位于仓库外的临时目录。
|
||||
- detector 输入输出均为 JSON;输入只有匿名候选 ID、候选和冻结到 `as_of` 的规划上下文,不含 arm 名、目标章 proxy 或其他评委结果。任一 `high` 严重度发现整组标记 `detector_blocked`,judge 调用数必须为 0;报告不合约时标记 `detector_invalid`。
|
||||
- detector 输入输出均为 JSON;输入只有匿名候选 ID、候选和公共冻结到 `as_of` 的规划上下文,不含 arm 名、`cardInjection`、`cardManifest`、任何臂特有卡内容、目标章 proxy 或其他评委结果。卡注入合法性只由确定性预检负责。任一 `high` 严重度发现整组标记 `detector_blocked`,judge 调用数必须为 0;报告不合约时标记 `detector_invalid`。
|
||||
- 两个 judge 使用不同 `judgeId` 和独立无会话进程。第二个 judge 的匿名候选顺序必须与第一个完全相反;任何 rubric 不合约标记 `judge_invalid`,任一同维差值大于 `0.5` 标记 `judge_unstable`,两者都不得标记 `completed`。
|
||||
- 只有三臂 schema、detector、双 judge rubric 和稳定性门全部通过,才去盲生成逐维 `B-A` / `C-A` 差值矩阵并标记 `completed`。`--detector-bin`、`--judge-primary-bin`、`--judge-secondary-bin` 可分别指定本地 runner;未指定时复用 `--planner-bin`,测试只能使用 fake binary。
|
||||
- `scripts/write_report.py`:从 `run_result.json` 生成安全摘要;它不会读取候选正文,也不会把候选路径以外的原始响应写入报告。
|
||||
- planner、detector、judge 子进程统一受 `--timeout-seconds` 限制,默认 300 秒;任一超时分别落盘 `planner_timeout`、`detector_timeout`、`judge_timeout`,不得继续进入后续阶段或标记 `completed`。
|
||||
- `scripts/write_report.py`:从 `run_result.json` 生成独立严格 schema 的安全摘要,只接受受限标识符、枚举、数字、短安全摘要和 SHA-256;不会读取候选正文,也不会把候选路径以外的原始响应写入报告。
|
||||
|
||||
真实作品运行前必须先从权威来源取得不可变授权快照。数据库没有该字段时,使用 dry-run 证明机制并保持 `blocked_authorization`,不得用本地配置或口头许可伪造放行。
|
||||
|
||||
@ -19,6 +19,8 @@ ALLOWED_FINDING_CATEGORIES = frozenset(
|
||||
}
|
||||
)
|
||||
ALLOWED_COVERAGE_CATEGORIES = frozenset({"frozen_context_gap"})
|
||||
FINDING_FIELDS = frozenset({"category", "severity", "location", "evidenceSummary"})
|
||||
COVERAGE_FINDING_FIELDS = frozenset({"category", "summary"})
|
||||
|
||||
|
||||
def build_detector_request(
|
||||
@ -28,10 +30,9 @@ def build_detector_request(
|
||||
as_of_chapter: int,
|
||||
frozen_snapshot: Mapping[str, Any],
|
||||
common_context: Mapping[str, Any],
|
||||
card_context: list[Any],
|
||||
sources: list[Any],
|
||||
) -> dict[str, Any]:
|
||||
"""构造不含 arm、目标章 proxy 或卡策略标签的盲检输入。"""
|
||||
"""只用公共冻结事实和匿名候选构造真正不识别评测臂的盲检输入。"""
|
||||
|
||||
return {
|
||||
"protocol": DETECTOR_PROTOCOL,
|
||||
@ -41,7 +42,6 @@ def build_detector_request(
|
||||
"planningContext": {
|
||||
"frozenSnapshot": dict(frozen_snapshot),
|
||||
"commonContext": dict(common_context),
|
||||
"cardInjection": list(card_context),
|
||||
"sources": list(sources),
|
||||
},
|
||||
"rules": {
|
||||
@ -88,6 +88,12 @@ def validate_detector_report(
|
||||
if not isinstance(finding, Mapping):
|
||||
errors.append(f"{section}[{index}] 必须是对象")
|
||||
continue
|
||||
allowed_fields = FINDING_FIELDS if section == "findings" else COVERAGE_FINDING_FIELDS
|
||||
unexpected_fields = sorted(set(finding) - allowed_fields)
|
||||
if unexpected_fields:
|
||||
errors.append(
|
||||
f"{section}[{index}] 包含未登记字段: {','.join(unexpected_fields)}"
|
||||
)
|
||||
category = finding.get("category")
|
||||
if not isinstance(category, str) or category not in allowed_categories:
|
||||
errors.append(f"{section}[{index}].category 未登记")
|
||||
@ -99,6 +105,10 @@ def validate_detector_report(
|
||||
value = finding.get(field)
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
errors.append(f"findings[{index}].{field} 必须是非空字符串")
|
||||
else:
|
||||
summary = finding.get("summary")
|
||||
if not isinstance(summary, str) or not summary.strip():
|
||||
errors.append(f"coverageFindings[{index}].summary 必须是非空字符串")
|
||||
|
||||
high_count = sum(
|
||||
1
|
||||
|
||||
@ -21,6 +21,8 @@ DIMENSIONS = (
|
||||
PROSE_DIMENSIONS = frozenset(
|
||||
{"style_fit", "readability", "文风一致性", "文笔", "pacing_tension", "information_density"}
|
||||
)
|
||||
SCORE_FIELDS = frozenset({"score", "evidence"})
|
||||
EVALUATION_FIELDS = frozenset({"candidateId", "scores", "summary"})
|
||||
|
||||
|
||||
def validate_scores(scores: Mapping[str, Any]) -> list[str]:
|
||||
@ -43,6 +45,11 @@ def validate_scores(scores: Mapping[str, Any]) -> list[str]:
|
||||
if not isinstance(value, Mapping):
|
||||
errors.append(f"维度必须包含 score/evidence 对象: {dimension}")
|
||||
continue
|
||||
unexpected_fields = sorted(set(value) - SCORE_FIELDS)
|
||||
if unexpected_fields:
|
||||
errors.append(
|
||||
f"维度包含未登记字段: {dimension}:{','.join(unexpected_fields)}"
|
||||
)
|
||||
score = value.get("score")
|
||||
if isinstance(score, bool) or not isinstance(score, (int, float)) or not 1 <= score <= 5:
|
||||
errors.append(f"分数必须在 1-5: {dimension}")
|
||||
@ -130,6 +137,11 @@ def validate_report(
|
||||
if not isinstance(evaluation, Mapping):
|
||||
errors.append(f"evaluations[{index}] 必须是对象")
|
||||
continue
|
||||
unexpected_evaluation_fields = sorted(set(evaluation) - EVALUATION_FIELDS)
|
||||
if unexpected_evaluation_fields:
|
||||
errors.append(
|
||||
f"evaluations[{index}] 包含未登记字段: {','.join(unexpected_evaluation_fields)}"
|
||||
)
|
||||
errors.extend(
|
||||
f"evaluations[{index}]: {error}"
|
||||
for error in validate_scores(evaluation.get("scores", {}))
|
||||
|
||||
@ -9,12 +9,15 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
from build_snapshot import build_snapshot, normalize_chapter, sha256_value
|
||||
from build_snapshot import SnapshotError, build_snapshot, normalize_chapter, sha256_value
|
||||
from audit_leakage import audit_snapshot
|
||||
from check_snapshot import (
|
||||
STATUS_READY,
|
||||
@ -30,6 +33,7 @@ REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
SKILL_PATH = REPO_ROOT / ".claude/skills/fine-outline/SKILL.md"
|
||||
PLANNER_PATH = REPO_ROOT / ".claude/agents/planner.md"
|
||||
JUDGE_IDS = ("judge-primary", "judge-secondary")
|
||||
DEFAULT_TIMEOUT_SECONDS = 300.0
|
||||
|
||||
|
||||
class ReplayRunError(ValueError):
|
||||
@ -44,6 +48,10 @@ class RunnerOutputError(ReplayRunError):
|
||||
"""外部 runner 返回的内容不符合机器合同。"""
|
||||
|
||||
|
||||
class RunnerTimeoutError(ReplayRunError):
|
||||
"""外部 runner 超过允许的最长执行时间。"""
|
||||
|
||||
|
||||
def _read_json(path: Path) -> Any:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
@ -52,6 +60,15 @@ def _safe_json(value: Any) -> str:
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def _timeout_stdout(error: subprocess.TimeoutExpired) -> str:
|
||||
"""规范化超时前捕获的标准输出,供临时目录留痕和哈希审计。"""
|
||||
|
||||
output = error.stdout or ""
|
||||
if isinstance(output, bytes):
|
||||
return output.decode("utf-8", errors="replace")
|
||||
return output
|
||||
|
||||
|
||||
def _require_mapping(config: Mapping[str, Any], key: str) -> Mapping[str, Any]:
|
||||
value = config.get(key)
|
||||
if not isinstance(value, Mapping):
|
||||
@ -175,6 +192,7 @@ def _invoke_planner(
|
||||
model: str,
|
||||
output_path: Path,
|
||||
max_budget_usd: float,
|
||||
timeout_seconds: float,
|
||||
) -> Mapping[str, Any]:
|
||||
"""用无工具、无会话持久化的 Claude print 模式运行 planner。"""
|
||||
|
||||
@ -197,7 +215,16 @@ def _invoke_planner(
|
||||
prompt,
|
||||
]
|
||||
try:
|
||||
completed = subprocess.run(command, text=True, capture_output=True, check=False)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
output_path.write_text(_timeout_stdout(error), encoding="utf-8")
|
||||
raise RunnerTimeoutError(f"planner 调用超时,限制={timeout_seconds:g}秒") from error
|
||||
except OSError as error:
|
||||
raise RunnerInvocationError(f"planner 启动失败: {error}") from error
|
||||
output_path.write_text(completed.stdout, encoding="utf-8")
|
||||
@ -215,6 +242,7 @@ def _invoke_structured_agent(
|
||||
output_path: Path,
|
||||
max_budget_usd: float,
|
||||
identity: str,
|
||||
timeout_seconds: float,
|
||||
) -> Mapping[str, Any]:
|
||||
"""以独立无会话进程调用 detector/judge,并保存仓库外原始响应。"""
|
||||
|
||||
@ -237,7 +265,16 @@ def _invoke_structured_agent(
|
||||
_safe_json(request),
|
||||
]
|
||||
try:
|
||||
completed = subprocess.run(command, text=True, capture_output=True, check=False)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
output_path.write_text(_timeout_stdout(error), encoding="utf-8")
|
||||
raise RunnerTimeoutError(f"{agent} 调用超时,限制={timeout_seconds:g}秒") from error
|
||||
except OSError as error:
|
||||
raise RunnerInvocationError(f"{agent} 启动失败: {error}") from error
|
||||
output_path.write_text(completed.stdout, encoding="utf-8")
|
||||
@ -380,7 +417,26 @@ def _aggregate_evaluation(
|
||||
def _write_result(output_dir: Path, result: Mapping[str, Any]) -> None:
|
||||
"""每个阶段都覆盖写入可恢复的结构化运行状态。"""
|
||||
|
||||
(output_dir / "run_result.json").write_text(_safe_json(result) + "\n", encoding="utf-8")
|
||||
result_path = output_dir / "run_result.json"
|
||||
temporary_path: Path | None = None
|
||||
try:
|
||||
# 唯一临时文件避免同目录并发写相互覆盖;同目录 replace 保证正式状态原子切换。
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
encoding="utf-8",
|
||||
dir=output_dir,
|
||||
prefix=".run_result.",
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
) as handle:
|
||||
temporary_path = Path(handle.name)
|
||||
handle.write(_safe_json(result) + "\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
temporary_path.replace(result_path)
|
||||
finally:
|
||||
if temporary_path is not None and temporary_path.exists():
|
||||
temporary_path.unlink()
|
||||
|
||||
|
||||
def run_replay(
|
||||
@ -394,67 +450,98 @@ def run_replay(
|
||||
judge_secondary_bin: str | None = None,
|
||||
model: str = "opus",
|
||||
max_budget_usd: float = 1.0,
|
||||
timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS,
|
||||
) -> dict[str, Any]:
|
||||
"""执行一次单目标三臂回放;任何前置门失败都不调用模型。"""
|
||||
|
||||
if mode not in {"dry_run", "execute"}:
|
||||
raise ReplayRunError("mode 只能是 dry_run 或 execute")
|
||||
output_dir = output_dir.resolve()
|
||||
if output_dir.is_relative_to(REPO_ROOT.resolve()):
|
||||
raise ReplayRunError("原始候选运行目录不得位于仓库内")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
snapshot_config = _require_mapping(config, "snapshot")
|
||||
as_of = normalize_chapter(snapshot_config.get("asOfChapter"))
|
||||
target = normalize_chapter(config.get("targetChapter"))
|
||||
snapshot_version = str(snapshot_config.get("snapshotVersion") or "")
|
||||
if as_of is None or target is None:
|
||||
raise ReplayRunError("as_of/target 必须是明确正整数")
|
||||
if target != as_of + 1:
|
||||
raise ReplayRunError("targetChapter 必须等于 snapshot.asOfChapter+1")
|
||||
reference_work = _require_mapping(config, "referenceWork")
|
||||
authorization = _require_mapping(config, "authorization")
|
||||
sources = config.get("sources", [])
|
||||
if not isinstance(sources, list):
|
||||
raise ReplayRunError("sources 必须是数组")
|
||||
common_input = _require_mapping(config, "commonContext")
|
||||
arms = _require_mapping(config, "arms")
|
||||
if set(arms) != set(REQUIRED_ARMS):
|
||||
raise ReplayRunError("生产回放必须精确配置三臂")
|
||||
|
||||
arm_manifests = {
|
||||
name: _build_arm_manifest(
|
||||
name=name,
|
||||
arm=_require_mapping(arms, name),
|
||||
common_input=common_input,
|
||||
as_of=as_of,
|
||||
target=target,
|
||||
snapshot_version=snapshot_version,
|
||||
)
|
||||
for name in REQUIRED_ARMS
|
||||
}
|
||||
preflight = check_replay(
|
||||
authorization=authorization,
|
||||
as_of_chapter=as_of,
|
||||
target_chapter=target,
|
||||
planner_sources=sources,
|
||||
arm_manifests=arm_manifests,
|
||||
)
|
||||
result: dict[str, Any] = {
|
||||
"runId": str(config.get("runId") or "unassigned"),
|
||||
"mode": mode,
|
||||
"status": preflight["status"],
|
||||
"ok": preflight["ok"],
|
||||
"referenceWork": str(reference_work.get("id") or ""),
|
||||
"referenceWorkVersion": str(reference_work.get("version") or ""),
|
||||
"asOfChapter": as_of,
|
||||
"targetChapter": target,
|
||||
"snapshotVersion": snapshot_version,
|
||||
"preflight": {"status": preflight["status"], "errors": preflight["errors"], "warnings": preflight["warnings"]},
|
||||
"arms": arm_manifests,
|
||||
"status": "validating_config",
|
||||
"ok": False,
|
||||
"results": {},
|
||||
}
|
||||
_write_result(output_dir, result)
|
||||
|
||||
try:
|
||||
if mode not in {"dry_run", "execute"}:
|
||||
raise ReplayRunError("mode 只能是 dry_run 或 execute")
|
||||
if (
|
||||
isinstance(timeout_seconds, bool)
|
||||
or not isinstance(timeout_seconds, (int, float))
|
||||
or not math.isfinite(float(timeout_seconds))
|
||||
or timeout_seconds <= 0
|
||||
):
|
||||
raise ReplayRunError("timeout_seconds 必须是正数")
|
||||
snapshot_config = _require_mapping(config, "snapshot")
|
||||
as_of = normalize_chapter(snapshot_config.get("asOfChapter"))
|
||||
target = normalize_chapter(config.get("targetChapter"))
|
||||
snapshot_version = str(snapshot_config.get("snapshotVersion") or "")
|
||||
if as_of is None or target is None:
|
||||
raise ReplayRunError("as_of/target 必须是明确正整数")
|
||||
if target != as_of + 1:
|
||||
raise ReplayRunError("targetChapter 必须等于 snapshot.asOfChapter+1")
|
||||
reference_work = _require_mapping(config, "referenceWork")
|
||||
authorization = _require_mapping(config, "authorization")
|
||||
sources = config.get("sources", [])
|
||||
if not isinstance(sources, list):
|
||||
raise ReplayRunError("sources 必须是数组")
|
||||
common_input = _require_mapping(config, "commonContext")
|
||||
arms = _require_mapping(config, "arms")
|
||||
if set(arms) != set(REQUIRED_ARMS):
|
||||
raise ReplayRunError("生产回放必须精确配置三臂")
|
||||
|
||||
arm_manifests = {
|
||||
name: _build_arm_manifest(
|
||||
name=name,
|
||||
arm=_require_mapping(arms, name),
|
||||
common_input=common_input,
|
||||
as_of=as_of,
|
||||
target=target,
|
||||
snapshot_version=snapshot_version,
|
||||
)
|
||||
for name in REQUIRED_ARMS
|
||||
}
|
||||
preflight = check_replay(
|
||||
authorization=authorization,
|
||||
as_of_chapter=as_of,
|
||||
target_chapter=target,
|
||||
planner_sources=sources,
|
||||
arm_manifests=arm_manifests,
|
||||
)
|
||||
except ReplayRunError as error:
|
||||
result["status"] = "config_invalid"
|
||||
result["errors"] = [str(error)]
|
||||
_write_result(output_dir, result)
|
||||
raise
|
||||
|
||||
result.update(
|
||||
{
|
||||
"referenceWork": str(reference_work.get("id") or ""),
|
||||
"referenceWorkVersion": str(reference_work.get("version") or ""),
|
||||
"asOfChapter": as_of,
|
||||
"targetChapter": target,
|
||||
"snapshotVersion": snapshot_version,
|
||||
"preflight": {
|
||||
"status": preflight["status"],
|
||||
"errors": preflight["errors"],
|
||||
"warnings": preflight["warnings"],
|
||||
},
|
||||
"arms": arm_manifests,
|
||||
}
|
||||
)
|
||||
if preflight["ok"]:
|
||||
# 前置门通过不等于快照配置有效;深层冻结完成前始终保持非成功态。
|
||||
result["status"] = "validating_config"
|
||||
result["ok"] = False
|
||||
else:
|
||||
result["status"] = preflight["status"]
|
||||
result["ok"] = False
|
||||
_write_result(output_dir, result)
|
||||
if not preflight["ok"]:
|
||||
return result
|
||||
|
||||
@ -484,13 +571,20 @@ def run_replay(
|
||||
"armConfig": {"arms": list(REQUIRED_ARMS)},
|
||||
}
|
||||
snapshot_data = snapshot_config.get("data", {})
|
||||
frozen = build_snapshot(
|
||||
snapshot_data,
|
||||
as_of,
|
||||
snapshot_version,
|
||||
target_chapter=target,
|
||||
manifest_metadata=metadata,
|
||||
)
|
||||
try:
|
||||
frozen = build_snapshot(
|
||||
snapshot_data,
|
||||
as_of,
|
||||
snapshot_version,
|
||||
target_chapter=target,
|
||||
manifest_metadata=metadata,
|
||||
)
|
||||
except SnapshotError as error:
|
||||
result["status"] = "config_invalid"
|
||||
result["ok"] = False
|
||||
result["errors"] = [str(error)]
|
||||
_write_result(output_dir, result)
|
||||
raise
|
||||
|
||||
# 内容审计同时覆盖公共冻结快照和各臂卡注入区;卡不在公共区,不能因此逃过未来事实检查。
|
||||
audit_payload = {
|
||||
@ -546,6 +640,7 @@ def run_replay(
|
||||
model=model,
|
||||
output_path=raw_path,
|
||||
max_budget_usd=max_budget_usd,
|
||||
timeout_seconds=float(timeout_seconds),
|
||||
)
|
||||
candidate_path = output_dir / f"candidate_{name}.json"
|
||||
candidate_path.write_text(_safe_json(candidate) + "\n", encoding="utf-8")
|
||||
@ -558,6 +653,16 @@ def run_replay(
|
||||
"candidateSha256": sha256_value(candidate),
|
||||
"candidatePath": str(candidate_path),
|
||||
}
|
||||
except RunnerTimeoutError as error:
|
||||
result["results"][name] = {
|
||||
"status": "planner_timeout",
|
||||
"ok": False,
|
||||
"errors": [str(error)],
|
||||
"rawOutputSha256": sha256_value(raw_path.read_text(encoding="utf-8")) if raw_path.exists() else None,
|
||||
}
|
||||
result["status"] = "planner_timeout"
|
||||
_write_result(output_dir, result)
|
||||
return result
|
||||
except RunnerInvocationError as error:
|
||||
result["results"][name] = {
|
||||
"status": "planner_failed",
|
||||
@ -594,14 +699,12 @@ def run_replay(
|
||||
for assignment in assignments:
|
||||
arm = str(assignment["arm"])
|
||||
candidate_id = str(assignment["candidateId"])
|
||||
arm_config = _require_mapping(arms, arm)
|
||||
request = build_detector_request(
|
||||
candidate_id=candidate_id,
|
||||
candidate=assignment["candidate"],
|
||||
as_of_chapter=as_of,
|
||||
frozen_snapshot=_public_snapshot(frozen["snapshot"]),
|
||||
common_context=common_input,
|
||||
card_context=arm_config.get("cards", []),
|
||||
sources=sources,
|
||||
)
|
||||
detector_path = output_dir / f"detector_{candidate_id}.raw.json"
|
||||
@ -614,6 +717,7 @@ def run_replay(
|
||||
output_path=detector_path,
|
||||
max_budget_usd=max_budget_usd,
|
||||
identity="blind-detector",
|
||||
timeout_seconds=float(timeout_seconds),
|
||||
)
|
||||
detector_check = validate_detector_report(detector_report, candidate_id)
|
||||
detector_invalid = detector_invalid or not detector_check["ok"]
|
||||
@ -636,6 +740,22 @@ def run_replay(
|
||||
result["status"] = "detector_invalid"
|
||||
_write_result(output_dir, result)
|
||||
return result
|
||||
except RunnerTimeoutError as error:
|
||||
result["results"][arm]["detector"] = {
|
||||
"status": "timeout",
|
||||
"findingCount": 0,
|
||||
"coverageFindingCount": 0,
|
||||
"highSeverityCount": 0,
|
||||
"errors": [str(error)],
|
||||
"rawOutputSha256": (
|
||||
sha256_value(detector_path.read_text(encoding="utf-8"))
|
||||
if detector_path.exists()
|
||||
else None
|
||||
),
|
||||
}
|
||||
result["status"] = "detector_timeout"
|
||||
_write_result(output_dir, result)
|
||||
return result
|
||||
except RunnerInvocationError as error:
|
||||
result["results"][arm]["detector"] = {
|
||||
"status": "failed",
|
||||
@ -710,6 +830,7 @@ def run_replay(
|
||||
output_path=judge_path,
|
||||
max_budget_usd=max_budget_usd,
|
||||
identity=judge_id,
|
||||
timeout_seconds=float(timeout_seconds),
|
||||
)
|
||||
candidate_ids = tuple(str(item["candidateId"]) for item in ordered)
|
||||
judge_errors = validate_report(
|
||||
@ -728,6 +849,15 @@ def run_replay(
|
||||
return result
|
||||
else:
|
||||
judge_reports.append(judge_report)
|
||||
except RunnerTimeoutError:
|
||||
result["status"] = "judge_timeout"
|
||||
result["evaluation"] = {
|
||||
"profile": RUBRIC_PROFILE,
|
||||
"judgeIds": list(JUDGE_IDS),
|
||||
"status": "timeout",
|
||||
}
|
||||
_write_result(output_dir, result)
|
||||
return result
|
||||
except RunnerInvocationError:
|
||||
result["status"] = "judge_failed"
|
||||
result["evaluation"] = {
|
||||
@ -782,6 +912,7 @@ def _parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--judge-secondary-bin")
|
||||
parser.add_argument("--model", default="opus")
|
||||
parser.add_argument("--max-budget-usd", type=float, default=1.0)
|
||||
parser.add_argument("--timeout-seconds", type=float, default=DEFAULT_TIMEOUT_SECONDS)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@ -797,6 +928,7 @@ def main() -> int:
|
||||
judge_secondary_bin=args.judge_secondary_bin,
|
||||
model=args.model,
|
||||
max_budget_usd=args.max_budget_usd,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
)
|
||||
return 0 if result["ok"] else 2
|
||||
|
||||
|
||||
@ -80,6 +80,38 @@ class FineOutlineDetectorTest(unittest.TestCase):
|
||||
self.assertNotIn("要单列为资料覆盖发现", skill)
|
||||
self.assertIn("归 judge/eval", skill)
|
||||
|
||||
def test_nested_findings_reject_arm_and_card_manifest_leakage(self):
|
||||
base = {
|
||||
"protocol": "fine_outline_detector_v0",
|
||||
"candidateId": "blind-1",
|
||||
"findings": [
|
||||
{
|
||||
"category": "entity_state",
|
||||
"severity": "low",
|
||||
"location": "entities[0]",
|
||||
"evidenceSummary": "冻结状态核对",
|
||||
}
|
||||
],
|
||||
"coverageFindings": [
|
||||
{
|
||||
"category": "frozen_context_gap",
|
||||
"summary": "公共冻结事实缺字段",
|
||||
}
|
||||
],
|
||||
}
|
||||
leaking_finding = {
|
||||
**base,
|
||||
"findings": [{**base["findings"][0], "arm": "outline_plus_cards"}],
|
||||
}
|
||||
leaking_coverage = {
|
||||
**base,
|
||||
"coverageFindings": [
|
||||
{**base["coverageFindings"][0], "cardManifest": {"count": 1}}
|
||||
],
|
||||
}
|
||||
self.assertFalse(validate_detector_report(leaking_finding, "blind-1")["ok"])
|
||||
self.assertFalse(validate_detector_report(leaking_coverage, "blind-1")["ok"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@ -99,6 +99,45 @@ class FineOutlineRubricTest(unittest.TestCase):
|
||||
)
|
||||
)
|
||||
|
||||
def test_nested_evaluation_and_score_reject_arm_card_manifest_leakage(self):
|
||||
evaluation = {
|
||||
"candidateId": "blind-1",
|
||||
"scores": valid_scores(),
|
||||
"summary": "短安全摘要",
|
||||
}
|
||||
report = {
|
||||
"profile": RUBRIC_PROFILE,
|
||||
"judgeId": "judge-primary",
|
||||
"evaluations": [evaluation],
|
||||
}
|
||||
leaking_evaluation = {
|
||||
**report,
|
||||
"evaluations": [{**evaluation, "arm": "outline_only"}],
|
||||
}
|
||||
leaking_scores = valid_scores()
|
||||
leaking_scores[DIMENSIONS[0]] = {
|
||||
**leaking_scores[DIMENSIONS[0]],
|
||||
"cardManifest": {"count": 1},
|
||||
}
|
||||
leaking_score = {
|
||||
**report,
|
||||
"evaluations": [{**evaluation, "scores": leaking_scores}],
|
||||
}
|
||||
self.assertTrue(
|
||||
validate_report(
|
||||
leaking_evaluation,
|
||||
expected_judge_id="judge-primary",
|
||||
expected_candidate_ids=("blind-1",),
|
||||
)
|
||||
)
|
||||
self.assertTrue(
|
||||
validate_report(
|
||||
leaking_score,
|
||||
expected_judge_id="judge-primary",
|
||||
expected_candidate_ids=("blind-1",),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@ -9,9 +9,10 @@ import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
from run_replay import _planner_prompt, run_replay # noqa: E402
|
||||
from run_replay import _parse_args, _planner_prompt, run_replay # noqa: E402
|
||||
from write_report import render_report # noqa: E402
|
||||
|
||||
|
||||
@ -112,7 +113,7 @@ def write_fake_runner(directory, mode="stable"):
|
||||
run_result_path = directory / "run" / "run_result.json"
|
||||
runner.write_text(
|
||||
"#!/usr/bin/env python3\n"
|
||||
"import json, pathlib, sys\n"
|
||||
"import json, pathlib, sys, time\n"
|
||||
f"mode = {mode!r}\n"
|
||||
f"log_path = pathlib.Path({str(log_path)!r})\n"
|
||||
f"count_path = pathlib.Path({str(count_path)!r})\n"
|
||||
@ -123,6 +124,8 @@ def write_fake_runner(directory, mode="stable"):
|
||||
"run_state = json.loads(run_result_path.read_text()) if run_result_path.exists() else None\n"
|
||||
"with log_path.open('a', encoding='utf-8') as handle:\n"
|
||||
" handle.write(json.dumps({'agent': agent, 'prompt': prompt, 'runState': run_state}, ensure_ascii=False) + '\\n')\n"
|
||||
"if mode == f'{agent}_timeout':\n"
|
||||
" time.sleep(2)\n"
|
||||
"if mode == f'{agent}_exit':\n"
|
||||
" raise SystemExit(7)\n"
|
||||
"if mode == f'{agent}_invalid_json':\n"
|
||||
@ -170,6 +173,22 @@ def read_calls(log_path):
|
||||
return [json.loads(line) for line in log_path.read_text(encoding="utf-8").splitlines()]
|
||||
|
||||
|
||||
def nested_keys(value):
|
||||
"""收集嵌套 JSON 的全部字段名,供盲化边界测试使用。"""
|
||||
|
||||
if isinstance(value, dict):
|
||||
keys = set(value)
|
||||
for item in value.values():
|
||||
keys.update(nested_keys(item))
|
||||
return keys
|
||||
if isinstance(value, list):
|
||||
keys = set()
|
||||
for item in value:
|
||||
keys.update(nested_keys(item))
|
||||
return keys
|
||||
return set()
|
||||
|
||||
|
||||
class ReplayRunTest(unittest.TestCase):
|
||||
def test_public_planner_context_does_not_include_snapshot_cards(self):
|
||||
prompt = _planner_prompt(
|
||||
@ -200,6 +219,27 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertEqual(result["status"], "blocked_authorization")
|
||||
self.assertFalse((pathlib.Path(directory) / "snapshot_manifest.json").exists())
|
||||
|
||||
def test_reused_output_dir_bad_config_overwrites_previous_completed_state(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
output_dir = pathlib.Path(directory) / "run"
|
||||
output_dir.mkdir()
|
||||
result_path = output_dir / "run_result.json"
|
||||
result_path.write_text(
|
||||
json.dumps({"runId": "old-run", "status": "completed", "ok": True}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
bad = config()
|
||||
bad["snapshot"]["data"]["unknownSection"] = []
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "未登记顶层分区"):
|
||||
run_replay(bad, output_dir, mode="execute")
|
||||
|
||||
persisted = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
self.assertEqual(persisted["runId"], "smoke-001")
|
||||
self.assertEqual(persisted["status"], "config_invalid")
|
||||
self.assertFalse(persisted["ok"])
|
||||
self.assertNotEqual(persisted["status"], "completed")
|
||||
|
||||
def test_content_leak_stops_before_manifest(self):
|
||||
bad = config()
|
||||
bad["leakageAudit"]["targetFacts"]["forbiddenFacts"][0]["text"] = "安全历史"
|
||||
@ -238,6 +278,36 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertNotIn('"prompt":', report)
|
||||
self.assertNotIn('"payload":', report)
|
||||
|
||||
def test_report_rejects_text_injection_through_identifier_fields(self):
|
||||
base = run_replay(config(), pathlib.Path(tempfile.mkdtemp()), mode="dry_run")
|
||||
injections = {
|
||||
"runId": "safe-run\n完整目标细纲:第一幕到第三幕的全部事件原文",
|
||||
"referenceWork": "深空之影目标章原文与完整细纲",
|
||||
}
|
||||
for field, injected in injections.items():
|
||||
with self.subTest(field=field):
|
||||
unsafe = dict(base)
|
||||
unsafe[field] = injected
|
||||
with self.assertRaisesRegex(ValueError, field):
|
||||
render_report(unsafe)
|
||||
|
||||
def test_report_does_not_render_preflight_free_text(self):
|
||||
bad = config()
|
||||
injected = "目标章事实文本"
|
||||
bad["authorization"] = {**bad["authorization"], "sourceStatus": injected}
|
||||
result = run_replay(bad, pathlib.Path(tempfile.mkdtemp()), mode="dry_run")
|
||||
report = render_report(result)
|
||||
self.assertNotIn(injected, report)
|
||||
self.assertIn("授权前置门未通过", report)
|
||||
|
||||
def test_report_accepts_registered_target_source_failure_status(self):
|
||||
result = run_replay(config(), pathlib.Path(tempfile.mkdtemp()), mode="dry_run")
|
||||
result["results"] = {
|
||||
"outline_only": {"status": "target_source_forbidden", "ok": False}
|
||||
}
|
||||
report = render_report(result)
|
||||
self.assertIn("target_source_forbidden", report)
|
||||
|
||||
def test_execute_uses_external_planner_and_validates_each_candidate(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
fake, log_path = write_fake_runner(directory)
|
||||
@ -284,6 +354,23 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertFalse(result["ok"])
|
||||
self.assertEqual(len(calls), 1)
|
||||
|
||||
def test_planner_timeout_fails_closed_and_persists_timeout_status(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "planner_timeout")
|
||||
output_dir = pathlib.Path(directory) / "run"
|
||||
result = run_replay(
|
||||
config(),
|
||||
output_dir,
|
||||
mode="execute",
|
||||
planner_bin=str(runner),
|
||||
timeout_seconds=1.0,
|
||||
)
|
||||
persisted = json.loads((output_dir / "run_result.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual(result["status"], "planner_timeout")
|
||||
self.assertFalse(result["ok"])
|
||||
self.assertEqual(persisted["status"], "planner_timeout")
|
||||
self.assertEqual(len(read_calls(log_path)), 1)
|
||||
|
||||
def test_detector_nonzero_exit_stops_before_judges(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "detector_exit")
|
||||
@ -305,6 +392,22 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertEqual(len([call for call in calls if call["agent"] == "detector"]), 1)
|
||||
self.assertEqual(len([call for call in calls if call["agent"] == "judge"]), 0)
|
||||
|
||||
def test_detector_timeout_fails_closed_before_judges(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "detector_timeout")
|
||||
result = run_replay(
|
||||
config(),
|
||||
pathlib.Path(directory) / "run",
|
||||
mode="execute",
|
||||
planner_bin=str(runner),
|
||||
timeout_seconds=1.0,
|
||||
)
|
||||
calls = read_calls(log_path)
|
||||
self.assertEqual(result["status"], "detector_timeout")
|
||||
self.assertFalse(result["ok"])
|
||||
self.assertEqual(len([call for call in calls if call["agent"] == "detector"]), 1)
|
||||
self.assertEqual(len([call for call in calls if call["agent"] == "judge"]), 0)
|
||||
|
||||
def test_primary_judge_nonzero_exit_does_not_call_secondary(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "judge_exit")
|
||||
@ -323,6 +426,53 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertFalse(result["ok"])
|
||||
self.assertEqual(len(judge_calls), 1)
|
||||
|
||||
def test_primary_judge_timeout_does_not_call_secondary(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "judge_timeout")
|
||||
result = run_replay(
|
||||
config(),
|
||||
pathlib.Path(directory) / "run",
|
||||
mode="execute",
|
||||
planner_bin=str(runner),
|
||||
timeout_seconds=1.0,
|
||||
)
|
||||
judge_calls = [call for call in read_calls(log_path) if call["agent"] == "judge"]
|
||||
self.assertEqual(result["status"], "judge_timeout")
|
||||
self.assertFalse(result["ok"])
|
||||
self.assertEqual(len(judge_calls), 1)
|
||||
|
||||
def test_cli_accepts_subprocess_timeout_seconds(self):
|
||||
argv = [
|
||||
"run_replay.py",
|
||||
"--config",
|
||||
"/tmp/replay-config.json",
|
||||
"--output-dir",
|
||||
"/tmp/replay-output",
|
||||
"--timeout-seconds",
|
||||
"12.5",
|
||||
]
|
||||
with patch.object(sys, "argv", argv):
|
||||
args = _parse_args()
|
||||
self.assertEqual(args.timeout_seconds, 12.5)
|
||||
|
||||
def test_non_finite_timeout_is_persisted_as_config_invalid(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
for index, timeout_seconds in enumerate((float("nan"), float("inf"))):
|
||||
with self.subTest(timeout_seconds=timeout_seconds):
|
||||
output_dir = pathlib.Path(directory) / f"run-{index}"
|
||||
with self.assertRaisesRegex(ValueError, "timeout_seconds"):
|
||||
run_replay(
|
||||
config(),
|
||||
output_dir,
|
||||
mode="execute",
|
||||
timeout_seconds=timeout_seconds,
|
||||
)
|
||||
persisted = json.loads(
|
||||
(output_dir / "run_result.json").read_text(encoding="utf-8")
|
||||
)
|
||||
self.assertEqual(persisted["status"], "config_invalid")
|
||||
self.assertFalse(persisted["ok"])
|
||||
|
||||
def test_schema_failure_stops_before_all_detectors_and_judges(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "invalid_candidate")
|
||||
@ -345,6 +495,33 @@ class ReplayRunTest(unittest.TestCase):
|
||||
self.assertTrue(all("outline_plus" not in call["prompt"] for call in detector_calls))
|
||||
self.assertTrue(all("targetFacts" not in call["prompt"] for call in detector_calls))
|
||||
|
||||
def test_detector_requests_cannot_distinguish_arm_specific_card_identity(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "stable")
|
||||
result = run_replay(config(), pathlib.Path(directory) / "run", mode="execute", planner_bin=str(runner))
|
||||
self.assertEqual(result["status"], "completed")
|
||||
requests = [
|
||||
json.loads(call["prompt"])
|
||||
for call in read_calls(log_path)
|
||||
if call["agent"] == "detector"
|
||||
]
|
||||
self.assertEqual(len(requests), 3)
|
||||
for request in requests:
|
||||
self.assertTrue(
|
||||
nested_keys(request).isdisjoint({"arm", "cardInjection", "cardManifest"})
|
||||
)
|
||||
serialized = json.dumps(request, ensure_ascii=False)
|
||||
self.assertNotIn("正确卡", serialized)
|
||||
self.assertNotIn("错配卡", serialized)
|
||||
self.assertNotIn("card-correct-1", serialized)
|
||||
self.assertNotIn("card-placebo-1", serialized)
|
||||
public_parts = [
|
||||
{key: value for key, value in request.items() if key not in {"candidateId", "candidate"}}
|
||||
for request in requests
|
||||
]
|
||||
self.assertEqual(public_parts[0], public_parts[1])
|
||||
self.assertEqual(public_parts[1], public_parts[2])
|
||||
|
||||
def test_invalid_detector_report_fails_closed_before_judge(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
runner, log_path = write_fake_runner(directory, "invalid_detector")
|
||||
|
||||
@ -5,6 +5,8 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
@ -27,6 +29,79 @@ REPORT_FORBIDDEN_KEYS = frozenset(
|
||||
"完整目标细纲",
|
||||
}
|
||||
)
|
||||
IDENTIFIER_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,63}\Z")
|
||||
SHA256_PATTERN = re.compile(r"[0-9a-f]{64}\Z")
|
||||
ARM_NAMES = frozenset(
|
||||
{"outline_only", "outline_plus_cards", "outline_plus_placebo_cards"}
|
||||
)
|
||||
RUN_STATUSES = frozenset(
|
||||
{
|
||||
"unknown",
|
||||
"validating_config",
|
||||
"config_invalid",
|
||||
"blocked_authorization",
|
||||
"invalid_snapshot",
|
||||
"invalid_arm_diff",
|
||||
"target_source_forbidden",
|
||||
"invalid_audit_input",
|
||||
"blocked_leakage_audit",
|
||||
"ready",
|
||||
"running_planner",
|
||||
"planner_timeout",
|
||||
"planner_failed",
|
||||
"planner_invalid",
|
||||
"candidate_blocked",
|
||||
"running_detector",
|
||||
"detector_timeout",
|
||||
"detector_failed",
|
||||
"detector_invalid",
|
||||
"detector_blocked",
|
||||
"running_judge",
|
||||
"judge_timeout",
|
||||
"judge_failed",
|
||||
"judge_invalid",
|
||||
"judge_unstable",
|
||||
"completed",
|
||||
}
|
||||
)
|
||||
PREFLIGHT_STATUSES = frozenset(
|
||||
{"unknown", "ready", "blocked_authorization", "invalid_snapshot", "invalid_arm_diff", "target_source_forbidden"}
|
||||
)
|
||||
ARM_STATUSES = frozenset(
|
||||
{
|
||||
"not_run",
|
||||
"ready",
|
||||
"schema_invalid",
|
||||
"invalid_snapshot",
|
||||
"target_source_forbidden",
|
||||
"planner_timeout",
|
||||
"planner_failed",
|
||||
"planner_invalid",
|
||||
}
|
||||
)
|
||||
DETECTOR_STATUSES = frozenset(
|
||||
{"not_run", "passed", "blocked_high", "timeout", "failed", "invalid"}
|
||||
)
|
||||
CARD_STRATEGIES = frozenset({"unknown", "none", "correct", "placebo"})
|
||||
RUBRIC_PROFILES = frozenset({"unknown", "fine_outline_replay"})
|
||||
RUBRIC_DIMENSIONS = frozenset(
|
||||
{
|
||||
"structure_completeness",
|
||||
"direction_causality",
|
||||
"order_pacing",
|
||||
"entity_state",
|
||||
"foreshadowing_action",
|
||||
"handoff_hook",
|
||||
}
|
||||
)
|
||||
PREFLIGHT_SUMMARIES = {
|
||||
"unknown": "前置门状态不可用",
|
||||
"ready": "前置门通过",
|
||||
"blocked_authorization": "授权前置门未通过",
|
||||
"invalid_snapshot": "冻结快照前置门未通过",
|
||||
"invalid_arm_diff": "三臂公共输入一致性未通过",
|
||||
"target_source_forbidden": "发现目标章或未来来源",
|
||||
}
|
||||
|
||||
|
||||
def _assert_no_forbidden_keys(value: Any, path: str = "result") -> None:
|
||||
@ -42,13 +117,165 @@ def _assert_no_forbidden_keys(value: Any, path: str = "result") -> None:
|
||||
_assert_no_forbidden_keys(item, f"{path}[{index}]")
|
||||
|
||||
|
||||
def _arm_line(name: str, arm: Mapping[str, Any], result: Mapping[str, Any]) -> str:
|
||||
outcome = result.get(name) or {}
|
||||
detector = outcome.get("detector") or {}
|
||||
def _mapping(value: Any, path: str) -> Mapping[str, Any]:
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError(f"{path} 必须是对象")
|
||||
return value
|
||||
|
||||
|
||||
def _identifier(value: Any, path: str, default: str = "unknown") -> str:
|
||||
candidate = default if value is None or value == "" else value
|
||||
if not isinstance(candidate, str) or IDENTIFIER_PATTERN.fullmatch(candidate) is None:
|
||||
raise ValueError(f"{path} 必须是长度不超过 64 的安全标识符")
|
||||
return candidate
|
||||
|
||||
|
||||
def _enum(value: Any, allowed: frozenset[str], path: str, default: str) -> str:
|
||||
candidate = default if value is None or value == "" else value
|
||||
if not isinstance(candidate, str) or candidate not in allowed:
|
||||
raise ValueError(f"{path} 不是已登记枚举值")
|
||||
return candidate
|
||||
|
||||
|
||||
def _integer(value: Any, path: str, *, minimum: int = 0) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
|
||||
raise ValueError(f"{path} 必须是大于等于 {minimum} 的整数")
|
||||
return value
|
||||
|
||||
|
||||
def _number(value: Any, path: str) -> float:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
raise ValueError(f"{path} 必须是数字")
|
||||
number = float(value)
|
||||
if not math.isfinite(number):
|
||||
raise ValueError(f"{path} 必须是有限数字")
|
||||
return number
|
||||
|
||||
|
||||
def _hash(value: Any, path: str) -> str | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
if not isinstance(value, str) or SHA256_PATTERN.fullmatch(value) is None:
|
||||
raise ValueError(f"{path} 必须是 SHA-256")
|
||||
return value
|
||||
|
||||
|
||||
def _numeric_map(value: Any, path: str, allowed_keys: frozenset[str]) -> dict[str, float]:
|
||||
mapping = _mapping(value, path)
|
||||
unexpected = sorted(set(mapping) - allowed_keys)
|
||||
if unexpected:
|
||||
raise ValueError(f"{path} 包含未登记字段: {','.join(unexpected)}")
|
||||
return {str(key): _number(item, f"{path}.{key}") for key, item in mapping.items()}
|
||||
|
||||
|
||||
def _project_evaluation(value: Any) -> dict[str, Any]:
|
||||
if value is None or value == "" or value == {}:
|
||||
return {}
|
||||
evaluation = _mapping(value, "evaluation")
|
||||
projected: dict[str, Any] = {
|
||||
"profile": _enum(evaluation.get("profile"), RUBRIC_PROFILES, "evaluation.profile", "unknown")
|
||||
}
|
||||
stability = evaluation.get("stability")
|
||||
if stability is not None:
|
||||
stability_mapping = _mapping(stability, "evaluation.stability")
|
||||
stable = stability_mapping.get("stable")
|
||||
if not isinstance(stable, bool):
|
||||
raise ValueError("evaluation.stability.stable 必须是布尔枚举")
|
||||
projected["stabilityStatus"] = "stable" if stable else "unstable"
|
||||
projected["maxGaps"] = _numeric_map(
|
||||
stability_mapping.get("maxGaps", {}),
|
||||
"evaluation.stability.maxGaps",
|
||||
RUBRIC_DIMENSIONS,
|
||||
)
|
||||
arm_scores = _mapping(evaluation.get("armScores", {}), "evaluation.armScores")
|
||||
unexpected_arms = sorted(set(arm_scores) - ARM_NAMES)
|
||||
if unexpected_arms:
|
||||
raise ValueError(f"evaluation.armScores 包含未登记臂: {','.join(unexpected_arms)}")
|
||||
projected["armScores"] = {
|
||||
str(arm): _numeric_map(scores, f"evaluation.armScores.{arm}", RUBRIC_DIMENSIONS)
|
||||
for arm, scores in arm_scores.items()
|
||||
}
|
||||
deltas = _mapping(evaluation.get("deltas", {}), "evaluation.deltas")
|
||||
unexpected_deltas = sorted(set(deltas) - {"B-A", "C-A"})
|
||||
if unexpected_deltas:
|
||||
raise ValueError(f"evaluation.deltas 包含未登记比较: {','.join(unexpected_deltas)}")
|
||||
projected["deltas"] = {
|
||||
str(name): _numeric_map(scores, f"evaluation.deltas.{name}", RUBRIC_DIMENSIONS)
|
||||
for name, scores in deltas.items()
|
||||
}
|
||||
return projected
|
||||
|
||||
|
||||
def _project_report(result: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""从运行态提取独立安全 schema,未登记类型和值一律拒绝。"""
|
||||
|
||||
_assert_no_forbidden_keys(result)
|
||||
preflight = _mapping(result.get("preflight", {}), "preflight")
|
||||
preflight_status = _enum(
|
||||
preflight.get("status"),
|
||||
PREFLIGHT_STATUSES,
|
||||
"preflight.status",
|
||||
"unknown",
|
||||
)
|
||||
arms = _mapping(result.get("arms", {}), "arms")
|
||||
outcomes = _mapping(result.get("results", {}), "results")
|
||||
unexpected_arms = sorted((set(arms) | set(outcomes)) - ARM_NAMES)
|
||||
if unexpected_arms:
|
||||
raise ValueError(f"报告包含未登记评测臂: {','.join(unexpected_arms)}")
|
||||
projected_arms: dict[str, Any] = {}
|
||||
for name in sorted(arms):
|
||||
arm = _mapping(arms[name], f"arms.{name}")
|
||||
outcome = _mapping(outcomes.get(name, {}), f"results.{name}")
|
||||
detector = _mapping(outcome.get("detector", {}), f"results.{name}.detector")
|
||||
projected_arms[name] = {
|
||||
"status": _enum(outcome.get("status"), ARM_STATUSES, f"results.{name}.status", "not_run"),
|
||||
"candidateSha256": _hash(
|
||||
outcome.get("candidateSha256") or outcome.get("rawOutputSha256"),
|
||||
f"results.{name}.candidateSha256",
|
||||
),
|
||||
"detectorStatus": _enum(
|
||||
detector.get("status"),
|
||||
DETECTOR_STATUSES,
|
||||
f"results.{name}.detector.status",
|
||||
"not_run",
|
||||
),
|
||||
"cardStrategy": _enum(
|
||||
arm.get("cardStrategy"),
|
||||
CARD_STRATEGIES,
|
||||
f"arms.{name}.cardStrategy",
|
||||
"unknown",
|
||||
),
|
||||
"cardInjectionCount": _integer(
|
||||
arm.get("cardInjectionCount", 0),
|
||||
f"arms.{name}.cardInjectionCount",
|
||||
),
|
||||
}
|
||||
return {
|
||||
"runId": _identifier(result.get("runId"), "runId"),
|
||||
"status": _enum(result.get("status"), RUN_STATUSES, "status", "unknown"),
|
||||
"mode": _enum(result.get("mode"), frozenset({"unknown", "dry_run", "execute"}), "mode", "unknown"),
|
||||
"referenceWork": _identifier(result.get("referenceWork"), "referenceWork"),
|
||||
"referenceWorkVersion": _identifier(result.get("referenceWorkVersion"), "referenceWorkVersion"),
|
||||
"asOfChapter": _integer(result.get("asOfChapter"), "asOfChapter", minimum=1),
|
||||
"targetChapter": _integer(result.get("targetChapter"), "targetChapter", minimum=1),
|
||||
"snapshotVersion": _identifier(result.get("snapshotVersion"), "snapshotVersion"),
|
||||
"preflightStatus": preflight_status,
|
||||
"preflightSummary": PREFLIGHT_SUMMARIES[preflight_status],
|
||||
"arms": projected_arms,
|
||||
"snapshotManifestSha256": _hash(
|
||||
result.get("snapshotManifestSha256"), "snapshotManifestSha256"
|
||||
),
|
||||
"evaluation": _project_evaluation(result.get("evaluation")),
|
||||
}
|
||||
|
||||
|
||||
def _arm_line(name: str, arm: Mapping[str, Any]) -> str:
|
||||
return (
|
||||
f"- `{name}`: {outcome.get('status', 'not_run')},"
|
||||
f"候选哈希 `{outcome.get('candidateSha256') or outcome.get('rawOutputSha256') or 'n/a'}`,"
|
||||
f"detector `{detector.get('status', 'not_run')}`,"
|
||||
f"- `{name}`: {arm['status']},"
|
||||
f"候选哈希 `{arm['candidateSha256'] or 'n/a'}`,"
|
||||
f"detector `{arm['detectorStatus']}`,"
|
||||
f"卡策略 `{arm.get('cardStrategy', 'unknown')}`,卡数 `{arm.get('cardInjectionCount', 0)}`"
|
||||
)
|
||||
|
||||
@ -58,15 +285,14 @@ def _render_evaluation(lines: list[str], evaluation: Mapping[str, Any]) -> None:
|
||||
|
||||
if not evaluation:
|
||||
return
|
||||
stability = evaluation.get("stability") or {}
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 双盲评分",
|
||||
"",
|
||||
f"- profile: `{evaluation.get('profile', 'unknown')}`",
|
||||
f"- 评委稳定性: `{'stable' if stability.get('stable') else 'unstable'}`",
|
||||
f"- 最大维度差: `{json.dumps(stability.get('maxGaps') or {}, ensure_ascii=False, sort_keys=True)}`",
|
||||
f"- 评委稳定性: `{evaluation.get('stabilityStatus', 'not_available')}`",
|
||||
f"- 最大维度差: `{json.dumps(evaluation.get('maxGaps') or {}, ensure_ascii=False, sort_keys=True)}`",
|
||||
]
|
||||
)
|
||||
arm_scores = evaluation.get("armScores") or {}
|
||||
@ -85,32 +311,28 @@ def _render_evaluation(lines: list[str], evaluation: Mapping[str, Any]) -> None:
|
||||
def render_report(result: Mapping[str, Any]) -> str:
|
||||
"""只从运行结果的白名单字段渲染摘要。"""
|
||||
|
||||
_assert_no_forbidden_keys(result)
|
||||
safe = _project_report(result)
|
||||
lines = [
|
||||
"# 细纲回放摘要",
|
||||
"",
|
||||
f"- runId: `{result.get('runId', 'unknown')}`",
|
||||
f"- status: `{result.get('status', 'unknown')}`",
|
||||
f"- mode: `{result.get('mode', 'unknown')}`",
|
||||
f"- referenceWork: `{result.get('referenceWork', 'unknown')}@{result.get('referenceWorkVersion', 'unknown')}`",
|
||||
f"- freeze: 第 `{result.get('asOfChapter', 'unknown')}` 章,目标第 `{result.get('targetChapter', 'unknown')}` 章",
|
||||
f"- snapshot: `{result.get('snapshotVersion', 'unknown')}`",
|
||||
f"- runId: `{safe['runId']}`",
|
||||
f"- status: `{safe['status']}`",
|
||||
f"- mode: `{safe['mode']}`",
|
||||
f"- referenceWork: `{safe['referenceWork']}@{safe['referenceWorkVersion']}`",
|
||||
f"- freeze: 第 `{safe['asOfChapter'] if safe['asOfChapter'] is not None else 'unknown'}` 章,目标第 `{safe['targetChapter'] if safe['targetChapter'] is not None else 'unknown'}` 章",
|
||||
f"- snapshot: `{safe['snapshotVersion']}`",
|
||||
"",
|
||||
"## 前置门",
|
||||
"",
|
||||
f"- status: `{(result.get('preflight') or {}).get('status', 'unknown')}`",
|
||||
f"- status: `{safe['preflightStatus']}`",
|
||||
f"- 摘要: {safe['preflightSummary']}",
|
||||
]
|
||||
errors = (result.get("preflight") or {}).get("errors") or []
|
||||
for error in errors:
|
||||
lines.append(f"- 阻断: {error}")
|
||||
lines.extend(["", "## 三臂", ""])
|
||||
arms = result.get("arms") or {}
|
||||
outcomes = result.get("results") or {}
|
||||
for name in sorted(arms):
|
||||
lines.append(_arm_line(name, arms[name], outcomes))
|
||||
if result.get("snapshotManifestSha256"):
|
||||
lines.extend(["", f"- snapshotManifestSha256: `{result['snapshotManifestSha256']}`"])
|
||||
_render_evaluation(lines, result.get("evaluation") or {})
|
||||
for name, arm in safe["arms"].items():
|
||||
lines.append(_arm_line(name, arm))
|
||||
if safe["snapshotManifestSha256"]:
|
||||
lines.extend(["", f"- snapshotManifestSha256: `{safe['snapshotManifestSha256']}`"])
|
||||
_render_evaluation(lines, safe["evaluation"])
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
|
||||
@ -11,7 +11,7 @@
|
||||
## 执行状态(2026-07-19)
|
||||
|
||||
- 已落地并同步到 `agent-example/main`:`fbb262c`(冻结/细纲/评分合同)、`a54a3a4`(审查反馈收紧与回放编排)、`1243d9b`(外部 planner execute 链路测试)。
|
||||
- 已验证:replay-eval 58 个离线测试、fine-outline 合同 3 个测试全部通过;fake runner 已跑通三臂 planner、逐臂 detector、双 judge 反序盲评、稳定性门和去盲矩阵。测试只使用合成 fixture,不代表参考作品评测结果。
|
||||
- 已验证:replay-eval 70 个离线测试、fine-outline 合同 3 个测试全部通过;fake runner 已跑通三臂 planner、真正不识别臂卡内容的逐候选 detector、双 judge 反序盲评、子进程超时失败关闭、稳定性门和去盲矩阵。测试只使用合成 fixture,不代表参考作品评测结果。
|
||||
- 已新增并验证:`audit_leakage.py` 对公共快照和三臂卡注入区执行内容级事实审计;`load_reference_work.py` 通过 PostgreSQL 只读事务组装仓库外临时配置,候选卡标记为 `eval_draft`,不进入生产检索。
|
||||
- 已执行真实适配 smoke:work=8、冻结 488、目标 489,组装 31 个完整历史大纲窗、6 个近章细纲摘要、正确/错配卡各 2 张;送入回放仍为 `blocked_authorization`(`example_reference_work` 没有授权快照),未生成 `snapshot_manifest`,未调用模型。
|
||||
- 已执行真实数据库前置 smoke:深空之影 `work_id=8` 的 `example_reference_work` 表没有不可变授权快照/版权用途字段,结果为 `blocked_authorization`,三臂均未调用模型。
|
||||
@ -206,7 +206,7 @@ arm ∈ {outline_only, outline_plus_cards, outline_plus_placebo_cards}
|
||||
|
||||
### 6.1 审查智能体
|
||||
|
||||
审查使用现有 `detector`,但要新增“细纲候选”输入分支;它只看到冻结到 N 的 writer/planner 基线加检测增量,不看到目标章标准事实。检查:
|
||||
审查使用现有 `detector`,但要新增“细纲候选”输入分支;它只看到匿名候选和冻结到 N 的公共 writer/planner 基线,不看到 arm、`cardInjection`、`cardManifest`、任何臂特有卡内容或目标章标准事实。卡注入合法性留在确定性预检。检查:
|
||||
|
||||
- 细纲字段是否完整、事件顺序是否自洽;
|
||||
- 角色/势力/地点/能力是否违反 N 时点已知事实;
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user