zizi d80df3ed7c 治理: 剩余问题收口——humanization 规则/样例数据库权威 + PG 集成全通过 + 行为评测脚手架 + raw 冲突备忘录
范围(不含 design-story-foundation、docs/design、docs/write-chapter、
craft/、humanization/README.md 等进行中改动):

1. humanization 规则/样例运行时数据库权威
   - db/ddl/111:example_ai_flavor_rule / example_ai_flavor_sample /
     example_ai_flavor_rule_event(append-only 生命周期留痕),已应用到 muse-example
   - deai/load_db.py:数据库装载器,激活门/重复检测/指纹与文件装载器同源;
     数据库失败关闭,不静默回退 Git 文件资产
   - humanization/tools/seed_rules_db.py:YAML 种子单事务同步,幂等、
     变化留痕、--strict 对 db-only 行失败关闭;真实库已种入 26 规则/107 样例
   - prevent/diagnose/revise 生产路径切到数据库读取(--offline 显式读文件),
     落库前新鲜度检查与合同声明来源一致;四个 SKILL.md 数据库合同同步
   - 真实库验证:规则库指纹与文件种子一致(v-609bc40e21d0b5db),
     三个生产脚本端到端从库装载通过

2. PostgreSQL 集成:显式授权后 9/9 通过
   - 此前被依赖门阻断的 6 个 _db/smoke 测试全部通过
   - extract rollback 冒烟改为回滚事务内自给夹具(pending 窗/草稿缺失时自建),
     不再依赖瞬时生产状态;夹具残留核验为 0

3. Skill 行为评测脚手架(真实执行数量仍为 0)
   - harness/evals/skill_eval.py:场景合同、六类评测范畴、适配器和结构化裁决报告
   - diagnose-ai-flavor 参考场景 4 条 + 管道自测 7 项通过
   - 真实模型适配器未授权时以稳定码 EVAL_ADAPTER_UNAVAILABLE 失败关闭;
     清单登记 skill_behavior_eval 条目,默认被依赖门阻断

4. evaluate-frozen-replay raw 存储边界冲突
   - docs/2026-08-19 备忘录:平台 DB-first 合同(创始人批准)与回放链
     仓外 vault 强制的冲突事实、两个选项和裁决前约束;运行时合同未单方面改写

5. harness 自身修复
   - runner 对账语义:行为评测入口不参与测试资产双向等值,但登记文件必须存在;
     manifest 保留 skill_behavior_eval 布尔字段并校验类型
   - 新增 2 条对账回归用例

验证证据: harness 三组自测 15+15+7 通过;静态审计 32 Skill / 0 问题;
76 个非数据库条目通过;9 个 PostgreSQL 集成条目显式授权后通过;
行为评测条目默认阻断;py_compile 与 git diff --check 通过。
未调用真实模型、embedding 或额度;真实行为评测执行数量仍为 0。
2026-08-19 02:41:25 +08:00

5380 lines
255 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#!/usr/bin/env python3
"""extract-work-knowledge Skill:作品面知识抽取执行器(方案 v6,创始人 2026-07-14 认可)。
把参考书自己的实体/关系/大纲按「全实体统一生长」机制抽进正式候选层:
一实体一卡(不分全卡/轻卡),每窗流程=
⓪ 机械预扫(零 AI):全库已知名字在本窗正文精确扫描 → 在场已知实体子集
① 实体观察(AI×1):新名字(顺带产初卡)/ 已知实体新信息;纯出场由名称索引逐章机械登记
② 判重(机械+观察自判):别名表/留档表精确查 → 疑似别名转观察材料
③ 立卡门槛:跨章戏份才立卡(单章龙套进出场留档,跨窗合计跨章再补立)
④ 卡更新(AI×0-3):只对有新信息的卡,每批≤6 张,读全卡→只回变更字段→三类合并
⑤ 关系增量(AI×1):核心角色两两关系变化 → 滚进关系卡(甲乙锚草稿编号)
⑥ 机械收尾:出场留档/窗状态/卡水位/覆写审计
防膨胀(第四轮评审 G1/G5):观察调用只带「窗内命中」实体索引(不带全库);
窗切割限 12 章/3.5 万字双闸。防重跑自噬:同窗重跑先按审计撤销再重写。
连接三段式(洞①):每窗读→算→写切短连接,DB 连接绝不跨 LLM/嵌入 HTTP 存活;实体、关系和最终嵌入
各自在短事务内提交。嵌入:默认关(试跑期延后);开 --semantic-dedup 时,最终嵌入必须完整成功后才把
窗口置为 done。
命令:
windows --work-id N 机械切正文窗(幂等,from_chapter 锚)
run --work-id N [--max-windows K] [--max-calls M] 按窗顺序跑,断点续跑
status --work-id N 进度
"""
import hashlib
import json
import math
import pathlib
import re
import sys
import unicodedata
from collections import Counter
from copy import deepcopy
from numbers import Real
import click
import psycopg
# 复用章级管线的敏感降级链与 llm 入口(trust_env/重试/JSON 容错同源)。
# 拆分后 upgrade 与 parse-book 分属两个 skill:parse_llm(敏感降级链+llm 入口+DSN/TENANT)留
# parse-book 字节不动,本脚本单向跨 skill 引用(upgrade→parse-book,合法);upgrade_work_lock 与本脚本同目录。
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "deconstruct-book" / "scripts"))
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "call-content-model" / "scripts"))
from parse_llm import m3_json, SensitiveHardStop, IDENTITY, TENANT, DSN # noqa: E402
from upgrade_work_lock import UpgradeWorkLockUnavailable, upgrade_work_lock # noqa: E402
# 语义判重(P1)复用 embed-knowledge 的嵌入通道(同模型同维、与检索端语义对齐)——
# 只在开启 --semantic-dedup 时才真调,默认关(试跑期嵌入延后,见文件头注释);
# build_embed_text/MODEL/DIM/ACTOR 供最终嵌入短事务复用检索端同源文本构造器与列常量。
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "embed-knowledge" / "scripts"))
from embed_drafts import (_session as _embed_session, embed_texts, build_embed_text, # noqa: E402
MODEL as EMBED_MODEL, DIM as EMBED_DIM, ACTOR as EMBED_ACTOR)
# ── 窗切割参数(方案 §五B:3–5 万字/窗、10–15 章,取保守双闸防观察输出过载)──
WIN_MAX_CHARS = 35000 # 单窗正文字数上限
WIN_MAX_CHAPS = 12 # 单窗章数上限
UPDATE_BATCH = 6 # 卡更新每批张数上限(防长清单丢字段)
CROSS_TYPE_ALIAS_MAX_CANDIDATES = 8 # 跨型别名裁决单批上限,超出直接失败关闭
SOURCE_TYPE = "upgrade_book" # 独立来源标记:与范式卡 parse_book 隔离判重/检索/确认
# 墓碑 updater 只允许来自明确的恢复/撤销路径;上下文再从中选择自己的子集。
TOMBSTONE_UPDATER_ALLOWLIST = frozenset(("upgrade-reset", "upgrade-undo"))
# 作品面七型(六实体型+关系型;大纲卡全书收尾单独做,不在窗循环内)
ENTITY_TYPES = ("character", "location", "item", "faction", "power_system", "event")
RELATION_TYPE = "character_relation"
# 追加类字段白名单(值为数组的字段一律追加)。「演变历程」是里程碑对象数组,走独立对象
# 合并路径(MILESTONE_FIELDS);其余是字符串条目数组(历史上带 [窗N] 前缀)。「大事记」「经历」
# 是历史孤儿名(任何 schema 都没定义、会被合同守卫裁掉),保留仅为向后兼容旧数据、不再新用——
# 升格卡改造(2026-07-17)后已发生台阶统一记入「演变历程」。
APPEND_FIELDS = {"成长弧线", "演变轨迹", "大事记", "经历", "演变历程"}
# 里程碑对象数组字段:条目是 {章,台阶,周期} 结构化对象,去重按台阶内容、排序按真实章号
# (不用运行时窗号)——升格卡改造 P0 落点,区别于上面的字符串条目追加字段。
MILESTONE_FIELDS = {"演变历程"}
# 生命周期枚举(设计稿 §4.2):每条里程碑「周期」的取值域。
LIFECYCLE = ("登场", "成长", "高光", "退场", "结局")
# 语义判重(P1,设计稿 §8.2):召回同书近邻相似度 ≥ 此阈值才交 M3 终判。
# 0.78→0.60(2026-07-18 补3洞零落库小样实测校准):改名场景(洞① 病例)查询侧是初观察薄快照、
# 存储侧是长成的厚卡,文本不对称压低余弦——同实体改名对实测 0.63-0.66("联邦生物机甲技术"vs
# 卡「生物机甲」0.6565),0.78 永不触发语义层;无关对实测 ≤0.51,0.60 落在分离带内留边距。
# 放宽只增终判候选量,并卡仍须 M3 终判"同型同一实体"才发生(跨型/无关由终判把关)。
DEDUP_SIM_THRESHOLD = 0.60
CHAP_BIG = 10 ** 9 # 章号缺失/待人工的里程碑,排序时排到最后
STEP_MAX = 80 # 里程碑「台阶」机械字数上限(洞③):提示词目标 ≤40 字,机械上限 80,双层防跑飞长文
# 已知实体观察点由模型提供事实摘要;保留短提示词目标,同时给后续更新 prompt 留出足够上下文空间。
KNOWN_INFO_OBSERVATION_MAX = 600
OBSERVATION_MODEL_KEYS = ("新名字", "已知实体新信息")
OBSERVATION_INTERNAL_KEYS = OBSERVATION_MODEL_KEYS + ("纯出场",)
# 兼容已有私有测试 helper;生产 observation 调用必须使用 OBSERVATION_MODEL_KEYS。
OBSERVATION_KEYS = OBSERVATION_INTERNAL_KEYS
OBSERVATION_ITEM_ALLOWED_KEYS = {
"新名字": frozenset(("型", "名称", "正文原名", "别名", "一句话摘要", "字段", "出场章", "疑似别名指向")),
"已知实体新信息": frozenset(("名称", "型", "观察点", "出场章")),
"纯出场": frozenset(("名称", "出场章")),
}
OBSERVATION_MODEL_ITEM_ALLOWED_KEYS = {
key: OBSERVATION_ITEM_ALLOWED_KEYS[key]
for key in OBSERVATION_MODEL_KEYS
}
# 登场机械兜底只允许使用名称与型的中性描述,禁止把跨章摘要事实绑定到首章。
DEBUT_TYPE_LABELS = {
"character": "人物",
"location": "地点",
"item": "物件",
"faction": "组织",
"power_system": "能力体系",
"event": "事件",
}
class EmbeddingOwnershipConflict(RuntimeError):
"""同一嵌入唯一键被两个活跃 draft 的当前 payload 同时声明。"""
class UpgradeDraftWriteConflict(RuntimeError):
"""升格卡已离开可写状态,或锁定后的 revision 写入失败。"""
class AliasOwnershipConflict(RuntimeError):
"""同书同别名已唯一映射到另一张 canonical 卡。"""
class CompensationFenceConflict(RuntimeError):
"""窗提交后的 draft 已被外部改动,禁止补偿覆盖。"""
class RelationOutputConflict(RuntimeError):
"""关系模型对同一无序实体对给出内容不同的重复项。"""
class EntityUpdateOutputConflict(RuntimeError):
"""实体更新模型对同一 draft 的同一字段给出不同值。"""
COMPENSATION_FAILED_PREFIX = "compensation-failed:"
UPGRADE_FENCE_VERSION = "upgrade-fence:v1"
# 输入摘要必须绑定运行合同;修改抽取批量、卡来源或嵌入模型都会使旧窗口拒绝继续写入。
UPGRADE_CONTRACT = {
"windowProtocol": UPGRADE_FENCE_VERSION,
"sourceType": SOURCE_TYPE,
"updateBatch": UPDATE_BATCH,
"embeddingModel": EMBED_MODEL,
"embeddingDimensions": EMBED_DIM,
}
# 仅允许修复已核实的 work8 单卡;其他作品、draft 或名称必须显式失败关闭。
QUALITY_REPAIR_CONTRACT = "manual-quality-repair:v1"
QUALITY_REPAIR_UPDATER = "manual-quality-repair-work8"
QUALITY_REPAIR_WORK_ID = 8
QUALITY_REPAIR_DRAFT_ID = 25360
QUALITY_REPAIR_NAME = "神权之星"
QUALITY_REPAIR_OLD_TYPE = "item"
QUALITY_REPAIR_OLD_DRAFT_UPDATER = "upgrade"
QUALITY_REPAIR_PRESENCE_ID = 10879
QUALITY_REPAIR_PRESENCE_WINDOW = 57
QUALITY_REPAIR_PRESENCE_CHAPTER = 403
QUALITY_REPAIR_PRESENCE_OBSERVATION = (
"天启神国主星,巨型机械行星表面如巨大眼睛环绕棱形机械宝石,象征无上神权"
)
QUALITY_REPAIR_REVOKED_ALIAS_ID = 8738
QUALITY_REPAIR_ALIAS_CANONICAL = "天启神国·神之宫殿"
QUALITY_REPAIR_ALIAS_VALUE = "神权之星"
QUALITY_REPAIR_ALIAS_EVIDENCE_WINDOW = 86
QUALITY_REPAIR_ALIAS_VERDICT_BY = "semantic"
QUALITY_REPAIR_REVOKED_ALIAS_UPDATER = "manual-revoke-work8-w108"
QUALITY_REPAIR_CARD_WATERMARK = 109
QUALITY_REPAIR_OLD_APPEARANCES = (
566, 567, 569, 570, 576, 577, 580, 581, 583, 584, 586, 587,
)
QUALITY_REPAIR_TOP_LEVEL_KEYS = frozenset({
"type", "名称", "别名", "一句话摘要", "字段", "出场章", "来源", "状态",
"目标库", "可见范围", "_work_id",
})
QUALITY_REPAIR_ITEM_FIELDS = frozenset({
"类别", "品阶", "来历", "能力与限制", "当前持有者", "知情范围", "流转计划",
"演变历程", "演变概括", "前身", "后继",
})
QUALITY_REPAIR_LOCATION_FIELDS = frozenset({
"层级", "地理与环境", "归属势力", "规则特例", "演变历程",
})
# 仅允许清理已核实 work8 的十组 presence 冗余;execute 二次运行按精确快照失败关闭。
PRESENCE_DEDUPE_CONTRACT = "manual-presence-dedupe:v1"
PRESENCE_DEDUPE_WORK_ID = 8
PRESENCE_DEDUPE_GROUP_COUNT = 10
PRESENCE_DEDUPE_ROW_COUNT = 20
PRESENCE_DEDUPE_DELETE_IDS = frozenset({
11582, 11701, 11695, 11698, 11697,
11696, 11700, 11699, 11693, 11694,
})
def _sha256_json(value):
"""以稳定 JSON 编码计算摘要;数据库时间等值统一转字符串,避免驱动类型影响结果。"""
raw = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def _aggregate_window_chapter_texts(chapters):
"""按窗口送模规则把每章多个 block 以换行拼接,保持读段和摘要规则一致。"""
grouped = {}
if chapters and isinstance(chapters[0], dict):
for chapter in chapters:
grouped.setdefault(chapter["order_no"], []).append(
str(chapter.get("content_text") or "")
)
else:
for order_no, _, content in chapters or []:
grouped.setdefault(int(order_no), []).append(str(content or ""))
return {order_no: "\n".join(parts) for order_no, parts in grouped.items()}
def _script_sha256():
"""读取当前实际运行脚本的 SHA-256,避免只绑定 git HEAD 或人工版本字符串。"""
return hashlib.sha256(pathlib.Path(__file__).read_bytes()).hexdigest()
def _window_input_sha(window, chapters, schemas, work_title=None):
"""绑定实际送模输入、schema active 合同、运行合同和当前脚本内容。"""
return _sha256_json({
"window": window,
"workTitle": work_title,
"chapters": chapters,
"chapterPromptText": _aggregate_window_chapter_texts(chapters),
"schemas": schemas,
"contract": UPGRADE_CONTRACT,
"parseUpgradeSha256": _script_sha256(),
})
def _window_state_sha(state, current_window_id):
"""计算本书七域状态摘要;当前窗 status/error 是 marker 载体,不参与自引用 hash。"""
normalized = deepcopy(state)
windows = []
for row in normalized.get("windows", []):
if isinstance(row, dict):
item = deepcopy(row)
if item.get("id") == current_window_id:
item.pop("status", None)
item.pop("error_message", None)
windows.append(item)
elif row and row[0] == current_window_id and len(row) >= 7:
windows.append(tuple(row[:5]) + tuple(row[7:]))
else:
windows.append(row)
normalized["windows"] = windows
return _sha256_json(normalized)
def _upgrade_marker(stage, input_sha, state_sha):
"""生成只有 entity/relation 两种阶段的窗口恢复 marker。"""
if stage not in ("entity", "relation"):
raise CompensationFenceConflict(f"非法 fence stage:{stage}")
return f"{UPGRADE_FENCE_VERSION}:{stage}:{input_sha}:{state_sha}"
def _parse_upgrade_marker(marker):
"""严格解析 v1 marker;未知版本、阶段或非 SHA-256 值一律失败关闭。"""
match = re.fullmatch(
r"upgrade-fence:v1:(entity|relation):([0-9a-f]{64}):([0-9a-f]{64})",
str(marker or ""),
)
if not match:
raise CompensationFenceConflict(f"非法或未知窗口 marker:{marker}")
return match.groups()
def _recovery_decision(marker, current_input_sha, current_state_sha):
"""纯恢复判定:只有 marker/input/state 三者精确一致才允许撤销。"""
try:
_, expected_input, expected_state = _parse_upgrade_marker(marker)
except CompensationFenceConflict:
return "compensation-failed"
if expected_input != current_input_sha or expected_state != current_state_sha:
return "compensation-failed"
return "undo"
def _call_external_without_connection(tracker, external, *args):
"""离线可观测守卫:外部调用入口要求业务连接计数为零。"""
if tracker.get("connections", 0):
raise RuntimeError("外部调用期间仍持有业务数据库连接")
return external(*args)
def _apply_prepared_and_done(conn, prepared, *, apply_one, mark_done):
"""在调用者提供的同一事务内先写 prepared 向量,再标记 done。"""
for item in prepared:
apply_one(item)
mark_done()
conn.commit()
def _lock_upgrade_domains(conn):
"""固定锁仅跨短写事务;代价是短暂串行化不同作品,换取七域摘要无幻读。"""
conn.execute(
"LOCK TABLE muse_content_chapter, muse_content_block, muse_meta_schema, "
"muse_meta_schema_version IN SHARE MODE"
)
conn.execute(
"LOCK TABLE muse_knowledge_draft, example_upgrade_alias, example_upgrade_presence, "
"example_upgrade_card_state, example_upgrade_audit, example_upgrade_window, "
"example_knowledge_embedding IN SHARE ROW EXCLUSIVE MODE"
)
def _capture_window_input(conn, work_id, win_no, *, validate=True):
"""读取 inputSha 的完整组成;字段必须与实际送模正文和作品标题一致。"""
row = conn.execute(
"""SELECT id, window_no, from_chapter, to_chapter, deleted
FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND window_no=%s""",
(TENANT, work_id, win_no),
).fetchone()
if not row:
raise CompensationFenceConflict(f"窗口不存在:work={work_id},window={win_no}")
window = dict(zip(("id", "window_no", "from_chapter", "to_chapter", "deleted"), row))
work_title = conn.execute(
"""SELECT title FROM muse_content_work
WHERE tenant_id=%s AND id=%s AND deleted=FALSE""",
(TENANT, work_id),
).fetchone()
if not work_title:
raise CompensationFenceConflict(f"作品不存在或已删除:work={work_id}")
chapter_rows = conn.execute(
"""SELECT c.id, c.order_no, c.title, c.status, c.revision,
b.id, b.order_no, b.block_type, b.revision, COALESCE(b.content_text,'')
FROM muse_content_chapter c
JOIN muse_content_block b ON b.chapter_id=c.id AND b.deleted=FALSE
WHERE c.tenant_id=%s AND c.work_id=%s AND c.deleted=FALSE
AND c.order_no BETWEEN %s AND %s
ORDER BY c.order_no, b.order_no, b.id""",
(TENANT, work_id, window["from_chapter"], window["to_chapter"]),
).fetchall()
chapters = [
{
"chapter_id": row[0],
"order_no": row[1],
"title": row[2],
"status": row[3],
"revision": row[4],
"block_id": row[5],
"block_order_no": row[6],
"block_type": row[7],
"block_revision": row[8],
"content_text": row[9],
}
for row in chapter_rows
]
if validate:
expected_orders = set(range(window["from_chapter"], window["to_chapter"] + 1))
actual_orders = {chapter["order_no"] for chapter in chapters}
chapter_texts = _aggregate_window_chapter_texts(chapters)
if actual_orders != expected_orders:
raise CompensationFenceConflict(
f"窗口章域不完整:expected={min(expected_orders)}-{max(expected_orders)},"
f"actual={sorted(actual_orders)}"
)
empty_orders = sorted(
order_no for order_no in expected_orders
if not chapter_texts.get(order_no, "").strip()
)
if empty_orders:
raise CompensationFenceConflict(f"窗口正文为空:chapters={empty_orders}")
schemas = conn.execute(
"""SELECT s.schema_key, s.active_version_id, v.field_contract_snapshot
FROM muse_meta_schema s JOIN muse_meta_schema_version v ON v.id=s.active_version_id
WHERE s.tenant_id=%s AND s.schema_key=ANY(%s) ORDER BY s.schema_key""",
(TENANT, list(ENTITY_TYPES + (RELATION_TYPE,))),
).fetchall()
return window, chapters, schemas, work_title[0]
def _capture_window_state(conn, work_id, current_window_id):
"""读取本书七个受控域的完整行像,仅当前 marker 窗忽略 status/error。"""
queries = {
"drafts": ("SELECT d.id, encode(sha256(convert_to(to_jsonb(d)::text, 'UTF8')), 'hex') "
"FROM muse_knowledge_draft d WHERE d.tenant_id=%s AND d.work_id=%s "
"AND d.source_type=%s ORDER BY d.id", (TENANT, work_id, SOURCE_TYPE)),
"aliases": ("SELECT a.id, encode(sha256(convert_to(to_jsonb(a)::text, 'UTF8')), 'hex') "
"FROM example_upgrade_alias a WHERE a.tenant_id=%s AND a.work_id=%s "
"ORDER BY a.id", (TENANT, work_id)),
"presence": ("SELECT p.id, encode(sha256(convert_to(to_jsonb(p)::text, 'UTF8')), 'hex') "
"FROM example_upgrade_presence p WHERE p.tenant_id=%s AND p.work_id=%s "
"ORDER BY p.id", (TENANT, work_id)),
"card_state": ("SELECT s.draft_id, encode(sha256(convert_to(to_jsonb(s)::text, 'UTF8')), 'hex') "
"FROM example_upgrade_card_state s WHERE s.tenant_id=%s AND s.work_id=%s "
"ORDER BY s.draft_id", (TENANT, work_id)),
"audits": ("SELECT a.id, encode(sha256(convert_to(to_jsonb(a)::text, 'UTF8')), 'hex') "
"FROM example_upgrade_audit a JOIN muse_knowledge_draft d ON d.id=a.draft_id "
"WHERE a.tenant_id=%s AND d.work_id=%s AND d.source_type=%s ORDER BY a.id",
(TENANT, work_id, SOURCE_TYPE)),
"windows": ("SELECT w.id, encode(sha256(convert_to("
"CASE WHEN w.id=%s THEN (to_jsonb(w)-'status'-'error_message')::text "
"ELSE to_jsonb(w)::text END, 'UTF8')), 'hex') "
"FROM example_upgrade_window w WHERE w.tenant_id=%s "
"AND w.work_id=%s ORDER BY w.id",
(current_window_id, TENANT, work_id)),
"embeddings": ("SELECT e.id, encode(sha256("
"convert_to((to_jsonb(e)-'embedding')::text, 'UTF8') || "
"coalesce(vector_send(e.embedding), ''::bytea)), 'hex') "
"FROM example_knowledge_embedding e "
"JOIN muse_knowledge_draft d ON d.id=e.draft_id "
"WHERE e.tenant_id=%s AND d.tenant_id=%s AND d.work_id=%s "
"AND d.source_type=%s ORDER BY e.id",
(TENANT, TENANT, work_id, SOURCE_TYPE)),
}
return {name: conn.execute(sql, params).fetchall()
for name, (sql, params) in queries.items()}
def _window_state_domain_counts(state):
"""把七域行像压缩成可审计的逐域行数,供补偿漂移错误回显前后差异。"""
return {name: len(rows) for name, rows in state.items()}
# ── 库内合同(元数据驱动公理:prompt 与守卫同源,禁手写合同)──
def load_entity_contracts(conn):
"""加载作品面七型的字段合同(走 active_version_id,与章级管线同语义)。"""
contracts = {}
for t in ENTITY_TYPES + (RELATION_TYPE,):
snap = conn.execute(
"""SELECT v.field_contract_snapshot FROM muse_meta_schema_version v
JOIN muse_meta_schema s ON s.active_version_id=v.id
WHERE s.tenant_id=%s AND s.schema_key=%s""", (TENANT, t)).fetchone()[0]
contracts[t] = {"中文名": snap.get("中文名", t), "判据": snap.get("判据", ""),
"字段": [f for f in snap.get("特有字段", [])]}
return contracts
def render_entity_contracts(contracts, types):
"""合同渲染为 markdown 表(观察/更新 prompt 共用)。"""
parts = []
for t in types:
c = contracts[t]
rows = "\n".join(f"| {f['key']} | {f.get('说明', '')} |" for f in c["字段"])
parts.append(f"### {t}({c['中文名']})\n判据:{c['判据']}\n\n"
f"| 字段 key | 说明 |\n|---|---|\n{rows}")
return "\n\n".join(parts)
# ── 窗切割(机械,零 AI)──
def cut_windows(conn, work_id):
"""按章边界贪心切正文窗:累计超 3.5 万字或 12 章即断窗。幂等(from_chapter 锚)。"""
rows = conn.execute(
"""SELECT c.order_no, COALESCE(b.word_count, length(b.content_text))
FROM muse_content_chapter c
JOIN muse_content_block b ON b.chapter_id=c.id AND b.deleted=FALSE
WHERE c.tenant_id=%s AND c.work_id=%s AND c.deleted=FALSE
ORDER BY c.order_no""", (TENANT, work_id)).fetchall()
wins, cur, chars = [], [], 0
for order_no, wc in rows:
if cur and (chars + (wc or 0) > WIN_MAX_CHARS or len(cur) >= WIN_MAX_CHAPS):
wins.append((cur[0], cur[-1]))
cur, chars = [], 0
cur.append(order_no)
chars += (wc or 0)
if cur:
wins.append((cur[0], cur[-1]))
n = 0
for i, (a, b) in enumerate(wins, 1):
r = conn.execute(
"""INSERT INTO example_upgrade_window
(work_id, window_no, from_chapter, to_chapter, tenant_id)
VALUES (%s,%s,%s,%s,%s)
ON CONFLICT (tenant_id, work_id, from_chapter) DO NOTHING""",
(work_id, i, a, b, TENANT))
n += r.rowcount
conn.commit()
return len(wins), n
# ── 每窗材料与已知名加载 ──
def load_window_material(conn, work_id, a, b):
"""返回窗正文拼接文本与按绝对章号聚合的原始正文。"""
rows = conn.execute(
"""SELECT c.order_no, c.title, b2.content_text
FROM muse_content_chapter c
JOIN muse_content_block b2 ON b2.chapter_id=c.id AND b2.deleted=FALSE
WHERE c.tenant_id=%s AND c.work_id=%s AND c.order_no BETWEEN %s AND %s
AND c.deleted=FALSE ORDER BY c.order_no, b2.order_no, b2.id""",
(TENANT, work_id, a, b)).fetchall()
chapter_titles = {}
for order_no, chapter_title, content in rows:
chapter_titles.setdefault(int(order_no), chapter_title)
chapter_texts = _aggregate_window_chapter_texts(rows)
text = "\n\n".join(
f"## 第{order_no}章 {chapter_titles[order_no]}\n{chapter_texts[order_no]}"
for order_no in sorted(chapter_texts)
)
return text, chapter_texts
def load_window_text(conn, work_id, a, b):
"""兼容旧调用:只返回带章标题的窗内正文拼接文本。"""
return load_window_material(conn, work_id, a, b)[0]
def load_known(conn, work_id):
"""加载判重底册:名字索引、按卡完整合法别名集合与 presence 留档出场章。
aliases_by_draft 同时汇总 payload 与独立别名表,供后续正文实体命中过滤使用。"""
name_map = {}
aliases_by_draft = {}
revisions = {}
for did, entity_type, name, brief, aliases, revision in conn.execute(
"""SELECT id,
draft_payload->>'type',
draft_payload->>'名称',
COALESCE(draft_payload->>'一句话摘要',''),
COALESCE(draft_payload->'别名','[]'::jsonb),
revision
FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s
AND status='pending' AND deleted=FALSE""",
(TENANT, work_id, SOURCE_TYPE)).fetchall():
revisions[did] = revision
t, nm = entity_type or "", (name or "").strip()
if nm:
name_map[nm] = (did, t, brief)
for raw_alias in aliases or []:
alias = _clean_alias(raw_alias)
if alias:
name_map[alias] = (did, t, brief)
aliases_by_draft.setdefault(did, set()).add(alias)
for cn, al in conn.execute(
"SELECT canonical_name, alias FROM example_upgrade_alias "
"WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE",
(TENANT, work_id)).fetchall():
alias = _clean_alias(al)
if cn in name_map and alias:
target = name_map[cn]
name_map[alias] = target
aliases_by_draft.setdefault(target[0], set()).add(alias)
presence = {}
for t, nm, ch in conn.execute(
"SELECT entity_type, name, chapter_no FROM example_upgrade_presence "
"WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE",
(TENANT, work_id)).fetchall():
presence.setdefault((t, nm), set()).add(ch)
return name_map, presence, aliases_by_draft, revisions
def prescan(name_map, text):
"""机械预扫(G1 防膨胀核心):只把「本窗正文出现」的已知实体带进观察调用。"""
hit = {}
for nm, (did, t, brief) in name_map.items():
if nm and nm in text:
# 同卡多名字只留一条(正名优先:先插入的是正名)
hit.setdefault(did, (nm, t, brief))
return {nm: (did, t, brief) for did, (nm, t, brief) in hit.items()}
def mechanical_presence(onstage, name_map, aliases_by_draft, chapter_texts):
"""按索引机械生成已知卡的逐章出场,不让模型重抄 presence 清单。
``onstage`` 只决定本窗哪些已知 draft 需要检查;实际章号逐章扫描规范名与全部
合法 alias。这样即使同一 draft 同时有 known-info,presence 也不依赖模型提供完整
出场章;返回值只使用 planner 既有的内部 ``纯出场`` 形状。
"""
if not isinstance(onstage, dict):
raise RuntimeError("机械 presence 的 onstage 必须是对象")
if not isinstance(name_map, dict) or not isinstance(aliases_by_draft, dict):
raise RuntimeError("机械 presence 的名称索引必须是对象")
if not isinstance(chapter_texts, dict):
raise RuntimeError("机械 presence 的 chapter_texts 必须是对象")
drafts = {}
for matched_name, target in onstage.items():
if not isinstance(matched_name, str) or not matched_name.strip():
raise RuntimeError("机械 presence 命中的名称非法")
if not isinstance(target, (tuple, list)) or len(target) < 3:
raise RuntimeError(f"机械 presence owner 投影不完整:名称={matched_name}")
draft_id = target[0]
canonical = _resolve_canonical_name(name_map, aliases_by_draft, matched_name)
previous = drafts.setdefault(draft_id, canonical)
if previous != canonical:
raise RuntimeError(
f"机械 presence 同一 draft 规范名不唯一:draft={draft_id},"
f"names={previous!r},{canonical!r}"
)
chapter_numbers = sorted(_int_chaps(chapter_texts.keys()))
entries = []
for draft_id, canonical in sorted(drafts.items(), key=lambda item: (str(item[1]), item[0])):
aliases = sorted(aliases_by_draft.get(draft_id, set()))
chapters = sorted(
_filter_entity_chapters(canonical, aliases, chapter_numbers, chapter_texts)
)
if not chapters:
# onstage 已命中整窗正文,但逐章材料若不一致,不能伪造 presence;上层
# 的输入章域校验会负责发现正文漂移。
continue
entries.append({"名称": canonical, "出场章": chapters})
return entries
def populate_mechanical_presence(obs, onstage, name_map, aliases_by_draft, chapter_texts):
"""把机械 presence 写入内部 observation;模型不得提供该数组。"""
if not isinstance(obs, dict):
raise RuntimeError("实体观察结果必须是对象")
obs["纯出场"] = mechanical_presence(
onstage, name_map, aliases_by_draft, chapter_texts
)
return obs
def _observation_scan_names(item, key, name_map, aliases_by_draft):
"""返回 observation 项可用于逐章机械扫描的名称集合。
新名字只接受自身名称、通过别名准入的别名和模型给出的疑似别名指向;
已知实体则扩展到底册同 draft 的规范名与合法别名。模型提供的出场章
从不参与该集合或后续筛选。
"""
if not isinstance(item, dict):
raise RuntimeError(f"{key} observation 项必须是对象")
raw_name = item.get("名称")
name = raw_name.strip() if isinstance(raw_name, str) else ""
if not name:
raise RuntimeError(f"{key} observation 项缺少有效名称")
candidates = [name]
if key == "新名字":
aliases = item.get("别名", [])
if aliases is not None and not isinstance(aliases, list):
raise RuntimeError(f"{key}别名必须是字符串列表:名称={name}")
candidates.extend(
alias for alias in (aliases or [])
if isinstance(alias, str) and _clean_alias(alias)
)
hint = item.get("疑似别名指向")
if hint is not None:
if not isinstance(hint, str):
raise RuntimeError(f"{key}疑似别名指向必须是字符串:名称={name}")
if hint.strip():
candidates.append(hint.strip())
evidence_name = item.get("正文原名")
if evidence_name is not None:
if not isinstance(evidence_name, str):
raise RuntimeError(f"{key}正文原名必须是字符串:名称={name}")
if evidence_name.strip():
candidates.append(evidence_name.strip())
return candidates
target = name_map.get(name)
if not isinstance(target, (tuple, list)) or len(target) < 3:
return candidates
draft_id = target[0]
aliases = set(aliases_by_draft.get(draft_id, set()))
# name_map 也可能含有尚未回填 alias 表的合法称谓,按同 draft 投影一并扫描。
aliases.update(
candidate_name
for candidate_name, candidate_target in name_map.items()
if isinstance(candidate_target, (tuple, list))
and len(candidate_target) >= 1
and candidate_target[0] == draft_id
)
try:
candidates.append(_resolve_canonical_name(name_map, aliases_by_draft, name))
except RuntimeError:
# 底册无法解析规范名时仍保留直接命中的模型名称;无正文命中会被记录后过滤。
pass
candidates.extend(sorted(aliases))
return candidates
def populate_model_observation_chapters(obs, name_map, aliases_by_draft, chapter_texts):
"""用逐章正文索引覆盖模型 observation 的出场章。
出场章是确定性 presence 证据,不是模型事实。每个新名字/已知变化项都必须
至少命中一个正文章节;模型缺失、错误或恶意提供的出场章都会被覆盖,零命中
的不可验证候选记录到 stderr 后过滤,不进入 planner。
"""
if not isinstance(obs, dict):
raise RuntimeError("实体观察结果必须是对象")
if not isinstance(chapter_texts, dict):
raise RuntimeError("实体观察逐章正文必须是对象")
chapter_numbers = sorted(_int_chaps(chapter_texts.keys()))
dropped = []
for key in OBSERVATION_MODEL_KEYS:
items = obs.get(key)
if not isinstance(items, list):
# 顶层数组/项结构仍交给严格 observation validator;这里不静默补空数组。
continue
retained = []
for index, item in enumerate(items):
label = f"{key}[{index}]"
# 名称/字段结构错误必须保留给严格 validator;helper 只接管合法名称项的
# 出场章计算,避免把 unrelated structure error 误报成无正文命中。
if not isinstance(item, dict):
retained.append(item)
continue
raw_name = item.get("名称")
if not isinstance(raw_name, str) or not raw_name.strip():
retained.append(item)
continue
try:
names = _observation_scan_names(item, key, name_map, aliases_by_draft)
except RuntimeError:
retained.append(item)
continue
chapters = sorted(
_filter_entity_chapters(
names[0], names[1:], chapter_numbers, chapter_texts,
)
)
if not chapters:
dropped.append((label, item.get("名称")))
continue
# 无论模型是否带出场章,都只保留正文索引的结果。
item["出场章"] = chapters
retained.append(item)
obs[key] = retained
if dropped:
details = ", ".join(f"{label}={name!r}" for label, name in dropped[:12])
suffix = "" if len(dropped) <= 12 else f" …共{len(dropped)}项"
print(
f"[过滤] observation 无正文索引命中,丢弃不可验证语义候选:{details}{suffix}",
file=sys.stderr,
)
return obs
def normalize_model_observation_aliases(obs, contracts=None):
"""只归一模型常见的结构键别名,不放宽字段合同或改写语义值。"""
if not isinstance(obs, dict):
return obs
contract_fields = {"演变历程"}
structural_fields = {
"型", "名称", "正文原名", "别名", "一句话摘要", "字段", "出场章", "疑似别名指向",
}
for contract in (contracts or {}).values():
for field in (contract.get("字段", []) if isinstance(contract, dict) else []):
if isinstance(field, dict) and isinstance(field.get("key"), str):
contract_fields.add(field["key"])
for key in OBSERVATION_MODEL_KEYS:
items = obs.get(key)
if not isinstance(items, list):
continue
for item in items:
if not isinstance(item, dict):
continue
if "type" in item and "型" not in item:
raw_type = item.get("type")
if isinstance(raw_type, str) and raw_type.strip() in ENTITY_TYPES:
item["型"] = raw_type.strip()
item.pop("type", None)
if key == "新名字":
nested_fields = item.get("字段")
if nested_fields is None:
nested_fields = {}
item["字段"] = nested_fields
if isinstance(nested_fields, dict):
for field_name in sorted(contract_fields):
if field_name in structural_fields:
continue
if field_name in item:
if field_name not in nested_fields:
nested_fields[field_name] = item[field_name]
item.pop(field_name, None)
return obs
# ── 提示词(缓存友好:固定规则前置,正文窗次之且窗内多调用共享,任务尾置)──
def observe_prompt(contracts, title, a, b, text, onstage):
onstage_lines = "\n".join(f"- {t}|{nm}|{brief}" for nm, (_, t, brief) in onstage.items()) or "(无)"
return f"""【功能指令(parse-book 作品面升格·实体观察)】
通读本窗正文,只产出两类需要语义判断的变化结果(只输出一个 JSON 对象)。
1) 新名字:正文出现、但「在场已知实体」清单里没有的实体(六型:{"/".join(ENTITY_TYPES)})。每个给:型、规范名称、**正文原名**(正文中逐字出现的短名称/短语,不得改写)、别名、一句话摘要、按该型合同能填的字段(有正文证据才填)。若你怀疑它其实是清单中某已知实体的别名/改名/化名,填「疑似别名指向」。不要输出出场章,出场章由系统逐章扫描正文确定。**凡该型合同含「演变历程」字段的新实体,必须给「演变历程」的首条登场里程碑**(章=首次出场的真实章号,台阶=以什么身份/形态/状态登场的一句话 ≤40 字,周期=登场);本窗另见其进化台阶的,照常在「演变历程」里多给几条。
2) 已知实体新信息:清单中实体在本窗的实质新信息(境界变化/性格显露/重大经历/立场转变),每条给型、名称和一条观察点;建议≤60字,机械上限≤600字,超出按前600字截断。没有实质新信息的不要报。不要输出出场章,系统会按该名称及底册规范名/合法别名逐章扫描正文确定。
纯出场登记由系统根据名称索引、规范名/合法别名和逐章正文机械计算,不要输出「纯出场」数组,也不要为完整性重抄在场清单。
纪律:一次性龙套(单章无名或仅路过)不报进新名字;实体判据与字段以合同为准,无证据不填;不脑补。数值(战力/指数/排名等)必须正文原样出现才可写,禁止推算或编造。组织改组/合并产生的新组织是**新实体**(走新名字),不是旧组织的别名。「疑似别名指向」只在确为同一实体改名/化名时填。
里程碑纪律(凡填「演变历程」字段必守):每条是对象 {{"章": 正文原样出现的真实章号(整数如 420,跨多章连续事件用区间字符串如 "420-423"), "台阶": "进化到什么+靠什么关键事件的一句话(≤40 字,只说跃迁+关键事件,不展开过程)", "周期": 登场/成长/高光/退场/结局 之一, "证据": "所标章节正文原样连续短句(8–30字)"}};**证据必须逐字来自所填章号对应正文,系统会机械核验,错章、缺证据或改写证据均拒收入库;证据只用于校验,不写入卡体**。必须用真实章号,禁止 [窗N] 窗号、禁止"本窗/近期/前段"这类相对指代(卡会脱离运行环境被单独阅读);无法给出章号与原文证据时不要输出该里程碑;当前态字段(品阶/能力与限制/摘要等)只写"现在什么样"的干净值,历史进化流水一律进「演变历程」、不许塞进当前态字段。**体系级纪律(power_system/faction 这类体系/组织卡必守)**:其「演变历程」只收**体系级**变化(代际更替/规则改写/整体兴衰/版图重划);某台具体机甲的战斗、某名成员的晋升这类**个体单位事件**绝不写进体系/组织卡,只记进各自实体卡(无卡则走新名字或纯出场),防止体系卡被单位流水账污染。
【六型字段合同】
{render_entity_contracts(contracts, ENTITY_TYPES)}
【输出规则(只输出一个 JSON 对象)】
{{"新名字": [{{"型": "character", "名称": "", "正文原名": "", "别名": [], "一句话摘要": "", "字段": {{}}, "疑似别名指向": ""}}],
"已知实体新信息": [{{"名称": "", "型": "", "观察点": ""}}]}}
━━━ 本窗材料(每窗不同,非规则)━━━
《{title}》第 {a}–{b} 章正文:
{text}
【在场已知实体】(预扫命中,判重参照)
{onstage_lines}"""
def update_prompt(contracts, title, a, b, text, cards_with_obs):
cards_json = json.dumps([{"draft_id": d, "当前卡": p, "本窗观察点": o}
for d, p, o in cards_with_obs], ensure_ascii=False, indent=1)
return f"""【功能指令(parse-book 作品面升格·卡增量更新)】
下列每张卡给出「当前卡全文」与「本窗观察点」。对照本窗正文,**只输出需要变更的字段**:
- 覆写类字段(性格底色/说话方式/当前状态等标量):**必须输出该字段完整的新全量值**——旧值里仍然成立的信息要保留进新值,禁止只写"新增…"式增量(那会把旧信息抹掉);
- 里程碑字段(演变历程):每当实体发生境界/代际/形态/能力的跃迁,或到达登场/高光/退场/结局节点,**追加**一条里程碑对象 {{"章": 真实章号(整数如 420 或跨章区间字符串 "420-423"), "台阶": "进化到什么+靠什么关键事件的一句话(≤40 字,只说跃迁+关键事件,不展开过程)", "周期": 登场/成长/高光/退场/结局 之一, "证据": "所标章节正文原样连续短句(8–30字)"}}——只输出本窗**新增**里程碑(不重抄旧条目,不输出 _win 等内部键);证据必须逐字来自所填章号对应正文,系统机械核验后丢弃证据字段,错章、缺证据或改写证据均拒收;章必须是正文原样章号、**禁 [窗N] 与"本窗/近期"相对指代**;无法给出章号与证据时不要输出该里程碑;并入观察材料里的别名/判重「初卡材料」时,其中「登场」条目**若本卡演变历程已有登场里程碑则不再重复追加**,其余台阶照常判断追加;**体系级纪律**:power_system/faction 这类体系/组织卡的演变历程只收体系级变化(代际更替/规则改写/整体兴衰/版图重划),某台具体机甲的战斗、某名成员的晋升这类个体单位事件绝不写进体系/组织卡;
- 其他追加类字段(成长弧线=未来计划、演变轨迹等数组):只输出本窗新增条目(不要重抄旧条目,不要自己加 [窗N] 前缀,系统会加);
- **当前态字段干净纪律**:品阶/能力与限制/当前状态/一句话摘要等只写"现在是什么样"的全量当前值——历史进化流水一律进「演变历程」里程碑,**绝不把成长史塞进当前态字段**(这是老卡把升级线塞进「戏剧作用/流转计划/跨体系换算」污染字段语义的病根,务必避免);
- 没有变化的字段不要输出;整卡无实质变化则不输出该卡;
- 该实体的身份/处境已发生重大变化时,把「一句话摘要」也作为变更字段输出(写当前态全量)。
纪律:以正文为证据,不脑补;数值必须正文原样出现,禁止推算;字段 key 必须来自该型合同;**禁止「本窗/本段/近期」等相对时间指代**——写绝对状态或带章号(卡会脱离本窗被单独阅读);只写**该实体自己**的信息(他人的任命/心理/变化不得写入本卡);每卡别名有新发现可在「别名新增」里给——只收**该实体自己**的新别名(他人对它的称呼算,它对别的实体的称呼不算),且必须是可在正文原样出现的纯名字(禁带括号注释与说明文字,禁单字)。
【相关型字段合同】
{render_entity_contracts(contracts, sorted({p.get("type") for _, p, _ in cards_with_obs} & set(ENTITY_TYPES)))}
【输出规则(只输出一个 JSON 对象;即使本批所有卡都无变化,也必须输出 {{"更新": []}},不得省略"更新"键)】
{{"更新": [{{"draft_id": 数字, "变更字段": {{"字段key": "新值或新增条目数组"}}, "别名新增": []}}]}}
━━━ 本窗材料 ━━━
《{title}》第 {a}–{b} 章正文:
{text}
【待更新的卡】
{cards_json}"""
def relation_prompt(contracts, title, a, b, text, char_cards, existing_rels):
chars = "\n".join(f"- draft_id={d}|{p.get('名称')}|{p.get('一句话摘要', '')}"
for d, p in char_cards)
rels = "\n".join(f"- {r.get('甲方名称')} × {r.get('乙方名称')}:{r.get('关系类型', '')}"
for _, r in existing_rels) or "(暂无)"
return f"""【功能指令(parse-book 作品面升格·人物关系增量)】
基于本窗正文,报告下列核心角色**两两之间**的关系变化(新建立的关系 / 已有关系的演变)。
只报有正文证据的实质变化;没有变化输出空数组。
【character_relation 字段合同】
{render_entity_contracts(contracts, [RELATION_TYPE])}
【输出规则(只输出一个 JSON 对象;甲乙用 draft_id 指认)】
存储合同规定一对人物只对应一张关系卡:同一无序人物对(甲,乙)与(乙,甲)视为同一对,输出中同一无序人物对最多一条;完全相同的反向重复也不要重复输出。
如果同一人物对同时存在多个关系维度,必须合并为一条综合记录,统一写入一个「关系类型」、一个「本窗演变」和同一条「其他字段」对象,**不得按关系维度拆成多条**。
{{"关系": [{{"甲方": 数字, "乙方": 数字, "关系类型": "", "本窗演变": "≤60字", "其他字段": {{}}}}]}}
━━━ 本窗材料 ━━━
《{title}》第 {a}–{b} 章正文:
{text}
【核心角色】
{chars}
【已有关系(避免重报建立)】
{rels}"""
def relation_repair_prompt(contracts, title, a, b, text, char_cards, existing_rels,
first_relations):
"""构造关系冲突后的唯一一次专用重整提示,要求模型返回可直接替换的完整关系数组。"""
first_output = json.dumps(first_relations or [], ensure_ascii=False, indent=1)
return f"""{relation_prompt(contracts, title, a, b, text, char_cards, existing_rels)}
【关系专用重整】
首次关系输出如下,其中同一无序人物对出现了内容冲突的重复项:
{first_output}
请结合本窗正文与已有关系,对首次关系输出做一次完整重整后重新输出「关系」数组:
- 同一无序人物对最多保留一条关系记录;甲乙反向只算同一对。
- 同一人物对的多个关系维度必须合并进同一条综合记录的「关系类型」「本窗演变」「其他字段」,不得拆成多条。
- 不得选择性静默丢弃冲突信息,不得按首条或末条覆盖;合并时保留正文有证据的各维度信息。
- 除解决上述重复冲突外,仍遵守原有字段合同、正文证据和输出格式纪律。
"""
# ── 合并与审计(三类字段演进 + 撤销依据)──
WIN_PREFIX_RE = None # 延迟编译(模块顶部 import re 已有)
def _strip_prefix(s):
"""剥条目行首的窗号类前缀(可能多重堆叠;含模型自造的 [窗本窗] 等变体),返回纯内容。"""
import re as _re
global WIN_PREFIX_RE
if WIN_PREFIX_RE is None:
# [窗…]/[本窗…] 任意变体全剥(窗29实测模型自造 "[窗本窗]",仅数字版剥不掉;
# 深空实测又造 "[窗387-388]" 章号范围变体,7 字符超旧上限 6——放宽到 12)
WIN_PREFIX_RE = _re.compile(r"^(?:\[[窗本][^\]]{0,12}\]\s*)+")
return WIN_PREFIX_RE.sub("", str(s)).strip()
TAIL_DEBRIS_RE = None # 尾部 JSON 拼接残渣(延迟编译)
def _strip_tail(s):
"""剥条目尾部的 JSON 拼接残渣(批9 样张走查实证:李锋等 20 张卡 59 处条目
尾挂 ", / '] / "} 等符号——模型把结构化输出的收尾符号带进了条目文本)。反复剥直到干净。"""
import re as _re
global TAIL_DEBRIS_RE
if TAIL_DEBRIS_RE is None:
TAIL_DEBRIS_RE = _re.compile(r"""(?:",|'\]|"\]|"}|'}|',)\s*$""")
t = str(s).rstrip()
while True:
m = TAIL_DEBRIS_RE.search(t)
if not m:
return t
t = t[:m.start()].rstrip()
def _clean_alias(al):
"""别名机械准入(抽检#6):拒收含括号注释/超长的备忘录式别名——预扫精确匹配永不命中=死数据。
单字别名一并拒收(复检 M5:'新'字入表后预扫全文命中率爆炸,纯噪声)。"""
al = (al or "").strip()
if not al or len(al) < 2 or len(al) > 12:
return None
if any(c in al for c in "()()。,,:"):
return None
return al
def _claim_alias(conn, work_id, canonical_name, alias, win_no, verdict_by):
"""原子登记别名:仅活跃同 canonical 幂等,冲突或 tombstone 必须失败关闭。"""
alias = _clean_alias(alias)
if not alias or alias == canonical_name:
return None
row = conn.execute(
"""INSERT INTO example_upgrade_alias
(work_id, canonical_name, alias, evidence_window, verdict_by, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s)
ON CONFLICT (tenant_id, work_id, alias) DO UPDATE
SET canonical_name=EXCLUDED.canonical_name
WHERE example_upgrade_alias.canonical_name=EXCLUDED.canonical_name
AND example_upgrade_alias.deleted=FALSE
RETURNING canonical_name""",
(work_id, canonical_name, alias, win_no, verdict_by, TENANT),
).fetchone()
if row and row[0] == canonical_name:
return alias
owner = conn.execute(
"""SELECT canonical_name, deleted FROM example_upgrade_alias
WHERE tenant_id=%s AND work_id=%s AND alias=%s FOR UPDATE""",
(TENANT, work_id, alias),
).fetchone()
owner_name = owner[0] if owner else "<唯一行缺失>"
owner_state = "deleted" if owner and owner[1] else "active"
raise AliasOwnershipConflict(
f"别名唯一映射冲突:work={work_id},alias={alias},"
f"现有canonical={owner_name},现有状态={owner_state},请求canonical={canonical_name}"
)
def _is_garbage(text):
"""结构垃圾检测(抽检 H2 根治):窗75 实测模型把原始变更包字符串化塞进字段值,
7 张卡被 {'draft_id':...} 类程序结构污染。含结构特征的文本一律拒收留审计。"""
t = str(text)
return ("draft_id" in t or "变更字段" in t or "别名新增" in t
or t.lstrip().startswith(("{'", '{"', "[{")))
def _entry_win(x):
"""追加条目的窗号(无 [窗N] 前缀的初卡条目记 0,排最前)。"""
m = re.match(r"^\[窗(\d+)\]", str(x))
return int(m.group(1)) if m else 0
# ── 里程碑对象(升格卡改造 P0):{章,台阶,周期} 结构化条目的清洗/合并 ──
# 章级排序不用运行时窗号,而用真实章号(设计稿 §6.1「真实章号索引」的落点)。
# 内嵌真实章号抽取:正文/台阶里原样出现的「第X章 / X章 / Ch.X」——迁移与降级共用
# 只认"第X章"/"Ch.X"两种明确章号写法。刻意不收裸"N章"——它无法与"隔3章/花了3章篇幅"这类
# 数量词区分,误当章号会污染迁移(且被标成最高置信"内嵌"),违背人工精确化;裸写法一律落到待LLM重抽。
# 带"第"前缀时取首个数字,天然处理"第489-490章"→489(区间取首章、不取末章)。
INLINE_CHAP_RE = re.compile(r"第\s*(\d+)(?:\s*[-—~到]\s*\d+)?\s*章|(?:Ch|CH|ch)\.?\s*(\d+)")
def _extract_inline_chapter(text):
"""从台阶文本抽第一个内嵌真实章号(第X章/ChX/X章),抽不到返回 None。"""
m = INLINE_CHAP_RE.search(str(text))
if not m:
return None
return int(next(g for g in m.groups() if g))
def _chapter_sort_key(ch):
"""里程碑排序键:从「章」值(整数 / 区间字符串"420-423" / None)取起点章号;
缺失或非法排到最后(CHAP_BIG),保证"待人工"条目不插进正常时间线中间。"""
if isinstance(ch, int):
return ch
if isinstance(ch, str):
m = re.search(r"\d+", ch)
if m:
return int(m.group())
return CHAP_BIG
def _int_chaps(chaps):
"""出场章归一化为整数集合(窗113 实证 bug 修复):模型偶尔把章号输出成字符串(如 "508"),
与库内 int 章号一起 sorted 会炸('<' not supported between int and str,深空窗113 当场停书);
且 set 并集把 "508" 和 508 当两个值、会虚增跨章计数误判立卡门槛。这里把数字串强制转 int、
非数字(脏值/区间)丢弃——出场章只应是单章整数(区间只出现在里程碑「章」,不在出场章)。
bool 是 int 子类,显式排除以防 True/False 混入被当章号。"""
out = set()
for c in (chaps or []):
if isinstance(c, bool):
continue
if isinstance(c, int):
out.add(c)
elif isinstance(c, str) and c.strip().isdigit():
out.add(int(c.strip()))
return out
def _infer_lifecycle(text):
"""从台阶文本启发式推断生命周期枚举(模型未给或非法「周期」时的降级填充)。
诚实边界:这是关键词启发式、非精确判定;新抽取由提示词强制模型直接给枚举,此路仅兜底。"""
t = str(text)
if any(w in t for w in ("登场", "首次", "初次", "出场", "诞生", "创立", "问世", "面世")):
return "登场"
if any(w in t for w in ("退役", "封存", "陨落", "覆灭", "销毁", "谢幕", "退场", "解散", "湮灭")):
return "退场"
if any(w in t for w in ("结局", "终局", "最终", "决战", "了结", "落幕")):
return "结局"
if any(w in t for w in ("巅峰", "高光", "对决", "突破", "解放", "觉醒", "封神", "登顶", "碾压")):
return "高光"
return "成长"
def _milestone_chapters(value):
"""把单章或闭区间章号转为整数列表;含糊格式直接返回空列表。"""
if isinstance(value, int) and not isinstance(value, bool) and value > 0:
return [value]
if not isinstance(value, str):
return []
match = re.fullmatch(r"\s*(\d+)\s*(?:[-~—–至]\s*(\d+)\s*)?", value)
if not match:
return []
start = int(match.group(1))
end = int(match.group(2) or start)
if start <= 0 or end < start or end - start > 50:
return []
return list(range(start, end + 1))
def _normalize_evidence_text(value):
"""证据比对忽略排版空白和宽窄差异,但保留正文字符与标点。"""
return re.sub(r"\s+", "", unicodedata.normalize("NFKC", str(value or "")))
GENERIC_ADDRESS_ALIASES = frozenset({
"小姐", "少爷", "先生", "女士", "夫人", "大人", "老师", "师父", "师傅",
"老板", "医生",
# 军职:只禁裸职务词,带专名的完整名称仍可作为精确证据。
"队长", "舰长", "指挥官", "总指挥", "司令", "司令官", "舰队司令",
"统帅", "元帅", "将军", "统领", "军长", "师长", "旅长", "团长",
"营长", "连长", "排长", "班长", "参谋长",
# 组织与学校职务。
"首领", "会长", "副会长", "理事长", "董事长", "社长", "主任", "主管",
"经理", "部长", "局长", "处长", "科长", "厂长", "院长", "副院长",
"校长", "副校长", "系主任", "教授", "导师", "教官",
# 宗门、帮派与家族职务。
"宗主", "门主", "掌门", "掌门人", "长老", "大长老", "太上长老",
"护法", "教主", "帮主", "盟主", "峰主", "堂主", "宫主", "家主",
"族长", "少主", "圣子", "圣女",
# 领地与王室称谓。
"领主", "城主", "堡主", "庄主", "国王", "女王", "皇帝", "皇后",
"王后", "王爷", "王妃", "王子", "公主", "太子", "皇太子", "亲王",
"太后", "陛下", "殿下",
"父亲", "母亲", "爸爸", "妈妈", "哥哥", "姐姐", "弟弟", "妹妹",
"丈夫", "妻子", "夫君", "兄长", "师兄", "师姐", "师弟", "师妹",
"叔叔", "阿姨", "爷爷", "奶奶",
})
TOP_LEVEL_APPEARANCE_AUDIT_FIELD = "顶层:出场章"
# 关系卡顶层字段与卡水位不在 payload["字段"] 内,必须用明确 sentinel 让 undo 精确落回原位置。
TOP_LEVEL_ALIASES_AUDIT_FIELD = "顶层:别名"
TOP_LEVEL_SUMMARY_AUDIT_FIELD = "顶层:一句话摘要"
TOP_LEVEL_RELATION_TYPE_AUDIT_FIELD = "顶层:关系类型"
TOP_LEVEL_RELATION_EVOLUTION_AUDIT_FIELD = "顶层:演变轨迹"
CARD_STATE_WATERMARK_AUDIT_FIELD = "状态:watermark_window"
def _filter_entity_chapters(name, aliases, chaps, chapter_texts):
"""只保留实体规范名或合法别名在对应章节正文真实出现的模型出场章。
规范名按原样参与机械匹配;别名先经过既有准入规则,避免单字、括号注释等噪声
被当作实体证据。章号与正文都做既有宽窄、空白归一化,但不做语义猜测。"""
if not isinstance(chapter_texts, dict):
return set()
names = []
canonical = str(name or "").strip()
# 与立卡名称清洗保持一致:括号前是规范名,括号内容只有通过别名准入后才能成为匹配证据。
parenthetical = re.match(r"^(.+?)[((](.+?)[))]\s*$", canonical)
if parenthetical:
canonical = parenthetical.group(1).strip()
aliases = list(aliases or []) + [parenthetical.group(2)]
# 单字规范名在中文正文中假阳率极高,不得单独作为机械章证据;常规二至四字专名仍按原子串规则匹配。
if len(canonical) >= 2 and canonical not in GENERIC_ADDRESS_ALIASES:
names.append((canonical, False))
names.extend(
(alias, True)
for alias in (_clean_alias(value) for value in (aliases or []))
# 通用称谓/关系称呼即使在正文独立出现也无法唯一指向该卡,保守地完全禁作出场章证据。
if alias and alias not in GENERIC_ADDRESS_ALIASES
)
normalized_names = {
(unicodedata.normalize("NFKC", value), is_alias)
for value, is_alias in names
if value
}
if not normalized_names:
return set()
verified = set()
for chapter in _int_chaps(chaps):
if chapter not in chapter_texts:
continue
text = unicodedata.normalize("NFKC", str(chapter_texts[chapter] or ""))
for entity_name, _ in normalized_names:
if entity_name in text:
verified.add(chapter)
break
return verified
def _append_verified_appearance_chapters(
conn,
draft_id,
win_no,
payload,
appearance_chapters,
chapter_texts,
*,
known_aliases=None,
):
"""过滤并追加顶层出场章,同时审计完整旧值,供同窗撤销精确还原。"""
verified = _filter_entity_chapters(
payload.get("名称"),
list(payload.get("别名") or []) + list(known_aliases or []),
appearance_chapters,
chapter_texts,
)
if not verified:
return False
merged = sorted(_int_chaps(payload.get("出场章", [])) | verified)
if payload.get("出场章") == merged:
return False
old_value = json.dumps(payload.get("出场章"), ensure_ascii=False) \
if "出场章" in payload else None
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s)""",
(draft_id, win_no, TOP_LEVEL_APPEARANCE_AUDIT_FIELD, old_value,
json.dumps(merged, ensure_ascii=False), TENANT),
)
payload["出场章"] = merged
return True
def _read_upgrade_draft_snapshot(conn, draft_id):
"""读取送入模型的可信 payload/revision;不持锁跨模型调用。"""
row = conn.execute(
"""SELECT draft_payload, status, revision, source_type
FROM muse_knowledge_draft
WHERE id=%s AND tenant_id=%s AND deleted=FALSE""",
(draft_id, TENANT),
).fetchone()
if not row:
raise UpgradeDraftWriteConflict(f"升格 draft 不存在或已删除:draft={draft_id}")
payload, status, revision, source_type = row
if status != "pending" or source_type != SOURCE_TYPE:
raise UpgradeDraftWriteConflict(
f"升格 draft 不可写:draft={draft_id},status={status},source_type={source_type}"
)
return deepcopy(payload), revision
def _lock_upgrade_draft(conn, draft_id, expected_revision=None):
"""锁定当前 draft;状态、来源或模型所见 revision 漂移都失败关闭。"""
row = conn.execute(
"""SELECT draft_payload, status, revision, source_type
FROM muse_knowledge_draft
WHERE id=%s AND tenant_id=%s AND deleted=FALSE
FOR UPDATE""",
(draft_id, TENANT),
).fetchone()
if not row:
raise UpgradeDraftWriteConflict(f"升格 draft 不存在或已删除:draft={draft_id}")
payload, status, revision, source_type = row
if status != "pending" or source_type != SOURCE_TYPE:
raise UpgradeDraftWriteConflict(
f"升格 draft 不可写:draft={draft_id},status={status},source_type={source_type}"
)
if expected_revision is not None and revision != expected_revision:
raise UpgradeDraftWriteConflict(
f"升格 draft 模型版本冲突:draft={draft_id},"
f"expected_revision={expected_revision},current_revision={revision}"
)
return deepcopy(payload), revision
def _write_locked_upgrade_draft(conn, draft_id, payload, revision):
"""用锁后 revision 回写;任何状态或版本漂移都按确定性冲突失败关闭。"""
row = conn.execute(
"""UPDATE muse_knowledge_draft SET draft_payload=%s,
revision=revision+1, updater='upgrade'
WHERE id=%s AND tenant_id=%s AND deleted=FALSE
AND status='pending' AND source_type=%s AND revision=%s
RETURNING revision""",
(json.dumps(payload, ensure_ascii=False), draft_id, TENANT, SOURCE_TYPE, revision),
).fetchone()
if not row:
raise UpgradeDraftWriteConflict(
f"升格 draft 锁后写入冲突:draft={draft_id},expected_revision={revision}"
)
return row[0]
def _audit_relation_change(conn, draft_id, win_no, field_name, old_value, new_value):
"""记录关系卡一个真实落点的完整旧/新 JSON;值未变化时不制造空审计。"""
if old_value == new_value:
return False
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s)""",
(
draft_id,
win_no,
field_name,
json.dumps(old_value, ensure_ascii=False) if old_value is not None else None,
json.dumps(new_value, ensure_ascii=False) if new_value is not None else None,
TENANT,
),
)
return True
def _update_card_state(conn, draft_id, work_id, win_no):
"""推进实体或关系卡水位并审计旧值;undo 可精确恢复或删除新 state。"""
row = conn.execute(
"SELECT watermark_window FROM example_upgrade_card_state WHERE draft_id=%s",
(draft_id,),
).fetchone()
old_watermark = row[0] if row else None
_audit_relation_change(
conn,
draft_id,
win_no,
CARD_STATE_WATERMARK_AUDIT_FIELD,
old_watermark,
max(old_watermark or 0, win_no),
)
conn.execute(
"""INSERT INTO example_upgrade_card_state (draft_id, work_id, watermark_window, tenant_id)
VALUES (%s,%s,%s,%s)
ON CONFLICT (draft_id) DO UPDATE
SET watermark_window=GREATEST(example_upgrade_card_state.watermark_window,
EXCLUDED.watermark_window),
update_time=now()""",
(draft_id, work_id, win_no, TENANT),
)
def _update_relation_card(conn, work_id, win_no, draft_id, expected_revision, relation):
"""锁后按当前 payload 更新关系卡,并校验模型读取时的 revision。"""
payload, revision = _lock_upgrade_draft(conn, draft_id, expected_revision)
fields = payload.setdefault("字段", {})
# 历史数据可能把演变轨迹放在顶层;迁移本身也必须可撤销,不能只恢复字段侧。
old_field_evolution = deepcopy(fields.get("演变轨迹")) if "演变轨迹" in fields else None
legacy_present = "演变轨迹" in payload
legacy_evolution = deepcopy(payload.get("演变轨迹")) if legacy_present else None
if legacy_present:
_audit_relation_change(
conn,
draft_id,
win_no,
TOP_LEVEL_RELATION_EVOLUTION_AUDIT_FIELD,
legacy_evolution,
None,
)
payload.pop("演变轨迹", None)
evolution = fields.get("演变轨迹")
evolution = list(evolution) if isinstance(evolution, list) else ([evolution] if evolution else [])
if legacy_present:
evolution.extend(legacy_evolution if isinstance(legacy_evolution, list) else [legacy_evolution])
evolution_core = _strip_prefix(relation.get("本窗演变", ""))
if evolution_core and evolution_core not in {_strip_prefix(item) for item in evolution}:
evolution.append(f"[窗{win_no}] {evolution_core}")
if (legacy_present or evolution_core or "演变轨迹" in fields) \
and old_field_evolution != evolution:
_audit_relation_change(
conn, draft_id, win_no, "演变轨迹", old_field_evolution, evolution
)
fields["演变轨迹"] = evolution
# 关系合同的其他字段逐字段覆写,每个字段都记录完整旧/新值,供即时重试精确还原。
for field_name, new_value in (relation.get("其他字段") or {}).items():
if field_name == "演变轨迹":
continue
old_value = deepcopy(fields.get(field_name)) if field_name in fields else None
if _audit_relation_change(
conn, draft_id, win_no, field_name, old_value, new_value):
fields[field_name] = new_value
new_relation_type = relation.get("关系类型")
if new_relation_type:
old_relation_type = payload.get("关系类型") if "关系类型" in payload else None
if _audit_relation_change(
conn,
draft_id,
win_no,
TOP_LEVEL_RELATION_TYPE_AUDIT_FIELD,
old_relation_type,
new_relation_type):
payload["关系类型"] = new_relation_type
_write_locked_upgrade_draft(conn, draft_id, payload, revision)
_update_card_state(conn, draft_id, work_id, win_no)
return draft_id
def _normalize_relation_output(relations, valid_draft_ids):
"""整批过滤并规范化关系输出;冲突重复必须在任何关系卡写入前失败。"""
normalized = []
relation_by_pair = {}
for relation in relations or []:
if not isinstance(relation, dict):
continue
pair = _normalized_relation_pair(relation, valid_draft_ids)
if pair is None:
continue
item = deepcopy(relation)
item["甲方"], item["乙方"] = pair
if pair not in relation_by_pair:
relation_by_pair[pair] = item
normalized.append(item)
continue
if relation_by_pair[pair] != item:
raise RelationOutputConflict(
f"关系输出同一实体对存在冲突重复:甲方draft={pair[0]},乙方draft={pair[1]}"
)
return normalized
def _normalized_relation_pair(relation, valid_draft_ids):
"""返回合法关系的无序 pair;窗外实体和自关系沿用原规则直接过滤。"""
first, second = relation.get("甲方"), relation.get("乙方")
if first not in valid_draft_ids or second not in valid_draft_ids or first == second:
return None
return tuple(sorted((first, second)))
def _relation_value_parts(value):
"""把关系维度转为可稳定去重的非空文本片段,不丢弃非空列表成员。"""
if not _relation_value_is_nonempty(value):
return []
if isinstance(value, (list, tuple)):
parts = []
for item in value:
parts.extend(_relation_value_parts(item))
return parts
text = str(value).strip()
return [text] if text else []
def _relation_value_is_nonempty(value):
"""识别 JSON 维度是否携带信息;空字符串、空容器和 null 可安全忽略。"""
if value is None:
return False
if isinstance(value, str):
return bool(value.strip())
if isinstance(value, (list, tuple, dict, set)):
return bool(value)
return True
def _merge_repaired_relation_output(relations, valid_draft_ids):
"""只供第二次 repair 输出使用:按无序 pair 安全合并多维关系,不覆盖冲突值。"""
standard_keys = frozenset(("甲方", "乙方", "关系类型", "本窗演变", "其他字段"))
states = {}
for relation in relations or []:
if not isinstance(relation, dict):
continue
pair = _normalized_relation_pair(relation, valid_draft_ids)
if pair is None:
continue
unknown_keys = [
str(key)
for key, value in relation.items()
if key not in standard_keys and _relation_value_is_nonempty(value)
]
if unknown_keys:
raise RelationOutputConflict(
f"关系 repair 含非空未知顶层键:甲方draft={pair[0]},"
f"乙方draft={pair[1]},keys={sorted(unknown_keys)}"
)
if pair not in states:
states[pair] = {
"关系类型": [],
"本窗演变": [],
"其他字段": {},
}
state = states[pair]
for field_name in ("关系类型", "本窗演变"):
for part in _relation_value_parts(relation.get(field_name)):
if part not in state[field_name]:
state[field_name].append(part)
other_fields = relation.get("其他字段")
if other_fields is None:
other_fields = {}
if not isinstance(other_fields, dict):
raise RelationOutputConflict(
f"关系 repair 的其他字段不是对象:甲方draft={pair[0]},乙方draft={pair[1]}"
)
for field_name, value in other_fields.items():
if not _relation_value_parts(value):
continue
if field_name not in state["其他字段"]:
state["其他字段"][field_name] = deepcopy(value)
continue
if state["其他字段"][field_name] != value:
raise RelationOutputConflict(
f"关系 repair 同一字段值冲突:甲方draft={pair[0]},"
f"乙方draft={pair[1]},field={field_name}"
)
# 模型返回顺序不属于业务语义;输出前统一固定 pair、片段和字段键的顺序。
return [
{
"甲方": pair[0],
"乙方": pair[1],
"关系类型": " / ".join(sorted(states[pair]["关系类型"])),
"本窗演变": ";".join(sorted(states[pair]["本窗演变"])),
"其他字段": {
field_name: deepcopy(states[pair]["其他字段"][field_name])
for field_name in sorted(states[pair]["其他字段"])
},
}
for pair in sorted(states)
]
def _relation_pair_counts(relations, valid_draft_ids):
"""统计 repair 输出中每个合法 pair 的原始条数,供二次合并日志使用。"""
counts = {}
for relation in relations or []:
if not isinstance(relation, dict):
continue
pair = _normalized_relation_pair(relation, valid_draft_ids)
if pair is not None:
counts[pair] = counts.get(pair, 0) + 1
return counts
def _insert_relation_card(conn, work_id, win_no, payload):
"""新建关系卡并返回 draft id;逐字段审计和 state 任一步失败都会让窗事务回滚。"""
draft_id = conn.execute(
"""INSERT INTO muse_knowledge_draft
(work_id, draft_type, draft_payload, status, source_type,
source_id, creator, updater, tenant_id)
VALUES (%s,'entity',%s,'pending',%s,%s,'upgrade','upgrade',%s)
RETURNING id""",
(work_id, json.dumps(payload, ensure_ascii=False), SOURCE_TYPE, work_id, TENANT),
).fetchone()[0]
_audit_relation_change(
conn,
draft_id,
win_no,
TOP_LEVEL_RELATION_TYPE_AUDIT_FIELD,
None,
payload.get("关系类型"),
)
for field_name, value in (payload.get("字段") or {}).items():
_audit_relation_change(conn, draft_id, win_no, field_name, None, value)
_update_card_state(conn, draft_id, work_id, win_no)
return draft_id
def _milestone_evidence_matches(item, chapter_texts):
"""验证短原文证据确实存在于里程碑声明的章节正文。"""
if not isinstance(item, dict) or not isinstance(chapter_texts, dict):
return False
evidence = _normalize_evidence_text(item.get("证据"))
if not 8 <= len(evidence) <= 30:
return False
chapters = _milestone_chapters(item.get("章"))
if not chapters or any(chapter not in chapter_texts for chapter in chapters):
return False
return any(
evidence in _normalize_evidence_text(chapter_texts[chapter])
for chapter in chapters
)
def _walk_milestone_items(value, path="$"):
"""稳定遍历模型输出中的里程碑对象,并生成可回填的 JSON 路径。"""
if isinstance(value, dict):
if "章" in value and ("台阶" in value or "阶" in value):
yield path, value
return
for key, child in value.items():
yield from _walk_milestone_items(child, f"{path}.{key}")
elif isinstance(value, list):
for index, child in enumerate(value):
yield from _walk_milestone_items(child, f"{path}[{index}]")
def repair_missing_milestone_evidence(
model_output,
*,
title,
a,
b,
text,
chapter_texts,
call,
):
"""为本窗缺证据里程碑补原文短引;窗外重抄项不进入修复调用。"""
candidates = []
by_ref = {}
for ref, item in _walk_milestone_items(model_output):
if _milestone_evidence_matches(item, chapter_texts):
continue
chapters = _milestone_chapters(item.get("章"))
if not chapters or any(chapter not in chapter_texts for chapter in chapters):
continue
candidates.append(
{
"ref": ref,
"章": item.get("章"),
"台阶": item.get("台阶") or item.get("阶"),
}
)
by_ref[ref] = item
if not candidates:
return 0
prompt = f"""【功能指令(parse-book 里程碑证据修复)】
下列候选里程碑缺少可机械验证的证据。只为确实被正文支持的候选返回一条所标章节中的原文连续短句(8–30字);不得改章号、不得改台阶、不得概括或改写原文。找不到逐字证据的候选不要返回。
【输出规则(只输出一个 JSON 对象)】
{{"证据修复":[{{"ref":"候选ref原样回传","章":489,"证据":"正文原样连续短句"}}]}}
━━━ 候选里程碑 ━━━
{json.dumps(candidates, ensure_ascii=False)}
━━━ 《{title}》第 {a}–{b} 章正文 ━━━
{text}
"""
repaired, _ = call(prompt, ("证据修复",))
count = 0
for row in repaired.get("证据修复") or []:
if not isinstance(row, dict):
continue
ref = str(row.get("ref") or "")
item = by_ref.get(ref)
if item is None or _milestone_chapters(row.get("章")) != _milestone_chapters(item.get("章")):
continue
item["证据"] = row.get("证据")
if _milestone_evidence_matches(item, chapter_texts):
count += 1
else:
item.pop("证据", None)
return count
def _clean_milestone(item, win_no, *, chapter_texts=None, require_evidence=False):
"""规范化单个里程碑为 {章,台阶,周期,_win};垃圾/空台阶返回 None。
- dict 入参:取 章/台阶/周期;台阶剥前缀+剥尾残;缺章从台阶抽内嵌章号兜底;缺/非法周期启发式推断。
- 字符串入参(模型降级输出或存量迁移):整串当台阶,抽内嵌章号当章,推断周期。
- _win 盖当前窗号,仅作撤销溯源(undo_window 按它删本窗新增),不参与展示与排序。
降级保底(设计稿 §九 风险2 + 拍板#1):抽不出完整对象也退成"章号+一句话"最小对象,绝不整条丢。"""
if require_evidence and not _milestone_evidence_matches(item, chapter_texts):
return None
if isinstance(item, dict):
step = _strip_tail(_strip_prefix(str(item.get("台阶") or item.get("阶") or "")))
ch = item.get("章")
cycle = item.get("周期")
else:
step = _strip_tail(_strip_prefix(str(item)))
ch, cycle = None, None
if not step or _is_garbage(step):
return None
# 章号:对象已给(整数或含数字的区间字符串)就用;否则从台阶文本抽内嵌章号;再无则 None(待人工)
if not (isinstance(ch, int) or (isinstance(ch, str) and re.search(r"\d", ch))):
ch = _extract_inline_chapter(step)
if cycle not in LIFECYCLE:
cycle = _infer_lifecycle(step)
# 台阶机械守卫(洞③):超 STEP_MAX 字直接截断——提示词只是软目标(≤40 字),这里是硬闸(≤80 字)
# 防模型把整段流水写进一条台阶跑飞长文。截断必须放在章号抽取与周期推断**之后**:跑飞长文正是
# 依赖降级兜底的场景,藏在 80 字外的内嵌章号/周期关键词若先被截掉,兜底就瞎了(用全文抽、抽完再截)。
# 兜底登场台阶(_debut_milestone)同用 STEP_MAX,两处一致。
if len(step) > STEP_MAX:
step = step[:STEP_MAX]
return {"章": ch, "台阶": step, "周期": cycle, "_win": win_no}
def _merge_milestones(old, items, win_no, *, chapter_texts=None, require_evidence=False):
"""合并里程碑数组:去重按台阶内容、排序按真实章号。返回 (merged, rejected)。
old 中已有对象保留其原 _win(不被本窗覆盖);新对象由 _clean_milestone 盖当前 win_no。
rejected 为被判垃圾的原始条目,交调用方留审计(对齐字符串路径的垃圾拦截)。"""
kept = [m for m in (old or []) if isinstance(m, dict) and m.get("台阶")]
seen = {str(m.get("台阶", "")).strip() for m in kept}
rejected = []
for it in (items if isinstance(items, list) else [items]):
m = _clean_milestone(
it,
win_no,
chapter_texts=chapter_texts,
require_evidence=require_evidence,
)
if not m:
rejected.append(it)
continue
key = m["台阶"].strip()
if key and key not in seen:
kept.append(m)
seen.add(key)
return sorted(kept, key=lambda m: _chapter_sort_key(m.get("章"))), rejected
def _merge_material(ent, rest_limit=600):
"""归并材料构造(洞②:登场里程碑不丢,纯函数便于离线自测)。四条归并路径共用的「初卡材料」文本:
型/名称/一句话摘要 + 「演变历程」条目**完整保留不截断** + 其余字段 json 截断(其余部分上限 rest_limit)。
根治病象:原四路都传 json.dumps(ent)[:400](观察漏看路更只传摘要),会把里程碑尤其登场条目截掉——
并入既有卡后登场/进化台阶信息静默丢失。这里把演变历程整段拎出不截,只截其余字段防超长。"""
fields = ent.get("字段") or {}
ms = fields.get("演变历程")
milestones = ms if isinstance(ms, list) else ([ms] if ms else [])
rest = {k: v for k, v in fields.items() if k not in MILESTONE_FIELDS}
seg = [f"型={ent.get('型', '')}|名称={ent.get('名称', '')}|摘要:{ent.get('一句话摘要', '')}"]
if milestones: # 演变历程完整保留(登场/进化台阶是归并关键,绝不截断)
seg.append("演变历程:" + json.dumps(milestones, ensure_ascii=False))
if rest: # 其余字段可容忍截断(防超长撑爆更新 prompt)
seg.append("其余字段:" + json.dumps(rest, ensure_ascii=False)[:rest_limit])
return ";".join(seg)
def _debut_evidence_name(entity_name, aliases, chapter_text):
"""从 debut 正文选择唯一实际命中的合法名称;歧义或无命中时返回 None。"""
canonical = str(entity_name or "").strip()
candidates = []
if len(canonical) >= 2 and canonical not in GENERIC_ADDRESS_ALIASES:
candidates.append(canonical)
for raw_alias in aliases or []:
alias = _clean_alias(raw_alias)
if alias and alias not in GENERIC_ADDRESS_ALIASES and alias not in candidates:
candidates.append(alias)
normalized_text = unicodedata.normalize("NFKC", str(chapter_text or ""))
matched = [
name for name in candidates
if unicodedata.normalize("NFKC", name) in normalized_text
]
return matched[0] if len(matched) == 1 else None
def _debut_milestone(milestones, entity_name, entity_type, aliases, chaps, win_no, chapter_texts):
"""登场兜底(洞② 机械那一保险,纯函数便于离线自测):里程碑列表里若没有任何 周期=登场 的条目,
在头部补一条兜底登场里程碑;已有登场则原样返回(提示词软约束 + 此机械兜底=双保险)。
- 章:min(正文实证出场章)(仅取整数章号);无实证章则不补,真实性优先;
- 台阶:名称只可取 debut 正文唯一命中的合法规范名/别名;无唯一命中则退化为仅类型;
- 证据:debut 章必须存在非空正文;缺正文时不补,禁止把后续章事实倒灌到最早章;
- _win:盖当前窗号——带 _win 才能被同窗撤销(undo_window 按 _win 删本窗新增)识别,防重跑 double-append。
根治病象:实体首现走新名字路径时模型倾向只填当前态、漏建登场里程碑,机械补一条保成长线起点不缺。"""
if any(isinstance(m, dict) and m.get("周期") == "登场" for m in (milestones or [])):
return list(milestones or [])
ch_ints = sorted(_int_chaps(chaps))
if not ch_ints:
return list(milestones or [])
debut_chapter = ch_ints[0]
if not isinstance(chapter_texts, dict) or not str(chapter_texts.get(debut_chapter) or "").strip():
return list(milestones or [])
name = str(entity_name or "").strip()
label = DEBUT_TYPE_LABELS.get(entity_type)
if not name or not label:
return list(milestones or [])
evidence_name = _debut_evidence_name(name, aliases, chapter_texts[debut_chapter])
debut = {
"章": debut_chapter,
"台阶": (f"{label}「{evidence_name}」登场" if evidence_name else f"{label}登场")[:STEP_MAX],
"周期": "登场",
"_win": win_no,
}
return [debut] + list(milestones or [])
def _milestone_update_value_empty(value):
"""判断里程碑字段是否为空,可用于安全地用另一份更完整输出补字段。"""
if value is None:
return True
if isinstance(value, str):
return not value.strip()
if isinstance(value, (list, tuple, dict, set)):
return not value
return False
def _milestone_update_value_equal(field_name, left, right):
"""比较模型里程碑字段的语义值,兼容章号字符串/整数与排版空白差异。"""
if field_name == "章":
left_chapters = _milestone_chapters(left)
right_chapters = _milestone_chapters(right)
if left_chapters and right_chapters:
return left_chapters == right_chapters
if field_name in {"台阶", "阶", "周期", "证据"}:
return unicodedata.normalize("NFKC", str(left)).strip() == \
unicodedata.normalize("NFKC", str(right)).strip()
return left == right
def _milestone_update_step(item):
"""返回里程碑稳定的台阶身份;空台阶不参与跨条目匹配。"""
if not isinstance(item, dict):
return None
raw = item.get("台阶") or item.get("阶")
if not isinstance(raw, str) or not raw.strip():
return None
return unicodedata.normalize("NFKC", raw).strip()
def _milestone_update_event(item):
"""返回章+周期事件身份;只在两者均有值时使用,避免猜测条目关系。"""
if not isinstance(item, dict):
return None
chapters = _milestone_chapters(item.get("章"))
cycle = item.get("周期")
if not chapters or not isinstance(cycle, str) or not cycle.strip():
return None
return (tuple(chapters), unicodedata.normalize("NFKC", cycle).strip())
def _merge_compatible_milestone_dict(left, right):
"""合并同一里程碑的两份对象;非空字段冲突时返回 None。"""
merged = deepcopy(left)
for field_name, new_value in right.items():
if field_name not in merged or _milestone_update_value_empty(merged[field_name]):
if not _milestone_update_value_empty(new_value):
merged[field_name] = deepcopy(new_value)
elif field_name not in merged:
merged[field_name] = deepcopy(new_value)
continue
if _milestone_update_value_empty(new_value):
continue
if not _milestone_update_value_equal(field_name, merged[field_name], new_value):
return None
return merged
def _merge_compatible_milestone_updates(left, right):
"""合并同一实体重复返回的演变历程,只接受可证明兼容的条目。
台阶文本是首要事件身份,章+周期用于捕获同一事件但台阶互相冲突的情况。
无稳定身份的非全等条目不能证明是互补内容,直接返回 None 交调用方硬失败。
"""
if not isinstance(left, list) or not isinstance(right, list):
return None
merged = []
by_step = {}
by_event = {}
def ingest(item):
if not isinstance(item, dict):
# 模型协议要求对象;非对象只有完全相同才可去重。
if any(existing == item for existing in merged):
return True
if not merged:
merged.append(deepcopy(item))
return True
return False
step = _milestone_update_step(item)
event = _milestone_update_event(item)
index = by_step.get(step) if step is not None else None
if index is None and event is not None:
index = by_event.get(event)
if index is not None:
combined = _merge_compatible_milestone_dict(merged[index], item)
if combined is None:
return False
merged[index] = combined
# 同一条目可能在此前缺少 step/event,本次补齐后登记身份。
step = _milestone_update_step(combined)
event = _milestone_update_event(combined)
if step is not None:
by_step[step] = index
if event is not None:
by_event[event] = index
return True
if step is None and event is None:
# 没有任何稳定身份的对象,除完全相同外无法证明是互补条目。
if any(existing == item for existing in merged):
return True
if not merged:
merged.append(deepcopy(item))
return True
return False
# 台阶不同但章+周期相同,不能把两个可能互相矛盾的跃迁当成互补。
if event is not None and event in by_event:
return False
index = len(merged)
merged.append(deepcopy(item))
if step is not None:
by_step[step] = index
if event is not None:
by_event[event] = index
return True
for item in list(left) + list(right):
if not ingest(item):
return None
return merged
def _normalize_entity_updates(updates, valid_draft_ids):
"""整批合并有效实体更新;字段冲突必须在任何卡片或别名写入前失败。"""
normalized = []
update_by_draft = {}
aliases_by_draft = {}
for update in updates or []:
if not isinstance(update, dict):
continue
draft_id = update.get("draft_id")
if draft_id not in valid_draft_ids:
continue
changes = deepcopy(update.get("变更字段")) \
if isinstance(update.get("变更字段"), dict) else {}
# 兼容模型把别名误放进变更字段的既有降级路径,但统一在批量阶段过滤和去重。
raw_aliases = update.get("别名新增")
misplaced_aliases = changes.pop("别名新增", None)
alias_values = []
for value in (raw_aliases, misplaced_aliases):
if isinstance(value, str):
alias_values.append(value)
elif isinstance(value, list):
alias_values.extend(alias for alias in value if isinstance(alias, str))
if not changes and not alias_values:
continue
if draft_id not in update_by_draft:
merged = {"draft_id": draft_id, "变更字段": {}, "别名新增": []}
update_by_draft[draft_id] = merged
aliases_by_draft[draft_id] = set()
normalized.append(merged)
merged = update_by_draft[draft_id]
for field_name, new_value in changes.items():
if field_name in merged["变更字段"]:
old_value = merged["变更字段"][field_name]
if old_value == new_value:
continue
if field_name == "演变历程":
milestone_value = _merge_compatible_milestone_updates(
old_value, new_value
)
if milestone_value is not None:
merged["变更字段"][field_name] = milestone_value
continue
if old_value != new_value:
raise EntityUpdateOutputConflict(
f"实体更新输出同一字段存在冲突重复:draft_id={draft_id},field={field_name}"
)
continue
merged["变更字段"][field_name] = new_value
seen_aliases = aliases_by_draft[draft_id]
for alias in alias_values:
if alias not in seen_aliases:
merged["别名新增"].append(alias)
seen_aliases.add(alias)
return normalized
def merge_card(
conn,
draft_id,
win_no,
changes,
alias_new,
valid_keys=None,
*,
chapter_texts=None,
appearance_chapters=None,
known_aliases=None,
expected_revision=None,
):
"""按 5.1 三类规则合并变更字段:数组/白名单=追加(剥模型自带前缀+同文去重后带窗号),
标量=覆写留审计(增量式假全量拦截转追加——抽检#4 信息回退病)。
valid_keys:该型合同的合法字段 key 集——越合同 key 裁剪留审计(窗29实测模型把
整段条目文本误当字段 key 写入,无校验会把卡体字段区打烂)。"""
payload, revision = _lock_upgrade_draft(conn, draft_id, expected_revision)
fields = payload.setdefault("字段", {})
# 卡水位(该卡最后一次被更新的窗号):补跑迟到窗(如窗41在窗80后补跑)的覆写类字段
# 若直接落卡,会把书末态倒写回中期态(时间倒流污染,2026-07-15 补跑实测坐实)。
# 闸门:win_no < 水位 ⇒ 覆写只留审计不动卡体;追加类带窗号标签乱序无害,不拦。
wm_row = conn.execute("SELECT watermark_window FROM example_upgrade_card_state WHERE draft_id=%s",
(draft_id,)).fetchone()
watermark = wm_row[0] if wm_row else 0
changes = dict(changes or {})
# 模型偶把「别名新增」混进变更字段(窗2实测):摘出来并入别名流程,不落卡体字段
alias_new = list(alias_new or []) + \
[a for a in (changes.pop("别名新增", None) or []) if isinstance(a, str)]
# alias 表可能有尚未回填 payload 的合法别名;与本窗新别名一起参与过滤。统一 helper 同时写入可撤销审计。
_append_verified_appearance_chapters(
conn,
draft_id,
win_no,
payload,
appearance_chapters,
chapter_texts,
known_aliases=list(known_aliases or []) + list(alias_new),
)
for k, v in changes.items():
if valid_keys is not None and k not in valid_keys:
# 越合同字段:裁剪留审计(方案守卫条款),畸形长 key(条目误当 key)一并挡下
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, ("越合同:" + str(k))[:100],
json.dumps(v, ensure_ascii=False)[:2000], TENANT))
continue
if k == "一句话摘要":
# 摘要住卡顶层而非"字段"子对象——此前写进子对象成"影子摘要"且顶层摘要
# 从不更新(抽检 M4:600 章前的旧摘要一直挂着)。特判写顶层,同守水位闸。
if isinstance(v, str) and v.strip() and not _is_garbage(v):
if win_no < watermark:
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, "迟到覆写弃用:一句话摘要",
json.dumps(v, ensure_ascii=False)[:2000], TENANT))
else:
old_summary = deepcopy(payload.get("一句话摘要")) \
if "一句话摘要" in payload else None
if _audit_relation_change(
conn,
draft_id,
win_no,
TOP_LEVEL_SUMMARY_AUDIT_FIELD,
old_summary,
v.strip()):
payload["一句话摘要"] = v.strip()
continue
if k in MILESTONE_FIELDS:
# 里程碑对象数组(演变历程):走对象合并路径——去重按台阶内容、排序按真实章号,
# 不走下面处理字符串条目([窗N] 前缀)的老路径。迟到窗追加无害:对象自带章号,
# 乱序由 _chapter_sort_key 排序纠正,故不设水位闸(与字符串追加同策略)。
base = fields.get(k) if isinstance(fields.get(k), list) else ([fields[k]] if fields.get(k) else [])
merged, rejected = _merge_milestones(
base,
v,
win_no,
chapter_texts=chapter_texts,
require_evidence=True,
)
for rj in rejected:
# 垃圾/空台阶里程碑拒收留审计(对齐字符串路径的垃圾拦截守卫)
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, "垃圾拦截:演变历程",
json.dumps(rj, ensure_ascii=False)[:2000], TENANT))
fields[k] = merged
continue
is_append = k in APPEND_FIELDS or isinstance(v, list) or isinstance(fields.get(k), list)
# 覆写值以增量口吻开头=模型把增量当全量(抽检#4:新值会抹掉旧基线),转追加不覆写
if not is_append and isinstance(v, str) and \
any(v.lstrip().startswith(w) for w in ("新增", "本窗", "另外", "此外")):
is_append = True
if is_append:
items = v if isinstance(v, list) else [v]
old = fields.get(k) if isinstance(fields.get(k), list) else ([fields[k]] if fields.get(k) else [])
seen = {_strip_prefix(x) for x in old} # 同文去重(抽检#1 重复病)
for it in items:
# 巨型粘连拆分(唐灵窗51 实测:模型把整个历史弧线连成一条"…;[窗2]…;[窗3]…"
# 输出成单条目)——按";[窗N]"边界拆段,带原窗号的段保留原窗号,其余记本窗
for seg in re.split(r";\s*(?=\[[窗本])", str(it)):
m0 = re.match(r"^\[窗(\d+)\]\s*(.*)", seg.strip(), flags=re.S)
seg_win, body = (int(m0.group(1)), m0.group(2)) if m0 else (win_no, seg)
core = _strip_tail(_strip_prefix(body)) # 剥模型自带前缀+尾部拼接残渣
if core and _is_garbage(core):
# 结构垃圾条目拒收留审计(窗75 事故根治:变更包字符串化混进字段值)
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, ("垃圾拦截:" + str(k))[:100],
str(core)[:2000], TENANT))
continue
if core and core not in seen:
old.append(f"[窗{seg_win}] {core}")
seen.add(core)
# 窗序归位(抽检 M2:补跑/重试窗条目尾插致时间线倒流)——稳定排序,同窗保持原序
fields[k] = sorted(old, key=_entry_win)
elif isinstance(v, str) and _is_garbage(v):
# 结构垃圾覆写值拒收留审计(同窗75 事故根治)
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, ("垃圾拦截:" + str(k))[:100], v[:2000], TENANT))
elif win_no < watermark:
# 迟到覆写弃用:本窗时序早于卡已生长到的窗位,覆写会让卡态倒流——
# 只留审计(标记可查),卡体保持高窗态不动
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(draft_id, win_no, ("迟到覆写弃用:" + str(k))[:100],
json.dumps(v, ensure_ascii=False)[:2000], TENANT))
else:
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s)""",
(draft_id, win_no, k, json.dumps(fields.get(k), ensure_ascii=False)
if fields.get(k) is not None else None,
json.dumps(v, ensure_ascii=False), TENANT))
# 覆写值同样剥尾部拼接残渣(批9 样张走查同款病灶,覆写路一并守住)
fields[k] = _strip_tail(v) if isinstance(v, str) else v
old_aliases = deepcopy(payload.get("别名")) if "别名" in payload else None
merged_aliases = list(old_aliases) if isinstance(old_aliases, list) else []
work_id = payload.get("_work_id") or 0
canonical_name = payload.get("名称")
alias_added = False
for raw_alias in alias_new or []:
alias = _claim_alias(conn, work_id, canonical_name, raw_alias, win_no, "ai")
if alias and alias not in merged_aliases:
merged_aliases.append(alias)
alias_added = True
if alias_added and _audit_relation_change(
conn,
draft_id,
win_no,
TOP_LEVEL_ALIASES_AUDIT_FIELD,
old_aliases,
merged_aliases):
payload["别名"] = merged_aliases
_write_locked_upgrade_draft(conn, draft_id, payload, revision)
_update_card_state(conn, draft_id, work_id, win_no)
def _build_new_card_payload(work_id, win_no, ent, milestone_types=None, *,
chapter_texts=None, known_chapters=None):
"""纯内存构造初卡;planner 与短写共用,保证模型快照和最终落库内容一致。
milestone_types(洞②):含「演变历程」字段的型集合(据库内合同动态判定)——该型立卡时若抽取结果
无登场里程碑,机械兜底补一条登场(保成长型实体登场即有生命周期起点);None/空集则不触发兜底。
名称剥括号注(抽检#7 根因):「白色游魂(无名侦察兵)」这类名称使预扫精确匹配失明
——主名之外的括号内容若像名字则转别名,否则丢弃。"""
import re as _re
raw = ent["名称"].strip()
m = _re.match(r"^(.+?)[((](.+?)[))]\s*$", raw)
extra_alias = []
if m:
raw = m.group(1).strip()
note = _clean_alias(m.group(2))
if note:
extra_alias.append(note)
aliases = [
alias
for alias in ([_clean_alias(value) for value in ent.get("别名", [])] + extra_alias)
if alias
]
# 初卡字段过守卫(深空 4917 实测:立卡路不走 merge_card,粘连/自造前缀/垃圾
# 原样入库——更新路守了、立卡路漏了):列表值逐条拆分、剥前缀、垃圾拦截、带窗号
fields0 = {}
for k, v in (ent.get("字段", {}) or {}).items():
if k in MILESTONE_FIELDS:
# 里程碑字段(演变历程):初卡即走对象合并(登场/首个进化台阶),去重排序;
# 初卡尚无 draft_id 无法留审计,垃圾条目直接过滤(与下方字符串路径初卡同策略)。
merged, _ = _merge_milestones(
[],
v,
win_no,
chapter_texts=chapter_texts,
require_evidence=True,
)
fields0[k] = merged
continue
if not isinstance(v, list):
fields0[k] = v
continue
out, seen = [], set()
for it in v:
for seg in re.split(r";\s*(?=\[[窗本])", str(it)):
core = _strip_tail(_strip_prefix(seg))
if not core or _is_garbage(core) or core in seen:
continue
out.append(f"[窗{win_no}] {core}")
seen.add(core)
fields0[k] = out
# 当前窗模型章必须逐章命中实体名;历史留档章来自既有机械留档,单独并入,不能因本窗正文不含历史章而误删。
evidence_aliases = list(aliases)
raw_evidence_name = ent.get("正文原名", "")
if isinstance(raw_evidence_name, str) and raw_evidence_name.strip():
evidence_aliases.append(raw_evidence_name.strip())
chaps_int = sorted(
_filter_entity_chapters(raw, evidence_aliases, ent.get("出场章", []), chapter_texts)
| _int_chaps(known_chapters)
)
# 洞② 登场兜底:该型合同含「演变历程」但抽取结果无登场里程碑时,机械补一条登场(出场章 min + 摘要)——
# 模型倾向只填当前态、漏建登场,此为「提示词硬约束 + 机械兜底」双保险里的机械那一保险。
if milestone_types and ent.get("型") in milestone_types:
fields0["演变历程"] = _debut_milestone(
fields0.get("演变历程") or [], raw, ent.get("型"), aliases,
chaps_int, win_no, chapter_texts)
payload = {"type": ent["型"], "名称": raw,
"别名": aliases,
"一句话摘要": ent.get("一句话摘要", ""),
"字段": fields0,
"出场章": chaps_int,
"来源": f"升格@窗{win_no}", "状态": "草稿",
"目标库": "本书作品库", "可见范围": "本书私有",
"_work_id": work_id}
return payload
def new_card(
conn,
work_id,
win_no,
ent,
milestone_types=None,
*,
chapter_texts=None,
known_chapters=None,
):
"""立初卡:payload 全字段以旧值=NULL 入审计(G7,错认拆回可还原初始态)。"""
payload = _build_new_card_payload(
work_id, win_no, ent, milestone_types,
chapter_texts=chapter_texts, known_chapters=known_chapters,
)
# 先原子占用全部 alias;任一不同 canonical 冲突都会让窗事务在写 payload 前失败关闭。
for alias in payload["别名"]:
_claim_alias(conn, work_id, payload["名称"], alias, win_no, "init")
did, initial_revision = conn.execute(
"""INSERT INTO muse_knowledge_draft
(work_id, draft_type, draft_payload, status, source_type, source_id,
creator, updater, tenant_id)
VALUES (%s,'entity',%s,'pending',%s,%s,'upgrade','upgrade',%s)
RETURNING id, revision""",
(work_id, json.dumps(payload, ensure_ascii=False), SOURCE_TYPE, work_id,
TENANT)).fetchone()
for k, v in payload["字段"].items():
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(did, win_no, k, json.dumps(v, ensure_ascii=False), TENANT))
conn.execute(
"""INSERT INTO example_upgrade_card_state (draft_id, work_id, watermark_window, tenant_id)
VALUES (%s,%s,%s,%s) ON CONFLICT (draft_id) DO NOTHING""",
(did, work_id, win_no, TENANT))
return did, payload["名称"], tuple(payload["别名"]), initial_revision
def undo_window(conn, work_id, win_no):
"""按审计与本窗来源精确撤销,供失败补偿后立即重试。"""
# 与向量的互动(洞①,不改本函数逻辑,仅说明边界):软删卡的向量靠召回 SQL 的 JOIN d.deleted=FALSE
# 天然排除,无需在此动嵌入行;被回滚的更新卡向量暂时偏新(对应已撤销的内容),重跑后嵌段
# final apply 按当前 payload 复验后软删旧活行并 upsert,自愈到正确态。
# 还原覆写字段(倒序还原,先写的最后还原到最初旧值)
rows = conn.execute(
"""SELECT a.draft_id, a.field_name, a.old_value FROM example_upgrade_audit a
JOIN muse_knowledge_draft d ON d.id=a.draft_id
WHERE a.tenant_id=%s AND d.work_id=%s AND a.window_no=%s ORDER BY a.id DESC""",
(TENANT, work_id, win_no)).fetchall()
for did, fname, old in rows:
if fname == CARD_STATE_WATERMARK_AUDIT_FIELD:
# 水位是独立表状态,不得误还原进 payload["字段"];新关系旧值为空时直接删 state。
if old is None:
conn.execute("DELETE FROM example_upgrade_card_state WHERE draft_id=%s", (did,))
else:
watermark = json.loads(old) if isinstance(old, str) else old
conn.execute(
"""INSERT INTO example_upgrade_card_state
(draft_id, work_id, watermark_window, tenant_id)
VALUES (%s,%s,%s,%s)
ON CONFLICT (draft_id) DO UPDATE
SET watermark_window=EXCLUDED.watermark_window, update_time=now()""",
(did, work_id, watermark, TENANT),
)
continue
payload = conn.execute("SELECT draft_payload FROM muse_knowledge_draft WHERE id=%s",
(did,)).fetchone()[0]
if fname == TOP_LEVEL_APPEARANCE_AUDIT_FIELD:
# 顶层出场章不在「字段」对象内:旧值为空表示原键不存在,否则按审计原 JSON 精确恢复。
if old is None:
payload.pop("出场章", None)
else:
payload["出场章"] = json.loads(old) if isinstance(old, str) else old
elif fname == TOP_LEVEL_ALIASES_AUDIT_FIELD:
# 顶层别名与唯一表必须一起撤销;旧值为空表示原 payload 连该键都不存在。
if old is None:
payload.pop("别名", None)
else:
payload["别名"] = json.loads(old) if isinstance(old, str) else old
elif fname == TOP_LEVEL_SUMMARY_AUDIT_FIELD:
# 一句话摘要住顶层,禁止按普通字段恢复成 payload["字段"] 内的影子摘要。
if old is None:
payload.pop("一句话摘要", None)
else:
payload["一句话摘要"] = json.loads(old) if isinstance(old, str) else old
elif fname == TOP_LEVEL_RELATION_TYPE_AUDIT_FIELD:
# 关系类型住 payload 顶层;普通字段恢复会错误写到 payload["字段"]。
if old is None:
payload.pop("关系类型", None)
else:
payload["关系类型"] = json.loads(old) if isinstance(old, str) else old
elif fname == TOP_LEVEL_RELATION_EVOLUTION_AUDIT_FIELD:
# 兼容历史关系卡顶层演变轨迹迁移,undo 时恢复到原顶层位置。
if old is None:
payload.pop("演变轨迹", None)
else:
payload["演变轨迹"] = json.loads(old) if isinstance(old, str) else old
elif old is None:
payload.get("字段", {}).pop(fname, None)
else:
payload.setdefault("字段", {})[fname] = json.loads(old)
conn.execute(
"""UPDATE muse_knowledge_draft
SET draft_payload=%s, revision=revision+1, updater='upgrade-undo'
WHERE id=%s""",
(json.dumps(payload, ensure_ascii=False), did),
)
conn.execute("""DELETE FROM example_upgrade_audit WHERE tenant_id=%s AND window_no=%s
AND draft_id IN (SELECT id FROM muse_knowledge_draft WHERE work_id=%s)""",
(TENANT, win_no, work_id))
# 删本窗追加条目([窗N] 前缀)与初立于本窗的卡(audit 已删,靠 card_state 水位判初窗不可靠,
# 初卡以「来源=升格@窗N」标记识别)
for did, payload in conn.execute(
"""SELECT id, draft_payload FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s AND deleted=FALSE""",
(TENANT, work_id, SOURCE_TYPE)).fetchall():
if payload.get("来源") == f"升格@窗{win_no}":
conn.execute(
"""UPDATE muse_knowledge_draft
SET deleted=TRUE, revision=revision+1, updater='upgrade-undo'
WHERE id=%s""",
(did,),
)
# 同步清卡水位行(fable 复验实证:深空回滚删 400+ 初卡后 card_state 僵尸行
# 残留,按 draft JOIN 不过滤 deleted 的查询会捞出僵尸卡)
conn.execute("DELETE FROM example_upgrade_card_state WHERE draft_id=%s", (did,))
continue
tag, changed = f"[窗{win_no}] ", False
for k, v in list(payload.get("字段", {}).items()):
if isinstance(v, list):
# 字符串条目按 [窗N] 前缀删;里程碑对象(演变历程)按内部 _win 溯源键删——
# 对象无窗号前缀,撤销靠 _win 精确识别本窗新增,否则同窗重跑会 double-append。
nv = [x for x in v
if not (isinstance(x, str) and x.startswith(tag))
and not (isinstance(x, dict) and x.get("_win") == win_no)]
if len(nv) != len(v):
payload["字段"][k], changed = nv, True
if changed:
conn.execute(
"""UPDATE muse_knowledge_draft
SET draft_payload=%s, revision=revision+1, updater='upgrade-undo'
WHERE id=%s""",
(json.dumps(payload, ensure_ascii=False), did),
)
conn.execute("DELETE FROM example_upgrade_presence WHERE tenant_id=%s AND work_id=%s AND window_no=%s",
(TENANT, work_id, win_no))
conn.execute("""DELETE FROM example_upgrade_alias WHERE tenant_id=%s AND work_id=%s
AND evidence_window=%s""", (TENANT, work_id, win_no))
# ── 跨型别名裁决(机械候选 + 单次批量 LLM) ──
def _clean_candidate_aliases(ent):
"""校验候选别名字段并复用统一清洗规则;结构不合法直接失败关闭。"""
if "别名" not in ent:
return []
aliases = ent["别名"]
if not isinstance(aliases, list) or any(not isinstance(alias, str) for alias in aliases):
raise RuntimeError(f"候选别名必须是字符串列表:名称={ent.get('名称')}")
return [cleaned for raw in aliases if (cleaned := _clean_alias(raw))]
def _arbitrate_cross_type_aliases(obs, name_map, aliases_by_draft, text, call):
"""批量裁决候选自身/疑似别名命中的跨型 owner,并把结果写回 observation。
机械阶段只负责找唯一 owner;跨型是否同一实体必须由一次受治理的 JSON 调用裁决。
"""
cases = []
seen_case_keys = set()
for entity in obs.get("新名字", []):
if not isinstance(entity, dict):
continue
candidate_name = (entity.get("名称") or "").strip()
candidate_type = entity.get("型") or ""
if not isinstance(candidate_type, str):
raise RuntimeError(f"候选类型必须是字符串:名称={candidate_name}")
raw_hint = entity.get("疑似别名指向", "")
if raw_hint is None:
raw_hint = ""
if not isinstance(raw_hint, str):
raise RuntimeError(f"疑似别名指向必须是字符串:名称={candidate_name}")
hint = raw_hint.strip()
aliases = _clean_candidate_aliases(entity)
refs = []
if candidate_name and candidate_name in name_map:
refs.append(candidate_name)
if hint and hint in name_map:
if hint not in refs:
refs.append(hint)
refs.extend(alias for alias in aliases if alias in name_map and alias not in refs)
owners = {}
for matched in refs:
target = name_map.get(matched)
if not isinstance(target, (tuple, list)) or len(target) < 3:
raise RuntimeError(f"判重 owner 投影不完整:名称={candidate_name},命中={matched}")
draft_id, owner_type, owner_brief = target[:3]
owner = owners.setdefault(draft_id, {
"type": owner_type,
"brief": owner_brief,
"aliases": [],
})
if (owner["type"], owner["brief"]) != (owner_type, owner_brief):
raise RuntimeError(f"命中 owner 信息不唯一:名称={candidate_name},draft={draft_id}")
owner["aliases"].append(matched)
if len(owners) > 1:
# 精确候选名已经绑定同型 owner 时,附加别名撞到另一张同型卡不属于跨型
# 裁决:保留精确名称主权,交给 _classify_new_name 的 merge 分支处理。
exact_target = name_map.get(candidate_name)
exact_owner_id = (
exact_target[0]
if isinstance(exact_target, (tuple, list)) and len(exact_target) >= 3
else None
)
if (
candidate_name in name_map
and exact_owner_id in owners
and all(owner["type"] == candidate_type for owner in owners.values())
):
continue
raise RuntimeError(f"候选命中多个 owner,拒绝裁决:名称={candidate_name},owners={sorted(owners)}")
if not owners:
continue
draft_id, owner = next(iter(owners.items()))
if owner["type"] == candidate_type:
continue
canonical = _resolve_canonical_name(name_map, aliases_by_draft, refs[0])
case_key = (candidate_name, candidate_type, canonical, owner["type"])
if case_key in seen_case_keys:
raise RuntimeError(f"跨型候选裁决项重复:{case_key}")
seen_case_keys.add(case_key)
cases.append({
"entity": entity,
"name": candidate_name,
"type": candidate_type,
"brief": entity.get("一句话摘要", ""),
"fields": entity.get("字段") or {},
"chapters": sorted(_int_chaps(entity.get("出场章", []))),
"canonical": canonical,
"owner_type": owner["type"],
"owner_brief": owner["brief"],
"aliases": sorted(set(owner["aliases"])),
"name_exact_hit": candidate_name in name_map,
})
if not cases:
return 0
if len(cases) > CROSS_TYPE_ALIAS_MAX_CANDIDATES:
raise RuntimeError(
f"跨型候选超过单批裁决上限:count={len(cases)},"
f"max={CROSS_TYPE_ALIAS_MAX_CANDIDATES}"
)
rows = []
for case in cases:
rows.append({
"候选名称": case["name"],
"候选类型": case["type"],
"候选摘要": case["brief"],
"候选字段": case["fields"],
"已有规范名": case["canonical"],
"已有类型": case["owner_type"],
"已有摘要": case["owner_brief"],
"冲突别名": case["aliases"],
"候选出场章": case["chapters"],
})
prompt = f"""【功能指令(parse-book 作品面升格·跨型别名裁决)】
下面是本窗 observation 机械发现的跨型别名冲突。逐条判断候选与唯一已有 owner 是否同一实体。
只能输出 same_entity、different_entity 或 uncertain;不确定必须输出 uncertain,系统会失败关闭。
same_entity:候选是已有 owner 的改名/化名/完整称谓;different_entity:只是别名重名或不同实体。
输入只包含候选名称/类型/摘要/字段、已有规范名/类型/摘要、冲突别名和当前窗原文证据。
【待裁决候选】
{json.dumps(rows, ensure_ascii=False, indent=2)}
【当前窗原文证据】
{text}
【输出规则】
每个输入候选必须恰好输出一条,禁止漏项、多项或添加额外对象:
{{"跨型别名裁决": [{{"候选名称": "", "候选类型": "", "已有规范名": "", "已有类型": "", "冲突别名": [], "候选出场章": [], "裁决": "same_entity|different_entity|uncertain"}}]}}
"""
result, _ = call(prompt, ("跨型别名裁决",))
if not isinstance(result, dict) or set(result) != {"跨型别名裁决"}:
raise RuntimeError("跨型别名裁决输出结构非法")
decisions = result.get("跨型别名裁决")
if not isinstance(decisions, list):
raise RuntimeError("跨型别名裁决输出不是列表")
expected = {
(case["name"], case["type"], case["canonical"], case["owner_type"]): case
for case in cases
}
seen = {}
required_keys = {
"候选名称", "候选类型", "已有规范名", "已有类型", "冲突别名", "候选出场章", "裁决",
}
for decision in decisions:
if not isinstance(decision, dict) or set(decision) != required_keys:
raise RuntimeError("跨型别名裁决项结构非法")
key = (
decision.get("候选名称"), decision.get("候选类型"),
decision.get("已有规范名"), decision.get("已有类型"),
)
case = expected.get(key)
if case is None:
raise RuntimeError(f"跨型别名裁决项无法精确匹配:{key}")
if key in seen:
raise RuntimeError(f"跨型别名裁决项重复:{key}")
output_aliases = decision.get("冲突别名")
if not isinstance(output_aliases, list) or any(not isinstance(alias, str) for alias in output_aliases):
raise RuntimeError(f"跨型别名裁决 alias 字段非法:{key}")
cleaned_output = [_clean_alias(alias) for alias in output_aliases]
if (
any(alias is None for alias in cleaned_output)
or len(cleaned_output) != len(set(cleaned_output))
or set(cleaned_output) != set(case["aliases"])
):
raise RuntimeError(f"跨型别名裁决 alias 不精确:{key}")
if decision.get("候选出场章") != case["chapters"]:
raise RuntimeError(f"跨型别名裁决出场章不精确:{key}")
verdict = decision.get("裁决")
if verdict == "uncertain":
raise RuntimeError("跨型别名裁决不确定,拒绝继续")
if verdict not in {"same_entity", "different_entity"}:
raise RuntimeError(f"跨型别名裁决值非法:{key},verdict={verdict}")
seen[key] = verdict
if set(seen) != set(expected):
raise RuntimeError(
f"跨型别名裁决缺失候选:missing={sorted(set(expected) - set(seen))}"
)
for key, verdict in seen.items():
if verdict == "different_entity" and expected[key]["name_exact_hit"]:
raise RuntimeError(
f"跨型 different_entity 拒绝:候选名称已被底册占用:名称={expected[key]['name']}"
)
for key, verdict in seen.items():
case = expected[key]
entity = case["entity"]
if verdict == "same_entity":
entity["型"] = case["owner_type"]
entity["疑似别名指向"] = case["canonical"]
continue
conflict_aliases = set(case["aliases"])
entity["别名"] = [
raw_alias for raw_alias in entity.get("别名", [])
if _clean_alias(raw_alias) not in conflict_aliases
]
if entity.get("疑似别名指向", "").strip() in conflict_aliases:
entity.pop("疑似别名指向", None)
return len(cases)
def _observation_chapter_errors(value, label, a, b):
"""严格检查观察输出的出场章结构,允许既有数字字符串归一规则但不接受标量。"""
if not isinstance(value, list):
return [f"{label}出场章必须是列表"]
errors = []
for index, raw_chapter in enumerate(value):
if isinstance(raw_chapter, bool):
errors.append(f"{label}出场章[{index}]不是合法章号")
continue
if isinstance(raw_chapter, int):
chapter = raw_chapter
elif isinstance(raw_chapter, str) and raw_chapter.strip().isdigit():
chapter = int(raw_chapter.strip())
else:
errors.append(f"{label}出场章[{index}]不是合法章号")
continue
if chapter <= 0 or (a is not None and b is not None and not a <= chapter <= b):
errors.append(f"{label}出场章[{index}]超出当前窗范围")
return errors
def _observation_structure_errors(obs, name_map, a, b, *, keys=OBSERVATION_INTERNAL_KEYS):
"""返回实体观察 JSON 的结构错误;不改写输入,也不丢弃任何模型项。
默认三数组仅为旧私有 helper 兼容;生产 observation 传入两数组模型合同。
"""
if not isinstance(obs, dict):
return ["实体观察输出必须是 JSON 对象"]
keys = tuple(keys)
allowed_item_keys = (
OBSERVATION_MODEL_ITEM_ALLOWED_KEYS
if keys == OBSERVATION_MODEL_KEYS
else OBSERVATION_ITEM_ALLOWED_KEYS
)
errors = []
extra_keys = [key for key in obs if key not in keys]
if extra_keys:
errors.append(f"顶层存在额外键(只允许 observation keys):{extra_keys!r}")
for key in keys:
if key not in obs:
errors.append(f"缺少顶层数组:{key}")
elif not isinstance(obs[key], list):
errors.append(f"顶层字段必须是列表:{key}")
for key in keys:
items = obs.get(key)
if not isinstance(items, list):
continue
for index, item in enumerate(items):
label = f"{key}[{index}]"
if not isinstance(item, dict):
errors.append(f"{label}必须是对象")
continue
extra_item_keys = sorted(
set(item) - allowed_item_keys[key],
key=repr,
)
if extra_item_keys:
errors.append(f"{label}存在额外键:{extra_item_keys!r}")
raw_name = item.get("名称")
name = raw_name.strip() if isinstance(raw_name, str) else ""
if not name:
errors.append(f"{label}缺少有效名称")
if key == "新名字":
raw_type = item.get("型")
entity_type = raw_type.strip() if isinstance(raw_type, str) else ""
if entity_type not in ENTITY_TYPES:
errors.append(f"{label}缺少有效型")
if "别名" in item and (
not isinstance(item["别名"], list)
or any(not isinstance(alias, str) for alias in item["别名"])
):
errors.append(f"{label}别名必须是字符串列表")
if "字段" in item and not isinstance(item["字段"], dict):
errors.append(f"{label}字段必须是对象")
if "疑似别名指向" in item and not isinstance(item["疑似别名指向"], str):
errors.append(f"{label}疑似别名指向必须是字符串")
if keys != OBSERVATION_MODEL_KEYS:
errors.extend(_observation_chapter_errors(item.get("出场章"), label, a, b))
continue
if key == "已知实体新信息":
owner = name_map.get(name) if name else None
if not owner:
raw_type = item.get("型")
entity_type = raw_type.strip() if isinstance(raw_type, str) else ""
if entity_type not in ENTITY_TYPES:
errors.append(f"{label}缺少有效型")
elif "型" in item:
raw_type = item["型"]
entity_type = raw_type.strip() if isinstance(raw_type, str) else ""
if entity_type not in ENTITY_TYPES:
errors.append(f"{label}型无效")
raw_observation = item.get("观察点")
if not isinstance(raw_observation, str) or not raw_observation.strip():
errors.append(f"{label}观察点无效")
if keys != OBSERVATION_MODEL_KEYS:
errors.extend(_observation_chapter_errors(item.get("出场章"), label, a, b))
continue
errors.extend(_observation_chapter_errors(item.get("出场章"), label, a, b))
return errors
def _observation_repair_chapter_values(value):
"""提取可证明保留的正整数章号;允许 repair 做标量包列表和数字字符串归一化。"""
if isinstance(value, bool):
raw_values = []
elif isinstance(value, int):
raw_values = [value]
elif isinstance(value, str) and value.strip().isdigit():
raw_values = [value]
elif isinstance(value, list):
raw_values = value
else:
raw_values = []
chapters = []
for raw_chapter in raw_values:
if isinstance(raw_chapter, bool):
continue
if isinstance(raw_chapter, int):
chapter = raw_chapter
elif isinstance(raw_chapter, str) and raw_chapter.strip().isdigit():
chapter = int(raw_chapter.strip())
else:
continue
if chapter > 0:
chapters.append(chapter)
return chapters
def _observation_repair_json_value(value, label):
"""把 JSON 值编码成稳定事实指纹;无法精确表示时失败关闭。"""
try:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
allow_nan=False,
)
except (TypeError, ValueError) as exc:
raise RuntimeError(f"{label}含不可守恒的非 JSON 值") from exc
def _observation_repair_chapter_signature(value, label):
"""返回章节的稳定多重集合;只接受标量包列表和数字字符串归一化。"""
if isinstance(value, bool):
raise RuntimeError(f"{label}出场章无法作为结构修复输入")
elif isinstance(value, int):
raw_values = [value]
elif isinstance(value, str) and value.strip().isdigit():
raw_values = [value]
elif isinstance(value, list):
raw_values = value
else:
raise RuntimeError(f"{label}出场章无法作为结构修复输入")
chapters = []
for index, raw_chapter in enumerate(raw_values):
if isinstance(raw_chapter, bool):
raise RuntimeError(f"{label}出场章[{index}]不是可归一化章号")
if isinstance(raw_chapter, int):
chapter = raw_chapter
elif isinstance(raw_chapter, str) and raw_chapter.strip().isdigit():
chapter = int(raw_chapter.strip())
else:
raise RuntimeError(f"{label}出场章[{index}]不是可归一化章号")
if chapter <= 0:
raise RuntimeError(f"{label}出场章[{index}]不是正整数")
chapters.append(chapter)
return tuple(sorted(chapters))
def _observation_repair_record(item, key, index, stage, *, require_chapters=True):
"""为单个观察项建立跨数组可匹配的稳定记录。"""
label = f"{stage}{key}[{index}]"
if require_chapters and "出场章" not in item:
raise RuntimeError(f"{label}缺少出场章,无法建立守恒身份")
facts = tuple(sorted(
(
(
field,
_observation_repair_json_value(value, f"{label}.{field}"),
)
for field, value in item.items()
if field != "出场章"
),
key=lambda pair: repr(pair[0]),
))
if require_chapters:
chapters = _observation_repair_chapter_signature(item.get("出场章", []), label)
else:
# 模型协议里的出场章即使出现也不是事实;非法/缺失值只按空派生结构兼容,
# 生产路径随后会由正文索引重新覆盖。
try:
chapters = _observation_repair_chapter_signature(item.get("出场章", []), label)
except RuntimeError:
chapters = ()
return {
"array": key,
"name": item["名称"],
"facts": facts,
"chapters": chapters,
}
def _observation_repair_fact_map(record):
"""返回单个稳定记录的字段事实映射。"""
return dict(record["facts"])
def _observation_repair_type_matches(before_facts, after_facts):
"""已有型必须原值保留;只有原对象缺型时允许补合法型。"""
if "型" in before_facts:
return after_facts.get("型") == before_facts["型"]
if "型" not in after_facts:
return True
try:
entity_type = json.loads(after_facts["型"])
except (TypeError, ValueError):
return False
return isinstance(entity_type, str) and entity_type.strip() in ENTITY_TYPES
def _observation_repair_record_matches(before, after):
"""判断一个 repair 后对象是否只做了声明允许的结构变化。"""
if before["name"] != after["name"]:
return False
if (
before["array"] not in OBSERVATION_MODEL_KEYS
or after["array"] not in OBSERVATION_MODEL_KEYS
) and before["chapters"] != after["chapters"]:
return False
before_facts = _observation_repair_fact_map(before)
after_facts = _observation_repair_fact_map(after)
if not _observation_repair_type_matches(before_facts, after_facts):
return False
before_without_type = {
field: value for field, value in before_facts.items() if field != "型"
}
after_without_type = {
field: value for field, value in after_facts.items() if field != "型"
}
if before["array"] == after["array"]:
return before_without_type == after_without_type
# 只要数组切换没有改变任何事实,同样允许跨数组移动;schema 特有的
# 观察点/摘要映射和新名字默认值由下面的专用分支单独放行。
if before_without_type == after_without_type:
return True
if before["array"] != "已知实体新信息" or after["array"] != "新名字":
return False
observation = before_without_type.get("观察点")
if observation is None:
return False
expected = {
field: value
for field, value in before_without_type.items()
if field != "观察点"
}
expected["一句话摘要"] = observation
defaults = {
"别名": _observation_repair_json_value([], "repair默认别名"),
"字段": _observation_repair_json_value({}, "repair默认字段"),
}
for field, value in after_without_type.items():
if field in defaults:
if value != defaults[field]:
return False
elif expected.get(field) != value:
return False
return all(after_without_type.get(field) == value for field, value in expected.items())
def _observation_repair_records_match(before_records, after_records):
"""以多重集合方式匹配对象,允许数组移动但不允许事实漂移。"""
if len(before_records) != len(after_records):
return False
candidates = [
[
after_index
for after_index, after in enumerate(after_records)
if _observation_repair_record_matches(before, after)
]
for before in before_records
]
if any(not options for options in candidates):
return False
order = sorted(range(len(before_records)), key=lambda index: len(candidates[index]))
used = set()
def assign(position):
if position == len(order):
return True
before_index = order[position]
for after_index in candidates[before_index]:
if after_index in used:
continue
used.add(after_index)
if assign(position + 1):
return True
used.remove(after_index)
return False
return assign(0)
def _observation_repair_snapshot(obs, *, stage, keys=OBSERVATION_INTERNAL_KEYS):
"""建立 repair 守恒快照;身份不可靠时在模型调用前失败关闭。"""
if not isinstance(obs, dict):
raise RuntimeError(f"{stage}不是对象,无法建立守恒身份")
snapshot = {
"names": Counter(),
"count": 0,
"known_observations": Counter(),
"new_summaries": Counter(),
"chapters": Counter(),
"records": [],
# 出场章在模型协议中是派生值,允许 repair 响应省略或被错误重写;
# 旧三数组私有 helper 仍严格守恒该字段。
"track_chapters": tuple(keys) != OBSERVATION_MODEL_KEYS,
}
for key in tuple(keys):
if key not in obs:
continue
items = obs[key]
if not isinstance(items, list):
raise RuntimeError(f"{stage}{key}不是列表,无法建立守恒身份")
for index, item in enumerate(items):
label = f"{stage}{key}[{index}]"
if not isinstance(item, dict):
raise RuntimeError(f"{label}不是对象,无法建立守恒身份")
raw_name = item.get("名称")
if not isinstance(raw_name, str) or not raw_name.strip():
raise RuntimeError(f"{label}缺少有效名称,无法建立守恒身份")
name = raw_name
snapshot["names"][name] += 1
snapshot["count"] += 1
snapshot["records"].append(
_observation_repair_record(
item,
key,
index,
stage,
require_chapters=tuple(keys) != OBSERVATION_MODEL_KEYS,
)
)
if key == "已知实体新信息":
raw_observation = item.get("观察点")
if raw_observation is not None and not isinstance(raw_observation, str):
raise RuntimeError(f"{label}观察点不是字符串,无法建立事实守恒")
if isinstance(raw_observation, str) and raw_observation.strip():
snapshot["known_observations"][(name, raw_observation)] += 1
elif key == "新名字":
raw_summary = item.get("一句话摘要")
if raw_summary is not None and not isinstance(raw_summary, str):
raise RuntimeError(f"{label}一句话摘要不是字符串,无法建立事实守恒")
if isinstance(raw_summary, str) and raw_summary.strip():
snapshot["new_summaries"][(name, raw_summary)] += 1
for chapter in _observation_repair_chapter_values(item.get("出场章")):
snapshot["chapters"][(name, chapter)] += 1
return snapshot
def _observation_repair_conservation_errors(before, after):
"""比较 repair 前后身份、事实和合法章号,只允许声明过的结构归一化。"""
errors = []
if before["count"] != after["count"] or before["names"] != after["names"]:
errors.append(
"对象守恒失败:总数或名称多重集合变化,"
f"before_count={before['count']} after_count={after['count']},"
f"before_names={dict(before['names'])!r} after_names={dict(after['names'])!r}"
)
repaired_facts = after["known_observations"] + after["new_summaries"]
missing_observations = before["known_observations"] - repaired_facts
if missing_observations:
errors.append(f"首轮观察点未原样保留:{dict(missing_observations)!r}")
missing_summaries = before["new_summaries"] - after["new_summaries"]
if missing_summaries:
errors.append(f"首轮一句话摘要未原样保留:{dict(missing_summaries)!r}")
if before.get("track_chapters", True) and after.get("track_chapters", True) \
and before["chapters"] != after["chapters"]:
errors.append(
"合法出场章未守恒(只允许标量包列表或数字字符串归一化):"
f"before={dict(before['chapters'])!r} after={dict(after['chapters'])!r}"
)
if not _observation_repair_records_match(before["records"], after["records"]):
errors.append("首轮对象的所有既有事实未守恒(仅允许结构归一、缺型补值或已知信息迁移)")
return errors
def entity_observation_repair_prompt(
obs, *, title, a, b, text, onstage, name_map, errors,
keys=OBSERVATION_INTERNAL_KEYS,
):
"""构造唯一一次实体观察结构 repair prompt;不扩展事实。"""
owner_names = set(onstage or {})
keys = tuple(keys)
allowed_item_keys = (
OBSERVATION_MODEL_ITEM_ALLOWED_KEYS
if keys == OBSERVATION_MODEL_KEYS
else OBSERVATION_ITEM_ALLOWED_KEYS
)
for key in keys:
for item in (obs.get(key) if isinstance(obs, dict) else []) or []:
if isinstance(item, dict):
name = item.get("名称")
if isinstance(name, str) and name.strip() in name_map:
owner_names.add(name.strip())
for name in list(owner_names):
target = name_map.get(name)
if isinstance(target, (tuple, list)) and len(target) >= 3:
owner_names.update(
candidate_name
for candidate_name, candidate_target in name_map.items()
if candidate_target == target
)
owners = []
for name in sorted(owner_names):
target = name_map.get(name)
if isinstance(target, (tuple, list)) and len(target) >= 3:
owners.append({"名称": name, "draft_id": target[0], "型": target[1], "摘要": target[2]})
allowed_keys = {
key: sorted(allowed_item_keys[key])
for key in keys
}
raw_json = json.dumps(obs, ensure_ascii=False, indent=2, default=str)
if keys == OBSERVATION_MODEL_KEYS:
protocol = (
"模型协议只含「新名字」「已知实体新信息」两个列表;「纯出场」是系统机械索引产物,"
"不得输出、补回或重写;出场章不是模型事实,若首轮偶尔带出场章,系统会用正文索引覆盖。"
)
output_example = (
'{{"新名字": [{{"型": "character", "名称": "", "正文原名": "", "别名": [], '
'"一句话摘要": "", "字段": {{}}, '
'"疑似别名指向": ""}}], '
'"已知实体新信息": [{{"名称": "", "型": "", "观察点": ""}}]}}'
)
else:
protocol = "兼容旧私有 helper 时允许三数组;生产路径不得使用该协议。"
output_example = (
'{{"新名字": [{{"型": "character", "名称": "", "别名": [], '
'"一句话摘要": "", "字段": {{}}, "出场章": [章号]}},], '
'"已知实体新信息": [{{"名称": "", "型": "", "观察点": "", '
'"出场章": [章号]}}], "纯出场": [{{"名称": "", "出场章": [章号]}}]}}'
)
preserve_rule = (
"只可修复数组/对象结构;所有名称、对象数量和既有语义键值必须原样保留,"
"出场章即使存在也不属于守恒事实。"
if keys == OBSERVATION_MODEL_KEYS
else
"只可把单个章号包成列表、把数字字符串章号归一、把原有对象移到正确数组;"
"所有名称、对象数量、章节和既有键值必须原样保留。"
)
chapter_rule = (
"模型协议不要求出场章;若响应带有该键,生产路径会用正文逐章索引覆盖。"
if keys == OBSERVATION_MODEL_KEYS
else
f"所有出场章必须是当前窗 {a}-{b} 范围内的正整数列表;禁止区间、相对窗号和窗外章号。"
)
return f"""【功能指令(parse-book 作品面升格·实体观察结构 repair)】
实体观察首轮 JSON 只出现结构错误。只做一次、有界的、仅做结构修复,不重新分析正文,不新增事实,不删除或静默丢弃原有对象;只允许修复当前协议数组及其字段结构。
必须遵守:
1. {protocol}输出必须恰好是一个 JSON 对象,且只含当前协议列表;每个列表项必须是对象。
2. {preserve_rule}
3. 原对象缺少「型」时可补一个合法型:{"/".join(ENTITY_TYPES)};原对象已有「型」时禁止改写。已知实体名称命中下面 owner 索引时,后续仍由 name_map 规范化 owner 型。
4. 仅当「已知实体新信息」对象移到「新名字」时,才可把原「观察点」原样复制为「一句话摘要」,并补「别名」=[]、「字段」={{}}这两个新名字结构默认值;不得改写观察点。
5. 每个数组项只能使用下方允许键;额外键、任意新增事实键、删除已有事实键都会被系统失败关闭。
6. {chapter_rule}
【首轮结构错误】
{json.dumps(errors, ensure_ascii=False)}
【首轮 JSON】
{raw_json}
【name_map owner 索引(型不可被覆盖)】
{json.dumps(owners, ensure_ascii=False)}
【数组项允许键】
{json.dumps(allowed_keys, ensure_ascii=False)}
【当前窗原文(仅用于补齐已有对象的结构型信息)】
{text}
【输出规则】
{output_example}
"""
def _repair_entity_observation_structure(
obs, *, title, a, b, text, onstage, name_map, call,
keys=OBSERVATION_INTERNAL_KEYS,
):
"""需要时只调用一次结构 repair,repair 后再次严格验收,失败即向上关闭窗口。"""
keys = tuple(keys)
if keys == OBSERVATION_MODEL_KEYS and isinstance(obs, dict) and "纯出场" in obs:
raise RuntimeError("实体观察模型输出禁止包含纯出场字段")
errors = _observation_structure_errors(obs, name_map, a, b, keys=keys)
if not errors:
return obs
before = _observation_repair_snapshot(obs, stage="首轮", keys=keys)
prompt = entity_observation_repair_prompt(
obs,
title=title,
a=a,
b=b,
text=text,
onstage=onstage,
name_map=name_map,
errors=errors,
keys=keys,
)
repaired, _ = call(prompt, keys)
if keys == OBSERVATION_MODEL_KEYS and isinstance(repaired, dict) and "纯出场" in repaired:
raise RuntimeError("实体观察结构 repair 禁止包含纯出场字段")
repaired_errors = _observation_structure_errors(repaired, name_map, a, b, keys=keys)
if repaired_errors:
raise RuntimeError(
"实体观察结构 repair 仍非法:" + ";".join(repaired_errors[:12])
)
after = _observation_repair_snapshot(repaired, stage="repair后", keys=keys)
conservation_errors = _observation_repair_conservation_errors(before, after)
if conservation_errors:
raise RuntimeError(
"实体观察结构 repair 守恒失败:" + ";".join(conservation_errors[:12])
)
return repaired
def _normalize_known_entity_observations(obs, name_map):
"""严格校验 known-info,并把无法直接更新的项归一到新名字候选。"""
known_items = obs.get("已知实体新信息")
if not isinstance(known_items, list):
raise RuntimeError("已知实体新信息必须是列表")
new_items = obs.get("新名字")
if not isinstance(new_items, list):
raise RuntimeError("新名字必须是列表")
retained, converted = [], []
for item in known_items:
if not isinstance(item, dict):
raise RuntimeError("已知实体新信息项结构非法")
raw_name = item.get("名称")
name = raw_name.strip() if isinstance(raw_name, str) else ""
if not name:
raise RuntimeError("已知实体新信息缺少有效名称")
owner = name_map.get(name)
owner_type = owner[1] if isinstance(owner, (tuple, list)) and len(owner) > 1 else None
if owner:
entity_type = owner_type
else:
raw_type = item.get("型")
entity_type = raw_type.strip() if isinstance(raw_type, str) else ""
if entity_type not in ENTITY_TYPES:
raise RuntimeError(f"已知实体新信息缺少有效型:名称={name},型={raw_type}")
raw_observation = item.get("观察点")
observation = raw_observation.strip() if isinstance(raw_observation, str) else ""
if not observation:
raise RuntimeError(f"已知实体新信息观察点无效:名称={name}")
observation = observation[:KNOWN_INFO_OBSERVATION_MAX]
raw_chapters = item.get("出场章", [])
if not isinstance(raw_chapters, list):
raise RuntimeError(f"已知实体新信息出场章必须是列表:名称={name}")
chapters = []
for raw_chapter in raw_chapters:
if isinstance(raw_chapter, bool):
raise RuntimeError(f"已知实体新信息出场章无效:名称={name},章={raw_chapter}")
if isinstance(raw_chapter, int):
chapter = raw_chapter
elif isinstance(raw_chapter, str) and raw_chapter.strip().isdigit():
chapter = int(raw_chapter.strip())
else:
raise RuntimeError(f"已知实体新信息出场章无效:名称={name},章={raw_chapter}")
if chapter <= 0:
raise RuntimeError(f"已知实体新信息出场章无效:名称={name},章={chapter}")
chapters.append(chapter)
chapters = sorted(set(chapters))
normalized = dict(item)
normalized.update({"名称": name, "型": entity_type, "观察点": observation, "出场章": chapters})
if owner:
# known-info 的精确名称命中就是模型对身份的明确声明;型冲突按底册 owner 型归一,保留更新材料。
normalized["型"] = owner_type
retained.append(normalized)
continue
candidate = {
"型": entity_type,
"名称": name,
"别名": [],
"一句话摘要": observation,
"字段": {},
"出场章": chapters,
}
raw_evidence_name = item.get("正文原名")
if isinstance(raw_evidence_name, str) and raw_evidence_name.strip():
candidate["正文原名"] = raw_evidence_name.strip()
converted.append(candidate)
obs["已知实体新信息"] = retained
obs["新名字"].extend(converted)
def _validate_known_entity_observations(obs, name_map):
"""兼容旧私有入口,实际执行确定性 known-info 归一化。"""
return _normalize_known_entity_observations(obs, name_map)
# ── 机械判重分类(洞①):预判段与写段共用的纯函数,把判据抽出防两处漂移 ──
def _classify_new_name(ent, name_map, presence):
"""机械判重分类(纯函数,无副作用;洞①):把一个「新名字」实体归到判重分支之一,返回 (kind, key):
('empty', None) 名称为空 → 跳过
('merge', nm) nm 已在底册(观察漏看在场清单)→ 直接归并到 name_map[nm]
('alias', hint) 同型疑似别名指向或候选自身别名命中 → 并入 name_map[hint]
('substr', 既有名) 同型名称互为子串 → 并入 name_map[该既有名]
('new', None) 跨章(含跨窗合计 ≥2 章)新实体 → 立卡(语义判重在此候选上做)
('presence', None) 单章龙套 → 留档
判据与写段 ②-立卡门槛**完全一致**(名非空 / 不在 name_map / 无同型别名命中 / 无同型子串命中 /
跨章阈值 ≥2);预判段按此识别 'new' 候选做语义判重、写段按此走对应副作用分支,同一份判据防漂移。"""
nm = (ent.get("名称") or "").strip()
if not nm:
return "empty", None
t = ent.get("型", "")
if nm in name_map: # 观察漏看在场清单:直接归并
owner_type = name_map[nm][1]
if owner_type != t:
raise RuntimeError(
f"判重规范名跨型未裁决:名称={nm},候选型={t},owner型={owner_type}"
)
return "merge", nm
raw_hint = ent.get("疑似别名指向", "")
if raw_hint is None:
raw_hint = ""
if not isinstance(raw_hint, str):
raise RuntimeError(f"疑似别名指向必须是字符串:名称={nm}")
hint = raw_hint.strip()
alias_hits = []
if hint and hint in name_map:
alias_hits.append((hint, name_map[hint][0], name_map[hint][1]))
for alias in _clean_candidate_aliases(ent):
if alias in name_map and not any(alias == hit[0] for hit in alias_hits):
alias_hits.append((alias, name_map[alias][0], name_map[alias][1]))
hit_drafts = {draft_id for _, draft_id, _ in alias_hits}
if len(hit_drafts) > 1:
raise RuntimeError(
f"候选自身别名命中多个 draft,拒绝判重:名称={nm},命中别名={alias_hits}"
)
if alias_hits:
matched, _, owner_type = alias_hits[0]
if owner_type == t:
return "alias", matched
raise RuntimeError(
f"判重 alias 跨型未裁决:名称={nm},命中={matched},"
f"候选型={t},owner型={owner_type}"
)
sub_hit = next( # 同型名称互为子串(「果子」vs「开心果子」同人两卡)
(ex for ex, (d0, t0, _) in name_map.items()
if t0 == t and len(nm) >= 2 and len(ex) >= 2
and nm != ex and (nm in ex or ex in nm)), None)
if sub_hit:
return "substr", sub_hit
chaps = _int_chaps(ent.get("出场章", [])) # 归一化 int:避免 "508"/508 混算虚增跨章计数
hist = presence.get((t, nm), set())
if len(chaps | hist) >= 2: # 跨章(含跨窗合计)→ 立卡
return "new", None
return "presence", None # 单章龙套 → 留档
def _resolve_canonical_name(name_map, aliases_by_draft, matched_name):
"""把命中的规范名或别名解析为该 draft 当前真实规范名。
``name_map`` 同时以规范名和别名建索引,不能把命中的键直接当 ``canonical_name`` 落库。
真实规范名必须是同 draft 下不属于合法别名集合的活跃卡名称;若底册不完整到无法唯一解析,
直接失败关闭当前窗,避免写入 alias token 后仍把窗口标成完成。"""
target = name_map.get(matched_name)
if not target:
raise RuntimeError(f"判重命中项不在底册:{matched_name}")
draft_id = target[0]
aliases = set(aliases_by_draft.get(draft_id, set()))
canonical_names = [
name
for name, value in name_map.items()
if value[0] == draft_id and name not in aliases
]
if len(canonical_names) != 1:
raise RuntimeError(
f"draft={draft_id} 无法唯一解析真实规范名:候选={canonical_names},命中={matched_name}"
)
return canonical_names[0]
# ── 语义判重(P1,设计稿 §8.2):打开 v6 已设计、暂时关着的「嵌入近邻 + M3 终判」那级 ──
# 治病根 4:机械判重只比名字字符串,改名("影杀者"→"IV代纯机械机甲·影杀者")/跨型指代就漏并。
# 默认关(--semantic-dedup 开启):连接三段式(洞①)——嵌入/召回/终判在预判段(prejudge_semantic)无长
# 连接完成,写段只查预判 verdicts;final apply 后窗即可召回前窗刚长成的卡。
def _entity_embed_text(ent):
"""判重查询侧嵌入文本:【型】名称:摘要 + 关键字段摘选。
与存储侧 build_embed_text(embed-knowledge)格式同构同源——查询向量与库内卡向量落同一语义空间才可比;
差异仅在此处吃候选新实体 ent(键:型/名称/一句话摘要/字段),故下划线内部键(如里程碑 _win)在此
显式排除不进文本(build_embed_text 侧里程碑值 str() 后带 _win 噪音是已知递延项,两侧对称、本次不修)。"""
t = ent.get("型") or ent.get("type") or ""
fields = ent.get("字段") or {}
body = "\n".join(f"{k}:{v}" for k, v in fields.items()
if v and k not in ("名称", "一句话摘要") and not str(k).startswith("_"))
return f"【{t}】{ent.get('名称', '')}:{ent.get('一句话摘要', '')}\n{body}"[:4000]
def recall_neighbors(conn, vec, work_id, top=6):
"""用**现成向量**在 example_knowledge_embedding 召回同书 ≥阈值 近邻卡。
返回 [(did, 型, 名称, 摘要, 字段摘选, 相似度)…] 按相似度降序、过滤 <阈值。
字段摘选(2026-07-18 小样校准):终判证据不能只有一句话摘要——实测 M3 在"改名候选 vs 厚卡"上
因近邻证据太薄保守判无关(境界阶梯/力量来源这类字段才是同一实体的强证据),故召回时带出
字段 JSON 截断 300 字随近邻进终判 prompt(与候选侧字段 600 字对称)。
三段式连接(洞①核心红线):嵌入已在预判段批量算好并挪出——本函数只跑向量召回 SQL、不再发嵌入 HTTP,
调用方用短连接查完即关(不再沿用窗内长连接跨 LLM/嵌入调用存活,这正是本次要治的连接红线)。
嵌入表无同书向量(试跑期未 embed)时返回 []——优雅降级,判重回退纯机械,绝不阻断主流程。"""
qvec = json.dumps(vec)
rows = conn.execute(
"""SELECT d.id, d.draft_payload->>'type', d.draft_payload->>'名称',
d.draft_payload->>'一句话摘要', d.draft_payload->>'字段',
1 - (e.embedding <=> %s::vector) AS score
FROM example_knowledge_embedding e
JOIN muse_knowledge_draft d ON d.id = e.draft_id
WHERE e.tenant_id=%s AND e.deleted=FALSE AND d.deleted=FALSE
AND d.source_type=%s AND d.work_id=%s
ORDER BY score DESC LIMIT %s""",
(qvec, TENANT, SOURCE_TYPE, work_id, top)).fetchall()
return [(r[0], r[1], r[2], r[3], (r[4] or "")[:300], float(r[5])) for r in rows
if float(r[5]) >= DEDUP_SIM_THRESHOLD]
def dedup_judge_prompt(ent, neighbors):
"""M3 终判 prompt:候选新实体 vs 每个同书近邻,判 同一实体 / 前身 / 后继 / 无关。
近邻带字段摘选(小样校准):只给一句话摘要时模型证据不足会保守判无关,字段是强证据。"""
nb = "\n".join(f"{i + 1}. 卡号{did}|{t}|{nm}|{brief or ''}|字段摘选:{fs or '(无)'}"
for i, (did, t, nm, brief, fs, _s) in enumerate(neighbors))
return f"""【功能指令(parse-book 作品面升格·语义判重终判)】
下面是一个"候选新实体"和若干"同书既有卡"(向量召回的近邻)。逐一判断候选与每张近邻卡的关系,四选一:
- 同一实体:同一对象的改名/化名/不同侧面(如"影杀者"与"IV代纯机械机甲·影杀者")——**仅同型可判**。
- 前身:候选是该近邻卡的上一代/来源(同一条进化链的相邻代际,如"铁头(一代)"之于"铁卫(二代)")。
- 后继:候选是该近邻卡的下一代/继承者。
- 无关:只是题材相近,各自独立。
判据:看名称/摘要/字段是否指向同一对象或同一条演变链;**同型且名称互含(如"生物机甲"与"联邦生物机甲技术")通常是同一实体的简称/全称,除非字段证据明确指向不同对象**;名称不互含但摘要/字段描述同一套力量来源、同一条境界阶梯、同一批代表单位的,也应判同一实体(改名不改实质);**跨型(如具体机甲 item vs 整套体系 power_system)绝不判同一实体,最多判前身/后继**——一整套体系不等于其中一台机体。
【候选新实体】
型={ent.get("型", "")}|名称={ent.get("名称", "")}|摘要={ent.get("一句话摘要", "")}
字段:{json.dumps(ent.get("字段", {}), ensure_ascii=False)[:600]}
【同书近邻卡】
{nb}
【输出规则(只输出一个 JSON 对象)】
{{"判定": [{{"卡号": 数字, "关系": "同一实体|前身|后继|无关"}}]}}"""
def semantic_dedup(ent, neighbors, call):
"""语义判重终判(洞① 三段式:只吃**现成近邻** + M3,全程无连接)。返回 (verdict, data):
('merge', (did, 近邻名称)) 同型·同一实体 → 并卡(治改名漏并)
('chain', [(did, 关系, 名称)]) 前身后继 → 不并卡但记串链候选
('new', None) 无近邻或全判无关 → 各自立卡
近邻召回(recall_neighbors)已在预判段用短连接做完并挪出——本函数不碰连接、不发嵌入 HTTP,
只发 M3 终判且走 run 内 call() 包装(计入 calls 计数与 --max-calls 闸,与观察/更新同治理)。
同型才允许 merge;跨型即便 M3 判同一实体也降级为串链候选(设计稿 §8.2:跨型仅提示、不自动并)。
判重是增益非必需:无近邻 / 终判异常(敏感或 JSON 形状)一律保守返回 new(不并可后补,误并难回退)。"""
if not neighbors:
return "new", None
try:
data, _ = call(dedup_judge_prompt(ent, neighbors), ("判定",))
except (SensitiveHardStop, RuntimeError):
return "new", None
nb_type = {did: t for did, t, _, _, _, _ in neighbors}
nb_name = {did: nm for did, _, nm, _, _, _ in neighbors}
ent_type = ent.get("型", "")
chain = []
for j in [x for x in (data.get("判定") or []) if isinstance(x, dict)]:
did, rel = j.get("卡号"), j.get("关系")
if did not in nb_type:
continue
same_type = nb_type[did] == ent_type
if rel == "同一实体" and same_type:
return "merge", (did, nb_name[did]) # 近邻按相似度降序,第一个同型同一实体即采
if rel in ("前身", "后继"):
chain.append((did, rel, nb_name[did]))
elif rel == "同一实体": # 跨型判同一实体不可信 → 降级串链候选
chain.append((did, "前身后继待定", nb_name[did]))
return ("chain", chain) if chain else ("new", None)
def prejudge_semantic(obs, name_map, presence, embed_sess, work_id, call):
"""预判段(洞① 三段式连接):无长连接完成「粗筛将立卡候选 → 批量嵌入 → 短连接召回 → M3 终判」,
产出 verdicts 供写段查表替代原窗内 semantic_dedup(conn,…)。返回 (verdicts, merge_n, chain_n),
verdicts = {名称: (verdict, data)}。
连接纪律:嵌入 HTTP(b 段)与 M3 终判(d 段)全程无连接;仅召回 SQL(c 段)用短连接、查完即关。
近似超集:按 读1 时的 name_map 用 _classify_new_name 粗筛(与写段同判据),**不模拟**写段中途
new_card 对 name_map 的登记效应——name_map 只增不减,故本段候选恒 ⊇ 写段真正立卡集;多算的候选
写段自然走归并路径判定弃用(写段判据权威),此处宁多勿漏、不会漏判。"""
# a. 纯内存粗筛「将立卡」候选(与写段同判据,防两处漂移)
cands = [ent for ent in obs.get("新名字", [])
if _classify_new_name(ent, name_map, presence)[0] == "new"]
if not cands:
return {}, 0, 0
# b. 批量嵌入粗候选(HTTP,无连接)
vecs, bad = embed_texts(embed_sess, [_entity_embed_text(e) for e in cands])
# c. 短连接:逐候选用现成向量召回近邻(recall SQL 半),查完即关(三段式)
cand_nbrs = {}
with psycopg.connect(DSN) as conn:
for i, ent in enumerate(cands):
if i in bad or i >= len(vecs) or vecs[i] is None:
continue # 嵌入失败的候选优雅跳过(判重回退纯机械立卡)
nbrs = recall_neighbors(conn, vecs[i], work_id)
if nbrs:
cand_nbrs[i] = nbrs
# d. 逐「有近邻」候选发 M3 终判(无连接),走 call() 计数+闸;异常保守 new(semantic_dedup 内部兜)
verdicts, merge_n, chain_n = {}, 0, 0
for i, ent in enumerate(cands):
if i not in cand_nbrs:
continue # 无近邻→写段 verdicts.get 兜默认 new,不必发 M3
verdict, data = semantic_dedup(ent, cand_nbrs[i], call)
verdicts[(ent.get("名称") or "").strip()] = (verdict, data)
if verdict == "merge":
merge_n += 1
elif verdict == "chain":
chain_n += 1
return verdicts, merge_n, chain_n
def _assert_embedding_owner_available(conn, draft_id, content_hash, *, lock=False):
"""校验同 hash 唯一行可安全归当前 draft;写段可加行锁防预检后的并发竞态。"""
lock_clause = " FOR UPDATE OF e" if lock else ""
conflict = conn.execute(
"""SELECT e.draft_id, e.entity_id, e.deleted,
COALESCE(d.deleted, TRUE), d.draft_payload
FROM example_knowledge_embedding e
LEFT JOIN muse_knowledge_draft d ON d.id=e.draft_id
WHERE e.tenant_id=%s AND e.content_hash=%s AND e.model=%s""" + lock_clause,
(TENANT, content_hash, EMBED_MODEL),
).fetchone()
if not conflict:
return
owner_draft_id, owner_entity_id, _, owner_deleted, owner_payload = conflict
# 已确认实体拥有的向量不可重新降级归 draft;无论 draft 是否软删都必须失败关闭。
if owner_entity_id is not None:
raise EmbeddingOwnershipConflict(
f"同 hash 唯一行已归 entity:hash={content_hash},entity={owner_entity_id},"
f"candidate={draft_id}"
)
if owner_draft_id in (None, draft_id) or owner_deleted:
return
owner_text = build_embed_text(owner_payload or {})
owner_hash = hashlib.sha256(f"{owner_text}|{EMBED_MODEL}".encode()).hexdigest()
if owner_hash == content_hash:
raise EmbeddingOwnershipConflict(
f"活跃 draft 同 hash 争用:hash={content_hash},"
f"owner={owner_draft_id},candidate={draft_id}"
)
def _assert_draft_live_embeddings_mutable(conn, draft_id):
"""写前锁定同 draft 全部活向量;任一已归 entity 都禁止被 draft 更新链软删。"""
rows = conn.execute(
"""SELECT id, content_hash, entity_id FROM example_knowledge_embedding
WHERE tenant_id=%s AND draft_id=%s AND deleted=FALSE
FOR UPDATE""",
(TENANT, draft_id),
).fetchall()
entity_rows = [(row_id, content_hash, entity_id) for row_id, content_hash, entity_id in rows
if entity_id is not None]
if entity_rows:
raise EmbeddingOwnershipConflict(
f"draft={draft_id} 的旧活向量已归 entity,禁止软删:{entity_rows}"
)
def _assert_ready_draft_current(conn, draft_id, expected_hash):
"""锁定待写 draft,复验可信状态与当前 payload hash,阻断嵌入 HTTP 期间漂移。"""
row = conn.execute(
"""SELECT tenant_id, deleted, status, draft_payload
FROM muse_knowledge_draft WHERE id=%s FOR UPDATE""",
(draft_id,),
).fetchone()
if not row:
raise EmbeddingOwnershipConflict(f"待写 draft 不存在:draft={draft_id}")
tenant_id, deleted, status, payload = row
if tenant_id != TENANT or deleted or status != "pending":
raise EmbeddingOwnershipConflict(
f"待写 draft 状态非法:draft={draft_id},tenant={tenant_id},"
f"deleted={deleted},status={status}"
)
current_hash = _payload_embed_hash(payload)
if current_hash != expected_hash:
raise EmbeddingOwnershipConflict(
f"HTTP 期间 draft payload 已漂移:draft={draft_id},"
f"expected_hash={expected_hash},current_hash={current_hash}"
)
def _payload_embed_hash(payload):
"""按嵌入唯一键同源公式计算当前 payload hash。"""
text = build_embed_text(payload or {})
return hashlib.sha256(f"{text}|{EMBED_MODEL}".encode()).hexdigest()
def _quality_repair_validate_vector(vector):
"""校验单卡修复向量的维度、数值类型和有限性,拒绝静默截断或脏值落库。"""
if not isinstance(vector, list):
raise RuntimeError(
f"质量修复 embedding 必须是 list:actual={type(vector).__name__}"
)
if len(vector) != EMBED_DIM:
raise RuntimeError(
f"质量修复 embedding 维度错误:expected={EMBED_DIM},actual={len(vector)}"
)
for index, value in enumerate(vector):
if isinstance(value, bool) or not isinstance(value, Real):
raise RuntimeError(
f"质量修复 embedding 数值类型非法:index={index},"
f"actual={type(value).__name__}"
)
if not math.isfinite(float(value)):
raise RuntimeError(
f"质量修复 embedding 含非有限值:index={index},value={value!r}"
)
return vector
def _quality_repair_scope(work_id, draft_id, name):
"""校验单卡修复的固定目标,避免把人工修复扩成全局迁移。"""
if (work_id, draft_id, name) != (
QUALITY_REPAIR_WORK_ID, QUALITY_REPAIR_DRAFT_ID, QUALITY_REPAIR_NAME):
raise UpgradeDraftWriteConflict(
"质量修复目标不在已登记单卡范围:"
f"work={work_id},draft={draft_id},name={name!r}"
)
def _quality_repair_target_payload(payload, work_id, draft_id):
"""把已核实的 item 卡确定性映射为 location 卡,不调用模型或猜测字段。"""
if not isinstance(payload, dict):
raise UpgradeDraftWriteConflict("质量修复 draft_payload 必须是对象")
_quality_repair_scope(work_id, draft_id, payload.get("名称"))
if payload.get("type") != QUALITY_REPAIR_OLD_TYPE:
raise UpgradeDraftWriteConflict(
f"质量修复要求旧 type=item,实际为 {payload.get('type')!r}"
)
unexpected = set(payload) - QUALITY_REPAIR_TOP_LEVEL_KEYS
if unexpected:
raise UpgradeDraftWriteConflict(
f"质量修复遇到未登记顶层字段:{sorted(unexpected)}"
)
fields = payload.get("字段")
if not isinstance(fields, dict):
raise UpgradeDraftWriteConflict("质量修复旧卡字段必须是对象")
unknown_fields = set(fields) - QUALITY_REPAIR_ITEM_FIELDS
if unknown_fields:
raise UpgradeDraftWriteConflict(
f"质量修复遇到未登记 item 字段:{sorted(unknown_fields)}"
)
appearances = _int_chaps(payload.get("出场章"))
if appearances != set(QUALITY_REPAIR_OLD_APPEARANCES):
raise UpgradeDraftWriteConflict(
"质量修复旧卡出场章与已核实快照不一致:"
f"expected={list(QUALITY_REPAIR_OLD_APPEARANCES)},actual={sorted(appearances)}"
)
aliases = payload.get("别名", [])
if not isinstance(aliases, list) or any(not isinstance(alias, str) for alias in aliases):
raise UpgradeDraftWriteConflict("质量修复别名必须是字符串列表")
target = {
key: deepcopy(payload[key])
for key in QUALITY_REPAIR_TOP_LEVEL_KEYS
if key in payload and key not in {"type", "一句话摘要", "字段", "出场章"}
}
target.update({
"type": "location",
"名称": QUALITY_REPAIR_NAME,
"一句话摘要": (
"天启神国的主星·神权之星,是机械行星及其内部环形机械城,"
"兼具王冠宝石镜面护盾与主炮的兵器化防御;第587章遭撞击贯穿、表面崩碎。"
),
"字段": {
"层级": "主星/机械行星",
"地理与环境": "机械行星及其内部环形机械城;星上有十几亿居民。",
"归属势力": "天启神国",
"规则特例": (
"王冠宝石可形成镜面护盾并作为主炮,兼具兵器化防御;"
"王冠宝石已被击碎,神权之星后遭撞击贯穿、表面崩碎。"
),
"演变历程": [
{
"章": 403,
"台阶": "天启神国的主星·神权之星登场;机械行星内部有城市和居民。",
"周期": "登场",
},
{
"章": 566,
"台阶": "王冠宝石形成镜面护盾并作为主炮,神权之星兼具兵器化防御。",
"周期": "高光",
},
{
"章": 587,
"台阶": "神权之星遭撞击、被贯穿,表面崩碎,主星受到重创。",
"周期": "高光",
},
],
},
"出场章": sorted(appearances | {403}),
})
if set(target["字段"]) != QUALITY_REPAIR_LOCATION_FIELDS:
raise UpgradeDraftWriteConflict("质量修复 location 字段集合不完整")
return target
def _quality_repair_row_digest(row):
"""对关联行做稳定摘要;不把 embedding 向量原文写进确认输出。"""
return _sha256_json(row)
def _quality_repair_capture_snapshot(conn, work_id, draft_id, *, lock=False):
"""读取单卡及关联行快照;execute 在最终事务内用同一函数锁内复验。"""
_quality_repair_scope(work_id, draft_id, QUALITY_REPAIR_NAME)
lock_clause = " FOR UPDATE" if lock else ""
draft = conn.execute(
"""SELECT work_id, draft_payload, status, revision, source_type, updater, deleted
FROM muse_knowledge_draft
WHERE id=%s AND tenant_id=%s""" + lock_clause,
(draft_id, TENANT),
).fetchone()
if not draft:
raise UpgradeDraftWriteConflict(f"质量修复 draft 不存在:draft={draft_id}")
work, payload, status, revision, source_type, updater, deleted = draft
if work != work_id or status != "pending" or source_type != SOURCE_TYPE \
or updater != QUALITY_REPAIR_OLD_DRAFT_UPDATER or deleted:
raise UpgradeDraftWriteConflict(
"质量修复 draft 状态/来源非法:"
f"work={work},status={status},source_type={source_type},"
f"updater={updater},deleted={deleted}"
)
target_payload = _quality_repair_target_payload(payload, work_id, draft_id)
embedding_rows = conn.execute(
"""SELECT id, draft_id, content_hash, embed_text, model, dimensions, embedding,
entity_id, deleted
FROM example_knowledge_embedding
WHERE tenant_id=%s AND draft_id=%s
ORDER BY id""" + lock_clause,
(TENANT, draft_id),
).fetchall()
aliases = conn.execute(
"""SELECT id, canonical_name, alias, evidence_window, verdict_by, updater, deleted
FROM example_upgrade_alias
WHERE tenant_id=%s AND work_id=%s
AND (id=%s OR canonical_name=%s OR alias=%s)
ORDER BY id""" + lock_clause,
(
TENANT, work_id, QUALITY_REPAIR_REVOKED_ALIAS_ID,
QUALITY_REPAIR_ALIAS_VALUE, QUALITY_REPAIR_ALIAS_VALUE,
),
).fetchall()
presence = conn.execute(
"""SELECT id, window_no, chapter_no, entity_type, name, observation, deleted
FROM example_upgrade_presence
WHERE tenant_id=%s AND work_id=%s AND name=%s
ORDER BY id""" + lock_clause,
(TENANT, work_id, QUALITY_REPAIR_NAME),
).fetchall()
card_state = conn.execute(
"""SELECT draft_id, work_id, watermark_window, tenant_id
FROM example_upgrade_card_state
WHERE draft_id=%s""" + lock_clause,
(draft_id,),
).fetchall()
snapshot = {
"work_id": work_id,
"draft_id": draft_id,
"draft": {
"work_id": work,
"payload": deepcopy(payload),
"revision": revision,
"status": status,
"source_type": source_type,
"updater": updater,
"deleted": deleted,
},
"embeddings": [tuple(row) for row in embedding_rows],
"aliases": [tuple(row) for row in aliases],
"presence": [tuple(row) for row in presence],
"card_state": [tuple(row) for row in card_state],
"target_payload": target_payload,
}
active_embeddings = [row for row in snapshot["embeddings"] if not row[8]]
if len(active_embeddings) != 1:
raise EmbeddingOwnershipConflict(
f"质量修复要求恰有一条活跃 embedding:实际={len(active_embeddings)}"
)
active = active_embeddings[0]
if active[0] != 41253 or active[1] != draft_id or active[4] != EMBED_MODEL \
or active[5] != EMBED_DIM or active[7] is not None:
raise EmbeddingOwnershipConflict(
"质量修复 embedding 关联行不符合已核实 owner/model/entity 条件"
)
expected_alias = (
QUALITY_REPAIR_REVOKED_ALIAS_ID,
QUALITY_REPAIR_ALIAS_CANONICAL,
QUALITY_REPAIR_ALIAS_VALUE,
QUALITY_REPAIR_ALIAS_EVIDENCE_WINDOW,
QUALITY_REPAIR_ALIAS_VERDICT_BY,
QUALITY_REPAIR_REVOKED_ALIAS_UPDATER,
True,
)
if snapshot["aliases"] != [expected_alias]:
raise UpgradeDraftWriteConflict(
"质量修复要求 alias 关联集合精确唯一且 8738 已撤销:"
f"actual={snapshot['aliases']}"
)
expected_presence = (
QUALITY_REPAIR_PRESENCE_ID,
QUALITY_REPAIR_PRESENCE_WINDOW,
QUALITY_REPAIR_PRESENCE_CHAPTER,
"location",
QUALITY_REPAIR_NAME,
QUALITY_REPAIR_PRESENCE_OBSERVATION,
False,
)
if snapshot["presence"] != [expected_presence]:
raise UpgradeDraftWriteConflict(
"质量修复要求 presence 关联集合精确唯一且保持原 observation:"
f"actual={snapshot['presence']}"
)
if snapshot["card_state"] != [(draft_id, work_id, QUALITY_REPAIR_CARD_WATERMARK, TENANT)]:
raise UpgradeDraftWriteConflict("质量修复要求 card_state watermark=109 且关联行唯一")
return snapshot
def _quality_repair_confirmation_sha(snapshot):
"""按旧快照和确定性目标 payload 生成 execute 必须匹配的确认摘要。"""
draft = snapshot["draft"]
return _sha256_json({
"contract": QUALITY_REPAIR_CONTRACT,
"work_id": snapshot["work_id"],
"draft_id": snapshot["draft_id"],
"draft_revision": draft["revision"],
"draft_updater": draft["updater"],
"old_payload_sha256": _sha256_json(draft["payload"]),
"target_payload_sha256": _sha256_json(snapshot["target_payload"]),
"embeddings": [_quality_repair_row_digest(row) for row in snapshot["embeddings"]],
"aliases": [_quality_repair_row_digest(row) for row in snapshot["aliases"]],
"presence": [_quality_repair_row_digest(row) for row in snapshot["presence"]],
"card_state": [_quality_repair_row_digest(row) for row in snapshot["card_state"]],
})
def _quality_repair_output(snapshot, mode, *, new_revision=None, new_hash=None):
"""输出最小审计摘要;不复制旧 embedding 向量或窗口模型审计行。"""
active = next(row for row in snapshot["embeddings"] if not row[8])
target = snapshot["target_payload"]
return {
"command": "repair-card-quality",
"contract": QUALITY_REPAIR_CONTRACT,
"mode": mode,
"work_id": snapshot["work_id"],
"draft_id": snapshot["draft_id"],
"name": QUALITY_REPAIR_NAME,
"old_revision": snapshot["draft"]["revision"],
"old_draft_updater": snapshot["draft"]["updater"],
"new_revision": new_revision,
"old_payload_sha256": _sha256_json(snapshot["draft"]["payload"]),
"new_payload_sha256": _sha256_json(target),
"old_embedding_id": active[0],
"old_embedding_hash": active[2],
"new_embedding_hash": new_hash or _payload_embed_hash(target),
"presence_id": QUALITY_REPAIR_PRESENCE_ID,
"revoked_alias_id": QUALITY_REPAIR_REVOKED_ALIAS_ID,
"card_state_watermark": QUALITY_REPAIR_CARD_WATERMARK,
"audit_rows_written": 0,
"confirmation_sha": _quality_repair_confirmation_sha(snapshot),
"target_payload": deepcopy(target),
}
def _quality_repair_apply_locked(conn, work_id, draft_id, snapshot, vector, text, content_hash):
"""在已锁定快照内原子更新 payload/revision 与 embedding;关联行只读。"""
_quality_repair_validate_vector(vector)
active = next(row for row in snapshot["embeddings"] if not row[8])
if content_hash == active[2]:
raise EmbeddingOwnershipConflict("质量修复后的 embedding hash 未发生变化")
_assert_draft_live_embeddings_mutable(conn, draft_id)
_assert_embedding_owner_available(conn, draft_id, content_hash, lock=True)
new_payload = snapshot["target_payload"]
updated = conn.execute(
"""UPDATE muse_knowledge_draft SET draft_payload=%s, revision=revision+1,
updater=%s
WHERE id=%s AND tenant_id=%s AND work_id=%s AND deleted=FALSE
AND status='pending' AND source_type=%s AND revision=%s
RETURNING revision""",
(
json.dumps(new_payload, ensure_ascii=False), QUALITY_REPAIR_UPDATER,
draft_id, TENANT, work_id, SOURCE_TYPE, snapshot["draft"]["revision"],
),
).fetchone()
expected_revision = snapshot["draft"]["revision"] + 1
if not updated or updated[0] != expected_revision:
raise UpgradeDraftWriteConflict("质量修复 draft revision CAS 失败")
conn.execute(
"""UPDATE example_knowledge_embedding SET deleted=TRUE, updater=%s
WHERE tenant_id=%s AND draft_id=%s AND deleted=FALSE
AND entity_id IS NULL AND content_hash!=%s""",
(QUALITY_REPAIR_UPDATER, TENANT, draft_id, content_hash),
)
upserted = conn.execute(
"""INSERT INTO example_knowledge_embedding
(draft_id, content_hash, embed_text, model, dimensions, embedding,
creator, updater, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (tenant_id, content_hash, model)
DO UPDATE SET draft_id=EXCLUDED.draft_id, embed_text=EXCLUDED.embed_text,
dimensions=EXCLUDED.dimensions, embedding=EXCLUDED.embedding,
deleted=FALSE, updater=EXCLUDED.updater
WHERE example_knowledge_embedding.entity_id IS NULL
AND (example_knowledge_embedding.draft_id=EXCLUDED.draft_id
OR example_knowledge_embedding.deleted=TRUE
OR NOT EXISTS (SELECT 1 FROM muse_knowledge_draft owner
WHERE owner.id=example_knowledge_embedding.draft_id
AND owner.deleted=FALSE))
RETURNING draft_id""",
(
draft_id, content_hash, text, EMBED_MODEL, EMBED_DIM, json.dumps(vector),
QUALITY_REPAIR_UPDATER, QUALITY_REPAIR_UPDATER, TENANT,
),
).fetchone()
if not upserted or upserted[0] != draft_id:
raise EmbeddingOwnershipConflict("质量修复 embedding owner CAS 失败")
return expected_revision
def prepare_touched_cards(sess, touched):
"""短读后关闭连接,再发嵌入 HTTP;返回全部 touched 快照供 final 独立复验。"""
snapshots = {}
with psycopg.connect(DSN) as conn:
for did in sorted(touched):
row = conn.execute(
"""SELECT draft_payload, revision FROM muse_knowledge_draft
WHERE id=%s AND tenant_id=%s AND status='pending'
AND source_type=%s AND deleted=FALSE""",
(did, TENANT, SOURCE_TYPE),
).fetchone()
if not row:
raise EmbeddingOwnershipConflict(f"待嵌入 draft 不可写:draft={did}")
live = {h for (h,) in conn.execute(
"""SELECT content_hash FROM example_knowledge_embedding
WHERE tenant_id=%s AND draft_id=%s AND deleted=FALSE""",
(TENANT, did),
).fetchall()}
snapshots[did] = (row[0], row[1], live)
prepared = []
todo = []
for did, (payload, revision, live_hashes) in snapshots.items():
text, content_hash = build_embed_text(payload or {}), _payload_embed_hash(payload)
item = (did, text, content_hash, revision)
prepared.append((*item, None))
if content_hash not in live_hashes:
todo.append(item)
owners = {}
for did, _, content_hash, _, _ in prepared:
owners.setdefault(content_hash, []).append(did)
duplicates = {content_hash: ids for content_hash, ids in owners.items() if len(ids) > 1}
if duplicates:
raise EmbeddingOwnershipConflict(f"活跃 touched 卡出现同 hash 争用:{duplicates}")
if todo:
vecs, bad = embed_texts(sess, [text for _, text, _, _ in todo])
if bad or len(vecs) != len(todo) or any(vector is None for vector in vecs):
raise RuntimeError(
f"增量嵌入未完整返回:bad={sorted(bad)},"
f"expected={len(todo)},actual={len(vecs)}"
)
vectors = {item[0]: vecs[index] for index, item in enumerate(todo)}
prepared = [(*item[:4], vectors.get(item[0])) for item in prepared]
return prepared
def apply_prepared_cards(conn, work_id, prepared):
"""最终短事务内复验 payload/revision/owner 后写向量;调用者负责同事务标记 done。"""
for did, _, content_hash, revision, _ in sorted(prepared, key=lambda item: item[0]):
row = conn.execute(
"""SELECT work_id, draft_payload, status, revision, source_type, deleted
FROM muse_knowledge_draft WHERE id=%s AND tenant_id=%s FOR UPDATE""",
(did, TENANT),
).fetchone()
if not row or row[0] != work_id or row[2] != "pending" or row[3] != revision \
or row[4] != SOURCE_TYPE or row[5] or _payload_embed_hash(row[1]) != content_hash:
raise EmbeddingOwnershipConflict(f"最终嵌入 draft payload/revision 已漂移:draft={did}")
for did in sorted({item[0] for item in prepared}):
_assert_draft_live_embeddings_mutable(conn, did)
for did, _, content_hash, _, _ in sorted(prepared, key=lambda item: (item[2], item[0])):
_assert_embedding_owner_available(conn, did, content_hash, lock=True)
for did, text, content_hash, _, vector in prepared:
if vector is None:
continue
conn.execute(
"""UPDATE example_knowledge_embedding SET deleted=TRUE, updater=%s
WHERE tenant_id=%s AND draft_id=%s AND deleted=FALSE
AND entity_id IS NULL AND content_hash!=%s""",
(EMBED_ACTOR, TENANT, did, content_hash),
)
row = conn.execute(
"""INSERT INTO example_knowledge_embedding
(draft_id, content_hash, embed_text, model, dimensions, embedding,
creator, updater, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (tenant_id, content_hash, model)
DO UPDATE SET draft_id=EXCLUDED.draft_id, embed_text=EXCLUDED.embed_text,
dimensions=EXCLUDED.dimensions, embedding=EXCLUDED.embedding,
deleted=FALSE, updater=EXCLUDED.updater
WHERE example_knowledge_embedding.entity_id IS NULL
AND (example_knowledge_embedding.draft_id=EXCLUDED.draft_id
OR example_knowledge_embedding.deleted=TRUE
OR NOT EXISTS (SELECT 1 FROM muse_knowledge_draft owner
WHERE owner.id=example_knowledge_embedding.draft_id
AND owner.deleted=FALSE))
RETURNING draft_id""",
(did, content_hash, text, EMBED_MODEL, EMBED_DIM, json.dumps(vector),
EMBED_ACTOR, EMBED_ACTOR, TENANT),
).fetchone()
if not row or row[0] != did:
raise EmbeddingOwnershipConflict(
f"同 hash 唯一行未绑定当前 draft:hash={content_hash},candidate={did}"
)
return sum(item[4] is not None for item in prepared)
def embed_touched_cards(sess, work_id, touched):
"""兼容独立调用:严格 prepare/apply,普通异常也向上抛出,不再告警放行。"""
prepared = prepare_touched_cards(sess, touched)
with psycopg.connect(DSN) as conn:
conn.execute(
"LOCK TABLE muse_knowledge_draft, example_knowledge_embedding IN ROW EXCLUSIVE MODE"
)
done = apply_prepared_cards(conn, work_id, prepared)
conn.commit()
return done
@click.group()
def cli():
"""从已拆作品按正文窗抽取作品知识。"""
@click.group()
def maintenance_cli():
"""维护既有作品抽取状态;公开入口由维护 Skill 提供。"""
@cli.command()
@click.option("--work-id", type=int, required=True)
def windows(work_id):
"""机械切正文窗(幂等)。"""
try:
with upgrade_work_lock(DSN, TENANT, work_id):
with psycopg.connect(DSN) as conn:
total, new = cut_windows(conn, work_id)
click.echo(f"work={work_id} 切窗完成:全书 {total} 窗(本次新建 {new} 行)")
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
def _dedupe_presence_actions(actions):
"""按章/型/名收口 presence action;同观察幂等,不同观察失败关闭。"""
seen = {}
deduped = []
for action in actions:
if not isinstance(action, (tuple, list)) or not action or action[0] != "presence":
deduped.append(action)
continue
if len(action) != 6:
raise RuntimeError(f"presence action 结构非法:{action!r}")
_, _, chapter, entity_type, name, observation = action
key = (chapter, entity_type, name)
previous = seen.get(key)
if previous is None:
if not isinstance(observation, str):
raise RuntimeError(f"presence observation 必须是字符串:key={key!r}")
seen[key] = observation
deduped.append(action)
continue
if previous != observation:
raise RuntimeError(
"presence 同一逻辑 key 的 observation 冲突:"
f"key={key!r},first={previous!r},second={observation!r}"
)
# 完全相同的候选是模型重复回显,保留第一条即可。
return deduped
def _plan_window_entities(work_id, win_no, obs, name_map, presence, aliases_by_draft,
verdicts, semantic_on, milestone_types, chapter_texts):
"""纯 planner:只生成新卡/alias/presence/update 计划,不接收连接也不落库。"""
names, aliases = dict(name_map), {key: set(value) for key, value in aliases_by_draft.items()}
actions, payloads, to_update, appearances = [], {}, {}, {}
next_ref = -1
for ent in obs.get("新名字", []):
name = (ent.get("名称") or "").strip()
kind, key = _classify_new_name(ent, names, presence)
if kind == "empty":
continue
if kind in ("merge", "alias", "substr"):
ref = names[key][0]
if kind != "merge":
canonical = _resolve_canonical_name(names, aliases, key)
actions.append(("alias", ref, canonical, name,
"ai" if kind == "alias" else "substr"))
material = f"({'别名' if kind == 'alias' else '名称疑似同一实体'}「{name}」并入)"
else:
material = "(新名字归并)"
to_update.setdefault(ref, []).append(material + _merge_material(ent))
continue
chapters = _int_chaps(ent.get("出场章", []))
history = presence.get((ent.get("型", ""), name), set())
verdict, data = verdicts.get(name, ("new", None)) if semantic_on else ("new", None)
if kind == "new" and verdict == "merge":
ref, canonical = data
actions.append(("alias", ref, canonical, name, "semantic"))
to_update.setdefault(ref, []).append(
f"(语义判重·「{name}」并入同一实体)初卡材料:{_merge_material(ent)}"
)
continue
if kind == "new":
ref, next_ref = next_ref, next_ref - 1
payload = _build_new_card_payload(
work_id, win_no, ent, milestone_types,
chapter_texts=chapter_texts, known_chapters=history,
)
payloads[ref] = payload
actions.append(("new", ref, ent, history, data if verdict == "chain" else []))
target = (ref, ent.get("型", ""), ent.get("一句话摘要", ""))
names[payload["名称"]] = target
aliases.setdefault(ref, set()).update(payload["别名"])
for alias in payload["别名"]:
names[alias] = target
continue
for chapter in chapters:
actions.append(("presence", None, chapter, ent.get("型", ""), name,
ent.get("一句话摘要", "")))
for item in obs.get("已知实体新信息", []):
name = (item.get("名称") or "").strip()
if name in names and item.get("观察点"):
ref = names[name][0]
to_update.setdefault(ref, []).append(item["观察点"])
appearances.setdefault(ref, set()).update(_int_chaps(item.get("出场章")))
for item in obs.get("纯出场", []):
name = (item.get("名称") or "").strip()
if name in names:
appearances.setdefault(names[name][0], set()).update(_int_chaps(item.get("出场章")))
actions = _dedupe_presence_actions(actions)
return {"actions": actions, "payloads": payloads, "to_update": to_update,
"appearances": appearances, "aliases": aliases, "names": names}
def _compute_entity_updates(plan, expected_revisions, contracts, title, a, b, text,
chapter_texts, call):
"""分批短读 payload/revision,关闭连接后完成全部 update LLM、证据修复与归一化。"""
outputs, snapshots = [], {}
items = sorted(plan["to_update"].items())
for offset in range(0, len(items), UPDATE_BATCH):
batch = items[offset:offset + UPDATE_BATCH]
cards = []
with psycopg.connect(DSN) as conn:
for ref, points in batch:
if ref < 0:
payload, revision = deepcopy(plan["payloads"][ref]), 0
else:
payload, revision = _read_upgrade_draft_snapshot(conn, ref)
if revision != expected_revisions.get(ref):
raise UpgradeDraftWriteConflict(
f"实体 planner revision 漂移:draft={ref},expected={expected_revisions.get(ref)},"
f"current={revision}"
)
snapshots[ref] = (payload, revision)
cards.append((ref, payload, points))
try:
update, _ = call(update_prompt(contracts, title, a, b, text, cards), ("更新",))
except RuntimeError as exc:
if "缺少必需键" not in str(exc):
raise
print(f"[宽容] 窗{a}-{b} 更新批缺键按空批放行: {str(exc)[:80]}", file=sys.stderr)
update = {"更新": []}
repair_missing_milestone_evidence(
update, title=title, a=a, b=b, text=text, chapter_texts=chapter_texts, call=call,
)
valid = {ref for ref, _, _ in cards}
keys = {ref: {field["key"] for field in contracts.get(payload.get("type"), {}).get("字段", [])}
| {"一句话摘要"} for ref, payload, _ in cards}
for item in _normalize_entity_updates(update.get("更新") or [], valid):
item["valid_keys"] = keys[item["draft_id"]]
outputs.append(item)
return outputs, snapshots
def _assert_window_marker(conn, work_id, win_no, expected_stage, input_sha, state_sha):
"""锁内复验 marker、inputSha 与七域 stateSha;任何漂移都禁止继续写。"""
row = conn.execute(
"""SELECT id, status, error_message FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND window_no=%s FOR UPDATE""",
(TENANT, work_id, win_no),
).fetchone()
if not row or row[1] != "processing":
raise CompensationFenceConflict(f"窗口不在 processing:work={work_id},window={win_no}")
stage, marker_input, marker_state = _parse_upgrade_marker(row[2])
current_input = _window_input_sha(*_capture_window_input(conn, work_id, win_no))
current_state = _window_state_sha(
_capture_window_state(conn, work_id, row[0]), row[0]
)
if (stage, marker_input, marker_state) != (expected_stage, input_sha, state_sha) \
or current_input != input_sha or current_state != state_sha:
raise CompensationFenceConflict(
f"窗口 fence 漂移:stage={stage},input={current_input == input_sha},"
f"state={current_state == state_sha}"
)
return row[0], row[2]
def _set_processing_marker(conn, work_id, win_no, stage, input_sha, window_id):
"""写段末尾按当前七域状态生成 marker;status/error 更新不进入 stateSha。"""
conn.execute(
"""UPDATE example_upgrade_window SET status='processing', error_message=NULL, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s""",
(TENANT, work_id, win_no),
)
state_sha = _window_state_sha(
_capture_window_state(conn, work_id, window_id), window_id
)
marker = _upgrade_marker(stage, input_sha, state_sha)
conn.execute(
"""UPDATE example_upgrade_window SET error_message=%s, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s AND status='processing'""",
(marker, TENANT, work_id, win_no),
)
return state_sha
def _apply_entity_stage(conn, work_id, win_no, plan, updates, snapshots, expected_revisions,
milestone_types, chapter_texts, input_sha, window_id):
"""实体短写事务:固定锁后复验输入/revision,原子落 planner 与 entity marker。"""
# planner 已经收口一次;写段再做同一守卫,避免未来新增调用方绕过 planner
# 时把同章同型同名的冲突 observation 写成两行。
actions = _dedupe_presence_actions(plan["actions"])
_lock_upgrade_domains(conn)
start = conn.execute(
"""SELECT id, status, error_message FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND window_no=%s FOR UPDATE""",
(TENANT, work_id, win_no),
).fetchone()
if not start or start[0] != window_id or start[1] not in ("pending", "failed") \
or (start[1] == "failed"
and not str(start[2] or "").startswith("retryable-clean:")):
raise CompensationFenceConflict(f"实体提交前窗口状态非法:work={work_id},window={win_no}")
if _window_input_sha(*_capture_window_input(conn, work_id, win_no)) != input_sha:
raise CompensationFenceConflict("实体提交前 inputSha 漂移")
existing_refs = {ref for ref in plan["to_update"] if ref > 0}
existing_refs.update(action[1] for action in actions if action[0] == "alias" and action[1] > 0)
existing_refs.update(ref for ref in plan["appearances"] if ref > 0)
for ref in sorted(existing_refs):
_lock_upgrade_draft(conn, ref, expected_revisions[ref])
ref_map, new_ids, initial_revision_by_ref = {}, set(), {}
for action in actions:
if action[0] == "new":
_, ref, ent, history, chains = action
did, _, _, initial_revision = new_card(
conn, work_id, win_no, ent, milestone_types,
chapter_texts=chapter_texts, known_chapters=history,
)
ref_map[ref], new_ids = did, new_ids | {did}
initial_revision_by_ref[ref] = initial_revision
for other, relation, name in chains:
conn.execute(
"""INSERT INTO example_upgrade_audit
(draft_id, window_no, field_name, old_value, new_value, tenant_id)
VALUES (%s,%s,%s,NULL,%s,%s)""",
(did, win_no, ("串链候选:" + str(relation))[:100],
json.dumps({"对方卡号": other, "对方名称": name}, ensure_ascii=False), TENANT),
)
elif action[0] == "alias":
_, ref, canonical, alias, verdict = action
_claim_alias(conn, work_id, canonical, alias, win_no, verdict)
elif action[0] == "presence":
_, _, chapter, entity_type, name, observation = action
conn.execute(
"""INSERT INTO example_upgrade_presence
(work_id, window_no, chapter_no, entity_type, name, observation, tenant_id)
VALUES (%s,%s,%s,%s,%s,%s,%s)""",
(work_id, win_no, chapter, entity_type, name, observation, TENANT),
)
appearances = {ref_map.get(ref, ref): set(chapters)
for ref, chapters in plan["appearances"].items()}
aliases = {ref_map.get(ref, ref): values for ref, values in plan["aliases"].items()}
touched = set(new_ids)
for update in updates:
ref, actual = update["draft_id"], ref_map.get(update["draft_id"], update["draft_id"])
revision = initial_revision_by_ref[ref] if ref < 0 else snapshots[ref][1]
merge_card(
conn, actual, win_no, update["变更字段"], update["别名新增"],
valid_keys=update["valid_keys"], chapter_texts=chapter_texts,
appearance_chapters=appearances.pop(actual, set()),
known_aliases=aliases.get(actual, set()), expected_revision=revision,
)
touched.add(actual)
state_sha = _set_processing_marker(conn, work_id, win_no, "entity", input_sha, window_id)
conn.commit()
return ref_map, appearances, aliases, touched, state_sha
def _read_relation_snapshot(work_id, refs, onstage):
"""实体提交后短读人物、剩余出场卡与完整关系集合,连接关闭后才允许关系模型调用。"""
wanted = set(refs) | {did for _, (did, kind, _) in onstage.items() if kind == "character"}
snapshots, characters = {}, []
with psycopg.connect(DSN) as conn:
for did in sorted(wanted):
payload, revision = _read_upgrade_draft_snapshot(conn, did)
snapshots[did] = (payload, revision)
if payload.get("type") == "character" and len(characters) < 8:
characters.append((did, payload))
rows = conn.execute(
"""SELECT id, draft_payload, revision FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s AND deleted=FALSE
AND status='pending' AND draft_payload->>'type'=%s ORDER BY id""",
(TENANT, work_id, SOURCE_TYPE, RELATION_TYPE),
).fetchall()
return characters, rows, snapshots
def _compute_relations(contracts, title, a, b, text, characters, relation_rows, call):
"""无连接完成关系 LLM 与至多一次专用 repair。"""
if len(characters) < 2:
return []
relations = [(did, payload) for did, payload, _ in relation_rows]
output, _ = call(relation_prompt(contracts, title, a, b, text, characters, relations), ("关系",))
raw = output.get("关系") or []
valid_ids = {did for did, _ in characters}
try:
return _normalize_relation_output(raw, valid_ids)
except RelationOutputConflict:
repaired, _ = call(
relation_repair_prompt(contracts, title, a, b, text, characters, relations, raw),
("关系",),
)
repaired_raw = repaired.get("关系") or []
merged = _merge_repaired_relation_output(repaired_raw, valid_ids)
for pair, count in sorted(_relation_pair_counts(repaired_raw, valid_ids).items()):
if count > 1:
click.echo(
f"关系 repair 二次机械合并:pair={pair},合并数={count - 1}",
err=True,
)
return merged
def _apply_relation_stage(conn, work_id, win_no, relation_items, characters, relation_rows,
snapshots, appearances, aliases, chapter_texts, input_sha,
entity_state_sha, window_id):
"""关系短写事务:复验 entity marker、人物 revision 与完整关系集合后写关系和剩余出场。"""
_lock_upgrade_domains(conn)
_assert_window_marker(conn, work_id, win_no, "entity", input_sha, entity_state_sha)
for did, _ in characters:
_lock_upgrade_draft(conn, did, snapshots[did][1])
current_rows = conn.execute(
"""SELECT id, draft_payload, revision FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s AND deleted=FALSE
AND status='pending' AND draft_payload->>'type'=%s ORDER BY id""",
(TENANT, work_id, SOURCE_TYPE, RELATION_TYPE),
).fetchall()
if current_rows != relation_rows:
raise CompensationFenceConflict("关系提交前完整关系集合漂移")
existing = {tuple(sorted((payload.get("甲方draft"), payload.get("乙方draft")))): (did, revision)
for did, payload, revision in relation_rows
if payload.get("甲方draft") and payload.get("乙方draft")}
names = {did: payload.get("名称") for did, payload in characters}
touched = set()
for relation in relation_items:
first, second = relation.get("甲方"), relation.get("乙方")
key = tuple(sorted((first, second)))
if key in existing:
did, revision = existing[key]
touched.add(_update_relation_card(conn, work_id, win_no, did, revision, relation))
else:
evolution = _strip_prefix(relation.get("本窗演变", ""))
fields = dict(relation.get("其他字段") or {})
fields["演变轨迹"] = [f"[窗{win_no}] {evolution}"] if evolution else []
touched.add(_insert_relation_card(conn, work_id, win_no, {
"type": RELATION_TYPE, "名称": f"{names[first]}×{names[second]}",
"甲方draft": first, "乙方draft": second, "甲方名称": names[first],
"乙方名称": names[second], "关系类型": relation.get("关系类型", ""),
"字段": fields, "来源": f"升格@窗{win_no}", "状态": "草稿",
"目标库": "本书作品库", "可见范围": "本书私有", "_work_id": work_id,
}))
for did, chapters in appearances.items():
payload, revision = _lock_upgrade_draft(conn, did, snapshots[did][1])
if _append_verified_appearance_chapters(
conn, did, win_no, payload, chapters, chapter_texts,
known_aliases=aliases.get(did, set())):
_write_locked_upgrade_draft(conn, did, payload, revision)
touched.add(did)
state_sha = _set_processing_marker(conn, work_id, win_no, "relation", input_sha, window_id)
conn.commit()
return touched, state_sha
def _finalize_window(work_id, win_no, input_sha, relation_state_sha, prepared):
"""最终短事务原子完成向量写入与 done;semantic 关闭时 prepared 为空直接完成。"""
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
_, marker = _assert_window_marker(
conn, work_id, win_no, "relation", input_sha, relation_state_sha,
)
embedded = apply_prepared_cards(conn, work_id, prepared)
row = conn.execute(
"""UPDATE example_upgrade_window SET status='done', error_message=NULL, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s
AND status='processing' AND error_message=%s RETURNING status""",
(TENANT, work_id, win_no, marker),
).fetchone()
if not row:
raise CompensationFenceConflict("最终 done CAS 冲突")
conn.commit()
return embedded
def _window_artifact_counts(
conn,
work_id,
win_no,
allowed_tombstone_updaters=("upgrade-reset",),
):
"""统计阻断产物;墓碑豁免 updater 必须由调用上下文明确传入。"""
if isinstance(allowed_tombstone_updaters, (str, bytes)):
raise ValueError("allowed_tombstone_updaters 必须是 updater 序列")
allowed_tombstone_updaters = tuple(dict.fromkeys(allowed_tombstone_updaters))
unexpected = set(allowed_tombstone_updaters) - TOMBSTONE_UPDATER_ALLOWLIST
if unexpected:
raise ValueError(f"不允许的墓碑 updater:{sorted(unexpected)}")
allowed_updater_array = list(allowed_tombstone_updaters)
values = {}
statements = {
"audits": ("SELECT count(*) FROM example_upgrade_audit a JOIN muse_knowledge_draft d "
"ON d.id=a.draft_id WHERE a.tenant_id=%s AND d.work_id=%s AND a.window_no=%s",
(TENANT, work_id, win_no)),
"aliases": ("SELECT count(*) FROM example_upgrade_alias a WHERE a.tenant_id=%s "
"AND a.work_id=%s AND a.evidence_window=%s "
"AND NOT (a.deleted=TRUE AND a.updater = ANY(%s))",
(TENANT, work_id, win_no, allowed_updater_array)),
"presence": ("SELECT count(*) FROM example_upgrade_presence p WHERE p.tenant_id=%s "
"AND p.work_id=%s AND p.window_no=%s",
(TENANT, work_id, win_no)),
"new_cards": ("SELECT count(*) FROM muse_knowledge_draft d WHERE d.tenant_id=%s "
"AND d.work_id=%s AND d.source_type=%s AND d.draft_payload->>'来源'=%s "
"AND NOT (d.deleted=TRUE AND d.updater = ANY(%s) AND d.status='pending')",
(TENANT, work_id, SOURCE_TYPE, f"升格@窗{win_no}", allowed_updater_array)),
"card_state": ("SELECT count(*) FROM example_upgrade_card_state s "
"JOIN muse_knowledge_draft d ON d.id=s.draft_id "
"WHERE s.tenant_id=%s AND d.tenant_id=%s AND d.work_id=%s "
"AND d.source_type=%s AND s.watermark_window=%s",
(TENANT, TENANT, work_id, SOURCE_TYPE, win_no)),
"embeddings": ("SELECT count(*) FROM example_knowledge_embedding e JOIN muse_knowledge_draft d "
"ON d.id=e.draft_id WHERE e.tenant_id=%s AND d.tenant_id=%s "
"AND d.work_id=%s AND d.source_type=%s "
"AND d.draft_payload->>'来源'=%s "
"AND NOT (e.entity_id IS NULL AND e.deleted=TRUE "
"AND d.deleted=TRUE AND d.updater = ANY(%s) AND d.status='pending')",
(TENANT, TENANT, work_id, SOURCE_TYPE, f"升格@窗{win_no}",
allowed_updater_array)),
}
for name, (sql, params) in statements.items():
values[name] = conn.execute(sql, params).fetchone()[0]
return values
RETRYABLE_CLEAN_PREFIX = "retryable-clean:"
LEGACY_RECOVERY_CONTRACT = "recover-legacy-failed:v1"
def _legacy_recovery_confirmation_sha(snapshot):
"""按 preview 输出的完整快照生成 execute 必须精确匹配的确认摘要。"""
return _sha256_json({
"contract": LEGACY_RECOVERY_CONTRACT,
"work_id": snapshot["work_id"],
"window": snapshot["window"],
"artifact_counts": snapshot["artifact_counts"],
"processing_count": snapshot["processing_count"],
})
def _capture_legacy_failed_snapshot(conn, work_id, win_no, *, lock=False):
"""在当前连接读取 legacy failed 恢复快照;execute 调用方先固定表锁。"""
lock_clause = " FOR UPDATE" if lock else ""
row = conn.execute(
"""SELECT id, window_no, from_chapter, to_chapter, status, error_message, deleted
FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND window_no=%s""" + lock_clause,
(TENANT, work_id, win_no),
).fetchone()
if not row:
raise CompensationFenceConflict(f"目标窗口不存在:work={work_id},window={win_no}")
window = dict(zip(
("id", "window_no", "from_chapter", "to_chapter", "status", "error_message", "deleted"),
row,
))
if window["deleted"] or window["status"] != "failed":
raise CompensationFenceConflict(
f"目标窗口不是可恢复的 failed:status={window['status']},deleted={window['deleted']}"
)
error_message = str(window["error_message"] or "")
if error_message.startswith(("upgrade-fence:", COMPENSATION_FAILED_PREFIX, RETRYABLE_CLEAN_PREFIX)):
raise CompensationFenceConflict("目标窗口不是 fence 引入前的 legacy failed")
snapshot = {
"work_id": work_id,
"window": window,
"artifact_counts": _window_artifact_counts(conn, work_id, win_no),
"processing_count": conn.execute(
"""SELECT count(*) FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND status='processing' AND deleted=FALSE""",
(TENANT, work_id),
).fetchone()[0],
}
snapshot["confirmation_sha"] = _legacy_recovery_confirmation_sha(snapshot)
return snapshot
def _echo_legacy_recovery_snapshot(snapshot, mode):
"""以稳定 JSON 输出 preview/execute 结果,便于人工复制确认摘要。"""
click.echo(json.dumps({
"command": "recover-legacy-failed",
"mode": mode,
"window": snapshot["window"],
"artifact_counts": snapshot["artifact_counts"],
"processing_count": snapshot["processing_count"],
"confirmation_sha": snapshot["confirmation_sha"],
}, ensure_ascii=False, sort_keys=True))
@maintenance_cli.command("recover-legacy-failed")
@click.option("--work-id", type=int, required=True, help="目标作品 ID")
@click.option("--window-no", type=int, required=True, help="目标 legacy failed 窗号")
@click.option("--preview", is_flag=True, help="只读输出窗口、产物计数、processing 数和确认 SHA")
@click.option("--execute", is_flag=True, help="在确认参数满足后标记 retryable-clean,保持 failed")
@click.option("--confirmation-sha", help="必须精确等于 preview 输出的 confirmation_sha")
@click.option("--confirm-no-live-process", is_flag=True,
help="人工确认旧进程和数据库 backend 均已结束")
def recover_legacy_failed(work_id, window_no, preview, execute, confirmation_sha,
confirm_no_live_process):
"""恢复 fence 引入前的 legacy failed 窗;不调用模型或嵌入。"""
if preview == execute:
raise click.ClickException("必须且只能指定 --preview 或 --execute")
if preview:
with psycopg.connect(DSN) as conn:
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY")
snapshot = _capture_legacy_failed_snapshot(conn, work_id, window_no)
_echo_legacy_recovery_snapshot(snapshot, "preview")
return
if not confirmation_sha:
raise click.ClickException("--execute 必须提供 --confirmation-sha")
if not confirm_no_live_process:
raise click.ClickException("--execute 必须显式提供 --confirm-no-live-process")
try:
with upgrade_work_lock(DSN, TENANT, work_id):
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
snapshot = _capture_legacy_failed_snapshot(conn, work_id, window_no, lock=True)
if snapshot["confirmation_sha"] != confirmation_sha:
raise CompensationFenceConflict("confirmation-sha 与锁内重算结果不匹配")
if snapshot["processing_count"]:
raise CompensationFenceConflict("本书仍有 processing 窗,拒绝恢复")
if any(snapshot["artifact_counts"].values()):
raise CompensationFenceConflict(
f"本窗已有产物,拒绝恢复:{snapshot['artifact_counts']}"
)
changed = conn.execute(
"""UPDATE example_upgrade_window
SET status='failed', error_message=%s, updater='upgrade-recovery'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s
AND status='failed' AND error_message=%s
RETURNING status""",
(
f"{RETRYABLE_CLEAN_PREFIX} legacy failed recovered; "
f"confirmation-sha={confirmation_sha}",
TENANT,
work_id,
window_no,
snapshot["window"]["error_message"],
),
).fetchone()
if not changed:
raise CompensationFenceConflict("legacy failed 恢复 CAS 冲突")
conn.commit()
snapshot["window"]["error_message"] = (
f"{RETRYABLE_CLEAN_PREFIX} legacy failed recovered; confirmation-sha={confirmation_sha}"
)
_echo_legacy_recovery_snapshot(snapshot, "execute")
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
except CompensationFenceConflict as exc:
raise click.ClickException(str(exc)) from exc
@maintenance_cli.command("repair-card-quality")
@click.option("--work-id", type=int, required=True, help="目标作品 ID")
@click.option("--draft-id", type=int, required=True, help="目标升格 draft ID")
@click.option("--preview", is_flag=True, help="只读输出目标映射、关联快照和确认 SHA")
@click.option("--execute", is_flag=True, help="按确认 SHA 原子更新卡 payload 与 embedding")
@click.option("--confirmation-sha", help="必须精确等于 preview 输出的 confirmation_sha")
@click.option("--confirm-no-live-process", is_flag=True,
help="人工确认目标作品没有其他升格写进程")
def repair_card_quality(work_id, draft_id, preview, execute, confirmation_sha,
confirm_no_live_process):
"""修复已登记单卡的类型、字段、演变和 embedding 绑定。"""
if preview == execute:
raise click.ClickException("必须且只能指定 --preview 或 --execute")
try:
_quality_repair_scope(work_id, draft_id, QUALITY_REPAIR_NAME)
except UpgradeDraftWriteConflict as exc:
raise click.ClickException(str(exc)) from exc
if preview:
try:
with psycopg.connect(DSN) as conn:
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY")
snapshot = _quality_repair_capture_snapshot(conn, work_id, draft_id)
click.echo(json.dumps(
_quality_repair_output(snapshot, "preview"),
ensure_ascii=False, sort_keys=True,
))
return
except (UpgradeDraftWriteConflict, EmbeddingOwnershipConflict,
CompensationFenceConflict) as exc:
raise click.ClickException(str(exc)) from exc
if not confirmation_sha:
raise click.ClickException("--execute 必须提供 --confirmation-sha")
if not confirm_no_live_process:
raise click.ClickException("--execute 必须显式提供 --confirm-no-live-process")
try:
with upgrade_work_lock(DSN, TENANT, work_id):
with psycopg.connect(DSN) as conn:
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY")
snapshot = _quality_repair_capture_snapshot(conn, work_id, draft_id)
if _quality_repair_confirmation_sha(snapshot) != confirmation_sha:
raise CompensationFenceConflict("confirmation-sha 与执行前快照不匹配")
target = snapshot["target_payload"]
text = build_embed_text(target)
content_hash = _payload_embed_hash(target)
embed_sess = _embed_session()
try:
vectors, bad = embed_texts(embed_sess, [text])
finally:
close = getattr(embed_sess, "close", None)
if close:
close()
bad_indexes = sorted(bad) if isinstance(bad, (set, list, tuple)) else bad
if not isinstance(vectors, list) or not isinstance(bad, (set, list, tuple)) \
or bad or len(vectors) != 1 or vectors[0] is None:
raise RuntimeError(
f"质量修复 embedding 未完整返回:bad={bad_indexes},"
f"actual={len(vectors) if isinstance(vectors, list) else type(vectors).__name__}"
)
_quality_repair_validate_vector(vectors[0])
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
current = _quality_repair_capture_snapshot(conn, work_id, draft_id, lock=True)
if _quality_repair_confirmation_sha(current) != confirmation_sha:
raise CompensationFenceConflict("confirmation-sha 与锁内快照不匹配")
new_revision = _quality_repair_apply_locked(
conn, work_id, draft_id, current, vectors[0], text, content_hash,
)
conn.commit()
click.echo(json.dumps(
_quality_repair_output(
current, "execute", new_revision=new_revision, new_hash=content_hash,
),
ensure_ascii=False, sort_keys=True,
))
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
except (UpgradeDraftWriteConflict, EmbeddingOwnershipConflict,
CompensationFenceConflict, RuntimeError) as exc:
raise click.ClickException(str(exc)) from exc
except Exception as exc:
raise click.ClickException(
f"质量修复失败关闭:{type(exc).__name__}: {str(exc)[:300]}"
) from exc
def _presence_dedupe_capture_snapshot(conn, work_id, *, lock=False):
"""读取并严格验证 work8 的十组 presence 冗余;普通/额外组一律失败关闭。"""
if work_id != PRESENCE_DEDUPE_WORK_ID:
raise CompensationFenceConflict(
f"presence 冗余清理只允许 work={PRESENCE_DEDUPE_WORK_ID}:actual={work_id}"
)
lock_clause = " FOR UPDATE" if lock else ""
rows = conn.execute(
"""SELECT id, work_id, window_no, chapter_no, entity_type, name, observation,
creator, create_time, deleted, tenant_id
FROM example_upgrade_presence
WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE
ORDER BY id""" + lock_clause,
(TENANT, work_id),
).fetchall()
grouped = {}
for raw_row in rows:
row = tuple(raw_row)
if len(row) != 11 or row[1] != work_id or row[9] is not False or row[10] != TENANT:
raise CompensationFenceConflict(f"presence 行像不符合窄合同:row={row!r}")
key = (row[2], row[3], row[4], row[5])
grouped.setdefault(key, []).append(row)
duplicate_groups = []
for key, group_rows in sorted(grouped.items(), key=lambda item: repr(item[0])):
if len(group_rows) <= 1:
continue
if len(group_rows) != 2:
raise CompensationFenceConflict(
f"presence 冗余组不是恰好两行:key={key!r},count={len(group_rows)}"
)
ordered = sorted(group_rows, key=lambda row: row[0])
if ordered[0][6] == ordered[1][6]:
raise CompensationFenceConflict(f"presence 冗余组 observation 相同:key={key!r}")
if ordered[0][8] != ordered[1][8]:
raise CompensationFenceConflict(f"presence 冗余组 create_time 不同:key={key!r}")
ids = {row[0] for row in ordered}
delete_ids = ids & PRESENCE_DEDUPE_DELETE_IDS
if len(delete_ids) != 1:
raise CompensationFenceConflict(
f"presence 冗余组待删 ID 不精确:key={key!r},ids={sorted(ids)}"
)
delete_id = next(iter(delete_ids))
keep_id = min(ids)
if keep_id >= delete_id:
raise CompensationFenceConflict(
f"presence 冗余组保留 ID 不是较小者:key={key!r},ids={sorted(ids)}"
)
duplicate_groups.append({
"key": key,
"rows": ordered,
"keep_id": keep_id,
"delete_id": delete_id,
})
if len(duplicate_groups) != PRESENCE_DEDUPE_GROUP_COUNT:
raise CompensationFenceConflict(
"presence 冗余组数量不精确:"
f"expected={PRESENCE_DEDUPE_GROUP_COUNT},actual={len(duplicate_groups)}"
)
duplicate_rows = [row for group in duplicate_groups for row in group["rows"]]
if len(duplicate_rows) != PRESENCE_DEDUPE_ROW_COUNT:
raise CompensationFenceConflict(
"presence 冗余行数量不精确:"
f"expected={PRESENCE_DEDUPE_ROW_COUNT},actual={len(duplicate_rows)}"
)
actual_delete_ids = {group["delete_id"] for group in duplicate_groups}
if actual_delete_ids != PRESENCE_DEDUPE_DELETE_IDS:
raise CompensationFenceConflict(
"presence 待删 ID 集合不精确:"
f"expected={sorted(PRESENCE_DEDUPE_DELETE_IDS)},actual={sorted(actual_delete_ids)}"
)
duplicate_groups.sort(key=lambda group: repr(group["key"]))
duplicate_rows.sort(key=lambda row: row[0])
return {
"contract": PRESENCE_DEDUPE_CONTRACT,
"work_id": work_id,
"groups": duplicate_groups,
"rows": duplicate_rows,
"delete_ids": sorted(actual_delete_ids),
"keep_ids": sorted(group["keep_id"] for group in duplicate_groups),
}
def _presence_dedupe_confirmation_sha(snapshot):
"""确认摘要绑定十组逻辑 key、保留/删除 ID 与完整二十行内容。"""
return _sha256_json({
"contract": snapshot["contract"],
"work_id": snapshot["work_id"],
"groups": [
{
"key": list(group["key"]),
"keep_id": group["keep_id"],
"delete_id": group["delete_id"],
"row_digests": [_sha256_json(row) for row in group["rows"]],
}
for group in snapshot["groups"]
],
"rows_sha256": _sha256_json(snapshot["rows"]),
})
def _presence_dedupe_output(snapshot, mode, deleted_ids=None):
"""输出外部 receipt 所需摘要,不输出 observation 原文。"""
return {
"command": "repair-presence-duplicates",
"contract": snapshot["contract"],
"mode": mode,
"work_id": snapshot["work_id"],
"group_count": len(snapshot["groups"]),
"row_count": len(snapshot["rows"]),
"groups": [
{
"key": list(group["key"]),
"keep_id": group["keep_id"],
"delete_id": group["delete_id"],
"row_digests": [_sha256_json(row) for row in group["rows"]],
}
for group in snapshot["groups"]
],
"keep_ids": snapshot["keep_ids"],
"delete_ids": snapshot["delete_ids"],
"rows_sha256": _sha256_json(snapshot["rows"]),
"confirmation_sha": _presence_dedupe_confirmation_sha(snapshot),
"deleted_ids": sorted(deleted_ids) if deleted_ids is not None else None,
"audit_rows_written": 0,
}
@maintenance_cli.command("repair-presence-duplicates")
@click.option("--work-id", type=int, required=True, help="固定目标作品 ID(仅支持 work=8)")
@click.option("--preview", is_flag=True, help="RR 只读输出十组冗余的行摘要和确认 SHA")
@click.option("--execute", is_flag=True, help="按确认 SHA 软删十个精确冗余行")
@click.option("--confirmation-sha", help="必须精确等于 preview 输出的 confirmation_sha")
@click.option("--confirm-no-live-process", is_flag=True,
help="人工确认目标作品没有其他升格写进程")
def repair_presence_duplicates(work_id, preview, execute, confirmation_sha,
confirm_no_live_process):
"""固定 work8 十组 presence 冗余收口;execute 二次运行失败关闭。"""
if preview == execute:
raise click.ClickException("必须且只能指定 --preview 或 --execute")
if work_id != PRESENCE_DEDUPE_WORK_ID:
raise click.ClickException(
f"presence 冗余清理只允许 work={PRESENCE_DEDUPE_WORK_ID}"
)
if preview:
try:
with psycopg.connect(DSN) as conn:
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY")
snapshot = _presence_dedupe_capture_snapshot(conn, work_id)
click.echo(json.dumps(
_presence_dedupe_output(snapshot, "preview"),
ensure_ascii=False, sort_keys=True,
))
return
except CompensationFenceConflict as exc:
raise click.ClickException(str(exc)) from exc
if not confirmation_sha:
raise click.ClickException("--execute 必须提供 --confirmation-sha")
if not confirm_no_live_process:
raise click.ClickException("--execute 必须显式提供 --confirm-no-live-process")
try:
with upgrade_work_lock(DSN, TENANT, work_id):
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
snapshot = _presence_dedupe_capture_snapshot(conn, work_id, lock=True)
current_sha = _presence_dedupe_confirmation_sha(snapshot)
if current_sha != confirmation_sha:
raise CompensationFenceConflict(
"confirmation-sha 与锁内 presence 精确快照不匹配"
)
deleted = conn.execute(
"""UPDATE example_upgrade_presence SET deleted=TRUE
WHERE tenant_id=%s AND work_id=%s
AND id=ANY(%s) AND deleted=FALSE
RETURNING id""",
(TENANT, work_id, snapshot["delete_ids"]),
).fetchall()
deleted_ids = sorted(row[0] for row in deleted)
if deleted_ids != snapshot["delete_ids"]:
raise CompensationFenceConflict(
"presence 软删返回 ID 不精确:"
f"expected={snapshot['delete_ids']},actual={deleted_ids}"
)
conn.commit()
click.echo(json.dumps(
_presence_dedupe_output(snapshot, "execute", deleted_ids=deleted_ids),
ensure_ascii=False, sort_keys=True,
))
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
except CompensationFenceConflict as exc:
raise click.ClickException(str(exc)) from exc
except Exception as exc:
raise click.ClickException(
f"presence 冗余清理失败关闭:{type(exc).__name__}: {str(exc)[:300]}"
) from exc
def _create_smoke_window(conn, work_id):
"""冒烟自建 pending 窗夹具:选一个有正文、且未被任何窗锚定的章作单章窗。
夹具只在回滚事务内存在;序列值消耗是 PostgreSQL 回滚不回退的正常行为,
不构成公共行漂移。
"""
chapter_row = conn.execute(
"""SELECT c.order_no FROM muse_content_chapter c
JOIN muse_content_block b ON b.chapter_id=c.id AND b.deleted=FALSE
WHERE c.tenant_id=%s AND c.work_id=%s AND c.deleted=FALSE
AND COALESCE(b.content_text,'') <> ''
AND c.order_no NOT IN (
SELECT from_chapter FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s)
ORDER BY c.order_no LIMIT 1""",
(TENANT, work_id, TENANT, work_id),
).fetchone()
if not chapter_row:
raise RuntimeError(f"work={work_id} 没有可用作冒烟夹具的有正文章节,真实 smoke 明确失败")
order_no = chapter_row[0]
win_no = conn.execute(
"SELECT COALESCE(MAX(window_no), 0) + 1 FROM example_upgrade_window "
"WHERE tenant_id=%s AND work_id=%s",
(TENANT, work_id),
).fetchone()[0]
conn.execute(
"""INSERT INTO example_upgrade_window
(work_id, window_no, from_chapter, to_chapter, status, creator, tenant_id)
VALUES (%s, %s, %s, %s, 'pending', 'rollback-smoke', %s)""",
(work_id, win_no, order_no, order_no, TENANT),
)
return conn.execute(
"""SELECT id, window_no, status, error_message
FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND status='pending' AND deleted=FALSE
AND window_no=%s FOR UPDATE""",
(TENANT, work_id, win_no),
).fetchone()
def _create_smoke_draft(conn, work_id):
"""冒烟自建 upgrade_book 草稿夹具:只供 revision 漂移检查,随事务回滚。"""
conn.execute(
"""INSERT INTO muse_knowledge_draft
(work_id, draft_type, status, source_type, creator, tenant_id)
VALUES (%s, 'entity', 'pending', %s, 'rollback-smoke', %s)""",
(work_id, SOURCE_TYPE, TENANT),
)
return conn.execute(
"""SELECT id, revision FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s
AND creator='rollback-smoke' AND deleted=FALSE
ORDER BY id DESC LIMIT 1 FOR UPDATE""",
(TENANT, work_id, SOURCE_TYPE),
).fetchone()
def real_pg_rollback_smoke(work_id):
"""在真实 public 窗表内验证 marker 写入随后 rollback,且不改变公共行。
前置行(pending 窗、upgrade_book 草稿)缺失时在回滚事务内自建夹具:
冒烟不再依赖“生产恰好有 pending 窗”的瞬时状态;还原验证区分
夹具行必须消失与既有行必须还原。
"""
try:
with upgrade_work_lock(DSN, TENANT, work_id):
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
row = conn.execute(
"""SELECT id, window_no, status, error_message
FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND status='pending' AND deleted=FALSE
ORDER BY window_no LIMIT 1 FOR UPDATE""",
(TENANT, work_id),
).fetchone()
draft_row = conn.execute(
"""SELECT id, revision FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s
AND deleted=FALSE ORDER BY id LIMIT 1 FOR UPDATE""",
(TENANT, work_id, SOURCE_TYPE),
).fetchone()
fixture_window = row is None
fixture_draft = draft_row is None
if fixture_window:
row = _create_smoke_window(conn, work_id)
if fixture_draft:
draft_row = _create_smoke_draft(conn, work_id)
window_id, win_no, original_status, original_error = row
original_window = None if fixture_window else conn.execute(
"""SELECT to_jsonb(w) FROM example_upgrade_window w
WHERE w.tenant_id=%s AND w.work_id=%s AND w.window_no=%s""",
(TENANT, work_id, win_no),
).fetchone()[0]
draft_id, original_revision = draft_row
window, chapters, schemas, work_title = _capture_window_input(conn, work_id, win_no)
input_sha = _window_input_sha(window, chapters, schemas, work_title=work_title)
state_sha = _set_processing_marker(
conn, work_id, win_no, "entity", input_sha, window_id,
)
_assert_window_marker(conn, work_id, win_no, "entity", input_sha, state_sha)
conn.execute(
"""UPDATE muse_knowledge_draft SET revision=revision+1
WHERE tenant_id=%s AND id=%s AND source_type=%s""",
(TENANT, draft_id, SOURCE_TYPE),
)
try:
_assert_window_marker(conn, work_id, win_no, "entity", input_sha, state_sha)
except CompensationFenceConflict:
pass
else:
raise RuntimeError("draft revision 漂移未被 _assert_window_marker 拒绝")
conn.rollback()
with psycopg.connect(DSN) as verify_conn:
restored_window_row = verify_conn.execute(
"""SELECT to_jsonb(w) FROM example_upgrade_window w
WHERE w.tenant_id=%s AND w.work_id=%s AND w.window_no=%s""",
(TENANT, work_id, win_no),
).fetchone()
restored_window = restored_window_row[0] if restored_window_row else None
restored_draft_row = verify_conn.execute(
"""SELECT revision FROM muse_knowledge_draft
WHERE tenant_id=%s AND id=%s AND source_type=%s""",
(TENANT, draft_id, SOURCE_TYPE),
).fetchone()
restored_revision = restored_draft_row[0] if restored_draft_row else None
expected_window = None if fixture_window else original_window
expected_revision = None if fixture_draft else original_revision
if restored_window != expected_window or restored_revision != expected_revision:
raise RuntimeError(
f"rollback 后公共行漂移:window_equal={restored_window == expected_window},"
f"draft_revision={restored_revision}/{expected_revision}"
)
click.echo(
f"real PG rollback smoke 通过:work={work_id} window={win_no} "
f"status={original_status} fixture_window={fixture_window} "
f"fixture_draft={fixture_draft} 未提交 marker"
)
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
def _compensate_window(
work_id,
win_no,
error,
*,
expected_input_sha=None,
expected_state_sha=None,
expected_state_counts=None,
):
"""固定锁内 exact 补偿;有 baseline 时只接受本次 attempt 的零漂移状态。"""
has_pre_attempt_baseline = (
expected_input_sha is not None and expected_state_sha is not None
)
def compensation_failed_message(detail):
"""保留原始异常,同时把补偿自身的证据放入有限长度错误列。"""
original = f"{type(error).__name__}: {str(error)}"[:160]
return f"{COMPENSATION_FAILED_PREFIX} 原始异常={original}; {detail}"[:500]
with psycopg.connect(DSN) as conn:
_lock_upgrade_domains(conn)
row = conn.execute(
"""SELECT id, status, error_message FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND window_no=%s FOR UPDATE""",
(TENANT, work_id, win_no),
).fetchone()
if not row:
raise CompensationFenceConflict("补偿窗口不存在")
window_id, status, marker = row
if status == "processing":
try:
_, expected_input, expected_state = _parse_upgrade_marker(marker)
current_input = _window_input_sha(*_capture_window_input(conn, work_id, win_no))
current_state = _window_state_sha(
_capture_window_state(conn, work_id, window_id), window_id
)
if (current_input, current_state) != (expected_input, expected_state):
raise CompensationFenceConflict("补偿 input/state 漂移")
undo_window(conn, work_id, win_no)
except Exception as exc:
message = compensation_failed_message(
f"processing marker 补偿异常={type(exc).__name__}: {str(exc)[:160]}"
)
changed = conn.execute(
"""UPDATE example_upgrade_window SET status='failed', error_message=%s, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s
AND status='processing' AND error_message=%s RETURNING status""",
(message, TENANT, work_id, win_no, marker),
).fetchone()
if not changed:
raise CompensationFenceConflict("补偿失败 marker CAS 冲突") from exc
conn.commit()
raise CompensationFenceConflict(message) from exc
elif has_pre_attempt_baseline:
# 首次外部调用前已捕获 baseline;此处只接受 input/state 完全不变,历史软删不再按窗号猜来源。
current_input = _window_input_sha(*_capture_window_input(conn, work_id, win_no))
current_state_rows = _capture_window_state(conn, work_id, window_id)
current_state = _window_state_sha(current_state_rows, window_id)
current_state_counts = _window_state_domain_counts(current_state_rows)
if (current_input, current_state) != (expected_input_sha, expected_state_sha):
before_counts = expected_state_counts or {name: "unknown" for name in current_state_counts}
artifact_counts = _window_artifact_counts(
conn,
work_id,
win_no,
("upgrade-reset", "upgrade-undo"),
)
message = compensation_failed_message(
"baseline 漂移:"
f"input_sha={expected_input_sha}->{current_input}; "
f"state_sha={expected_state_sha}->{current_state}; "
f"artifact_counts={artifact_counts}; "
f"state_counts={before_counts}->{current_state_counts}"
)
changed = conn.execute(
"""UPDATE example_upgrade_window SET status='failed', error_message=%s, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s
AND status=%s AND error_message IS NOT DISTINCT FROM %s RETURNING status""",
(message, TENANT, work_id, win_no, status, marker),
).fetchone()
if not changed:
raise CompensationFenceConflict("baseline 补偿失败 CAS 冲突")
conn.commit()
raise CompensationFenceConflict(message)
elif any(_window_artifact_counts(
conn,
work_id,
win_no,
("upgrade-reset", "upgrade-undo"),
).values()):
message = f"{COMPENSATION_FAILED_PREFIX} 实体提交前无 marker,但本窗产物非零"[:500]
changed = conn.execute(
"""UPDATE example_upgrade_window SET status='failed', error_message=%s, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s
AND status=%s AND error_message IS NOT DISTINCT FROM %s RETURNING status""",
(message, TENANT, work_id, win_no, status, marker),
).fetchone()
if not changed:
raise CompensationFenceConflict("无 marker 补偿失败 CAS 冲突")
conn.commit()
raise CompensationFenceConflict(message)
conn.execute(
"""UPDATE example_upgrade_window SET status='failed', error_message=%s, updater='upgrade'
WHERE tenant_id=%s AND work_id=%s AND window_no=%s""",
(("retryable-clean: " + str(error))[:500], TENANT, work_id, win_no),
)
conn.commit()
def _recover_processing_windows(work_id):
"""启动时先恢复 processing;exact 才 undo,漂移或非法 marker 持久化后立即停止。"""
with psycopg.connect(DSN) as conn:
windows = conn.execute(
"""SELECT window_no FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND status='processing' AND deleted=FALSE
ORDER BY window_no""",
(TENANT, work_id),
).fetchall()
for (win_no,) in windows:
_compensate_window(work_id, win_no, "processing 启动恢复")
def _run(work_id, max_windows, max_calls, model, redo_window, semantic_on,
*, raise_on_second_failure=True, validate_window_input=True):
"""按窗顺序执行两阶段升格;默认强制章域校验并在二次失败时停止。"""
if redo_window:
raise click.ClickException(
"--redo-window 暂不支持;请先 backup,再按 reset/rebuild 人工恢复"
)
calls = {"n": 0}
embed_sess = _embed_session() if semantic_on else None
def call(prompt, need_keys):
calls["n"] += 1
return m3_json(prompt, model, need_keys, system=IDENTITY)
try:
_recover_processing_windows(work_id)
except CompensationFenceConflict as exc:
raise click.ClickException(str(exc)) from exc
with psycopg.connect(DSN) as conn:
title = conn.execute(
"SELECT title FROM muse_content_work WHERE id=%s", (work_id,)
).fetchone()[0]
contracts = load_entity_contracts(conn)
wins = conn.execute(
"""SELECT window_no, from_chapter, to_chapter, status, error_message
FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND status!='done' AND deleted=FALSE
ORDER BY from_chapter""",
(TENANT, work_id),
).fetchall()
for win_no, _, _, status, error_message in wins:
if status == "failed" and not str(error_message or "").startswith("retryable-clean:"):
raise click.ClickException(
f"legacy failed 窗{win_no} 禁止自动恢复;窗61等历史窗口须人工恢复:"
f"{str(error_message or '')[:200]}"
)
milestone_types = {
entity_type
for entity_type in ENTITY_TYPES
if any(
field.get("key") == "演变历程"
for field in contracts.get(entity_type, {}).get("字段", [])
)
}
done_n = 0
for win_no, a, b, _, _ in wins:
if max_windows and done_n >= max_windows:
break
if max_calls and calls["n"] >= max_calls:
click.echo(f"⏸ 调用闸 {max_calls} 已到,停在窗{win_no} 之前")
break
first_error = None
for attempt in range(2):
attempt_input_sha = None
attempt_state_sha = None
attempt_state_counts = None
try:
with psycopg.connect(DSN) as conn:
window, chapters, schemas, input_title = _capture_window_input(
conn, work_id, win_no, validate=validate_window_input,
)
text, chapter_texts = load_window_material(conn, work_id, a, b)
name_map, presence, aliases_by_draft, expected_revisions = load_known(
conn, work_id
)
if window["from_chapter"] != a or window["to_chapter"] != b:
raise CompensationFenceConflict(f"窗{win_no}章域在读取期间漂移")
captured_texts = _aggregate_window_chapter_texts(chapters)
if validate_window_input and captured_texts != chapter_texts:
raise CompensationFenceConflict(f"窗{win_no}正文在读取期间漂移")
input_sha = _window_input_sha(window, chapters, schemas, work_title=input_title)
window_id = window["id"]
model_title = input_title
pre_attempt_state = _capture_window_state(conn, work_id, window_id)
pre_attempt_state_sha = _window_state_sha(pre_attempt_state, window_id)
pre_attempt_state_counts = _window_state_domain_counts(pre_attempt_state)
attempt_input_sha = input_sha
attempt_state_sha = pre_attempt_state_sha
attempt_state_counts = pre_attempt_state_counts
onstage = prescan(name_map, text)
obs, _ = call(
observe_prompt(contracts, model_title, a, b, text, onstage),
OBSERVATION_MODEL_KEYS,
)
normalize_model_observation_aliases(obs, contracts)
# 出场章属于正文索引事实:先覆盖模型首轮缺失/错误值,避免仅因缺章
# 触发整批 repair;repair 后再次覆盖,防止 repair 响应带回模型章号。
populate_model_observation_chapters(
obs, name_map, aliases_by_draft, chapter_texts,
)
obs = _repair_entity_observation_structure(
obs,
title=model_title,
a=a,
b=b,
text=text,
onstage=onstage,
name_map=name_map,
call=call,
keys=OBSERVATION_MODEL_KEYS,
)
populate_model_observation_chapters(
obs, name_map, aliases_by_draft, chapter_texts,
)
_normalize_known_entity_observations(obs, name_map)
populate_mechanical_presence(
obs, onstage, name_map, aliases_by_draft, chapter_texts,
)
for entity in obs["新名字"]:
entity["出场章"] = sorted(
_filter_entity_chapters(
entity.get("名称"),
entity.get("别名"),
entity.get("出场章"),
chapter_texts,
)
)
repair_missing_milestone_evidence(
obs,
title=model_title,
a=a,
b=b,
text=text,
chapter_texts=chapter_texts,
call=call,
)
_arbitrate_cross_type_aliases(
obs,
name_map,
aliases_by_draft,
text,
call,
)
verdicts, dedup_merge_n, dedup_chain_n = {}, 0, 0
if semantic_on:
verdicts, dedup_merge_n, dedup_chain_n = prejudge_semantic(
obs, name_map, presence, embed_sess, work_id, call
)
plan = _plan_window_entities(
work_id,
win_no,
obs,
name_map,
presence,
aliases_by_draft,
verdicts,
semantic_on,
milestone_types,
chapter_texts,
)
updates, snapshots = _compute_entity_updates(
plan,
expected_revisions,
contracts,
model_title,
a,
b,
text,
chapter_texts,
call,
)
with psycopg.connect(DSN) as conn:
ref_map, appearances, aliases, entity_touched, entity_state_sha = (
_apply_entity_stage(
conn,
work_id,
win_no,
plan,
updates,
snapshots,
expected_revisions,
milestone_types,
chapter_texts,
input_sha,
window_id,
)
)
planned_refs = {
ref_map.get(ref, ref) for ref in plan["to_update"]
} | set(ref_map.values()) | set(appearances)
characters, relation_rows, relation_snapshots = _read_relation_snapshot(
work_id, planned_refs, onstage
)
relation_items = _compute_relations(
contracts,
model_title,
a,
b,
text,
characters,
relation_rows,
call,
)
with psycopg.connect(DSN) as conn:
relation_touched, relation_state_sha = _apply_relation_stage(
conn,
work_id,
win_no,
relation_items,
characters,
relation_rows,
relation_snapshots,
appearances,
aliases,
chapter_texts,
input_sha,
entity_state_sha,
window_id,
)
touched = entity_touched | relation_touched
prepared = prepare_touched_cards(embed_sess, touched) if semantic_on else []
embed_n = _finalize_window(
work_id, win_no, input_sha, relation_state_sha, prepared
)
except Exception as exc:
first_error = first_error or exc
try:
_compensate_window(
work_id,
win_no,
exc,
expected_input_sha=attempt_input_sha,
expected_state_sha=attempt_state_sha,
expected_state_counts=attempt_state_counts,
)
except CompensationFenceConflict as compensation_exc:
raise click.ClickException(str(compensation_exc)) from compensation_exc
if attempt == 0:
click.echo(
f" ↻ 窗{win_no}失败,clean 后当场重试:{str(exc)[:150]}"
)
continue
click.echo(
f" ⛔ 窗{win_no}第二次失败,已 clean 后停止:{str(exc)[:200]}"
)
if raise_on_second_failure:
raise click.ClickException(
f"窗{win_no}第二次失败,当前作品已停止;请人工再次运行以使用 retryable-clean 断点:"
f" {str(exc)[:200]}"
) from exc
return
new_n = len(obs.get("新名字", []))
extra = (
f" 判重:并{dedup_merge_n}/链{dedup_chain_n} 嵌入:{embed_n}"
if semantic_on
else ""
)
click.echo(
f" 窗{win_no}✓ ({a}-{b}章) 在场{len(onstage)} 新名字{new_n} "
f"更新卡{len(plan['to_update'])} 调用累计{calls['n']}{extra}"
)
done_n += 1
break
click.echo(f"《{title}》本次完成 {done_n} 窗,LLM 调用 {calls['n']} 次")
@cli.command("run")
@click.option("--work-id", type=int, required=True)
@click.option("--max-windows", type=click.IntRange(min=0), default=0,
help="本次最多跑几个窗(0=不限)")
@click.option("--max-calls", type=click.IntRange(min=0), default=0,
help="本次 LLM 调用上限(0=不限,含敏感失败)")
@click.option("--model", default="MiniMax-M3", show_default=True)
@click.option("--redo-window", type=int, default=0,
help="暂不支持;历史窗口须人工 backup/reset/rebuild")
@click.option("--semantic-dedup", "semantic_on", is_flag=True,
help="开启语义判重(P1):立卡前召回同书近邻+M3终判治改名/跨型漏并;"
"最终短事务严格写入完整向量;默认关")
def run(work_id, max_windows, max_calls, model, redo_window, semantic_on):
"""持有同书会话锁后执行升格,锁连接可安全跨越模型调用。"""
if redo_window:
raise click.ClickException(
"--redo-window 暂不支持;请先 backup,再按 reset/rebuild 人工恢复"
)
try:
with upgrade_work_lock(DSN, TENANT, work_id):
return _run(
work_id, max_windows, max_calls, model, redo_window, semantic_on,
)
except UpgradeWorkLockUnavailable as exc:
raise click.ClickException(str(exc)) from exc
@cli.command()
@click.option("--work-id", type=int, required=True)
def status(work_id):
"""升格进度:窗状态/卡数/留档数。"""
with psycopg.connect(DSN) as conn:
title = conn.execute("SELECT title FROM muse_content_work WHERE id=%s",
(work_id,)).fetchone()[0]
w = conn.execute(
"""SELECT count(*) FILTER (WHERE status='done'), count(*) FILTER (WHERE status='failed'),
count(*) FROM example_upgrade_window
WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE""",
(TENANT, work_id)).fetchone()
cards = conn.execute(
"""SELECT draft_payload->>'type', count(*) FROM muse_knowledge_draft
WHERE tenant_id=%s AND work_id=%s AND source_type=%s AND deleted=FALSE
GROUP BY 1 ORDER BY 2 DESC""", (TENANT, work_id, SOURCE_TYPE)).fetchall()
pres = conn.execute(
"SELECT count(*) FROM example_upgrade_presence WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE",
(TENANT, work_id)).fetchone()[0]
ali = conn.execute(
"SELECT count(*) FROM example_upgrade_alias WHERE tenant_id=%s AND work_id=%s AND deleted=FALSE",
(TENANT, work_id)).fetchone()[0]
click.echo(f"《{title}》窗 {w[0]}done/{w[1]}failed/{w[2]}total | "
f"卡 {', '.join(f'{t}:{n}' for t, n in cards) or '0'} | 留档{pres} 别名{ali}")
if __name__ == "__main__":
cli()