实现: 为升格全书重建建立可恢复备份
在统一作品锁下导出并校验七域状态，绑定正文、窗口、字段合同与代码身份；提供事务回滚演练和双确认恢复入口，防止 reset 前后出现并发混写或不可恢复状态。
This commit is contained in:
zizi
parent
fa922f8cc5
commit
daac544211
@ -60,6 +60,25 @@ disable-model-invocation: true
|
||||
|
||||
**作品面升格执行器 `scripts/parse_upgrade.py`（与上面范式拆书管线并行的另一条线,命令 `windows`/`run`/`status`）**：把参考书正文按窗抽成「会随剧情长大的实体卡」(升格卡,`source_type=upgrade_book`),设计见 `docs/2026-07-16-升格卡改造设计.md`。里程碑除真实章号外，模型输出必须临时携带所标章节正文短引；系统机械核验后删除证据，错章、缺证据或改写证据均拒收入库并留审计。模型给出的顶层出场章也必须由实体规范名或合法别名在对应章节正文中的实际出现机械证明；合法别名集合同时取卡内 payload 与独立 alias 表，单字规范名禁作章证据，规范名和别名均禁止通用称谓/关系称呼（如队长、舰长）作证据，常规二至四字专名保持精确子串命中。未命中章不得参与立卡、登场兜底或既有卡追加，无实证章时不补登场里程碑；既有卡每窗新增出场章必须审计完整旧值，redo/undo 精确恢复后再按新正文重算。显式 redo **仅允许当前 active 末窗**：执行前机械校验窗口号连续、章域合法且首尾相接，并校验目标窗逐章正文齐全且非空；任一失败必须在快照、undo 与任何写入前非零退出。历史窗修正必须全书前滚重建；后缀级联重算属于 P1，当前不支持。合法末窗 redo 仍须在撤销事务中清除所有 active `upgrade_book` 卡的本窗章域，保留窗外章并为历史无审计数据补旧值审计；redo 前建立完整恢复点（卡 payload/revision/deleted、别名、presence、水位、审计与窗状态），首次失败重新清理再试，最终失败完整恢复且重试初建卡软删。`--max-calls` 不得截断已经开始的即时重试，非显式 redo 不做该全局清理。`run` 默认**不发嵌入**;加 `--semantic-dedup` 开语义判重(治改名/跨型漏并)时,每窗按**读→算→写三段式短连接**跑——观察/近邻召回/M3 终判都在**无长连接**段发 LLM 与嵌入 HTTP(不再持窗级连接跨调用存活),写段只查预判结果落库;并**边抽边嵌**：本窗新建/更新卡在窗事务提交后增量嵌入落库(软删旧向量+upsert 新行),**后窗即可语义召回前窗刚长成的卡**,不再依赖"同书须预先全量 embed"。嵌段失败只告警、不牵连已 done 的窗。
|
||||
|
||||
**同书命令互斥**：`parse_upgrade.py run/windows`、`reset_upgrade_work.py` 的预览/执行，以及 `backup_upgrade_work.py backup/rehearse/restore` 均须先取得 `scripts/upgrade_work_lock.py` 的同租户同作品 PostgreSQL session advisory lock；失败必须在任何业务 SQL、文件 verify/写入、嵌入或 LLM 调用前非零退出。锁由独立 autocommit 连接持有到命令结束，该连接只执行加锁/解锁 SQL；`status` 只读且不取锁。所有调用方必须导入同一个 `upgrade_work_lock(...)` context manager，禁止另造不兼容锁键。
|
||||
|
||||
**升格 work reset 前备份（只读）**：任何 `reset_upgrade_work.py --execute` 前，先用单个 `REPEATABLE READ READ ONLY` 事务导出该作品七域状态；目录必须是 `/private/tmp` 下尚不存在的新目录。input 摘要在同一快照中绑定不含密码的数据库 identity、Canonical 章节/block 内容、active 窗边界、升格七型 active 字段合同，以及计划参数 `MiniMax-M3 + semantic-dedup=true`；同时绑定本次实际执行的 `backup_upgrade_work.py`、`parse_upgrade.py`、`llm.py` 和本 SKILL 的 fileSha。`gitCommit` 只记录 HEAD；当代码未提交时不得把它当执行代码身份，确认以 manifest 的 `inputSha + codeFiles + confirmationSha` 为准。SHA 仅用于发现传输、落盘或误操作造成的意外损坏，不宣称抵抗能同时改写工件和摘要的恶意篡改。命令默认不写数据库、不调用模型，生成后会关闭数据库连接并从磁盘独立复验；任一失败删除本次新建目录并非零退出。
|
||||
|
||||
```bash
|
||||
# 创建 work 8 七域一致性备份；expected 值进入 input 摘要并机械核对 594 章/116 窗
|
||||
.venv/bin/python .claude/skills/parse-book/scripts/backup_upgrade_work.py backup --work-id 8 --expected-chapters 594 --expected-windows 116
|
||||
# 显式指定 /private/tmp 下的新目录
|
||||
.venv/bin/python .claude/skills/parse-book/scripts/backup_upgrade_work.py backup --work-id 8 --expected-chapters 594 --expected-windows 116 --output-dir /private/tmp/muse-upgrade-work8-before-reset
|
||||
# 已有备份纯离线复验（不连接数据库、不调用模型）
|
||||
.venv/bin/python .claude/skills/parse-book/scripts/backup_upgrade_work.py verify --backup-dir /private/tmp/muse-upgrade-work8-before-reset
|
||||
# 在真实 public 表的单事务中应用七域恢复、逐域读回，最后强制 ROLLBACK；不调整 sequence
|
||||
.venv/bin/python .claude/skills/parse-book/scripts/backup_upgrade_work.py rehearse --work-id 8 --backup-dir /private/tmp/muse-upgrade-work8-before-reset --confirmation-sha <manifest-confirmationSha>
|
||||
# 演练通过并取得本次明确确认后才允许提交；backup-id 必须与 manifest 精确一致
|
||||
.venv/bin/python .claude/skills/parse-book/scripts/backup_upgrade_work.py restore --work-id 8 --backup-dir /private/tmp/muse-upgrade-work8-before-reset --execute --backup-id <manifest-backup_id> --confirmation-sha <manifest-confirmationSha>
|
||||
```
|
||||
|
||||
backup 在同一只读快照内机械验证 active `window_no=1..expected-windows`、各窗章域无缝覆盖 Canonical 连续章域，并要求每章至少一个非空 block；任一不符不产备份。恢复前固定先离线 `verify`，核对 `confirmationSha`，并在事务内按 manifest 的 expected 值重算 input；work、tenant、数据库 identity 或 Canonical/窗/合同/代码任一漂移均拒绝。恢复内核按备份 ID exact upsert drafts/windows/embeddings；额外 active 窗直接拒绝，reset 后新增升格 draft 及其额外 embedding 只软删；aliases/presence/card_state/audits 按 work 边界先清后插。为避免 `updated_at` trigger 改写备份时间戳，恢复写段使用事务局部 replica trigger 模式，仍按 drafts→windows→aliases/presence→card_state/audits→embeddings 的父子顺序执行，事务结束自动复原。`rehearse` 永不调用非事务性的 sequence 调整并强制回滚；`restore --execute` 还必须同时精确匹配 `backup_id + confirmationSha`，逐域读回通过后才把 sequence 仅向前推进并提交。
|
||||
|
||||
**范式线失败关闭（2026-07-21 假绿复盘后现行）**：`parse_outline.py window`、`parse_llm.py cards`、`parse_outline.py check` 是同一本作品的顺序流水线。普通网络/429/5xx 由 `chat_governed` 在单模型内短重试后沿 `MiniMax-M3 → MiniMax-M2.7 → glm-5.2 → deepseek-v4-flash` 降级；整条链耗尽，或功能层结构输出有限重试仍失败，必须**立即停止当前作品并保留断点**，禁止跳窗/跳型继续。`cards` 和 `check` 开始前机械验证大纲窗对全书章域无头部、中部、尾部缝隙；任一缝隙非零退出。补插窗后所有终检按 `from_order/to_order` 绝对章号排序并重新编号阶段，`window_no` 只作运行展示号，不得参与时序判断。
|
||||
|
||||
## 步骤(自底向上,与创作期规划的自顶向下互为镜像)
|
||||
|
||||
1333
.claude/skills/parse-book/scripts/backup_upgrade_work.py
Normal file
1333
.claude/skills/parse-book/scripts/backup_upgrade_work.py
Normal file
File diff suppressed because it is too large
Load Diff
@ -37,6 +37,7 @@ import psycopg
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "llm" / "scripts"))
|
||||
from parse_llm import m3_json, SensitiveHardStop, IDENTITY, TENANT, DSN # noqa: E402
|
||||
from upgrade_work_lock import UpgradeWorkLockUnavailable, upgrade_work_lock # noqa: E402
|
||||
# 语义判重（P1）复用 embed skill 的嵌入通道（同模型同维、与检索端语义对齐）——
|
||||
# 只在开启 --semantic-dedup 时才真调，默认关（试跑期嵌入延后，见文件头注释）；
|
||||
# build_embed_text/MODEL/DIM/ACTOR 供嵌段（洞①边抽边嵌）复用检索端同源文本构造器与列常量。
|
||||
@ -1460,24 +1461,16 @@ def cli():
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
def windows(work_id):
|
||||
"""机械切正文窗（幂等）。"""
|
||||
with psycopg.connect(DSN) as conn:
|
||||
total, new = cut_windows(conn, work_id)
|
||||
click.echo(f"work={work_id} 切窗完成：全书 {total} 窗（本次新建 {new} 行）")
|
||||
try:
|
||||
with upgrade_work_lock(DSN, TENANT, work_id):
|
||||
with psycopg.connect(DSN) as conn:
|
||||
total, new = cut_windows(conn, work_id)
|
||||
click.echo(f"work={work_id} 切窗完成：全书 {total} 窗（本次新建 {new} 行）")
|
||||
except UpgradeWorkLockUnavailable as exc:
|
||||
raise click.ClickException(str(exc)) from exc
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
@click.option("--max-windows", type=click.IntRange(min=0), default=0,
|
||||
help="本次最多跑几个窗（0=不限）")
|
||||
@click.option("--max-calls", type=click.IntRange(min=0), default=0,
|
||||
help="本次 LLM 调用上限（0=不限，含敏感失败）")
|
||||
@click.option("--model", default="MiniMax-M3", show_default=True)
|
||||
@click.option("--redo-window", type=int, default=0,
|
||||
help="仅允许当前末窗强制重跑；历史窗须全书前滚重建，后缀级联重算(P1)暂不支持")
|
||||
@click.option("--semantic-dedup", "semantic_on", is_flag=True,
|
||||
help="开启语义判重(P1)：立卡前召回同书近邻+M3终判治改名/跨型漏并；"
|
||||
"边抽边嵌（本窗新建/更新卡窗事务后增量嵌入落库，后窗即可召回前窗的卡）；默认关")
|
||||
def run(work_id, max_windows, max_calls, model, redo_window, semantic_on):
|
||||
def _run(work_id, max_windows, max_calls, model, redo_window, semantic_on):
|
||||
"""按窗顺序跑升格；redo 仅限当前末窗，历史窗须全书前滚重建。"""
|
||||
calls = {"n": 0} # 调用计数（含敏感失败换模型的次数由 m3_json 内部消化，此处计成功轮次）
|
||||
# 语义判重嵌入会话（仅开启时建；禁系统代理，走内网直连）
|
||||
@ -1945,6 +1938,28 @@ def run(work_id, max_windows, max_calls, model, redo_window, semantic_on):
|
||||
click.echo(f"《{title}》本次完成 {done_n} 窗，LLM 调用 {calls['n']} 次")
|
||||
|
||||
|
||||
@cli.command("run")
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
@click.option("--max-windows", type=click.IntRange(min=0), default=0,
|
||||
help="本次最多跑几个窗（0=不限）")
|
||||
@click.option("--max-calls", type=click.IntRange(min=0), default=0,
|
||||
help="本次 LLM 调用上限（0=不限，含敏感失败）")
|
||||
@click.option("--model", default="MiniMax-M3", show_default=True)
|
||||
@click.option("--redo-window", type=int, default=0,
|
||||
help="仅允许当前末窗强制重跑；历史窗须全书前滚重建，后缀级联重算(P1)暂不支持")
|
||||
@click.option("--semantic-dedup", "semantic_on", is_flag=True,
|
||||
help="开启语义判重(P1)：立卡前召回同书近邻+M3终判治改名/跨型漏并；"
|
||||
"边抽边嵌（本窗新建/更新卡窗事务后增量嵌入落库，后窗即可召回前窗的卡）；默认关")
|
||||
def run(work_id, max_windows, max_calls, model, redo_window, semantic_on):
|
||||
"""持有同书会话锁后执行升格，锁连接可安全跨越模型调用。"""
|
||||
|
||||
try:
|
||||
with upgrade_work_lock(DSN, TENANT, work_id):
|
||||
return _run(work_id, max_windows, max_calls, model, redo_window, semantic_on)
|
||||
except UpgradeWorkLockUnavailable as exc:
|
||||
raise click.ClickException(str(exc)) from exc
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
def status(work_id):
|
||||
|
||||
@ -30,14 +30,14 @@ import psycopg
|
||||
# 复用管线的连接与租户常量（与 parse_upgrade 同源）
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
from parse_llm import DSN, TENANT # noqa: E402
|
||||
from upgrade_work_lock import UpgradeWorkLockUnavailable, upgrade_work_lock # noqa: E402
|
||||
|
||||
SOURCE_TYPE = "upgrade_book"
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
@click.option("--execute", is_flag=True, help="真执行（默认只预览统计）")
|
||||
def main(work_id, execute):
|
||||
def _reset(work_id, execute):
|
||||
"""在调用方已持有同书锁时预览或执行重抽清理。"""
|
||||
|
||||
with psycopg.connect(DSN) as conn:
|
||||
title = conn.execute("SELECT title FROM muse_content_work WHERE id=%s",
|
||||
(work_id,)).fetchone()[0]
|
||||
@ -101,5 +101,18 @@ def main(work_id, execute):
|
||||
click.echo(f" ✅ 清后复核：活升格卡 {left}（应0）｜ 窗 pending {pend[0]}/{pend[1]}（应全 pending）")
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--work-id", type=int, required=True)
|
||||
@click.option("--execute", is_flag=True, help="真执行（默认只预览统计）")
|
||||
def main(work_id, execute):
|
||||
"""先获取同书会话锁，再预览或执行升格全量重抽准备。"""
|
||||
|
||||
try:
|
||||
with upgrade_work_lock(DSN, TENANT, work_id):
|
||||
return _reset(work_id, execute)
|
||||
except UpgradeWorkLockUnavailable as exc:
|
||||
raise click.ClickException(str(exc)) from exc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@ -0,0 +1,650 @@
|
||||
#!/usr/bin/env python3
|
||||
"""升格作品备份脚本的纯离线回归测试：不连接数据库、不调用模型。"""
|
||||
|
||||
import datetime as dt
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from decimal import Decimal
|
||||
from unittest import mock
|
||||
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import backup_upgrade_work as backup # noqa: E402
|
||||
|
||||
|
||||
class _Result:
|
||||
"""模拟 psycopg 查询结果，仅暴露脚本使用的读取接口。"""
|
||||
|
||||
def __init__(self, rows):
|
||||
self.rows = rows
|
||||
|
||||
def fetchone(self):
|
||||
return self.rows[0] if self.rows else None
|
||||
|
||||
def fetchall(self):
|
||||
return list(self.rows)
|
||||
|
||||
|
||||
class _OfflineConnection:
|
||||
"""按表名返回固定数据，并记录完整 SQL 与连接关闭时机。"""
|
||||
|
||||
def __init__(self):
|
||||
self.closed = False
|
||||
self.executions = []
|
||||
self.rows = {
|
||||
"drafts": [
|
||||
{
|
||||
"id": 2,
|
||||
"work_id": 8,
|
||||
"source_type": "upgrade_book",
|
||||
"confidence": Decimal("0.8200"),
|
||||
"draft_payload": {"名称": "乙"},
|
||||
"create_time": dt.datetime(2026, 7, 21, 9, 1, 2, 3000),
|
||||
"deleted": True,
|
||||
"tenant_id": 1,
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"work_id": 8,
|
||||
"source_type": "upgrade_book",
|
||||
"confidence": Decimal("0.90"),
|
||||
"draft_payload": {"名称": "甲"},
|
||||
"create_time": dt.datetime(2026, 7, 21, 9, 1, 1),
|
||||
"deleted": False,
|
||||
"tenant_id": 1,
|
||||
},
|
||||
],
|
||||
"windows": [
|
||||
{"id": 12, "work_id": 8, "window_no": 2, "from_chapter": 2,
|
||||
"to_chapter": 2, "deleted": False, "tenant_id": 1},
|
||||
{"id": 11, "work_id": 8, "window_no": 1, "from_chapter": 1,
|
||||
"to_chapter": 1, "deleted": False, "tenant_id": 1},
|
||||
],
|
||||
"aliases": [{"id": 21, "work_id": 8, "alias": "阿甲", "tenant_id": 1}],
|
||||
"presence": [{"id": 31, "work_id": 8, "name": "路人", "tenant_id": 1}],
|
||||
"card_state": [{"draft_id": 2, "work_id": 8, "watermark_window": 2,
|
||||
"tenant_id": 1}],
|
||||
"audits": [{"id": 41, "draft_id": 2, "window_no": 2, "tenant_id": 1}],
|
||||
"embeddings": [{"id": 51, "draft_id": 2, "embedding": "[0.1,0.20]",
|
||||
"content_hash": "a" * 64, "tenant_id": 1}],
|
||||
}
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, traceback):
|
||||
self.closed = True
|
||||
|
||||
def execute(self, sql, params=None):
|
||||
normalized = " ".join(sql.split()).lower()
|
||||
self.executions.append((normalized, params))
|
||||
if normalized.startswith("begin") or normalized in {"commit", "rollback"}:
|
||||
return _Result([])
|
||||
if "current_database()" in normalized:
|
||||
return _Result([{"database": "muse-example", "server_addr": "100.64.0.8",
|
||||
"server_port": 5433, "server_version": "160000"}])
|
||||
if "from muse_content_chapter c" in normalized:
|
||||
return _Result([
|
||||
{"chapter_id": 1, "chapter_order": 1, "chapter_title": "一",
|
||||
"chapter_revision": 1, "block_id": 101, "block_order": 1,
|
||||
"block_type": "scene", "block_title": None, "content_doc": None,
|
||||
"content_text": "正文一", "block_revision": 1},
|
||||
{"chapter_id": 2, "chapter_order": 2, "chapter_title": "二",
|
||||
"chapter_revision": 1, "block_id": 102, "block_order": 1,
|
||||
"block_type": "scene", "block_title": None, "content_doc": None,
|
||||
"content_text": "正文二", "block_revision": 1},
|
||||
])
|
||||
if "from muse_meta_schema s" in normalized:
|
||||
return _Result([
|
||||
{"schema_key": key, "active_version_id": index,
|
||||
"field_contract_snapshot": {"key": key}}
|
||||
for index, key in enumerate(sorted(backup.UPGRADE_SCHEMA_KEYS), start=1)
|
||||
])
|
||||
if "from muse_content_work" in normalized:
|
||||
return _Result([{"title": "离线测试书"}])
|
||||
table_to_domain = {
|
||||
"from muse_knowledge_draft d": "drafts",
|
||||
"from example_upgrade_window w": "windows",
|
||||
"from example_upgrade_alias a": "aliases",
|
||||
"from example_upgrade_presence p": "presence",
|
||||
"from example_upgrade_card_state s": "card_state",
|
||||
"from example_upgrade_audit a": "audits",
|
||||
"from example_knowledge_embedding e": "embeddings",
|
||||
}
|
||||
for marker, domain in table_to_domain.items():
|
||||
if marker in normalized:
|
||||
return _Result(self.rows[domain])
|
||||
raise AssertionError(f"未覆盖的离线 SQL：{normalized}")
|
||||
|
||||
|
||||
class _RestoreConnection:
|
||||
"""记录恢复事务 SQL；查询默认返回空集合，便于只验证恢复边界。"""
|
||||
|
||||
def __init__(self, sequence_last=100, table_max=50, extra_window=False):
|
||||
self.closed = False
|
||||
self.executions = []
|
||||
self.sequence_last = sequence_last
|
||||
self.table_max = table_max
|
||||
self.extra_window = extra_window
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, traceback):
|
||||
self.closed = True
|
||||
|
||||
def execute(self, sql, params=None):
|
||||
normalized = " ".join(sql.split()).lower()
|
||||
self.executions.append((normalized, params))
|
||||
if "from pg_sequences" in normalized:
|
||||
return _Result([{"last_value": self.sequence_last}])
|
||||
if "select max(id) as max_id" in normalized:
|
||||
return _Result([{"max_id": self.table_max}])
|
||||
if "pg_get_serial_sequence" in normalized:
|
||||
table = params[0] if params else "unknown"
|
||||
return _Result([{"sequence_name": f"public.{table}_id_seq"}])
|
||||
if "from example_upgrade_window w" in normalized and "not (w.id = any" in normalized:
|
||||
return _Result([{"id": 999}] if self.extra_window else [])
|
||||
return _Result([])
|
||||
|
||||
|
||||
def _new_backup_dir():
|
||||
"""返回 /private/tmp 下尚不存在的测试目录。"""
|
||||
|
||||
return pathlib.Path("/private/tmp") / f"backup-upgrade-offline-{uuid.uuid4()}"
|
||||
|
||||
|
||||
def _code_identity():
|
||||
"""离线测试固定代码身份，避免读取工作区状态影响摘要。"""
|
||||
|
||||
return {
|
||||
"gitCommit": "0123456789abcdef",
|
||||
"gitHeadCoversCodeFiles": False,
|
||||
"codeFiles": {
|
||||
"backup_upgrade_work.py": "b" * 64,
|
||||
"parse_upgrade.py": "u" * 64,
|
||||
"parse_llm.py": "p" * 64,
|
||||
"embed_drafts.py": "e" * 64,
|
||||
"upgrade_work_lock.py": "k" * 64,
|
||||
"llm.py": "l" * 64,
|
||||
"parse-book/SKILL.md": "s" * 64,
|
||||
},
|
||||
"headMatches": {},
|
||||
}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _no_lock(*_args, **_kwargs):
|
||||
"""离线测试锁替身，不连接 PostgreSQL。"""
|
||||
|
||||
yield (1, 2)
|
||||
|
||||
|
||||
def _check(name, condition):
|
||||
"""沿用 parse-book 离线测试的简明输出格式。"""
|
||||
|
||||
if not condition:
|
||||
raise AssertionError(name)
|
||||
print(f"[PASS] {name}")
|
||||
|
||||
|
||||
def test_stable_sort_and_json_types():
|
||||
"""相同主键集合不受查询返回顺序影响，特殊数据库类型可稳定 JSON 化。"""
|
||||
|
||||
rows = [
|
||||
{"id": 10, "at": dt.datetime(2026, 7, 21, 8, 0), "score": Decimal("1.2300")},
|
||||
{"id": 2, "at": dt.date(2026, 7, 20), "score": Decimal("0.10")},
|
||||
]
|
||||
ordered = backup.sort_rows("drafts", rows)
|
||||
_check("排序稳定-按域主键升序", [row["id"] for row in ordered] == [2, 10])
|
||||
encoded = json.loads(backup.canonical_json(ordered[1]))
|
||||
_check("JSON类型-datetime转ISO字符串", encoded["at"] == "2026-07-21T08:00:00")
|
||||
_check("JSON类型-Decimal保留精度字符串", encoded["score"] == "1.2300")
|
||||
vector = json.loads(backup.canonical_json({"embedding": [Decimal("0.10"), Decimal("-2.0")]}))
|
||||
_check("JSON类型-vector元素稳定序列化", vector["embedding"] == ["0.10", "-2.0"])
|
||||
|
||||
|
||||
def test_path_guards():
|
||||
"""输出只能是 /private/tmp 下的新目录。"""
|
||||
|
||||
for invalid in ("/tmp/not-explicit", "/private/tmp", "/private/tmp/../etc/backup"):
|
||||
try:
|
||||
backup.validate_new_output_dir(pathlib.Path(invalid))
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError(f"路径越界未拒绝：{invalid}")
|
||||
_check("路径越界-全部拒绝", True)
|
||||
|
||||
existing = _new_backup_dir()
|
||||
existing.mkdir(mode=0o700)
|
||||
try:
|
||||
try:
|
||||
backup.validate_new_output_dir(existing)
|
||||
except FileExistsError:
|
||||
_check("重复目录-拒绝覆盖", True)
|
||||
else:
|
||||
raise AssertionError("已存在目录未拒绝")
|
||||
finally:
|
||||
existing.rmdir()
|
||||
|
||||
|
||||
def test_backup_transaction_queries_and_round_trip():
|
||||
"""七域在单个只读一致性事务中读取，关库后再从磁盘复验。"""
|
||||
|
||||
output = _new_backup_dir()
|
||||
conn = _OfflineConnection()
|
||||
original_verify = backup.verify_backup
|
||||
|
||||
def verify_after_close(path):
|
||||
if not conn.closed:
|
||||
raise AssertionError("数据库连接尚未关闭就开始磁盘复验")
|
||||
return original_verify(path)
|
||||
|
||||
try:
|
||||
with mock.patch.object(backup, "verify_backup", side_effect=verify_after_close):
|
||||
manifest = backup.backup_work(
|
||||
work_id=8,
|
||||
output_dir=output,
|
||||
tenant=1,
|
||||
dsn="postgresql://user:secret@example.invalid/db",
|
||||
connect=lambda *_args, **_kwargs: conn,
|
||||
git_commit="0123456789abcdef",
|
||||
code_identity=_code_identity(),
|
||||
expected_chapters=2,
|
||||
expected_windows=2,
|
||||
lock_context=_no_lock,
|
||||
)
|
||||
|
||||
_check("事务-连接关闭后复验", conn.closed)
|
||||
statements = [sql for sql, _params in conn.executions]
|
||||
_check("事务-仅一个REPEATABLE READ READ ONLY", statements.count(
|
||||
"begin isolation level repeatable read read only") == 1)
|
||||
_check("事务-正常提交一次", statements.count("commit") == 1)
|
||||
_check("事务-无数据库写语句", not any(
|
||||
sql.startswith(("insert ", "update ", "delete ", "alter ", "drop "))
|
||||
for sql in statements
|
||||
))
|
||||
|
||||
_check("七域-文件与清单齐全", set(manifest["domains"]) == set(backup.DOMAIN_SPECS))
|
||||
# 作品标题一条 SELECT，加七域各一条 SELECT；draft 子查询不应被误算成独立域查询。
|
||||
_check("七域与input-各查询只执行一次",
|
||||
sum(sql.startswith("select") for sql in statements) == 11)
|
||||
sql_by_domain = {
|
||||
name: next(sql for sql in statements if f"from {spec.table} " in sql)
|
||||
for name, spec in backup.DOMAIN_SPECS.items()
|
||||
}
|
||||
_check("查询边界-drafts含软删", "d.deleted" not in sql_by_domain["drafts"])
|
||||
_check("查询边界-windows仅active", "w.deleted = false" in sql_by_domain["windows"])
|
||||
_check("查询边界-audits指向目标draft", "a.draft_id in" in sql_by_domain["audits"]
|
||||
and "source_type" in sql_by_domain["audits"])
|
||||
_check("查询边界-embeddings指向目标draft", "e.draft_id in" in sql_by_domain["embeddings"]
|
||||
and "source_type" in sql_by_domain["embeddings"])
|
||||
_check("清单-身份与边界摘要齐全", manifest["work"] == 8
|
||||
and manifest["tenant"] == 1
|
||||
and manifest["title"] == "离线测试书"
|
||||
and len(manifest["windowBoundarySha"]) == 64
|
||||
and len(manifest["inputSha"]) == 64)
|
||||
_check("input-同快照绑定Canonical窗合同代码与计划",
|
||||
manifest["input"]["canonicalContent"]["chapterCount"] == 2
|
||||
and manifest["input"]["windowBoundary"]["count"] == 2
|
||||
and manifest["input"]["activeFieldContracts"]["count"] == 7
|
||||
and manifest["input"]["codeFiles"]["llm.py"] == "l" * 64
|
||||
and manifest["input"]["plan"] == {
|
||||
"model": "MiniMax-M3", "semanticDedup": True})
|
||||
_check("清单-不泄露数据库密码", "secret" not in json.dumps(manifest, ensure_ascii=False))
|
||||
_check("排序稳定-JSONL按主键", [
|
||||
json.loads(line)["id"]
|
||||
for line in (output / "drafts.jsonl").read_text(encoding="utf-8").splitlines()
|
||||
] == [1, 2])
|
||||
_check("原子写-无临时文件残留", not list(output.glob("*.tmp")))
|
||||
_check("离线复验-七域校验通过", original_verify(output)["backup_id"] == manifest["backup_id"])
|
||||
finally:
|
||||
shutil.rmtree(output, ignore_errors=True)
|
||||
|
||||
|
||||
def test_accidental_corruption_detection():
|
||||
"""任一 JSONL 意外损坏后离线 verify 必须失败；不模拟恶意重写摘要。"""
|
||||
|
||||
output = _new_backup_dir()
|
||||
conn = _OfflineConnection()
|
||||
try:
|
||||
backup.backup_work(
|
||||
work_id=8,
|
||||
output_dir=output,
|
||||
tenant=1,
|
||||
dsn="unused",
|
||||
connect=lambda *_args, **_kwargs: conn,
|
||||
git_commit="fedcba9876543210",
|
||||
code_identity=_code_identity(),
|
||||
expected_chapters=2,
|
||||
expected_windows=2,
|
||||
lock_context=_no_lock,
|
||||
)
|
||||
with (output / "aliases.jsonl").open("ab") as stream:
|
||||
stream.write(b'{"id":999}\n')
|
||||
try:
|
||||
backup.verify_backup(output)
|
||||
except backup.VerificationError:
|
||||
_check("意外损坏检测-文件变化被拒绝", True)
|
||||
else:
|
||||
raise AssertionError("意外损坏后的备份仍通过 verify")
|
||||
finally:
|
||||
shutil.rmtree(output, ignore_errors=True)
|
||||
|
||||
|
||||
def test_dict_row_and_tuple_rejection():
|
||||
"""真实连接必须请求 dict_row，tuple 行不得被误转成 dict。"""
|
||||
|
||||
with mock.patch("psycopg.connect") as connect:
|
||||
backup._default_connect("postgresql://example.invalid/db", autocommit=True)
|
||||
from psycopg.rows import dict_row
|
||||
_check("连接-dict_row", connect.call_args.kwargs["row_factory"] is dict_row)
|
||||
|
||||
try:
|
||||
backup.sort_rows("drafts", [(1, 8)])
|
||||
except backup.BackupError:
|
||||
_check("真实rows-tuple拒绝", True)
|
||||
else:
|
||||
raise AssertionError("tuple 数据行被错误接受")
|
||||
|
||||
|
||||
def _restore_manifest():
|
||||
"""构造恢复 SQL 测试所需的最小已验证 manifest。"""
|
||||
|
||||
input_summary = {
|
||||
"databaseIdentity": {"database": "muse-example", "serverAddr": "100.64.0.8",
|
||||
"serverPort": 5433, "serverVersion": "160000"},
|
||||
"canonicalContent": {"chapterCount": 594, "blockCount": 594, "sha": "c" * 64},
|
||||
"windowBoundary": {"count": 116, "sha": "w" * 64},
|
||||
"activeFieldContracts": {"count": 7, "sha": "f" * 64},
|
||||
"codeFiles": {
|
||||
"backup_upgrade_work.py": "b" * 64,
|
||||
"parse_upgrade.py": "u" * 64,
|
||||
"parse_llm.py": "p" * 64,
|
||||
"embed_drafts.py": "e" * 64,
|
||||
"upgrade_work_lock.py": "k" * 64,
|
||||
"llm.py": "l" * 64,
|
||||
"parse-book/SKILL.md": "s" * 64,
|
||||
},
|
||||
"plan": {"model": "MiniMax-M3", "semanticDedup": True},
|
||||
"expectedChapters": 594,
|
||||
"expectedWindows": 116,
|
||||
"sourceType": "upgrade_book",
|
||||
"tenant": 1,
|
||||
"work": 8,
|
||||
}
|
||||
manifest = {
|
||||
"backup_id": "11111111-1111-4111-8111-111111111111",
|
||||
"work": 8,
|
||||
"tenant": 1,
|
||||
"input": input_summary,
|
||||
"inputSha": backup._sha256(backup.canonical_json(input_summary).encode("utf-8")),
|
||||
}
|
||||
confirmation = {
|
||||
"backup_id": manifest["backup_id"],
|
||||
"codeFiles": input_summary["codeFiles"],
|
||||
"inputSha": manifest["inputSha"],
|
||||
"tenant": manifest["tenant"],
|
||||
"work": manifest["work"],
|
||||
}
|
||||
manifest["confirmationSha"] = backup._sha256(
|
||||
backup.canonical_json(confirmation).encode("utf-8")
|
||||
)
|
||||
return manifest
|
||||
|
||||
|
||||
def _restore_rows():
|
||||
"""七域各给一行，驱动 upsert 与清理顺序测试。"""
|
||||
|
||||
return {
|
||||
"drafts": [{"id": 1, "work_id": 8, "source_type": "upgrade_book",
|
||||
"draft_payload": {"名称": "甲"}, "deleted": False, "tenant_id": 1}],
|
||||
"windows": [{"id": 11, "work_id": 8, "window_no": 1, "from_chapter": 1,
|
||||
"to_chapter": 5, "deleted": False, "tenant_id": 1}],
|
||||
"aliases": [{"id": 21, "work_id": 8, "canonical_name": "甲", "alias": "阿甲",
|
||||
"tenant_id": 1}],
|
||||
"presence": [{"id": 31, "work_id": 8, "window_no": 1, "chapter_no": 1,
|
||||
"entity_type": "character", "name": "甲", "tenant_id": 1}],
|
||||
"card_state": [{"draft_id": 1, "work_id": 8, "watermark_window": 1,
|
||||
"tenant_id": 1}],
|
||||
"audits": [{"id": 41, "draft_id": 1, "window_no": 1, "field_name": "名称",
|
||||
"tenant_id": 1}],
|
||||
"embeddings": [{"id": 51, "draft_id": 1, "content_hash": "a" * 64,
|
||||
"embed_text": "甲", "model": "Qwen", "dimensions": 2,
|
||||
"embedding": "[0.1,0.2]", "tenant_id": 1}],
|
||||
}
|
||||
|
||||
|
||||
def test_input_identity_and_drift_guard():
|
||||
"""input 必须绑定完整身份，任一 Canonical 输入漂移均拒绝恢复。"""
|
||||
|
||||
manifest = _restore_manifest()
|
||||
expected = manifest["input"]
|
||||
_check("input-模型与语义判重绑定", expected["plan"] == {
|
||||
"model": "MiniMax-M3", "semanticDedup": True})
|
||||
_check("input-594章与116窗绑定", expected["canonicalContent"]["chapterCount"] == 594
|
||||
and expected["windowBoundary"]["count"] == 116)
|
||||
changed = json.loads(json.dumps(expected))
|
||||
changed["canonicalContent"]["sha"] = "d" * 64
|
||||
try:
|
||||
backup.assert_restore_input(manifest, changed, work_id=8, tenant=1)
|
||||
except backup.BackupError:
|
||||
_check("input漂移-拒绝恢复", True)
|
||||
else:
|
||||
raise AssertionError("Canonical input 漂移未拒绝")
|
||||
wrong_database = json.loads(json.dumps(expected))
|
||||
wrong_database["databaseIdentity"]["database"] = "other-db"
|
||||
try:
|
||||
backup.assert_restore_input(manifest, wrong_database, work_id=8, tenant=1)
|
||||
except backup.BackupError:
|
||||
_check("数据库identity漂移-拒绝恢复", True)
|
||||
else:
|
||||
raise AssertionError("数据库 identity 不符未拒绝")
|
||||
for work_id, tenant in ((9, 1), (8, 2)):
|
||||
try:
|
||||
backup.assert_restore_input(manifest, expected, work_id=work_id, tenant=tenant)
|
||||
except backup.BackupError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("work/tenant 不符未拒绝")
|
||||
_check("恢复范围-work与tenant不符拒绝", True)
|
||||
|
||||
|
||||
def test_restore_sql_boundaries_and_rehearse_rollback():
|
||||
"""共享恢复内核按七域边界操作，演练无条件回滚。"""
|
||||
|
||||
manifest = _restore_manifest()
|
||||
rows = _restore_rows()
|
||||
conn = _RestoreConnection()
|
||||
with mock.patch.object(backup, "verify_backup", return_value=manifest), \
|
||||
mock.patch.object(backup, "read_backup_rows", return_value=rows), \
|
||||
mock.patch.object(backup, "capture_input_snapshot", return_value=manifest["input"]), \
|
||||
mock.patch.object(backup, "verify_restored_domains"):
|
||||
backup.restore_backup(
|
||||
pathlib.Path("/private/tmp/offline-restore"), work_id=8, tenant=1,
|
||||
dsn="unused", connect=lambda *_args, **_kwargs: conn, execute=False,
|
||||
confirmation_sha=manifest["confirmationSha"], lock_context=_no_lock,
|
||||
)
|
||||
statements = [sql for sql, _params in conn.executions]
|
||||
_check("rehearse-强制ROLLBACK", statements[-1] == "rollback" and "commit" not in statements)
|
||||
_check("exact恢复-事务局部关闭更新时间触发器",
|
||||
"set local session_replication_role to replica" in statements)
|
||||
_check("恢复边界-额外卡软删", any(
|
||||
sql.startswith("update muse_knowledge_draft") and "not (d.id = any" in sql
|
||||
for sql in statements
|
||||
))
|
||||
_check("恢复边界-额外向量软删", any(
|
||||
sql.startswith("update example_knowledge_embedding") and "not (e.id = any" in sql
|
||||
for sql in statements
|
||||
))
|
||||
for table in ("example_upgrade_alias", "example_upgrade_presence",
|
||||
"example_upgrade_card_state", "example_upgrade_audit"):
|
||||
if not any(sql.startswith(f"delete from {table}") for sql in statements):
|
||||
raise AssertionError(f"恢复前未清理 {table}")
|
||||
_check("恢复边界-四域先清再插", True)
|
||||
_check("恢复边界-windows拒绝额外active", any(
|
||||
"from example_upgrade_window" in sql and "not (w.id = any" in sql
|
||||
for sql in statements
|
||||
))
|
||||
_check("恢复边界-七域均有upsert", sum("on conflict" in sql for sql in statements) == 7)
|
||||
|
||||
rejected = _RestoreConnection(extra_window=True)
|
||||
try:
|
||||
backup.apply_restore_kernel(rejected, manifest, rows)
|
||||
except backup.BackupError:
|
||||
_check("恢复边界-实际拒绝额外active窗且写前失败", not any(
|
||||
sql.startswith(("insert ", "update ", "delete "))
|
||||
for sql, _params in rejected.executions
|
||||
))
|
||||
else:
|
||||
raise AssertionError("额外 active 窗未阻断恢复")
|
||||
|
||||
|
||||
def test_restore_execute_confirmation_and_sequence_forward_only():
|
||||
"""execute 必须双确认；sequence 当前值更大时不得 setval 回退。"""
|
||||
|
||||
manifest = _restore_manifest()
|
||||
for supplied in (None, "22222222-2222-4222-8222-222222222222"):
|
||||
try:
|
||||
backup.validate_execute_confirmation(
|
||||
True, supplied, manifest["backup_id"],
|
||||
manifest["confirmationSha"], manifest["confirmationSha"],
|
||||
)
|
||||
except backup.BackupError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("execute 缺少匹配 backup_id 仍被允许")
|
||||
backup.validate_execute_confirmation(
|
||||
True, manifest["backup_id"], manifest["backup_id"],
|
||||
manifest["confirmationSha"], manifest["confirmationSha"],
|
||||
)
|
||||
try:
|
||||
backup.validate_execute_confirmation(
|
||||
False, None, manifest["backup_id"], "wrong", manifest["confirmationSha"]
|
||||
)
|
||||
except backup.BackupError:
|
||||
_check("confirmation确认-rehearse也必须精确匹配", True)
|
||||
else:
|
||||
raise AssertionError("rehearse confirmationSha 不匹配仍被允许")
|
||||
_check("execute确认-必须显式flag与匹配backup_id", True)
|
||||
|
||||
conn = _RestoreConnection(sequence_last=100, table_max=50)
|
||||
backup.advance_sequences_forward(conn)
|
||||
_check("序列-不回退", not any("setval" in sql for sql, _params in conn.executions))
|
||||
|
||||
forward = _RestoreConnection(sequence_last=100, table_max=150)
|
||||
backup.advance_sequences_forward(forward)
|
||||
_check("序列-仅在需要时向前", sum(
|
||||
sql.startswith("select setval") for sql, _params in forward.executions
|
||||
) == len(backup.SEQUENCE_TABLES))
|
||||
|
||||
execute_conn = _RestoreConnection()
|
||||
with mock.patch.object(backup, "verify_backup", return_value=manifest), \
|
||||
mock.patch.object(backup, "read_backup_rows", return_value=_restore_rows()), \
|
||||
mock.patch.object(backup, "capture_input_snapshot", return_value=manifest["input"]), \
|
||||
mock.patch.object(backup, "verify_restored_domains"), \
|
||||
mock.patch.object(backup, "advance_sequences_forward"):
|
||||
backup.restore_backup(
|
||||
pathlib.Path("/private/tmp/offline-restore"), work_id=8, tenant=1,
|
||||
dsn="unused", connect=lambda *_args, **_kwargs: execute_conn, execute=True,
|
||||
backup_id=manifest["backup_id"], code_identity=_code_identity(),
|
||||
confirmation_sha=manifest["confirmationSha"], lock_context=_no_lock,
|
||||
)
|
||||
execute_statements = [sql for sql, _params in execute_conn.executions]
|
||||
_check("execute确认-匹配后才COMMIT", execute_statements[-1] == "commit")
|
||||
|
||||
|
||||
def test_expected_scope_code_dependencies_and_lock_fail_closed():
|
||||
"""期望范围、依赖代码与同书锁均进入机械门。"""
|
||||
|
||||
chapters = [
|
||||
{"chapter_id": 1, "chapter_order": 1, "block_id": 101, "content_text": "甲"},
|
||||
{"chapter_id": 2, "chapter_order": 2, "block_id": 102, "content_text": "乙"},
|
||||
]
|
||||
windows = [
|
||||
{"id": 11, "window_no": 1, "from_chapter": 1, "to_chapter": 1},
|
||||
{"id": 12, "window_no": 2, "from_chapter": 2, "to_chapter": 2},
|
||||
]
|
||||
backup.validate_expected_scope(chapters, windows, expected_chapters=2, expected_windows=2)
|
||||
_check("expected-章节窗口连续且每章非空", True)
|
||||
for bad_chapters, bad_windows in (
|
||||
(chapters[:1], windows),
|
||||
([dict(chapters[0], content_text=""), chapters[1]], windows),
|
||||
(chapters, [windows[0], dict(windows[1], window_no=3)]),
|
||||
(chapters, [windows[0], dict(windows[1], from_chapter=3)]),
|
||||
):
|
||||
try:
|
||||
backup.validate_expected_scope(
|
||||
bad_chapters, bad_windows, expected_chapters=2, expected_windows=2
|
||||
)
|
||||
except backup.BackupError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("expected 范围异常未拒绝")
|
||||
_check("expected-数量非空与连续性异常均拒绝", True)
|
||||
|
||||
identity = backup.capture_code_identity("HEAD")
|
||||
_check("代码身份-补齐实际依赖SHA", {
|
||||
"parse_llm.py", "embed_drafts.py", "upgrade_work_lock.py"
|
||||
}.issubset(identity["codeFiles"]))
|
||||
|
||||
business_calls = {"count": 0}
|
||||
|
||||
@contextmanager
|
||||
def unavailable_lock(*_args, **_kwargs):
|
||||
raise backup.UpgradeWorkLockUnavailable(1, 8)
|
||||
yield
|
||||
|
||||
def forbidden_connect(*_args, **_kwargs):
|
||||
business_calls["count"] += 1
|
||||
raise AssertionError("锁失败后仍创建业务连接")
|
||||
|
||||
output = _new_backup_dir()
|
||||
try:
|
||||
backup.backup_work(
|
||||
8, output, tenant=1, dsn="unused", connect=forbidden_connect,
|
||||
code_identity=_code_identity(), expected_chapters=594, expected_windows=116,
|
||||
lock_context=unavailable_lock,
|
||||
)
|
||||
except backup.UpgradeWorkLockUnavailable:
|
||||
_check("advisory-lock-失败时零业务SQL", business_calls["count"] == 0)
|
||||
else:
|
||||
raise AssertionError("同书锁失败未阻断 backup")
|
||||
|
||||
with mock.patch.object(backup, "verify_backup") as offline_verify:
|
||||
try:
|
||||
backup.restore_backup(
|
||||
pathlib.Path("/private/tmp/not-read"), work_id=8, tenant=1, dsn="unused",
|
||||
connect=forbidden_connect, confirmation_sha="unused",
|
||||
lock_context=unavailable_lock,
|
||||
)
|
||||
except backup.UpgradeWorkLockUnavailable:
|
||||
_check("advisory-lock-恢复演练也在离线verify前阻断", not offline_verify.called)
|
||||
else:
|
||||
raise AssertionError("同书锁失败未阻断 restore/rehearse")
|
||||
|
||||
|
||||
def main():
|
||||
"""顺序运行全部离线测试，便于直接以 Python 执行。"""
|
||||
|
||||
test_stable_sort_and_json_types()
|
||||
test_path_guards()
|
||||
test_backup_transaction_queries_and_round_trip()
|
||||
test_accidental_corruption_detection()
|
||||
test_dict_row_and_tuple_rejection()
|
||||
test_input_identity_and_drift_guard()
|
||||
test_restore_sql_boundaries_and_rehearse_rollback()
|
||||
test_restore_execute_confirmation_and_sequence_forward_only()
|
||||
test_expected_scope_code_dependencies_and_lock_fail_closed()
|
||||
print("\n全部离线自测通过：49 项（未连库、未发任何 LLM 调用）")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -16,6 +16,7 @@
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
from contextlib import nullcontext
|
||||
from copy import deepcopy
|
||||
from unittest.mock import patch
|
||||
|
||||
@ -628,7 +629,8 @@ def test_run_redo_preflight_rejects_unsafe_targets():
|
||||
text = "\n".join(str(chapter_texts.get(chapter, "")) for chapter in range(a, b + 1))
|
||||
return text, chapter_texts
|
||||
|
||||
with patch.object(pu.psycopg, "connect", side_effect=fake_connect), \
|
||||
with patch.object(pu, "upgrade_work_lock", return_value=nullcontext()), \
|
||||
patch.object(pu.psycopg, "connect", side_effect=fake_connect), \
|
||||
patch.object(pu, "load_entity_contracts", return_value=_fake_contracts()), \
|
||||
patch.object(pu, "load_window_material", side_effect=fake_material), \
|
||||
patch.object(pu, "_snapshot_redo_window", side_effect=lambda *args: snapshot_calls.append(args)), \
|
||||
@ -714,7 +716,7 @@ def _run_with_failures(*, max_calls, failures):
|
||||
patch.object(pu, "m3_json", side_effect=fake_m3_json), \
|
||||
patch.object(pu, "_snapshot_redo_window", return_value=snapshot), \
|
||||
patch.object(pu, "_restore_redo_window", side_effect=fake_restore):
|
||||
pu.run.callback(
|
||||
pu._run(
|
||||
work_id=8,
|
||||
max_windows=0,
|
||||
max_calls=max_calls,
|
||||
@ -850,7 +852,7 @@ def test_run_presence_paths_accept_numeric_strings():
|
||||
return_value=({"安若雪": (701, "character", "旧摘要")}, {}, {}),
|
||||
), \
|
||||
patch.object(pu, "m3_json", side_effect=fake_m3_json):
|
||||
pu.run.callback(
|
||||
pu._run(
|
||||
work_id=8,
|
||||
max_windows=0,
|
||||
max_calls=0,
|
||||
|
||||
@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""升格命令锁接线的纯离线测试：锁失败时禁止任何业务调用。"""
|
||||
|
||||
import contextlib
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
|
||||
SCRIPT_DIR = pathlib.Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
import parse_upgrade as parse # noqa: E402
|
||||
import reset_upgrade_work as reset # noqa: E402
|
||||
from upgrade_work_lock import UpgradeWorkLockUnavailable # noqa: E402
|
||||
|
||||
|
||||
def _lock_failure(*_args, **_kwargs):
|
||||
"""模拟同书已有命令持锁。"""
|
||||
|
||||
raise UpgradeWorkLockUnavailable(tenant_id=1, work_id=8)
|
||||
|
||||
|
||||
class UpgradeCommandLockOfflineTest(unittest.TestCase):
|
||||
"""验证锁位于全部业务 SQL、写入和模型调用之前。"""
|
||||
|
||||
def setUp(self):
|
||||
self.runner = CliRunner()
|
||||
|
||||
def test_parse_run_lock_failure_skips_business(self):
|
||||
with patch.object(parse, "upgrade_work_lock", side_effect=_lock_failure) as lock, \
|
||||
patch.object(parse, "_run") as business:
|
||||
result = self.runner.invoke(parse.cli, ["run", "--work-id", "8"])
|
||||
|
||||
self.assertNotEqual(result.exit_code, 0)
|
||||
self.assertIn("已有升格命令正在处理", result.output)
|
||||
lock.assert_called_once_with(parse.DSN, parse.TENANT, 8)
|
||||
business.assert_not_called()
|
||||
|
||||
def test_parse_windows_lock_failure_skips_database(self):
|
||||
with patch.object(parse, "upgrade_work_lock", side_effect=_lock_failure) as lock, \
|
||||
patch.object(parse.psycopg, "connect") as connect:
|
||||
result = self.runner.invoke(parse.cli, ["windows", "--work-id", "8"])
|
||||
|
||||
self.assertNotEqual(result.exit_code, 0)
|
||||
self.assertIn("已有升格命令正在处理", result.output)
|
||||
lock.assert_called_once_with(parse.DSN, parse.TENANT, 8)
|
||||
connect.assert_not_called()
|
||||
|
||||
def test_parse_status_does_not_request_lock(self):
|
||||
lock = Mock(side_effect=AssertionError("status 不应请求锁"))
|
||||
with patch.object(parse, "upgrade_work_lock", lock), \
|
||||
patch.object(parse.psycopg, "connect", side_effect=RuntimeError("离线中止")):
|
||||
result = self.runner.invoke(parse.cli, ["status", "--work-id", "8"])
|
||||
|
||||
self.assertNotEqual(result.exit_code, 0)
|
||||
lock.assert_not_called()
|
||||
|
||||
def test_reset_preview_lock_failure_skips_database(self):
|
||||
self._assert_reset_lock_failure_skips_database(execute=False)
|
||||
|
||||
def test_reset_execute_lock_failure_skips_database(self):
|
||||
self._assert_reset_lock_failure_skips_database(execute=True)
|
||||
|
||||
def _assert_reset_lock_failure_skips_database(self, *, execute):
|
||||
args = ["--work-id", "8"] + (["--execute"] if execute else [])
|
||||
with patch.object(reset, "upgrade_work_lock", side_effect=_lock_failure) as lock, \
|
||||
patch.object(reset.psycopg, "connect") as connect:
|
||||
result = self.runner.invoke(reset.main, args)
|
||||
|
||||
self.assertNotEqual(result.exit_code, 0)
|
||||
self.assertIn("已有升格命令正在处理", result.output)
|
||||
lock.assert_called_once_with(reset.DSN, reset.TENANT, 8)
|
||||
connect.assert_not_called()
|
||||
|
||||
def test_lock_success_wraps_parse_run_business(self):
|
||||
events = []
|
||||
|
||||
@contextlib.contextmanager
|
||||
def fake_lock(*_args, **_kwargs):
|
||||
events.append("lock-enter")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
events.append("lock-exit")
|
||||
|
||||
def fake_business(*_args, **_kwargs):
|
||||
events.append("business")
|
||||
|
||||
with patch.object(parse, "upgrade_work_lock", fake_lock), \
|
||||
patch.object(parse, "_run", side_effect=fake_business):
|
||||
result = self.runner.invoke(parse.cli, ["run", "--work-id", "8"])
|
||||
|
||||
self.assertEqual(result.exit_code, 0, result.output)
|
||||
self.assertEqual(events, ["lock-enter", "business", "lock-exit"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env python3
|
||||
"""升格作品会话锁的纯离线测试：不连接数据库，不调用模型。"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
|
||||
SCRIPT_DIR = pathlib.Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
from upgrade_work_lock import ( # noqa: E402
|
||||
UpgradeWorkLockUnavailable,
|
||||
advisory_lock_keys,
|
||||
upgrade_work_lock,
|
||||
)
|
||||
|
||||
|
||||
class _Result:
|
||||
"""模拟 psycopg 查询结果。"""
|
||||
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
def fetchone(self):
|
||||
return (self.value,)
|
||||
|
||||
|
||||
class _LockServer:
|
||||
"""在内存中模拟 PostgreSQL 会话级 advisory lock。"""
|
||||
|
||||
def __init__(self):
|
||||
self.held = {}
|
||||
self.connections = []
|
||||
|
||||
def connect(self, dsn, *, autocommit):
|
||||
connection = _LockConnection(self, dsn, autocommit)
|
||||
self.connections.append(connection)
|
||||
return connection
|
||||
|
||||
|
||||
class _LockConnection:
|
||||
"""每个实例代表一条独立数据库会话，只接受锁相关 SQL。"""
|
||||
|
||||
def __init__(self, server, dsn, autocommit):
|
||||
self.server = server
|
||||
self.dsn = dsn
|
||||
self.autocommit = autocommit
|
||||
self.owned = set()
|
||||
self.statements = []
|
||||
self.closed = False
|
||||
|
||||
def execute(self, sql, params):
|
||||
normalized = " ".join(sql.split())
|
||||
key = tuple(params)
|
||||
self.statements.append((normalized, key))
|
||||
if normalized == "SELECT pg_try_advisory_lock(%s, %s)":
|
||||
owner = self.server.held.get(key)
|
||||
if owner is not None and owner is not self:
|
||||
return _Result(False)
|
||||
self.server.held[key] = self
|
||||
self.owned.add(key)
|
||||
return _Result(True)
|
||||
if normalized == "SELECT pg_advisory_unlock(%s, %s)":
|
||||
owned = key in self.owned
|
||||
if owned:
|
||||
self.owned.remove(key)
|
||||
self.server.held.pop(key, None)
|
||||
return _Result(owned)
|
||||
raise AssertionError(f"锁连接禁止执行业务 SQL：{normalized}")
|
||||
|
||||
def close(self):
|
||||
for key in list(self.owned):
|
||||
self.server.held.pop(key, None)
|
||||
self.owned.clear()
|
||||
self.closed = True
|
||||
|
||||
|
||||
class UpgradeWorkLockOfflineTest(unittest.TestCase):
|
||||
"""覆盖锁键稳定性、互斥语义和 finally 释放。"""
|
||||
|
||||
def test_keys_are_stable_signed_int32_pair(self):
|
||||
first = advisory_lock_keys(tenant_id=1, work_id=8)
|
||||
second = advisory_lock_keys(tenant_id=1, work_id=8)
|
||||
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(len(first), 2)
|
||||
self.assertTrue(all(-(2 ** 31) <= value < 2 ** 31 for value in first))
|
||||
self.assertNotEqual(first, advisory_lock_keys(tenant_id=1, work_id=9))
|
||||
self.assertNotEqual(first, advisory_lock_keys(tenant_id=2, work_id=8))
|
||||
|
||||
def test_same_work_concurrent_session_is_rejected(self):
|
||||
server = _LockServer()
|
||||
|
||||
with upgrade_work_lock("postgresql://offline", 1, 8, connect=server.connect):
|
||||
with self.assertRaises(UpgradeWorkLockUnavailable):
|
||||
with upgrade_work_lock("postgresql://offline", 1, 8, connect=server.connect):
|
||||
self.fail("同书第二条会话不得进入受保护区")
|
||||
|
||||
self.assertEqual(server.held, {})
|
||||
self.assertTrue(all(connection.closed for connection in server.connections))
|
||||
|
||||
def test_exception_releases_lock_and_closes_autocommit_session(self):
|
||||
server = _LockServer()
|
||||
|
||||
with self.assertRaisesRegex(RuntimeError, "业务失败"):
|
||||
with upgrade_work_lock("postgresql://offline", 1, 8, connect=server.connect):
|
||||
raise RuntimeError("业务失败")
|
||||
|
||||
self.assertEqual(server.held, {})
|
||||
self.assertEqual(len(server.connections), 1)
|
||||
connection = server.connections[0]
|
||||
self.assertTrue(connection.autocommit)
|
||||
self.assertTrue(connection.closed)
|
||||
self.assertEqual(
|
||||
[sql for sql, _ in connection.statements],
|
||||
["SELECT pg_try_advisory_lock(%s, %s)", "SELECT pg_advisory_unlock(%s, %s)"],
|
||||
)
|
||||
|
||||
def test_different_work_does_not_conflict(self):
|
||||
server = _LockServer()
|
||||
entered = []
|
||||
|
||||
with upgrade_work_lock("postgresql://offline", 1, 8, connect=server.connect):
|
||||
with upgrade_work_lock("postgresql://offline", 1, 9, connect=server.connect):
|
||||
entered.append(True)
|
||||
|
||||
self.assertEqual(entered, [True])
|
||||
self.assertEqual(server.held, {})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
65
.claude/skills/parse-book/scripts/upgrade_work_lock.py
Normal file
65
.claude/skills/parse-book/scripts/upgrade_work_lock.py
Normal file
@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""升格作品命令共用的 PostgreSQL 会话级 advisory lock。
|
||||
|
||||
锁连接是独立 autocommit 会话，只执行加锁和解锁 SQL，不参与任何业务事务。调用方可让该
|
||||
context manager 跨越 LLM/嵌入调用持续存活；退出受保护区时，无论成功或异常都会显式解锁并
|
||||
关闭会话。backup 的 restore/rehearse 可直接导入 ``upgrade_work_lock`` 复用同一互斥边界。
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import struct
|
||||
from contextlib import contextmanager
|
||||
|
||||
import psycopg
|
||||
|
||||
|
||||
LOCK_NAMESPACE = b"muse:upgrade-work-lock:v1"
|
||||
|
||||
|
||||
class UpgradeWorkLockUnavailable(RuntimeError):
|
||||
"""同租户、同作品已有另一条升格命令持锁。"""
|
||||
|
||||
def __init__(self, tenant_id, work_id):
|
||||
self.tenant_id = tenant_id
|
||||
self.work_id = work_id
|
||||
super().__init__(
|
||||
f"作品 tenant={tenant_id} work={work_id} 已有升格命令正在处理；"
|
||||
"本次未执行任何业务操作"
|
||||
)
|
||||
|
||||
|
||||
def advisory_lock_keys(tenant_id, work_id):
|
||||
"""把租户与作品稳定映射为 PostgreSQL 接受的两个有符号 int32 锁键。"""
|
||||
|
||||
identity = f"{int(tenant_id)}:{int(work_id)}".encode("ascii")
|
||||
digest = hashlib.sha256(LOCK_NAMESPACE + b"\0" + identity).digest()
|
||||
return struct.unpack(">ii", digest[:8])
|
||||
|
||||
|
||||
@contextmanager
|
||||
def upgrade_work_lock(dsn, tenant_id, work_id, *, connect=None):
|
||||
"""非阻塞获取同书会话锁，失败立即抛错，finally 中释放并关闭专用连接。
|
||||
|
||||
``connect`` 仅供离线测试注入；生产默认使用 ``psycopg.connect``。返回的两个锁键可用于
|
||||
调试记录，调用方不得使用该专用连接执行任何业务 SQL。
|
||||
"""
|
||||
|
||||
connector = connect or psycopg.connect
|
||||
keys = advisory_lock_keys(tenant_id, work_id)
|
||||
connection = connector(dsn, autocommit=True)
|
||||
acquired = False
|
||||
try:
|
||||
row = connection.execute(
|
||||
"SELECT pg_try_advisory_lock(%s, %s)", keys
|
||||
).fetchone()
|
||||
acquired = bool(row and row[0] is True)
|
||||
if not acquired:
|
||||
raise UpgradeWorkLockUnavailable(tenant_id, work_id)
|
||||
yield keys
|
||||
finally:
|
||||
try:
|
||||
if acquired:
|
||||
connection.execute("SELECT pg_advisory_unlock(%s, %s)", keys)
|
||||
finally:
|
||||
# 即使显式解锁 SQL 失败，关闭会话也会由 PostgreSQL 释放会话级锁。
|
||||
connection.close()
|
||||
Loading…
x
Reference in New Issue
Block a user