feat(p1r): add Dify S1 live acceptance

This commit is contained in:
zizi 2026-07-07 17:10:40 +08:00
parent 13b305d4da
commit 6f2629302f
9 changed files with 651 additions and 13 deletions

View File

@ -6,21 +6,26 @@
| 依赖 | 地址 | 关键事实 |
|---|---|---|
| New-API(LLM 网关) | `http://100.64.0.8:3000`,OpenAI 兼容 `/v1/chat/completions` | 已验收模型 `MiniMax-M2.5`;容器 `new-api` 端口 3000;其 DB 在 `infra-postgres`(宿主 `100.64.0.8:5433`) |
| RAGFlow(知识运行时) | `http://100.64.0.8`(及 `:9380`) | `/api/v1/datasets``/api/v1/retrieval`;健康检查 `/v1/system/healthz` |
| RAGFlow(1.0 历史知识运行时) | `http://100.64.0.8`(及 `:9380`) | `/api/v1/datasets``/api/v1/retrieval`;健康检查 `/v1/system/healthz`;2.0 下线由用户手工执行,不阻塞 S1 |
| Dify 1.15.0(2.0 单人版目标外部服务) | `http://100.64.0.8:18080` | 官方 compose 部署在 `mini-infra:/home/qingse/dify-1.15.0/docker`,compose project=`muse-dify`;API health 返回 `version=1.15.0` |
| 开发 PG / Redis | 见下方凭据来源 | 远端 PG **15**(用户确认可用,不强制 PG16);Redis `100.64.0.8:6379` |
**Dify 状态(2026-06-28 盘点)**:当前 Muse 1.0.0 运行链路没有接入 Dify,infra 主机也未部署 Dify 相关容器/端口。仓内只剩设计映射与示例测试提及 Dify;因此它不是当前生产运行时依赖。若产品后续要求 Dify 工作流能力,应按新集成从契约、配置、失败闭环、live 验收补齐,不能把它当成已存在基础设施
**Dify 状态(2026-07-07 S1)**:Dify 已作为 2.0 单人版目标外部服务完成基础设施部署与 live 契约验收,但生产运行时切换仍按 S2+ 后续步骤推进,不要把 S1 等同于 Muse runtime 已全面改走 Dify。当前 `muse-dify` 是一套独立基础设施栈,包含自己的 `db/redis/weaviate/nginx/sandbox/plugin_daemon/worker/web` 等容器;它不复用、不替换 Muse 既有 PostgreSQL/Redis
**凭据来源(明文不入库,只记位置)**:
- 外部验收:`muse-cloud/scripts/dev/p1r-external-acceptance.env`(`set -a; . 该文件; set +a` 加载;含 New-API base/token、`MUSE_AI_NEW_API_DEFAULT_MODEL_KEY=MiniMax-M2.5`、RAGFlow base/key、GraphRAG 开关)。
- 开发基础设施:`~/.config/muse-repo/infra.env`(PG/Redis 连接 + 真实凭据)。
- ⚠️ **安全**:上述令牌仅限当前内网验收上下文;**仓库若同步到更大范围,必须先轮换 New-API 令牌**。
- Dify 控制台/模型/Datasets 凭据:`mini-infra:/home/qingse/.config/muse-repo/dify.env` 与本仓 `muse-cloud/scripts/dev/p1r-external-acceptance.env``MUSE_AI_DIFY_*``MUSE_KNOWLEDGE_DIFY_*` 块。仓内 env 按项目内网约定用于验收,日志与文档只输出长度/hash 前缀。
- ⚠️ **安全**:上述令牌仅限当前内网验收上下文;本轮 agent 工具输出曾出现 New-API 与 Dify 明文 key,**用户需手工轮换 New-API token 与 Dify app/dataset keys**。
**Live 验收 harness(opt-in,默认跳过,`MUSE_P1R_EXTERNAL_ACCEPTANCE=true` 才真跑)**:
- `muse-module-ai/.../application/muse/facade/P1rNewApiLiveAcceptanceIT.java`(New-API)
- `muse-module-ai/.../application/muse/P1rImportLlmNewApiLiveAcceptanceIT.java`(New-API 完整导入 LLM 解析)
- `muse-module-knowledge/.../application/muse/facade/P1rRagFlowLiveAcceptanceIT.java`(RAGFlow)
- `muse-module-ai/.../application/muse/facade/P1rDifyChatLiveAcceptanceIT.java`(Dify chat app)
- `muse-module-knowledge/.../application/muse/facade/P1rDifyDatasetsContractLiveIT.java`(Dify Datasets create/upload/index/retrieve)
- 另:`muse-server/.../framework/api/P1rAiRuntimeEndToEndLiveAcceptanceIT.java``P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT.java`
- 输出脱敏(只打 endpoint/模型/id/状态/耗时/usage/key 长度 + sha256 前 12);RAGFlow smoke 会留存 `p1r-live-<时间戳>` dataset 不删。
- 输出脱敏(只打 endpoint/模型/id/状态/耗时/usage/key 长度 + sha256 前 12);RAGFlow smoke 会留存 `p1r-live-<时间戳>` dataset 不删,Dify Datasets smoke 会留存 `dify-contract-<时间戳>` dataset 不删
- **runtime client live ≠ coverage completed**:仍需 Muse API 触发 + 落库 + 审计 + 失败路径证据(见 [golden-journey](../skills/golden-journey-vertical-slice.md))。
## 二、外部集成兼容坑(实证)
@ -31,6 +36,12 @@
- **RAGFlow 检索 metadata 字段名**:RAGFlow 官方契约是 `metadata_condition`。2026-06-28 已将 Muse HTTP 客户端从旧的 `metadata_filter` 修为 `metadata_condition`,并用请求体单测防回退。即便字段已修,跨可见性安全隔离仍不得依赖运行时 metadata 过滤,market KB 继续使用公开副本物理隔离。
- **GraphRAG attribution**:默认 fail-closed(`ATTRIBUTION_NOT_CONFIGURED`);真跑需 `MUSE_KNOWLEDGE_RAGFLOW_GRAPHRAG_ATTRIBUTION_READY=true` + `..._GRAPHRAG_DATASET_ID`
- **New-API/RAGFlow Java 客户端代理污染**:2026-06-28 盘点时,未清代理的 Java/Python 探针曾返回 502,但 `curl --noproxy '*'` 与清代理 live IT 均证明 New-API/RAGFlow 可用。运行时代码已在 New-API 文本生成、导入解析、RAGFlow HTTP 客户端上显式使用 direct `ProxySelector`,避免 macOS/环境代理把 tailnet 内网请求转到本地代理导致假故障。
- **Dify 官方 compose 钉版**:S1 使用 `langgenius/dify` 官方仓库 tag `1.15.0``docker/` 目录,远端源码备份在 `/home/qingse/dify-1.15.0`,compose project 固定为 `muse-dify`。控制台初始化后 `/console/api/setup` 返回 `step=finished`,API health 返回 `{"status":"ok","version":"1.15.0"}`
- **Dify 存储类型坑**:官方 `.env` 默认 `STORAGE_TYPE=opendal` + `OPENDAL_FS_ROOT=storage` 在当前 compose 部署下会出现 API 上传成功、worker 索引时报 `File not found`。已改为 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage` 并重启 `api/api_websocket/worker/worker_beat/nginx`;回滚备份为 `mini-infra:/home/qingse/dify-1.15.0/docker/.env.s1-storage-before-local`
- **Dify 模型 provider**:通过 marketplace 插件 `langgenius/openai_api_compatible/openai_api_compatible` 连接 New-API。`MiniMax-M2.5` 作为 LLM,`Qwen/Qwen3-Embedding-8B` 作为 text-embedding,`Qwen/Qwen3-Reranker-8B` 作为 rerank;创建 high_quality dataset 前必须先配置 text-embedding,否则 dataset/create 会缺默认 embedding。
- **Dify Datasets key 类型**:工作区级 key 从 `/console/api/datasets/api-keys` 创建,前缀 `dataset-`,可调用 `/v1/datasets``/v1/datasets/{id}/document/create-by-file``/v1/datasets/{id}/retrieve`;单 dataset key 从 `/console/api/datasets/{id}/api-keys` 创建,前缀通常为 `ds-`,用于限定某个 dataset。S1 执行计划要求的是工作区级 Datasets API key。app key 前缀 `app-`,只能打 app/workflow API,不能打 Datasets。
- **Dify Datasets retrieve 契约**:Dify 1.15 的 `/v1/datasets/{id}/retrieve``retrieval_model` 中必填 `search_method``reranking_enable`;只传 `top_k/score_threshold_enabled` 会 400。S1 live IT 固定 `search_method=semantic_search``reranking_enable=false`
- **Dify 控制台 API 坑**:`INIT_PASSWORD` 最大长度 30;控制台登录 password 字段实际传 base64(password),不是 RSA;控制台 API 写操作需 cookie + `X-CSRF-Token`,不要拿 app/dataset key 调控制台端点。
## 三、前端 / 构建坑速查
- **studio(Vite + React + TS6.0)**:`erasableSyntaxOnly` 禁用 constructor 参数属性;`exactOptionalPropertyTypes` 下可选属性需规避写法;采纳/Diff 用 **Coarse-to-Fine 分级 Diff** 规避 3000+ 字正文上 O(N×M) LCS 性能爆炸;MSW 仅 `import.meta.env.DEV` 启用。

View File

@ -25,6 +25,8 @@
**2.0.0 单人版改造 S0 门禁基线(2026-07-07)**:已坐实并修复 local 覆盖门基线红项。初跑 `bash muse-cloud/scripts/run-p1r-verification.sh local` 先暴露本机 Maven 绑定 JDK17 与沙箱写 `~/.m2` 的环境问题;显式 `JAVA_HOME=/Users/qingse/Library/Java/JavaVirtualMachines/corretto-21.0.7/Contents/Home` 后真跑坐实 5 个失败,均为跨域 gate 仍按旧 `completedOperations=241 / AI=47` 断言,而覆盖 JSON 已是 `242/242` 且 AI 为 48。新增 AI operation 为 `adminArchiveAgent`,已有 `MuseAgentServiceTest``AdminMuseAgentControllerAnnotationTest` 两个真实 `testFiles` 证据。修正 `P1rMarketRealApiGateTest``P1rKnowledgeRealApiGateTest``P1rEventsRealApiGateTest` 的 summary/AI 计数后,复跑 `run-p1r-verification.sh local` **61/0F/0E/0S, BUILD SUCCESS**。本步未改覆盖 JSON、OpenAPI、业务实现或数据库迁移S1 Dify 部署仍未开始。
**2.0.0 单人版改造 S1 Dify 基础设施与 live 契约(2026-07-07)**:已在 `mini-infra` 用官方 Dify `1.15.0` compose 独立部署 `muse-dify`,入口 `http://100.64.0.8:18080`API health 返回 `version=1.15.0`;该栈带自己的 `db/redis/weaviate/nginx/sandbox/plugin_daemon/worker/web` 等容器,不替换 Muse 既有 PG/Redis。Dify 绑定 New-API OpenAI-compatible provider已接入 `MiniMax-M2.5``Qwen/Qwen3-Embedding-8B``Qwen/Qwen3-Reranker-8B`,并创建写作 chat app、全书解析 workflow app、工作区级 Datasets API key。S1 真实验收新增 `P1rDifyChatLiveAcceptanceIT``P1rDifyDatasetsContractLiveIT`,默认 `MUSE_P1R_EXTERNAL_ACCEPTANCE` 未启用时 honest skipped显式 opt-in 后真实调用 Difychat app 返回 `status=success / messageIdPresent=true / summarySha256Prefix=565339bc4d33`Datasets 完成 create/upload/index/retrieve保留证据 dataset `cce33d41-c738-42d2-97b6-2d08a69965f2``indexingStatus=completed``score=0.7442479133605957`。踩坑已沉淀:Dify 官方 `.env``STORAGE_TYPE=opendal` 在本部署下会导致 worker `File not found`,已切到 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage` 并保留远端备份 `.env.s1-storage-before-local`Dify 1.15 `/retrieve``retrieval_model` 必填 `search_method``reranking_enable`。回归证据:targeted live 2/0F/0E/0S、默认跳过 2/0F/0E/2S、`run-p1r-verification.sh local` 61/0F/0E/0S。RAGFlow 实例下线与已暴露 key 轮换仍为手工事项,不阻塞后续 S2。
**E1 content 创作闭环切片(2026-06-27)**:已补 AI suggestion 采纳归档、前端“改后合并”入口、IndexedDB 草稿键对账、旧知识草稿失效、工作台知识/导入/导出/记录入口、Block 版本历史最小 API/UI。Content merge 写 Canonical 与来源归因后,必须由 AI owner 写 `accepted` 状态、accepted decision archive、AI command、business auditAI owner 不可用时整笔 merge 回滚,避免 Canonical 已写但候选仍 pending。Content 正文变更后通过 Knowledge owner API 将关联 pending draft 标为 `conflicted/needs_recheck` 并写 Knowledge draft decision archive通知失败不回滚 Canonical 主写Knowledge confirm 端仍按来源状态 fail-closed。Studio `CandidatePanel` 支持编辑最终正文并按 `accept_as_is`/`modify_then_merge` 提交IndexedDB 草稿键改为 `workId+blockId+revision` 防跨作品/版本污染。`saveBlock`/`mergeBlockSuggestion` 现在写 `muse_content_block_revision_snapshot`工作台“历史”Tab 只读展示 Canonical revision 快照;导入可创建真实任务,导出支持范围/格式选择、任务查询与下载凭证消费,知识/记录入口跳转对应工作台。fresh 证据:后端局部单测 `ContentSourceServiceTest` 25/0F/0E、`ContentAppServiceTest` 19/0F/0E、`MuseKnowledgeDraftInvalidationServiceTest` 3/0F/0E、`AiSuggestionMergeProjectionFacadeTest` 7/0F/0E契约/覆盖门 `ContractFirstGateTest` 4/0F/0E + `P1rApiCoverageReportTest` 8/0F/0Ereal-PG `P1rContentCoreCompletedApprovalIT` 13/0F/0E/0S + `P1rContentMergeSuggestionIT` 5/0F/0E/0S + `P1rContentMergeGeneratedSuggestionIT` 1/0F/0E/0Sstudio `tsc -b --force` 通过、lint 0 errors、Vitest 12/12追加 `AIPanel.contract.test.tsx` + `sse.test.ts` 22/0F/0E导出 UI 追加 `useWorks.test.tsx`+`ExportWorkModal.test.tsx` 11/0F/0E、目标 ESLint 0 errors。共享 PG 写入闸门批准后已补跑 MSW-off Playwright 真后端 `accept-suggestion.spec.ts` 2/0F/0E:正路真 New-API 生成 suggestionId=79、authz `rpe-local-*`、Block revision 160→161、AI owner accepted decision archive 非空;负路 stale revision 业务冲突。边界:`muse-studio/src/types/content.ts` 生成类型因 openapi-typescript 版本漂移未同步hook 内暂维护 `BlockRevision` 最小类型;完整导入向导已在 RC 后补,见下方记录;真后端导出下载 e2e 已在 2026-06-28 补跑通过FileApi 异常仍按后端合同 fail-closed。
**E2 ai 智能体生命周期与候选处置闭环(2026-06-27)**:已补用户自建 Agent update/archive、初始版本自动创建、版本列表/激活/归档非当前版本、作品槽位 unbind、Studio reject 调后端 AI owner 决策归档。Agent update 会创建下一 active version 并更新 `current_version_id`archive Agent 同步归档仍 active 的槽位绑定slot unbind 将绑定行 revision+1 且 `status=archived`运行时回到默认能力WorkspacePage “放弃修改”不再只清本地候选,而是真打 `POST /suggestions/{id}/reject``rejected` 与 decision archive。启动修红:最新单体启动时暴露 `muse.codegen.importEnable` 缺省,已在 `muse-server/src/main/resources/application.yaml``import-enable:false`,随后 48080 成功启动并 `GET /app-api/muse/agents` smoke 返回 `code=0`。fresh 证据:后端 AI targeted `MuseAgentServiceTest`+`MuseAgentSlotServiceTest`+Controller annotation 63/0F/0E契约门 `ContractFirstGateTest` 4/0F/0Estudio `tsc -b --force` 通过、目标 ESLint 0 errors、Vitest 29/0F/0EMSW-off Playwright 真后端 `agent-create.spec.ts`+`agent-slot-bind.spec.ts`+`accept-suggestion.spec.ts` 5/0F/0E:采纳真生成 suggestionId=80、Block revision 161→162、accepted decision archive 非空;拒绝真生成 suggestionId=81、`status=rejected`、decision archive 非空Agent 生命周期 DB 核验 v1/v2 current 切换与 v2 archivedSlot unbind DB 核验 revision 2→3、status archived、响应 `sourceStatus=unbound`。边界:本轮未跑 studio 全量 e2e。

File diff suppressed because one or more lines are too long

View File

@ -225,6 +225,18 @@ muse:
max-attempts: ${MUSE_AI_NEW_API_MAX_ATTEMPTS:3}
retry-backoff-seconds: ${MUSE_AI_NEW_API_RETRY_BACKOFF_SECONDS:1,2,4}
propagate-muse-context: ${MUSE_AI_NEW_API_PROPAGATE_MUSE_CONTEXT:true}
dify:
enabled: ${MUSE_AI_DIFY_ENABLED:false}
base-url: ${MUSE_AI_DIFY_BASE_URL:}
console-base-urls: ${MUSE_AI_DIFY_CONSOLE_BASE_URLS:}
credentials:
- ref: ${MUSE_AI_DIFY_WRITING_CREDENTIAL_REF:dify-writing-s1}
api-key: ${MUSE_AI_DIFY_WRITING_API_KEY:}
- ref: ${MUSE_AI_DIFY_PARSER_CREDENTIAL_REF:dify-parser-s1}
api-key: ${MUSE_AI_DIFY_PARSER_API_KEY:}
connect-timeout-seconds: ${MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS:5}
non-stream-read-timeout-seconds: ${MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS:90}
total-timeout-seconds: ${MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS:180}
--- #################### 芋道相关配置 ####################

View File

@ -0,0 +1,215 @@
package cn.iocoder.muse.module.ai.application.muse.facade;
import cn.iocoder.muse.framework.common.util.json.JsonUtils;
import cn.iocoder.muse.module.ai.framework.ai.config.MuseAiProperties;
import org.junit.jupiter.api.Test;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Instant;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeTrue;
/**
* S1 Dify chat app 外部 live acceptance 入口
*
* <p>默认必须跳过只有显式设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 才会真实调用外部 Dify
* 输出证据只保留 endpointappmessage id 状态和 key 指纹不能打印完整 key provider 原始回答</p>
*/
class P1rDifyChatLiveAcceptanceIT {
private static final String ACCEPTANCE_ENV = "MUSE_P1R_EXTERNAL_ACCEPTANCE";
private static final String BASE_URL_ENV = "MUSE_AI_DIFY_BASE_URL";
private static final String APP_ID_ENV = "MUSE_AI_DIFY_WRITING_APP_ID";
private static final String API_KEY_ENV = "MUSE_AI_DIFY_WRITING_API_KEY";
private static final String CREDENTIAL_REF_ENV = "MUSE_AI_DIFY_WRITING_CREDENTIAL_REF";
private static final String DEFAULT_CREDENTIAL_REF = "dify-writing-s1";
private static final String CHAT_MESSAGES_PATH = "/chat-messages";
@Test
void shouldCallDifyChatThroughMuseRuntimeClientAndPrintRedactedEvidence() {
assumeTrue(externalAcceptanceEnabled(), "P1R 外部验收未启用,设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 后才运行");
String baseUrl = requiredEnv(BASE_URL_ENV);
String appId = requiredEnv(APP_ID_ENV);
String apiKey = requiredEnv(API_KEY_ENV);
String credentialRef = envOrDefault(CREDENTIAL_REF_ENV, DEFAULT_CREDENTIAL_REF);
MuseAiRuntimeClient.RuntimeCommand command = liveCommand(appId, credentialRef);
RealDifyMuseAiRuntimeClient client = new RealDifyMuseAiRuntimeClient(difyProperties(baseUrl, apiKey,
credentialRef));
MuseAiRuntimeClient.RuntimeResponse response = client.execute(command);
assertNull(response.failure(), () -> "Dify chat live acceptance failed: "
+ redactedFailure(response.failure()));
assertNotNull(response.result(), "Dify chat live acceptance 必须返回 success result");
assertEquals("success", response.result().status());
assertNotNull(response.result().outputSummary(), "Dify chat outputSummary 不能为空");
Object summary = response.result().outputSummary().get("summary");
assertTrue(summary != null && !String.valueOf(summary).isBlank(), "Dify chat summary 不能为空");
assertNoSensitiveText(String.valueOf(summary), "summary");
System.out.println(JsonUtils.toJsonString(redactedEvidence(baseUrl, apiKey, appId, credentialRef,
response.result(), String.valueOf(summary))));
}
private MuseAiRuntimeClient.RuntimeCommand liveCommand(String appId, String credentialRef) {
String correlationId = "s1-dify-chat-live-" + Instant.now().toEpochMilli();
return new MuseAiRuntimeClient.RuntimeCommand(
"cmd-" + correlationId,
"s1-dify-chat-task",
"s1-dify-chat-job",
correlationId,
1,
"dify-writing",
"s1-dify-chat-live-v1",
Map.of("taskType", "s1_dify_chat_live_acceptance",
"intent", "external_dify_chat_smoke",
"runtimeProvider", "dify",
"providerRef", Map.of("dify", Map.of(
"appType", "agent",
"appId", appId,
"credentialRef", credentialRef)),
"outputContract", Map.of("format", "plain_text")),
"s1-dify-chat-source-snapshot",
"s1-dify-chat-runtime-envelope",
"只输出 OK",
timeoutPolicy(),
MuseAiRuntimeClient.RetryPolicy.defaults());
}
private MuseAiProperties difyProperties(String baseUrl, String apiKey, String credentialRef) {
MuseAiProperties properties = new MuseAiProperties();
MuseAiProperties.Dify dify = new MuseAiProperties.Dify();
dify.setEnabled(true);
dify.setBaseUrl(baseUrl);
dify.setConnectTimeoutSeconds(intEnv("MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS", 5));
dify.setNonStreamReadTimeoutSeconds(intEnv("MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS", 90));
dify.setTotalTimeoutSeconds(intEnv("MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS", 180));
MuseAiProperties.Dify.Credential credential = new MuseAiProperties.Dify.Credential();
credential.setRef(credentialRef);
credential.setApiKey(apiKey);
dify.setCredentials(List.of(credential));
properties.setDify(dify);
return properties;
}
private MuseAiRuntimeClient.TimeoutPolicy timeoutPolicy() {
return new MuseAiRuntimeClient.TimeoutPolicy(
intEnv("MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS", 5),
intEnv("MUSE_AI_DIFY_FIRST_BYTE_TIMEOUT_SECONDS", 15),
intEnv("MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS", 90),
intEnv("MUSE_AI_DIFY_STREAM_IDLE_TIMEOUT_SECONDS", 30),
intEnv("MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS", 180));
}
private Map<String, Object> redactedEvidence(String baseUrl, String apiKey, String appId, String credentialRef,
MuseAiRuntimeClient.RuntimeResult result, String summary) {
Map<String, Object> evidence = new LinkedHashMap<>();
evidence.put("acceptance", "s1-dify-chat-live");
evidence.put("endpoint", endpointSummary(baseUrl, CHAT_MESSAGES_PATH));
evidence.put("credentialRef", credentialRef);
evidence.put("apiKey", secretFingerprint(apiKey));
evidence.put("appId", appId);
evidence.put("correlationId", result.correlationId());
evidence.put("status", result.status());
evidence.put("messageIdPresent", result.providerRequestId() != null && !result.providerRequestId().isBlank());
evidence.put("summaryLength", summary.length());
evidence.put("summarySha256Prefix", sha256Prefix(summary, 12));
evidence.put("finishReason", result.finishReason());
evidence.put("usage", result.tokenUsage());
evidence.put("completedAt", result.completedAt());
return evidence;
}
private Map<String, Object> endpointSummary(String baseUrl, String path) {
URI uri = URI.create(trimRight(baseUrl, "/") + path);
Map<String, Object> endpoint = new LinkedHashMap<>();
endpoint.put("scheme", uri.getScheme());
endpoint.put("host", uri.getHost());
endpoint.put("port", uri.getPort());
endpoint.put("path", uri.getPath());
return endpoint;
}
private Map<String, Object> secretFingerprint(String secret) {
Map<String, Object> fingerprint = new LinkedHashMap<>();
fingerprint.put("length", secret == null ? 0 : secret.length());
fingerprint.put("sha256Prefix", sha256Prefix(secret == null ? "" : secret, 12));
return fingerprint;
}
private String redactedFailure(MuseAiRuntimeClient.RuntimeFailure failure) {
if (failure == null) {
return "none";
}
Map<String, Object> summary = new LinkedHashMap<>();
summary.put("failureType", failure.failureType());
summary.put("providerStatusCode", failure.providerStatusCode());
summary.put("errorCode", failure.errorCode());
summary.put("retryable", failure.retryable());
summary.put("message", failure.sanitizedMessage());
return JsonUtils.toJsonString(summary);
}
private void assertNoSensitiveText(String value, String label) {
String lower = value == null ? "" : value.toLowerCase(Locale.ROOT);
assertFalse(lower.contains("token"), label + " 不应包含 token 字样");
assertFalse(lower.contains("key"), label + " 不应包含 key 字样");
assertFalse(lower.contains("authorization"), label + " 不应包含 authorization 字样");
}
private boolean externalAcceptanceEnabled() {
return "true".equalsIgnoreCase(System.getenv(ACCEPTANCE_ENV));
}
private String requiredEnv(String name) {
String value = System.getenv(name);
assertTrue(value != null && !value.isBlank(), "缺少外部验收环境变量:" + name);
return value;
}
private String envOrDefault(String name, String defaultValue) {
String value = System.getenv(name);
return value == null || value.isBlank() ? defaultValue : value;
}
private int intEnv(String name, int defaultValue) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
return defaultValue;
}
return Integer.parseInt(value);
}
private String sha256Prefix(String value, int length) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
String hex = HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
return hex.substring(0, Math.min(length, hex.length()));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("JDK 缺少 SHA-256 摘要算法", e);
}
}
private String trimRight(String value, String suffix) {
String result = value == null ? "" : value.trim();
while (result.endsWith(suffix)) {
result = result.substring(0, result.length() - suffix.length());
}
return result;
}
}

View File

@ -3,7 +3,8 @@
> 模块跨会话记忆。真实现状权威见 [现状基线 §5.3](../../docs/agent-specs/2026-06-13-项目目标与模块现状基线.md);进度见 [总账](../../docs/mvp/进度总账.md)。本文件只记目标/边界/out-of-scope/现状指针/TODO,过时即更。
- **目标(owner 职责)**:Local KB / User KB / 全局知识处理 / Knowledge Draft / Knowledge Source Binding / 投影与索引状态。
- **边界(不可违反)**:**进 Local KB 唯一入口=用户显式确认草稿或手动修正**;全书解析章节确认只产/推 Draft,不写正式知识;来源优先级 Local KB > 用户绑定 > 市场/全局;依赖 RAGFlow GraphRAG(不可用时 Unavailable 优雅降级);守 BC 边界不碰他域 `.dal`([bc-boundaries](../../.agents/rules/bc-boundaries.md))。
- **边界(不可违反)**:**进 Local KB 唯一入口=用户显式确认草稿或手动修正**;全书解析章节确认只产/推 Draft,不写正式知识;来源优先级 Local KB > 用户绑定 > 市场/全局;1.0 运行时仍有 RAGFlow GraphRAG 历史链路,2.0 单人版迁移目标为 Dify Datasets(不可用时 Unavailable 优雅降级);守 BC 边界不碰他域 `.dal`([bc-boundaries](../../.agents/rules/bc-boundaries.md))。
- **out-of-scope**:正文写入(归 content)。
- **现状**:只读评估 78%;后端最完整模块之一(67 端点对齐契约),RAGFlow 真 HTTP + SSRF 防护;installed-KB 删除记录回显 bug 已修(2026-06-19:`MuseKnowledgeBindingMapperTest` + `MuseInstalledKnowledgeBaseServiceTest` 10/10,删除时写 `deleted=true` 且列表排除 `binding_status=deleted`);impact preview 已按 `muse_knowledge_source_binding_projection.owner_user_id` 输出 installed/source-binding owner 审计计数(2026-06-19:`MuseKnowledgeBaseServiceTest` + `MuseKnowledgeSourceBindingProjectionMapperTest` 30/30);source binding projection 回填闭环已修(2026-06-19:绑定确认为权威回填点写读模型行、unbind 按 bindingId 作用域撤销不误伤同源跨作品投影、唯一键补 work_id 修跨作品复用误判冲突V24;`MuseKnowledgeBindingServiceTest` + `MuseKnowledgeSourceBindingProjectionMapperTest` 11/11、整套件 214/214 绿);**读回端到端坐实 + unbind 真软删修复(2026-06-19)**:嵌入式 DB(H2)往返证 bind 写投影→`selectActiveByWorkId` 读回该来源、unbind 后读回消失、同源他作品投影存活;并修 unbind 软删真 bug——`deleted` 是 `@TableLogic`,实体 `setDeleted(true)` 被 MP 普通 update 剥离成空操作,改 `setSql("deleted = true")` 才真软删(`MuseKnowledgeSourceBindingProjectionRoundTripTest` 2/2、整套件 216/216 绿;新增 knowledge 模块首套嵌入式 DB 测试基建)。**GET bindings 读端点已补(2026-06-19)**:`AppMuseKnowledgeBindingController` 加 `GET /muse/works/{workId}/knowledge-bindings`→`listKnowledgeBindings`(requireWorkOwner 防 IDOR + selectActiveByWorkId 读回投影,id→String;happy-path + 越权 fail-closed 单测,整套件 218/218 绿);**bindings 读回后端腿(写→读→端点)已闭环,仅余 FE hook**。**FE 读回面板已加(2026-06-19)**:studio `useKnowledgeBindings` hook + `KnowledgeBindingsPanel`(镜像 DraftPanel)接入 `KnowledgePage`,契约+组件测试经 vitest 验证(全 studio 50/50、tsc 干净);live playwright e2e 需全栈 app(env 受限)未跑。bindings 写→读→端点→FE 读回展示链(除 live e2e)全通。**live e2e 已写 ready 未跑(2026-06-19)**:`muse-studio/e2e/knowledge-bindings.spec.ts`(镜像 graph spec,`playwright --list` 编译有效);活体受阻=远端 PG 宿主 100.64.0.8 离线(不在 tailnet)、无本地 PG 兜底,PG 宿主恢复后按 [.agents/knowledge §六] 起栈可跑。 **文本内容安全扫描已实现 + 摄入 scanning 关卡打通(2026-06-23,commit ec7637d)**:`KnowledgeContentScanService`(同步纯文本字符扫描——L1 非法 UTF-8/null/危险 C0-C1 控制符/Bidi 覆盖/体量硬阻断,L2 NFC+去零宽+换行规整,L3 prompt 注入软标记不阻断;特殊字符判定全用十六进制码点避免源码字面不可见字符),`KnowledgeFileFacade.materialize` 接入(passed→materialized+规整内容入 RAG、blocked→隔离带 reasonCode、扫描异常→fail-closed),替换原恒 `scanBlocked` 占位;`MaterializedFile.scanBlocked` 加 reasonCode/message(连带改 P1r live IT 构造签名)。单测 ScanService 9 + Facade 接线 3 + DocumentService 17 + P-B RetrievalApiImpl 11 全绿。**端到端真验(48080 + muse_slice_live + RAGFlow/New-API env 配齐)**:上传纯文本 document 4 → `scan_status=passed` → 处理链推进 → RAGFlow dataset(5950e9ee)+document(595cb63e)创建成功,坐实 scan 关卡 + muse→RAGFlow 集成贯通(对比 document 2 旧 jar 卡 SCAN_SERVICE_UNAVAILABLE、document 3 RAGFlow 未配 CONFIG_MISSING)。**E3 studio 与检索闭环已完成(2026-06-27)**:新增主动检索 App 端点与 `KnowledgeRetrievalPanel`no_dataset 显式治理提示,自建 KB 上传真实后端读回market KB handoff 正负路真 e2eAI 生成 e2e 反查 `source_summary.contextAssembly` 证实 Knowledge Source 被真实消费。fresh 证据:knowledge targeted 25/0F/0E/0S契约/迁移门 21/0F/0E/0Sstudio Vitest 19/0F/0E、ESLint/tsc 通过MSW-off Playwright knowledge 6/0F/0E + ai-generation 1/0F/0E。
- **2.0.0 S1 Dify Datasets live 契约(2026-07-07)**:Dify 1.15.0 已创建工作区级 Datasets API key(`dataset-` 类型,非 app key、非单 dataset `ds-` key),并通过 `P1rDifyDatasetsContractLiveIT` 真跑 create dataset / upload file / indexing-status poll / retrieve。验收 dataset `cce33d41-c738-42d2-97b6-2d08a69965f2` 保留在 Dify 作为证据,`indexingStatus=completed`、retrieval `score=0.7442479133605957`;默认未 opt-in 时 honest skipped。Dify 1.15 `/v1/datasets/{id}/retrieve` 的 `retrieval_model` 必须显式包含 `search_method=semantic_search` 与 `reranking_enable=false`,只传 `top_k/score_threshold_enabled` 会 400。Dify 官方 compose 在本机部署必须用 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage`,否则 upload 成功后 worker 取文件会报 `File not found`。
- **关键风险 / TODO**:前端图谱/发布深面仍薄;Task4 剩余 owner 缺口已收窄为 **document owner count / export_task_owner** 仍不能伪造,需后续 owner 事实或外部服务接入;已物化 KB 的召回触达/检索停用已在 E4 闭环。**2026-06-27:E4 market KB 召回触达已补并真验**:Knowledge 新增 `KnowledgeMarketSourceStatusChangedConsumer` 消费 Market `MarketAssetSourceStatusChangedEvent`,将本地 installed_ref KB 的 source binding projection 更新为 `recalled/blocked`,检索来源状态门返回 `omittedSources` 而不再送入 RAGFlowMSW-off Playwright `market-kb-recall.spec.ts` 1/0F/0E后端单测 `MuseKnowledgeSourceEventServiceTest`+`KnowledgeMarketSourceStatusChangedConsumerTest` 9/0F/0E。副本资源回收/更细补偿仍后置。**2026-06-27:E1 旧知识草稿失效 owner API 已补**:`MuseKnowledgeDraftInvalidationApi` 由 knowledge-api 定义、knowledge-server owner 实现Content 正文变更只传 work/chapter/block/revision/commandId,不跨 BC 传正文。Knowledge 将匹配的 pending draft 更新为 `conflicted` + `source_action_policy=needs_recheck`,并写 `muse_knowledge_draft_decision(decision_type=recheck)` 作为审计归档invalid command 跳过并 warnContent 通知失败不回滚 Canonical 主写confirm 端仍按 `needs_recheck` fail-closed。验证:`MuseKnowledgeDraftInvalidationServiceTest` 3/0F/0E`P1rContentMergeSuggestionIT` real-PG 5/0F/0E/0S 穿透验证采纳后旧 draft 失效与 replay 不重复。**2026-06-28:导入章节确认 Draft owner API 已补**:`MuseKnowledgeDraftCreationApi` 由 knowledge-api 定义、knowledge-server owner 实现Content/AI 只传章节 Shadow 结果与 sourceRefKnowledge 负责创建 pending draft 与幂等 command不写 Local KB Canonical仍需用户后续显式确认草稿才进入正式知识。验证:`MuseKnowledgeDraftCreationApiImplTest` 与完整导入链路 targeted 后端测试通过。**parse 轮询 worker 已补齐并端到端真验(2026-06-23,commit 635045c)**:`MuseKnowledgeParseStatusPollWorker`(@Scheduled + 开关 `muse.knowledge.parse-poll-worker.enabled`,跨租户捞 parsing 任务轮询 RAGFlow 文档状态→run=done 推 version processingStatus=searchable、failed 标败、processing 保持、RAGFlow 不可达/异常不误判)+ `ProcessingTaskService.markRagflowParseCompleted/markRagflowParsePolling` + `Mapper.selectParsingForPoll`;worker 接线 7 测试 + 回归 40 全绿;启用开关后 document 4 自动 parsing→searchable(isSearchable=true),摄入链 **scan→RAGFlow→worker→searchable** 全程贯通(此前"RAGFlow backend 故障"系本机 HTTP 代理误诊,RAGFlow 实始终正常,真因=muse 缺 parse 轮询 worker)。

View File

@ -0,0 +1,372 @@
package cn.iocoder.muse.module.knowledge.application.muse.facade;
import cn.iocoder.muse.framework.common.util.json.JsonUtils;
import com.fasterxml.jackson.databind.JsonNode;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.ProxySelector;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Duration;
import java.time.Instant;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeTrue;
/**
* S1 Dify Datasets API 契约 live IT
*
* <p>默认必须跳过只有显式设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 才会真实调用外部 Dify
* 本测试会保留新建 dataset 作为验收证据不做删除失败与输出证据只打印脱敏摘要</p>
*/
class P1rDifyDatasetsContractLiveIT {
private static final String ACCEPTANCE_ENV = "MUSE_P1R_EXTERNAL_ACCEPTANCE";
private static final String BASE_URL_ENV = "MUSE_KNOWLEDGE_DIFY_BASE_URL";
private static final String API_KEY_ENV = "MUSE_KNOWLEDGE_DIFY_DATASET_API_KEY";
private static final Set<String> INDEXING_STATUSES = Set.of(
"waiting", "parsing", "cleaning", "splitting", "indexing", "completed", "paused", "error");
@Test
void shouldPinDifyDatasetsContractAndPrintRedactedEvidence() throws Exception {
assumeTrue(externalAcceptanceEnabled(), "P1R 外部验收未启用,设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 后才运行");
String baseUrl = requiredEnv(BASE_URL_ENV);
String apiKey = requiredEnv(API_KEY_ENV);
Duration timeout = Duration.ofSeconds(intEnv("MUSE_KNOWLEDGE_DIFY_TIMEOUT_SECONDS", 60));
HttpClient client = HttpClient.newBuilder()
.connectTimeout(timeout)
// Dify 部署在内网时必须直连避免 JVM 读取系统代理导致假 502
.proxy(ProxySelector.of(null))
.build();
String suffix = String.valueOf(Instant.now().toEpochMilli());
String datasetName = "dify-contract-" + suffix;
JsonNode dataset = createDataset(client, baseUrl, apiKey, timeout, datasetName);
String datasetId = requiredText(dataset, "id", "create dataset response");
assertEquals(datasetName, requiredText(dataset, "name", "create dataset response"));
assertEquals("high_quality", requiredText(dataset, "indexing_technique", "create dataset response"));
JsonNode upload = uploadDocument(client, baseUrl, apiKey, timeout, datasetId, suffix);
String documentId = requiredText(upload.path("document"), "id", "upload document response.document");
String batch = requiredText(upload, "batch", "upload document response");
JsonNode indexing = waitUntilCompleted(client, baseUrl, apiKey, timeout, datasetId, batch);
JsonNode retrieve = retrieve(client, baseUrl, apiKey, timeout, datasetId, suffix);
JsonNode firstRecord = firstRecord(retrieve);
String content = requiredText(firstRecord.path("segment"), "content", "retrieve response records[0].segment");
assertTrue(firstRecord.path("score").isNumber(), "retrieve response records[0].score 必须是 number");
assertFalse(content.isBlank(), "retrieve response records[0].segment.content 不能为空");
assertTrue(content.contains("Dify contract marker " + suffix),
"retrieve response 必须命中本轮上传文档的 marker避免旧数据误判为通过");
System.out.println(JsonUtils.toJsonString(redactedEvidence(baseUrl, apiKey, datasetName, datasetId,
documentId, batch, indexing, content, firstRecord.path("score").asDouble())));
}
private JsonNode createDataset(HttpClient client, String baseUrl, String apiKey, Duration timeout,
String datasetName) throws IOException, InterruptedException {
Map<String, Object> body = new LinkedHashMap<>();
body.put("name", datasetName);
body.put("permission", "only_me");
body.put("indexing_technique", "high_quality");
return sendJson(client, request(baseUrl, apiKey, "/v1/datasets", timeout)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(JsonUtils.toJsonString(body)))
.build(), "create dataset", apiKey);
}
private JsonNode uploadDocument(HttpClient client, String baseUrl, String apiKey, Duration timeout,
String datasetId, String suffix) throws IOException, InterruptedException {
Map<String, Object> data = new LinkedHashMap<>();
data.put("indexing_technique", "high_quality");
data.put("process_rule", Map.of("mode", "automatic"));
String fileName = "dify-contract-" + suffix + ".txt";
byte[] content = liveDocumentContent(suffix);
MultipartPayload payload = multipartPayload(JsonUtils.toJsonString(data), fileName, content);
return sendJson(client, request(baseUrl, apiKey,
"/v1/datasets/" + datasetId + "/document/create-by-file", timeout)
.header("Content-Type", payload.contentType())
.POST(HttpRequest.BodyPublishers.ofByteArray(payload.body()))
.build(), "upload document", apiKey);
}
private JsonNode waitUntilCompleted(HttpClient client, String baseUrl, String apiKey, Duration timeout,
String datasetId, String batch) throws IOException, InterruptedException {
long deadline = System.nanoTime()
+ Duration.ofSeconds(intEnv("MUSE_P1R_DIFY_INDEX_WAIT_SECONDS", 180)).toNanos();
JsonNode last = null;
while (System.nanoTime() < deadline) {
last = sendJson(client, request(baseUrl, apiKey,
"/v1/datasets/" + datasetId + "/documents/" + batch + "/indexing-status", timeout)
.GET()
.build(), "poll indexing status", apiKey);
String status = indexingStatus(last);
assertTrue(INDEXING_STATUSES.contains(status),
() -> "Dify indexing-status 返回未知状态:" + safeSnippet(status));
if ("completed".equals(status)) {
return last;
}
JsonNode current = last;
assertFalse("error".equals(status), () -> "Dify indexing-status 进入 error"
+ redactedJsonSummary(current, apiKey));
Thread.sleep(2_000L);
}
throw new AssertionError("Dify indexing-status 未在等待窗口内 completedlast=" + redactedJsonSummary(last, apiKey));
}
private JsonNode retrieve(HttpClient client, String baseUrl, String apiKey, Duration timeout,
String datasetId, String suffix) throws IOException, InterruptedException {
Map<String, Object> body = new LinkedHashMap<>();
body.put("query", "Dify contract marker " + suffix + " 是什么?");
body.put("retrieval_model", Map.of(
"search_method", "semantic_search",
"reranking_enable", false,
"top_k", 3,
"score_threshold_enabled", false));
return sendJson(client, request(baseUrl, apiKey, "/v1/datasets/" + datasetId + "/retrieve", timeout)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(JsonUtils.toJsonString(body)))
.build(), "retrieve dataset", apiKey);
}
private HttpRequest.Builder request(String baseUrl, String apiKey, String path, Duration timeout) {
return HttpRequest.newBuilder()
.uri(endpoint(baseUrl, path))
.timeout(timeout)
.header("Authorization", "Bearer " + apiKey)
.header("Accept", "application/json");
}
private JsonNode sendJson(HttpClient client, HttpRequest request, String operation, String apiKey)
throws IOException, InterruptedException {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() < 200 || response.statusCode() >= 300) {
throw new AssertionError(operation + " HTTP failed: " + httpFailureSummary(response, apiKey));
}
return assertDoesNotThrow(() -> JsonUtils.parseTree(response.body()),
operation + " response 必须是 JSON");
}
private JsonNode firstRecord(JsonNode retrieve) {
JsonNode records = retrieve.path("records");
assertTrue(records.isArray() && !records.isEmpty(), "retrieve response records 不能为空");
return records.get(0);
}
private String indexingStatus(JsonNode response) {
JsonNode data = response.path("data");
if (data.isArray() && !data.isEmpty()) {
return data.get(0).path("indexing_status").asText(data.get(0).path("status").asText(""));
}
if (data.isObject()) {
return data.path("indexing_status").asText(data.path("status").asText(""));
}
return response.path("indexing_status").asText(response.path("status").asText(""));
}
private String requiredText(JsonNode node, String field, String label) {
String value = node.path(field).asText(null);
assertTrue(value != null && !value.isBlank(), label + "." + field + " 不能为空");
return value;
}
private byte[] liveDocumentContent(String suffix) {
String text = """
Dify contract marker %s.
This document is intentionally small and kept in Dify Datasets as external acceptance evidence.
Muse Knowledge Dify contract smoke verifies dataset creation, upload, indexing status and retrieval.
""".formatted(suffix);
return text.getBytes(StandardCharsets.UTF_8);
}
private MultipartPayload multipartPayload(String dataJson, String fileName, byte[] content) throws IOException {
String boundary = "----MuseDify" + UUID.randomUUID().toString().replace("-", "");
ByteArrayOutputStream body = new ByteArrayOutputStream();
writeAscii(body, "--" + boundary + "\r\n");
writeAscii(body, "Content-Disposition: form-data; name=\"data\"\r\n");
writeAscii(body, "Content-Type: application/json; charset=UTF-8\r\n\r\n");
body.write(dataJson.getBytes(StandardCharsets.UTF_8));
writeAscii(body, "\r\n");
writeAscii(body, "--" + boundary + "\r\n");
writeAscii(body, "Content-Disposition: form-data; name=\"file\"; filename=\"" + fileName + "\"\r\n");
writeAscii(body, "Content-Type: text/plain; charset=UTF-8\r\n\r\n");
body.write(content);
writeAscii(body, "\r\n");
writeAscii(body, "--" + boundary + "--\r\n");
return new MultipartPayload(body.toByteArray(), "multipart/form-data; boundary=" + boundary);
}
private void writeAscii(ByteArrayOutputStream body, String value) throws IOException {
body.write(value.getBytes(StandardCharsets.US_ASCII));
}
private Map<String, Object> redactedEvidence(String baseUrl, String apiKey, String datasetName,
String datasetId, String documentId, String batch,
JsonNode indexing, String content, double score) {
Map<String, Object> evidence = new LinkedHashMap<>();
evidence.put("acceptance", "s1-dify-datasets-contract-live");
evidence.put("endpoint", endpointSummary(baseUrl, "/v1/datasets"));
evidence.put("apiKey", secretFingerprint(apiKey));
evidence.put("datasetName", datasetName);
evidence.put("datasetId", datasetId);
evidence.put("documentId", documentId);
evidence.put("batch", batch);
evidence.put("indexingStatus", indexingStatus(indexing));
evidence.put("retrievedContentLength", content.length());
evidence.put("retrievedContentSha256Prefix", sha256Prefix(content, 12));
evidence.put("score", score);
evidence.put("datasetRetained", true);
return evidence;
}
private Map<String, Object> endpointSummary(String baseUrl, String path) {
URI uri = endpoint(baseUrl, path);
Map<String, Object> endpoint = new LinkedHashMap<>();
endpoint.put("scheme", uri.getScheme());
endpoint.put("host", uri.getHost());
endpoint.put("port", uri.getPort());
endpoint.put("path", uri.getPath());
return endpoint;
}
private URI endpoint(String baseUrl, String path) {
String normalizedBase = trimRight(baseUrl, "/");
String normalizedPath = path.startsWith("/") ? path : "/" + path;
if (normalizedBase.endsWith("/v1") && normalizedPath.startsWith("/v1/")) {
normalizedPath = normalizedPath.substring("/v1".length());
}
return URI.create(normalizedBase + normalizedPath);
}
private String httpFailureSummary(HttpResponse<String> response, String apiKey) {
Map<String, Object> summary = new LinkedHashMap<>();
summary.put("status", response.statusCode());
JsonNode root = parseTreeQuietly(response.body());
if (root != null) {
summary.put("code", safeSnippet(firstText(root.path("code"), root.path("error"), root.path("error_code")),
apiKey));
summary.put("message", safeSnippet(firstText(root.path("message"), root.path("error_description"),
root.path("detail")), apiKey));
} else {
summary.put("code", "");
summary.put("message", safeSnippet(response.body(), apiKey));
}
return JsonUtils.toJsonString(summary);
}
private JsonNode parseTreeQuietly(String body) {
if (body == null || body.isBlank()) {
return JsonUtils.parseTree("{}");
}
try {
return JsonUtils.parseTree(body);
} catch (RuntimeException ex) {
return null;
}
}
private String redactedJsonSummary(JsonNode node, String apiKey) {
if (node == null || node.isMissingNode() || node.isNull()) {
return "{}";
}
Map<String, Object> summary = new LinkedHashMap<>();
summary.put("status", safeSnippet(indexingStatus(node), apiKey));
summary.put("code", safeSnippet(firstText(node.path("code"), node.path("error"), node.path("error_code")),
apiKey));
summary.put("message", safeSnippet(firstText(node.path("message"), node.path("error_description"),
node.path("detail")), apiKey));
return JsonUtils.toJsonString(summary);
}
private String firstText(JsonNode... nodes) {
for (JsonNode node : nodes) {
if (node != null && !node.isMissingNode() && !node.isNull()) {
String value = node.asText(null);
if (value != null && !value.isBlank()) {
return value;
}
}
}
return null;
}
private String safeSnippet(String value, String... secrets) {
String normalized = value == null ? "" : value.replaceAll("\\s+", " ").trim();
for (String secret : secrets) {
if (secret != null && !secret.isBlank()) {
normalized = normalized.replace(secret, "<redacted-secret>");
}
}
String lower = normalized.toLowerCase(Locale.ROOT);
if (lower.contains("authorization") || lower.contains("bearer ")
|| lower.contains("token") || lower.contains("api_key") || lower.contains("apikey")) {
return "<redacted>";
}
return normalized.length() <= 200 ? normalized : normalized.substring(0, 200);
}
private Map<String, Object> secretFingerprint(String secret) {
Map<String, Object> fingerprint = new LinkedHashMap<>();
fingerprint.put("length", secret == null ? 0 : secret.length());
fingerprint.put("sha256Prefix", sha256Prefix(secret == null ? "" : secret, 12));
return fingerprint;
}
private boolean externalAcceptanceEnabled() {
return "true".equalsIgnoreCase(System.getenv(ACCEPTANCE_ENV));
}
private String requiredEnv(String name) {
String value = System.getenv(name);
assertTrue(value != null && !value.isBlank(), "缺少外部验收环境变量:" + name);
return value;
}
private int intEnv(String name, int defaultValue) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
return defaultValue;
}
return Integer.parseInt(value);
}
private String sha256Prefix(String value, int length) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
String hex = HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
return hex.substring(0, Math.min(length, hex.length()));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("JDK 缺少 SHA-256 摘要算法", e);
}
}
private String trimRight(String value, String suffix) {
String result = value == null ? "" : value.trim();
while (result.endsWith(suffix)) {
result = result.substring(0, result.length() - suffix.length());
}
return result;
}
private record MultipartPayload(byte[] body, String contentType) {
}
}

View File

@ -1,5 +1,5 @@
# P1R 外部真实验收依赖。
# 该文件按用户授权明文提交,仅用于内网 P1R-4 New-API 与 P1R-5 RAGFlow 验收。
# 该文件按用户授权明文提交,仅用于内网 New-API / RAGFlow / Dify 外部验收。
# 使用方式set -a && . muse-cloud/scripts/dev/p1r-external-acceptance.env && set +a
# New-API 系统管理令牌,用于需要管理接口或渠道配置核查的场景。
@ -27,3 +27,27 @@ MUSE_KNOWLEDGE_RAGFLOW_API_KEY=ragflow-icnZ7zUcIz3oGCrpyFB2aosK-WTbvTQRaNUBPu6Rt
MUSE_KNOWLEDGE_RAGFLOW_TIMEOUT_SECONDS=30
MUSE_KNOWLEDGE_RAGFLOW_RETRY_BUDGET=0
MUSE_KNOWLEDGE_RAGFLOW_GRAPHRAG_ATTRIBUTION_READY=false
# --- DIFY_S1_BEGIN ---
# Dify 1.15.0 外部验收入口。
# S1 起 Dify 是 2.0.0 单人版唯一新增外部服务app key 与 dataset key 不可混用。
MUSE_AI_DIFY_ENABLED=true
MUSE_AI_DIFY_BASE_URL=http://100.64.0.8:18080/v1
MUSE_AI_DIFY_CONSOLE_BASE_URLS=http://100.64.0.8:18080
MUSE_AI_DIFY_WRITING_APP_ID=75f105e8-dbca-4aa6-a296-2113a65ad005
MUSE_AI_DIFY_WRITING_API_KEY=app-Un5DDoOiK7eQc617ahdn7ku2
MUSE_AI_DIFY_WRITING_CREDENTIAL_REF=dify-writing-s1
MUSE_AI_DIFY_PARSER_APP_ID=fed4d25c-1289-4090-9771-a811c89aba44
MUSE_AI_DIFY_PARSER_API_KEY=app-It4V7WukHohP8rhWV0gOn2H5
MUSE_AI_DIFY_PARSER_CREDENTIAL_REF=dify-parser-s1
MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS=5
MUSE_AI_DIFY_FIRST_BYTE_TIMEOUT_SECONDS=15
MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS=90
MUSE_AI_DIFY_STREAM_IDLE_TIMEOUT_SECONDS=30
MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS=180
MUSE_KNOWLEDGE_DIFY_BASE_URL=http://100.64.0.8:18080/v1
MUSE_KNOWLEDGE_DIFY_DATASET_API_KEY=dataset-Nyf7yWhF2CMFsD5TAqD39WHP
MUSE_KNOWLEDGE_DIFY_TIMEOUT_SECONDS=60
MUSE_KNOWLEDGE_DIFY_RETRY_BUDGET=0
MUSE_P1R_DIFY_INDEX_WAIT_SECONDS=180
# --- DIFY_S1_END ---

View File

@ -1,7 +1,7 @@
#!/usr/bin/env bash
# P1R 分层验收入口。
#
# WHY: 默认 Maven CI 只能证明纯本地层绿,不能证明真实 PostgreSQL、RAGFlow 或 New-API 链路。
# WHY: 默认 Maven CI 只能证明纯本地层绿,不能证明真实 PostgreSQL、New-API、RAGFlow 或 Dify 链路。
# 本脚本把三类证据显式分层,避免把台账门禁、默认跳过的 live IT 或 mock 绿混成"真验证"。
set -euo pipefail
@ -11,9 +11,9 @@ REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." >/dev/null 2>&1 && pwd)"
CLOUD_DIR="${REPO_ROOT}/muse-cloud"
LOCAL_TESTS="P1rApiCoverageReportTest,P1r*RealApiGateTest,AgentsInfraIntegrityTest,ContractFirstGateTest,BcBoundaryArchTest,AiGrantRuntimeBoundaryArchTest"
REAL_PG_TESTS="${P1R_REAL_PG_TESTS:-P1r*IT,!P1rAiRuntimeEndToEndLiveAcceptanceIT,!P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,!P1rMarketKbForkMaterializationIT,!P1rNewApiLiveAcceptanceIT,!P1rRagFlowLiveAcceptanceIT}"
REAL_PG_TESTS="${P1R_REAL_PG_TESTS:-P1r*IT,!P1rAiRuntimeEndToEndLiveAcceptanceIT,!P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,!P1rMarketKbForkMaterializationIT,!P1rImportLlmNewApiLiveAcceptanceIT,!P1rNewApiLiveAcceptanceIT,!P1rRagFlowLiveAcceptanceIT,!P1rDifyChatLiveAcceptanceIT,!P1rDifyDatasetsContractLiveIT}"
EXTERNAL_LIVE_TESTS="${P1R_EXTERNAL_LIVE_TESTS:-P1rAiRuntimeEndToEndLiveAcceptanceIT,P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,P1rMarketKbForkMaterializationIT}"
MODULE_LIVE_TESTS="${P1R_MODULE_LIVE_TESTS:-P1rNewApiLiveAcceptanceIT,P1rRagFlowLiveAcceptanceIT}"
MODULE_LIVE_TESTS="${P1R_MODULE_LIVE_TESTS:-P1rNewApiLiveAcceptanceIT,P1rImportLlmNewApiLiveAcceptanceIT,P1rRagFlowLiveAcceptanceIT,P1rDifyChatLiveAcceptanceIT,P1rDifyDatasetsContractLiveIT}"
usage() {
cat <<'EOF'
@ -26,7 +26,7 @@ usage() {
分层:
local 纯本地门禁: coverage/RealApiGate/ArchUnit/契约结构。无 DB、无外部服务。
real-pg 非 live P1r*IT: 真实 PostgreSQL _test 库,会 flyway.clean()
external-live live P1r*IT + module live IT: 真实 PG + New-API/RAGFlow,必须显式 opt-in。
external-live live P1r*IT + module live IT: 真实 PG + New-API/RAGFlow/Dify,必须显式 opt-in。
all 依次运行 local、real-pg、external-live。
real-pg / external-live 必需环境变量:
@ -36,7 +36,7 @@ real-pg / external-live 必需环境变量:
external-live 额外要求:
MUSE_P1R_EXTERNAL_ACCEPTANCE=true
已加载 muse-cloud/scripts/dev/p1r-external-acceptance.env 或等价 New-API/RAGFlow 环境变量。
已加载 muse-cloud/scripts/dev/p1r-external-acceptance.env 或等价 New-API/RAGFlow/Dify 环境变量。
示例:
set -a && . "$HOME/.config/muse-repo/infra.env" && set +a
@ -184,7 +184,7 @@ live_arg_line() {
run_local() {
echo "== P1R local gates: ${LOCAL_TESTS}"
(cd "${CLOUD_DIR}" && mvn -B -pl muse-server -am test \
(cd "${CLOUD_DIR}" && MUSE_P1R_EXTERNAL_ACCEPTANCE=false mvn -B -pl muse-server -am test \
-Dtest="${LOCAL_TESTS}" \
-DfailIfNoSpecifiedTests=false \
-Dsurefire.failIfNoSpecifiedTests=false)
@ -220,7 +220,7 @@ run_external_live() {
-DargLine="$(live_arg_line)")
echo "== P1R external-live module IT: ${MODULE_LIVE_TESTS}"
(cd "${CLOUD_DIR}" && mvn -B -pl muse-module-ai/muse-module-ai-server,muse-module-knowledge/muse-module-knowledge-server test \
(cd "${CLOUD_DIR}" && mvn -B -pl muse-module-ai/muse-module-ai-server,muse-module-knowledge/muse-module-knowledge-server -am test \
-Dtest="${MODULE_LIVE_TESTS}" \
-DfailIfNoSpecifiedTests=false \
-Dsurefire.failIfNoSpecifiedTests=false \