feat(p1r): add Dify S1 live acceptance
This commit is contained in:
parent
13b305d4da
commit
6f2629302f
@ -6,21 +6,26 @@
|
||||
| 依赖 | 地址 | 关键事实 |
|
||||
|---|---|---|
|
||||
| New-API(LLM 网关) | `http://100.64.0.8:3000`,OpenAI 兼容 `/v1/chat/completions` | 已验收模型 `MiniMax-M2.5`;容器 `new-api` 端口 3000;其 DB 在 `infra-postgres`(宿主 `100.64.0.8:5433`) |
|
||||
| RAGFlow(知识运行时) | `http://100.64.0.8`(及 `:9380`) | `/api/v1/datasets`、`/api/v1/retrieval`;健康检查 `/v1/system/healthz` |
|
||||
| RAGFlow(1.0 历史知识运行时) | `http://100.64.0.8`(及 `:9380`) | `/api/v1/datasets`、`/api/v1/retrieval`;健康检查 `/v1/system/healthz`;2.0 下线由用户手工执行,不阻塞 S1 |
|
||||
| Dify 1.15.0(2.0 单人版目标外部服务) | `http://100.64.0.8:18080` | 官方 compose 部署在 `mini-infra:/home/qingse/dify-1.15.0/docker`,compose project=`muse-dify`;API health 返回 `version=1.15.0` |
|
||||
| 开发 PG / Redis | 见下方凭据来源 | 远端 PG **15**(用户确认可用,不强制 PG16);Redis `100.64.0.8:6379` |
|
||||
|
||||
**Dify 状态(2026-06-28 盘点)**:当前 Muse 1.0.0 运行链路没有接入 Dify,infra 主机也未部署 Dify 相关容器/端口。仓内只剩设计映射与示例测试提及 Dify;因此它不是当前生产运行时依赖。若产品后续要求 Dify 工作流能力,应按新集成从契约、配置、失败闭环、live 验收补齐,不能把它当成已存在基础设施。
|
||||
**Dify 状态(2026-07-07 S1)**:Dify 已作为 2.0 单人版目标外部服务完成基础设施部署与 live 契约验收,但生产运行时切换仍按 S2+ 后续步骤推进,不要把 S1 等同于 Muse runtime 已全面改走 Dify。当前 `muse-dify` 是一套独立基础设施栈,包含自己的 `db/redis/weaviate/nginx/sandbox/plugin_daemon/worker/web` 等容器;它不复用、不替换 Muse 既有 PostgreSQL/Redis。
|
||||
|
||||
**凭据来源(明文不入库,只记位置)**:
|
||||
- 外部验收:`muse-cloud/scripts/dev/p1r-external-acceptance.env`(`set -a; . 该文件; set +a` 加载;含 New-API base/token、`MUSE_AI_NEW_API_DEFAULT_MODEL_KEY=MiniMax-M2.5`、RAGFlow base/key、GraphRAG 开关)。
|
||||
- 开发基础设施:`~/.config/muse-repo/infra.env`(PG/Redis 连接 + 真实凭据)。
|
||||
- ⚠️ **安全**:上述令牌仅限当前内网验收上下文;**仓库若同步到更大范围,必须先轮换 New-API 令牌**。
|
||||
- Dify 控制台/模型/Datasets 凭据:`mini-infra:/home/qingse/.config/muse-repo/dify.env` 与本仓 `muse-cloud/scripts/dev/p1r-external-acceptance.env` 的 `MUSE_AI_DIFY_*`、`MUSE_KNOWLEDGE_DIFY_*` 块。仓内 env 按项目内网约定用于验收,日志与文档只输出长度/hash 前缀。
|
||||
- ⚠️ **安全**:上述令牌仅限当前内网验收上下文;本轮 agent 工具输出曾出现 New-API 与 Dify 明文 key,**用户需手工轮换 New-API token 与 Dify app/dataset keys**。
|
||||
|
||||
**Live 验收 harness(opt-in,默认跳过,`MUSE_P1R_EXTERNAL_ACCEPTANCE=true` 才真跑)**:
|
||||
- `muse-module-ai/.../application/muse/facade/P1rNewApiLiveAcceptanceIT.java`(New-API)
|
||||
- `muse-module-ai/.../application/muse/P1rImportLlmNewApiLiveAcceptanceIT.java`(New-API 完整导入 LLM 解析)
|
||||
- `muse-module-knowledge/.../application/muse/facade/P1rRagFlowLiveAcceptanceIT.java`(RAGFlow)
|
||||
- `muse-module-ai/.../application/muse/facade/P1rDifyChatLiveAcceptanceIT.java`(Dify chat app)
|
||||
- `muse-module-knowledge/.../application/muse/facade/P1rDifyDatasetsContractLiveIT.java`(Dify Datasets create/upload/index/retrieve)
|
||||
- 另:`muse-server/.../framework/api/P1rAiRuntimeEndToEndLiveAcceptanceIT.java`、`P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT.java`
|
||||
- 输出脱敏(只打 endpoint/模型/id/状态/耗时/usage/key 长度 + sha256 前 12);RAGFlow smoke 会留存 `p1r-live-<时间戳>` dataset 不删。
|
||||
- 输出脱敏(只打 endpoint/模型/id/状态/耗时/usage/key 长度 + sha256 前 12);RAGFlow smoke 会留存 `p1r-live-<时间戳>` dataset 不删,Dify Datasets smoke 会留存 `dify-contract-<时间戳>` dataset 不删。
|
||||
- **runtime client live ≠ coverage completed**:仍需 Muse API 触发 + 落库 + 审计 + 失败路径证据(见 [golden-journey](../skills/golden-journey-vertical-slice.md))。
|
||||
|
||||
## 二、外部集成兼容坑(实证)
|
||||
@ -31,6 +36,12 @@
|
||||
- **RAGFlow 检索 metadata 字段名**:RAGFlow 官方契约是 `metadata_condition`。2026-06-28 已将 Muse HTTP 客户端从旧的 `metadata_filter` 修为 `metadata_condition`,并用请求体单测防回退。即便字段已修,跨可见性安全隔离仍不得依赖运行时 metadata 过滤,market KB 继续使用公开副本物理隔离。
|
||||
- **GraphRAG attribution**:默认 fail-closed(`ATTRIBUTION_NOT_CONFIGURED`);真跑需 `MUSE_KNOWLEDGE_RAGFLOW_GRAPHRAG_ATTRIBUTION_READY=true` + `..._GRAPHRAG_DATASET_ID`。
|
||||
- **New-API/RAGFlow Java 客户端代理污染**:2026-06-28 盘点时,未清代理的 Java/Python 探针曾返回 502,但 `curl --noproxy '*'` 与清代理 live IT 均证明 New-API/RAGFlow 可用。运行时代码已在 New-API 文本生成、导入解析、RAGFlow HTTP 客户端上显式使用 direct `ProxySelector`,避免 macOS/环境代理把 tailnet 内网请求转到本地代理导致假故障。
|
||||
- **Dify 官方 compose 钉版**:S1 使用 `langgenius/dify` 官方仓库 tag `1.15.0` 的 `docker/` 目录,远端源码备份在 `/home/qingse/dify-1.15.0`,compose project 固定为 `muse-dify`。控制台初始化后 `/console/api/setup` 返回 `step=finished`,API health 返回 `{"status":"ok","version":"1.15.0"}`。
|
||||
- **Dify 存储类型坑**:官方 `.env` 默认 `STORAGE_TYPE=opendal` + `OPENDAL_FS_ROOT=storage` 在当前 compose 部署下会出现 API 上传成功、worker 索引时报 `File not found`。已改为 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage` 并重启 `api/api_websocket/worker/worker_beat/nginx`;回滚备份为 `mini-infra:/home/qingse/dify-1.15.0/docker/.env.s1-storage-before-local`。
|
||||
- **Dify 模型 provider**:通过 marketplace 插件 `langgenius/openai_api_compatible/openai_api_compatible` 连接 New-API。`MiniMax-M2.5` 作为 LLM,`Qwen/Qwen3-Embedding-8B` 作为 text-embedding,`Qwen/Qwen3-Reranker-8B` 作为 rerank;创建 high_quality dataset 前必须先配置 text-embedding,否则 dataset/create 会缺默认 embedding。
|
||||
- **Dify Datasets key 类型**:工作区级 key 从 `/console/api/datasets/api-keys` 创建,前缀 `dataset-`,可调用 `/v1/datasets`、`/v1/datasets/{id}/document/create-by-file`、`/v1/datasets/{id}/retrieve`;单 dataset key 从 `/console/api/datasets/{id}/api-keys` 创建,前缀通常为 `ds-`,用于限定某个 dataset。S1 执行计划要求的是工作区级 Datasets API key。app key 前缀 `app-`,只能打 app/workflow API,不能打 Datasets。
|
||||
- **Dify Datasets retrieve 契约**:Dify 1.15 的 `/v1/datasets/{id}/retrieve` 在 `retrieval_model` 中必填 `search_method` 与 `reranking_enable`;只传 `top_k/score_threshold_enabled` 会 400。S1 live IT 固定 `search_method=semantic_search`、`reranking_enable=false`。
|
||||
- **Dify 控制台 API 坑**:`INIT_PASSWORD` 最大长度 30;控制台登录 password 字段实际传 base64(password),不是 RSA;控制台 API 写操作需 cookie + `X-CSRF-Token`,不要拿 app/dataset key 调控制台端点。
|
||||
|
||||
## 三、前端 / 构建坑速查
|
||||
- **studio(Vite + React + TS6.0)**:`erasableSyntaxOnly` 禁用 constructor 参数属性;`exactOptionalPropertyTypes` 下可选属性需规避写法;采纳/Diff 用 **Coarse-to-Fine 分级 Diff** 规避 3000+ 字正文上 O(N×M) LCS 性能爆炸;MSW 仅 `import.meta.env.DEV` 启用。
|
||||
|
||||
@ -25,6 +25,8 @@
|
||||
|
||||
**2.0.0 单人版改造 S0 门禁基线(2026-07-07)**:已坐实并修复 local 覆盖门基线红项。初跑 `bash muse-cloud/scripts/run-p1r-verification.sh local` 先暴露本机 Maven 绑定 JDK17 与沙箱写 `~/.m2` 的环境问题;显式 `JAVA_HOME=/Users/qingse/Library/Java/JavaVirtualMachines/corretto-21.0.7/Contents/Home` 后真跑坐实 5 个失败,均为跨域 gate 仍按旧 `completedOperations=241 / AI=47` 断言,而覆盖 JSON 已是 `242/242` 且 AI 为 48。新增 AI operation 为 `adminArchiveAgent`,已有 `MuseAgentServiceTest` 与 `AdminMuseAgentControllerAnnotationTest` 两个真实 `testFiles` 证据。修正 `P1rMarketRealApiGateTest`、`P1rKnowledgeRealApiGateTest`、`P1rEventsRealApiGateTest` 的 summary/AI 计数后,复跑 `run-p1r-verification.sh local` **61/0F/0E/0S, BUILD SUCCESS**。本步未改覆盖 JSON、OpenAPI、业务实现或数据库迁移;S1 Dify 部署仍未开始。
|
||||
|
||||
**2.0.0 单人版改造 S1 Dify 基础设施与 live 契约(2026-07-07)**:已在 `mini-infra` 用官方 Dify `1.15.0` compose 独立部署 `muse-dify`,入口 `http://100.64.0.8:18080`,API health 返回 `version=1.15.0`;该栈带自己的 `db/redis/weaviate/nginx/sandbox/plugin_daemon/worker/web` 等容器,不替换 Muse 既有 PG/Redis。Dify 绑定 New-API OpenAI-compatible provider,已接入 `MiniMax-M2.5`、`Qwen/Qwen3-Embedding-8B`、`Qwen/Qwen3-Reranker-8B`,并创建写作 chat app、全书解析 workflow app、工作区级 Datasets API key。S1 真实验收新增 `P1rDifyChatLiveAcceptanceIT` 与 `P1rDifyDatasetsContractLiveIT`,默认 `MUSE_P1R_EXTERNAL_ACCEPTANCE` 未启用时 honest skipped;显式 opt-in 后真实调用 Dify,chat app 返回 `status=success / messageIdPresent=true / summarySha256Prefix=565339bc4d33`,Datasets 完成 create/upload/index/retrieve,保留证据 dataset `cce33d41-c738-42d2-97b6-2d08a69965f2`,`indexingStatus=completed`、`score=0.7442479133605957`。踩坑已沉淀:Dify 官方 `.env` 的 `STORAGE_TYPE=opendal` 在本部署下会导致 worker `File not found`,已切到 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage` 并保留远端备份 `.env.s1-storage-before-local`;Dify 1.15 `/retrieve` 的 `retrieval_model` 必填 `search_method` 与 `reranking_enable`。回归证据:targeted live 2/0F/0E/0S、默认跳过 2/0F/0E/2S、`run-p1r-verification.sh local` 61/0F/0E/0S。RAGFlow 实例下线与已暴露 key 轮换仍为手工事项,不阻塞后续 S2。
|
||||
|
||||
**E1 content 创作闭环切片(2026-06-27)**:已补 AI suggestion 采纳归档、前端“改后合并”入口、IndexedDB 草稿键对账、旧知识草稿失效、工作台知识/导入/导出/记录入口、Block 版本历史最小 API/UI。Content merge 写 Canonical 与来源归因后,必须由 AI owner 写 `accepted` 状态、accepted decision archive、AI command、business audit;AI owner 不可用时整笔 merge 回滚,避免 Canonical 已写但候选仍 pending。Content 正文变更后通过 Knowledge owner API 将关联 pending draft 标为 `conflicted/needs_recheck` 并写 Knowledge draft decision archive;通知失败不回滚 Canonical 主写,Knowledge confirm 端仍按来源状态 fail-closed。Studio `CandidatePanel` 支持编辑最终正文并按 `accept_as_is`/`modify_then_merge` 提交,IndexedDB 草稿键改为 `workId+blockId+revision` 防跨作品/版本污染。`saveBlock`/`mergeBlockSuggestion` 现在写 `muse_content_block_revision_snapshot`,工作台“历史”Tab 只读展示 Canonical revision 快照;导入可创建真实任务,导出支持范围/格式选择、任务查询与下载凭证消费,知识/记录入口跳转对应工作台。fresh 证据:后端局部单测 `ContentSourceServiceTest` 25/0F/0E、`ContentAppServiceTest` 19/0F/0E、`MuseKnowledgeDraftInvalidationServiceTest` 3/0F/0E、`AiSuggestionMergeProjectionFacadeTest` 7/0F/0E;契约/覆盖门 `ContractFirstGateTest` 4/0F/0E + `P1rApiCoverageReportTest` 8/0F/0E;real-PG `P1rContentCoreCompletedApprovalIT` 13/0F/0E/0S + `P1rContentMergeSuggestionIT` 5/0F/0E/0S + `P1rContentMergeGeneratedSuggestionIT` 1/0F/0E/0S;studio `tsc -b --force` 通过、lint 0 errors、Vitest 12/12,追加 `AIPanel.contract.test.tsx` + `sse.test.ts` 22/0F/0E;导出 UI 追加 `useWorks.test.tsx`+`ExportWorkModal.test.tsx` 11/0F/0E、目标 ESLint 0 errors。共享 PG 写入闸门批准后已补跑 MSW-off Playwright 真后端 `accept-suggestion.spec.ts` 2/0F/0E:正路真 New-API 生成 suggestionId=79、authz `rpe-local-*`、Block revision 160→161、AI owner accepted decision archive 非空;负路 stale revision 业务冲突。边界:`muse-studio/src/types/content.ts` 生成类型因 openapi-typescript 版本漂移未同步,hook 内暂维护 `BlockRevision` 最小类型;完整导入向导已在 RC 后补,见下方记录;真后端导出下载 e2e 已在 2026-06-28 补跑通过,FileApi 异常仍按后端合同 fail-closed。
|
||||
|
||||
**E2 ai 智能体生命周期与候选处置闭环(2026-06-27)**:已补用户自建 Agent update/archive、初始版本自动创建、版本列表/激活/归档非当前版本、作品槽位 unbind、Studio reject 调后端 AI owner 决策归档。Agent update 会创建下一 active version 并更新 `current_version_id`,archive Agent 同步归档仍 active 的槽位绑定;slot unbind 将绑定行 revision+1 且 `status=archived`,运行时回到默认能力;WorkspacePage “放弃修改”不再只清本地候选,而是真打 `POST /suggestions/{id}/reject` 写 `rejected` 与 decision archive。启动修红:最新单体启动时暴露 `muse.codegen.importEnable` 缺省,已在 `muse-server/src/main/resources/application.yaml` 补 `import-enable:false`,随后 48080 成功启动并 `GET /app-api/muse/agents` smoke 返回 `code=0`。fresh 证据:后端 AI targeted `MuseAgentServiceTest`+`MuseAgentSlotServiceTest`+Controller annotation 63/0F/0E;契约门 `ContractFirstGateTest` 4/0F/0E;studio `tsc -b --force` 通过、目标 ESLint 0 errors、Vitest 29/0F/0E;MSW-off Playwright 真后端 `agent-create.spec.ts`+`agent-slot-bind.spec.ts`+`accept-suggestion.spec.ts` 5/0F/0E:采纳真生成 suggestionId=80、Block revision 161→162、accepted decision archive 非空;拒绝真生成 suggestionId=81、`status=rejected`、decision archive 非空;Agent 生命周期 DB 核验 v1/v2 current 切换与 v2 archived;Slot unbind DB 核验 revision 2→3、status archived、响应 `sourceStatus=unbound`。边界:本轮未跑 studio 全量 e2e。
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -225,6 +225,18 @@ muse:
|
||||
max-attempts: ${MUSE_AI_NEW_API_MAX_ATTEMPTS:3}
|
||||
retry-backoff-seconds: ${MUSE_AI_NEW_API_RETRY_BACKOFF_SECONDS:1,2,4}
|
||||
propagate-muse-context: ${MUSE_AI_NEW_API_PROPAGATE_MUSE_CONTEXT:true}
|
||||
dify:
|
||||
enabled: ${MUSE_AI_DIFY_ENABLED:false}
|
||||
base-url: ${MUSE_AI_DIFY_BASE_URL:}
|
||||
console-base-urls: ${MUSE_AI_DIFY_CONSOLE_BASE_URLS:}
|
||||
credentials:
|
||||
- ref: ${MUSE_AI_DIFY_WRITING_CREDENTIAL_REF:dify-writing-s1}
|
||||
api-key: ${MUSE_AI_DIFY_WRITING_API_KEY:}
|
||||
- ref: ${MUSE_AI_DIFY_PARSER_CREDENTIAL_REF:dify-parser-s1}
|
||||
api-key: ${MUSE_AI_DIFY_PARSER_API_KEY:}
|
||||
connect-timeout-seconds: ${MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS:5}
|
||||
non-stream-read-timeout-seconds: ${MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS:90}
|
||||
total-timeout-seconds: ${MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS:180}
|
||||
|
||||
--- #################### 芋道相关配置 ####################
|
||||
|
||||
|
||||
@ -0,0 +1,215 @@
|
||||
package cn.iocoder.muse.module.ai.application.muse.facade;
|
||||
|
||||
import cn.iocoder.muse.framework.common.util.json.JsonUtils;
|
||||
import cn.iocoder.muse.module.ai.framework.ai.config.MuseAiProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.net.URI;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.junit.jupiter.api.Assumptions.assumeTrue;
|
||||
|
||||
/**
|
||||
* S1 Dify chat app 外部 live acceptance 入口。
|
||||
*
|
||||
* <p>默认必须跳过,只有显式设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 才会真实调用外部 Dify。
|
||||
* 输出证据只保留 endpoint、app、message id 状态和 key 指纹,不能打印完整 key 或 provider 原始回答。</p>
|
||||
*/
|
||||
class P1rDifyChatLiveAcceptanceIT {
|
||||
|
||||
private static final String ACCEPTANCE_ENV = "MUSE_P1R_EXTERNAL_ACCEPTANCE";
|
||||
private static final String BASE_URL_ENV = "MUSE_AI_DIFY_BASE_URL";
|
||||
private static final String APP_ID_ENV = "MUSE_AI_DIFY_WRITING_APP_ID";
|
||||
private static final String API_KEY_ENV = "MUSE_AI_DIFY_WRITING_API_KEY";
|
||||
private static final String CREDENTIAL_REF_ENV = "MUSE_AI_DIFY_WRITING_CREDENTIAL_REF";
|
||||
private static final String DEFAULT_CREDENTIAL_REF = "dify-writing-s1";
|
||||
private static final String CHAT_MESSAGES_PATH = "/chat-messages";
|
||||
|
||||
@Test
|
||||
void shouldCallDifyChatThroughMuseRuntimeClientAndPrintRedactedEvidence() {
|
||||
assumeTrue(externalAcceptanceEnabled(), "P1R 外部验收未启用,设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 后才运行");
|
||||
|
||||
String baseUrl = requiredEnv(BASE_URL_ENV);
|
||||
String appId = requiredEnv(APP_ID_ENV);
|
||||
String apiKey = requiredEnv(API_KEY_ENV);
|
||||
String credentialRef = envOrDefault(CREDENTIAL_REF_ENV, DEFAULT_CREDENTIAL_REF);
|
||||
MuseAiRuntimeClient.RuntimeCommand command = liveCommand(appId, credentialRef);
|
||||
|
||||
RealDifyMuseAiRuntimeClient client = new RealDifyMuseAiRuntimeClient(difyProperties(baseUrl, apiKey,
|
||||
credentialRef));
|
||||
MuseAiRuntimeClient.RuntimeResponse response = client.execute(command);
|
||||
|
||||
assertNull(response.failure(), () -> "Dify chat live acceptance failed: "
|
||||
+ redactedFailure(response.failure()));
|
||||
assertNotNull(response.result(), "Dify chat live acceptance 必须返回 success result");
|
||||
assertEquals("success", response.result().status());
|
||||
assertNotNull(response.result().outputSummary(), "Dify chat outputSummary 不能为空");
|
||||
Object summary = response.result().outputSummary().get("summary");
|
||||
assertTrue(summary != null && !String.valueOf(summary).isBlank(), "Dify chat summary 不能为空");
|
||||
assertNoSensitiveText(String.valueOf(summary), "summary");
|
||||
|
||||
System.out.println(JsonUtils.toJsonString(redactedEvidence(baseUrl, apiKey, appId, credentialRef,
|
||||
response.result(), String.valueOf(summary))));
|
||||
}
|
||||
|
||||
private MuseAiRuntimeClient.RuntimeCommand liveCommand(String appId, String credentialRef) {
|
||||
String correlationId = "s1-dify-chat-live-" + Instant.now().toEpochMilli();
|
||||
return new MuseAiRuntimeClient.RuntimeCommand(
|
||||
"cmd-" + correlationId,
|
||||
"s1-dify-chat-task",
|
||||
"s1-dify-chat-job",
|
||||
correlationId,
|
||||
1,
|
||||
"dify-writing",
|
||||
"s1-dify-chat-live-v1",
|
||||
Map.of("taskType", "s1_dify_chat_live_acceptance",
|
||||
"intent", "external_dify_chat_smoke",
|
||||
"runtimeProvider", "dify",
|
||||
"providerRef", Map.of("dify", Map.of(
|
||||
"appType", "agent",
|
||||
"appId", appId,
|
||||
"credentialRef", credentialRef)),
|
||||
"outputContract", Map.of("format", "plain_text")),
|
||||
"s1-dify-chat-source-snapshot",
|
||||
"s1-dify-chat-runtime-envelope",
|
||||
"只输出 OK",
|
||||
timeoutPolicy(),
|
||||
MuseAiRuntimeClient.RetryPolicy.defaults());
|
||||
}
|
||||
|
||||
private MuseAiProperties difyProperties(String baseUrl, String apiKey, String credentialRef) {
|
||||
MuseAiProperties properties = new MuseAiProperties();
|
||||
MuseAiProperties.Dify dify = new MuseAiProperties.Dify();
|
||||
dify.setEnabled(true);
|
||||
dify.setBaseUrl(baseUrl);
|
||||
dify.setConnectTimeoutSeconds(intEnv("MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS", 5));
|
||||
dify.setNonStreamReadTimeoutSeconds(intEnv("MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS", 90));
|
||||
dify.setTotalTimeoutSeconds(intEnv("MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS", 180));
|
||||
MuseAiProperties.Dify.Credential credential = new MuseAiProperties.Dify.Credential();
|
||||
credential.setRef(credentialRef);
|
||||
credential.setApiKey(apiKey);
|
||||
dify.setCredentials(List.of(credential));
|
||||
properties.setDify(dify);
|
||||
return properties;
|
||||
}
|
||||
|
||||
private MuseAiRuntimeClient.TimeoutPolicy timeoutPolicy() {
|
||||
return new MuseAiRuntimeClient.TimeoutPolicy(
|
||||
intEnv("MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS", 5),
|
||||
intEnv("MUSE_AI_DIFY_FIRST_BYTE_TIMEOUT_SECONDS", 15),
|
||||
intEnv("MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS", 90),
|
||||
intEnv("MUSE_AI_DIFY_STREAM_IDLE_TIMEOUT_SECONDS", 30),
|
||||
intEnv("MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS", 180));
|
||||
}
|
||||
|
||||
private Map<String, Object> redactedEvidence(String baseUrl, String apiKey, String appId, String credentialRef,
|
||||
MuseAiRuntimeClient.RuntimeResult result, String summary) {
|
||||
Map<String, Object> evidence = new LinkedHashMap<>();
|
||||
evidence.put("acceptance", "s1-dify-chat-live");
|
||||
evidence.put("endpoint", endpointSummary(baseUrl, CHAT_MESSAGES_PATH));
|
||||
evidence.put("credentialRef", credentialRef);
|
||||
evidence.put("apiKey", secretFingerprint(apiKey));
|
||||
evidence.put("appId", appId);
|
||||
evidence.put("correlationId", result.correlationId());
|
||||
evidence.put("status", result.status());
|
||||
evidence.put("messageIdPresent", result.providerRequestId() != null && !result.providerRequestId().isBlank());
|
||||
evidence.put("summaryLength", summary.length());
|
||||
evidence.put("summarySha256Prefix", sha256Prefix(summary, 12));
|
||||
evidence.put("finishReason", result.finishReason());
|
||||
evidence.put("usage", result.tokenUsage());
|
||||
evidence.put("completedAt", result.completedAt());
|
||||
return evidence;
|
||||
}
|
||||
|
||||
private Map<String, Object> endpointSummary(String baseUrl, String path) {
|
||||
URI uri = URI.create(trimRight(baseUrl, "/") + path);
|
||||
Map<String, Object> endpoint = new LinkedHashMap<>();
|
||||
endpoint.put("scheme", uri.getScheme());
|
||||
endpoint.put("host", uri.getHost());
|
||||
endpoint.put("port", uri.getPort());
|
||||
endpoint.put("path", uri.getPath());
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
private Map<String, Object> secretFingerprint(String secret) {
|
||||
Map<String, Object> fingerprint = new LinkedHashMap<>();
|
||||
fingerprint.put("length", secret == null ? 0 : secret.length());
|
||||
fingerprint.put("sha256Prefix", sha256Prefix(secret == null ? "" : secret, 12));
|
||||
return fingerprint;
|
||||
}
|
||||
|
||||
private String redactedFailure(MuseAiRuntimeClient.RuntimeFailure failure) {
|
||||
if (failure == null) {
|
||||
return "none";
|
||||
}
|
||||
Map<String, Object> summary = new LinkedHashMap<>();
|
||||
summary.put("failureType", failure.failureType());
|
||||
summary.put("providerStatusCode", failure.providerStatusCode());
|
||||
summary.put("errorCode", failure.errorCode());
|
||||
summary.put("retryable", failure.retryable());
|
||||
summary.put("message", failure.sanitizedMessage());
|
||||
return JsonUtils.toJsonString(summary);
|
||||
}
|
||||
|
||||
private void assertNoSensitiveText(String value, String label) {
|
||||
String lower = value == null ? "" : value.toLowerCase(Locale.ROOT);
|
||||
assertFalse(lower.contains("token"), label + " 不应包含 token 字样");
|
||||
assertFalse(lower.contains("key"), label + " 不应包含 key 字样");
|
||||
assertFalse(lower.contains("authorization"), label + " 不应包含 authorization 字样");
|
||||
}
|
||||
|
||||
private boolean externalAcceptanceEnabled() {
|
||||
return "true".equalsIgnoreCase(System.getenv(ACCEPTANCE_ENV));
|
||||
}
|
||||
|
||||
private String requiredEnv(String name) {
|
||||
String value = System.getenv(name);
|
||||
assertTrue(value != null && !value.isBlank(), "缺少外部验收环境变量:" + name);
|
||||
return value;
|
||||
}
|
||||
|
||||
private String envOrDefault(String name, String defaultValue) {
|
||||
String value = System.getenv(name);
|
||||
return value == null || value.isBlank() ? defaultValue : value;
|
||||
}
|
||||
|
||||
private int intEnv(String name, int defaultValue) {
|
||||
String value = System.getenv(name);
|
||||
if (value == null || value.isBlank()) {
|
||||
return defaultValue;
|
||||
}
|
||||
return Integer.parseInt(value);
|
||||
}
|
||||
|
||||
private String sha256Prefix(String value, int length) {
|
||||
try {
|
||||
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||
String hex = HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
|
||||
return hex.substring(0, Math.min(length, hex.length()));
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new IllegalStateException("JDK 缺少 SHA-256 摘要算法", e);
|
||||
}
|
||||
}
|
||||
|
||||
private String trimRight(String value, String suffix) {
|
||||
String result = value == null ? "" : value.trim();
|
||||
while (result.endsWith(suffix)) {
|
||||
result = result.substring(0, result.length() - suffix.length());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
}
|
||||
@ -3,7 +3,8 @@
|
||||
> 模块跨会话记忆。真实现状权威见 [现状基线 §5.3](../../docs/agent-specs/2026-06-13-项目目标与模块现状基线.md);进度见 [总账](../../docs/mvp/进度总账.md)。本文件只记目标/边界/out-of-scope/现状指针/TODO,过时即更。
|
||||
|
||||
- **目标(owner 职责)**:Local KB / User KB / 全局知识处理 / Knowledge Draft / Knowledge Source Binding / 投影与索引状态。
|
||||
- **边界(不可违反)**:**进 Local KB 唯一入口=用户显式确认草稿或手动修正**;全书解析章节确认只产/推 Draft,不写正式知识;来源优先级 Local KB > 用户绑定 > 市场/全局;依赖 RAGFlow GraphRAG(不可用时 Unavailable 优雅降级);守 BC 边界不碰他域 `.dal`([bc-boundaries](../../.agents/rules/bc-boundaries.md))。
|
||||
- **边界(不可违反)**:**进 Local KB 唯一入口=用户显式确认草稿或手动修正**;全书解析章节确认只产/推 Draft,不写正式知识;来源优先级 Local KB > 用户绑定 > 市场/全局;1.0 运行时仍有 RAGFlow GraphRAG 历史链路,2.0 单人版迁移目标为 Dify Datasets(不可用时 Unavailable 优雅降级);守 BC 边界不碰他域 `.dal`([bc-boundaries](../../.agents/rules/bc-boundaries.md))。
|
||||
- **out-of-scope**:正文写入(归 content)。
|
||||
- **现状**:只读评估 78%;后端最完整模块之一(67 端点对齐契约),RAGFlow 真 HTTP + SSRF 防护;installed-KB 删除记录回显 bug 已修(2026-06-19:`MuseKnowledgeBindingMapperTest` + `MuseInstalledKnowledgeBaseServiceTest` 10/10,删除时写 `deleted=true` 且列表排除 `binding_status=deleted`);impact preview 已按 `muse_knowledge_source_binding_projection.owner_user_id` 输出 installed/source-binding owner 审计计数(2026-06-19:`MuseKnowledgeBaseServiceTest` + `MuseKnowledgeSourceBindingProjectionMapperTest` 30/30);source binding projection 回填闭环已修(2026-06-19:绑定确认为权威回填点写读模型行、unbind 按 bindingId 作用域撤销不误伤同源跨作品投影、唯一键补 work_id 修跨作品复用误判冲突〔V24〕;`MuseKnowledgeBindingServiceTest` + `MuseKnowledgeSourceBindingProjectionMapperTest` 11/11、整套件 214/214 绿);**读回端到端坐实 + unbind 真软删修复(2026-06-19)**:嵌入式 DB(H2)往返证 bind 写投影→`selectActiveByWorkId` 读回该来源、unbind 后读回消失、同源他作品投影存活;并修 unbind 软删真 bug——`deleted` 是 `@TableLogic`,实体 `setDeleted(true)` 被 MP 普通 update 剥离成空操作,改 `setSql("deleted = true")` 才真软删(`MuseKnowledgeSourceBindingProjectionRoundTripTest` 2/2、整套件 216/216 绿;新增 knowledge 模块首套嵌入式 DB 测试基建)。**GET bindings 读端点已补(2026-06-19)**:`AppMuseKnowledgeBindingController` 加 `GET /muse/works/{workId}/knowledge-bindings`→`listKnowledgeBindings`(requireWorkOwner 防 IDOR + selectActiveByWorkId 读回投影,id→String;happy-path + 越权 fail-closed 单测,整套件 218/218 绿);**bindings 读回后端腿(写→读→端点)已闭环,仅余 FE hook**。**FE 读回面板已加(2026-06-19)**:studio `useKnowledgeBindings` hook + `KnowledgeBindingsPanel`(镜像 DraftPanel)接入 `KnowledgePage`,契约+组件测试经 vitest 验证(全 studio 50/50、tsc 干净);live playwright e2e 需全栈 app(env 受限)未跑。bindings 写→读→端点→FE 读回展示链(除 live e2e)全通。**live e2e 已写 ready 未跑(2026-06-19)**:`muse-studio/e2e/knowledge-bindings.spec.ts`(镜像 graph spec,`playwright --list` 编译有效);活体受阻=远端 PG 宿主 100.64.0.8 离线(不在 tailnet)、无本地 PG 兜底,PG 宿主恢复后按 [.agents/knowledge §六] 起栈可跑。 **文本内容安全扫描已实现 + 摄入 scanning 关卡打通(2026-06-23,commit ec7637d)**:`KnowledgeContentScanService`(同步纯文本字符扫描——L1 非法 UTF-8/null/危险 C0-C1 控制符/Bidi 覆盖/体量硬阻断,L2 NFC+去零宽+换行规整,L3 prompt 注入软标记不阻断;特殊字符判定全用十六进制码点避免源码字面不可见字符),`KnowledgeFileFacade.materialize` 接入(passed→materialized+规整内容入 RAG、blocked→隔离带 reasonCode、扫描异常→fail-closed),替换原恒 `scanBlocked` 占位;`MaterializedFile.scanBlocked` 加 reasonCode/message(连带改 P1r live IT 构造签名)。单测 ScanService 9 + Facade 接线 3 + DocumentService 17 + P-B RetrievalApiImpl 11 全绿。**端到端真验(48080 + muse_slice_live + RAGFlow/New-API env 配齐)**:上传纯文本 document 4 → `scan_status=passed` → 处理链推进 → RAGFlow dataset(5950e9ee)+document(595cb63e)创建成功,坐实 scan 关卡 + muse→RAGFlow 集成贯通(对比 document 2 旧 jar 卡 SCAN_SERVICE_UNAVAILABLE、document 3 RAGFlow 未配 CONFIG_MISSING)。**E3 studio 与检索闭环已完成(2026-06-27)**:新增主动检索 App 端点与 `KnowledgeRetrievalPanel`,no_dataset 显式治理提示,自建 KB 上传真实后端读回,market KB handoff 正负路真 e2e;AI 生成 e2e 反查 `source_summary.contextAssembly` 证实 Knowledge Source 被真实消费。fresh 证据:knowledge targeted 25/0F/0E/0S,契约/迁移门 21/0F/0E/0S,studio Vitest 19/0F/0E、ESLint/tsc 通过,MSW-off Playwright knowledge 6/0F/0E + ai-generation 1/0F/0E。
|
||||
- **2.0.0 S1 Dify Datasets live 契约(2026-07-07)**:Dify 1.15.0 已创建工作区级 Datasets API key(`dataset-` 类型,非 app key、非单 dataset `ds-` key),并通过 `P1rDifyDatasetsContractLiveIT` 真跑 create dataset / upload file / indexing-status poll / retrieve。验收 dataset `cce33d41-c738-42d2-97b6-2d08a69965f2` 保留在 Dify 作为证据,`indexingStatus=completed`、retrieval `score=0.7442479133605957`;默认未 opt-in 时 honest skipped。Dify 1.15 `/v1/datasets/{id}/retrieve` 的 `retrieval_model` 必须显式包含 `search_method=semantic_search` 与 `reranking_enable=false`,只传 `top_k/score_threshold_enabled` 会 400。Dify 官方 compose 在本机部署必须用 `STORAGE_TYPE=local` + `STORAGE_LOCAL_PATH=storage`,否则 upload 成功后 worker 取文件会报 `File not found`。
|
||||
- **关键风险 / TODO**:前端图谱/发布深面仍薄;Task4 剩余 owner 缺口已收窄为 **document owner count / export_task_owner** 仍不能伪造,需后续 owner 事实或外部服务接入;已物化 KB 的召回触达/检索停用已在 E4 闭环。**2026-06-27:E4 market KB 召回触达已补并真验**:Knowledge 新增 `KnowledgeMarketSourceStatusChangedConsumer` 消费 Market `MarketAssetSourceStatusChangedEvent`,将本地 installed_ref KB 的 source binding projection 更新为 `recalled/blocked`,检索来源状态门返回 `omittedSources` 而不再送入 RAGFlow;MSW-off Playwright `market-kb-recall.spec.ts` 1/0F/0E,后端单测 `MuseKnowledgeSourceEventServiceTest`+`KnowledgeMarketSourceStatusChangedConsumerTest` 9/0F/0E。副本资源回收/更细补偿仍后置。**2026-06-27:E1 旧知识草稿失效 owner API 已补**:`MuseKnowledgeDraftInvalidationApi` 由 knowledge-api 定义、knowledge-server owner 实现;Content 正文变更只传 work/chapter/block/revision/commandId,不跨 BC 传正文。Knowledge 将匹配的 pending draft 更新为 `conflicted` + `source_action_policy=needs_recheck`,并写 `muse_knowledge_draft_decision(decision_type=recheck)` 作为审计归档;invalid command 跳过并 warn,Content 通知失败不回滚 Canonical 主写,confirm 端仍按 `needs_recheck` fail-closed。验证:`MuseKnowledgeDraftInvalidationServiceTest` 3/0F/0E;`P1rContentMergeSuggestionIT` real-PG 5/0F/0E/0S 穿透验证采纳后旧 draft 失效与 replay 不重复。**2026-06-28:导入章节确认 Draft owner API 已补**:`MuseKnowledgeDraftCreationApi` 由 knowledge-api 定义、knowledge-server owner 实现;Content/AI 只传章节 Shadow 结果与 sourceRef,Knowledge 负责创建 pending draft 与幂等 command,不写 Local KB Canonical,仍需用户后续显式确认草稿才进入正式知识。验证:`MuseKnowledgeDraftCreationApiImplTest` 与完整导入链路 targeted 后端测试通过。**parse 轮询 worker 已补齐并端到端真验(2026-06-23,commit 635045c)**:`MuseKnowledgeParseStatusPollWorker`(@Scheduled + 开关 `muse.knowledge.parse-poll-worker.enabled`,跨租户捞 parsing 任务轮询 RAGFlow 文档状态→run=done 推 version processingStatus=searchable、failed 标败、processing 保持、RAGFlow 不可达/异常不误判)+ `ProcessingTaskService.markRagflowParseCompleted/markRagflowParsePolling` + `Mapper.selectParsingForPoll`;worker 接线 7 测试 + 回归 40 全绿;启用开关后 document 4 自动 parsing→searchable(isSearchable=true),摄入链 **scan→RAGFlow→worker→searchable** 全程贯通(此前"RAGFlow backend 故障"系本机 HTTP 代理误诊,RAGFlow 实始终正常,真因=muse 缺 parse 轮询 worker)。
|
||||
|
||||
@ -0,0 +1,372 @@
|
||||
package cn.iocoder.muse.module.knowledge.application.muse.facade;
|
||||
|
||||
import cn.iocoder.muse.framework.common.util.json.JsonUtils;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.ProxySelector;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.junit.jupiter.api.Assumptions.assumeTrue;
|
||||
|
||||
/**
|
||||
* S1 Dify Datasets API 契约 live IT。
|
||||
*
|
||||
* <p>默认必须跳过,只有显式设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 才会真实调用外部 Dify。
|
||||
* 本测试会保留新建 dataset 作为验收证据,不做删除;失败与输出证据只打印脱敏摘要。</p>
|
||||
*/
|
||||
class P1rDifyDatasetsContractLiveIT {
|
||||
|
||||
private static final String ACCEPTANCE_ENV = "MUSE_P1R_EXTERNAL_ACCEPTANCE";
|
||||
private static final String BASE_URL_ENV = "MUSE_KNOWLEDGE_DIFY_BASE_URL";
|
||||
private static final String API_KEY_ENV = "MUSE_KNOWLEDGE_DIFY_DATASET_API_KEY";
|
||||
private static final Set<String> INDEXING_STATUSES = Set.of(
|
||||
"waiting", "parsing", "cleaning", "splitting", "indexing", "completed", "paused", "error");
|
||||
|
||||
@Test
|
||||
void shouldPinDifyDatasetsContractAndPrintRedactedEvidence() throws Exception {
|
||||
assumeTrue(externalAcceptanceEnabled(), "P1R 外部验收未启用,设置 MUSE_P1R_EXTERNAL_ACCEPTANCE=true 后才运行");
|
||||
|
||||
String baseUrl = requiredEnv(BASE_URL_ENV);
|
||||
String apiKey = requiredEnv(API_KEY_ENV);
|
||||
Duration timeout = Duration.ofSeconds(intEnv("MUSE_KNOWLEDGE_DIFY_TIMEOUT_SECONDS", 60));
|
||||
HttpClient client = HttpClient.newBuilder()
|
||||
.connectTimeout(timeout)
|
||||
// Dify 部署在内网时必须直连,避免 JVM 读取系统代理导致假 502。
|
||||
.proxy(ProxySelector.of(null))
|
||||
.build();
|
||||
|
||||
String suffix = String.valueOf(Instant.now().toEpochMilli());
|
||||
String datasetName = "dify-contract-" + suffix;
|
||||
JsonNode dataset = createDataset(client, baseUrl, apiKey, timeout, datasetName);
|
||||
String datasetId = requiredText(dataset, "id", "create dataset response");
|
||||
assertEquals(datasetName, requiredText(dataset, "name", "create dataset response"));
|
||||
assertEquals("high_quality", requiredText(dataset, "indexing_technique", "create dataset response"));
|
||||
|
||||
JsonNode upload = uploadDocument(client, baseUrl, apiKey, timeout, datasetId, suffix);
|
||||
String documentId = requiredText(upload.path("document"), "id", "upload document response.document");
|
||||
String batch = requiredText(upload, "batch", "upload document response");
|
||||
|
||||
JsonNode indexing = waitUntilCompleted(client, baseUrl, apiKey, timeout, datasetId, batch);
|
||||
JsonNode retrieve = retrieve(client, baseUrl, apiKey, timeout, datasetId, suffix);
|
||||
JsonNode firstRecord = firstRecord(retrieve);
|
||||
String content = requiredText(firstRecord.path("segment"), "content", "retrieve response records[0].segment");
|
||||
assertTrue(firstRecord.path("score").isNumber(), "retrieve response records[0].score 必须是 number");
|
||||
assertFalse(content.isBlank(), "retrieve response records[0].segment.content 不能为空");
|
||||
assertTrue(content.contains("Dify contract marker " + suffix),
|
||||
"retrieve response 必须命中本轮上传文档的 marker,避免旧数据误判为通过");
|
||||
|
||||
System.out.println(JsonUtils.toJsonString(redactedEvidence(baseUrl, apiKey, datasetName, datasetId,
|
||||
documentId, batch, indexing, content, firstRecord.path("score").asDouble())));
|
||||
}
|
||||
|
||||
private JsonNode createDataset(HttpClient client, String baseUrl, String apiKey, Duration timeout,
|
||||
String datasetName) throws IOException, InterruptedException {
|
||||
Map<String, Object> body = new LinkedHashMap<>();
|
||||
body.put("name", datasetName);
|
||||
body.put("permission", "only_me");
|
||||
body.put("indexing_technique", "high_quality");
|
||||
return sendJson(client, request(baseUrl, apiKey, "/v1/datasets", timeout)
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString(JsonUtils.toJsonString(body)))
|
||||
.build(), "create dataset", apiKey);
|
||||
}
|
||||
|
||||
private JsonNode uploadDocument(HttpClient client, String baseUrl, String apiKey, Duration timeout,
|
||||
String datasetId, String suffix) throws IOException, InterruptedException {
|
||||
Map<String, Object> data = new LinkedHashMap<>();
|
||||
data.put("indexing_technique", "high_quality");
|
||||
data.put("process_rule", Map.of("mode", "automatic"));
|
||||
String fileName = "dify-contract-" + suffix + ".txt";
|
||||
byte[] content = liveDocumentContent(suffix);
|
||||
MultipartPayload payload = multipartPayload(JsonUtils.toJsonString(data), fileName, content);
|
||||
return sendJson(client, request(baseUrl, apiKey,
|
||||
"/v1/datasets/" + datasetId + "/document/create-by-file", timeout)
|
||||
.header("Content-Type", payload.contentType())
|
||||
.POST(HttpRequest.BodyPublishers.ofByteArray(payload.body()))
|
||||
.build(), "upload document", apiKey);
|
||||
}
|
||||
|
||||
private JsonNode waitUntilCompleted(HttpClient client, String baseUrl, String apiKey, Duration timeout,
|
||||
String datasetId, String batch) throws IOException, InterruptedException {
|
||||
long deadline = System.nanoTime()
|
||||
+ Duration.ofSeconds(intEnv("MUSE_P1R_DIFY_INDEX_WAIT_SECONDS", 180)).toNanos();
|
||||
JsonNode last = null;
|
||||
while (System.nanoTime() < deadline) {
|
||||
last = sendJson(client, request(baseUrl, apiKey,
|
||||
"/v1/datasets/" + datasetId + "/documents/" + batch + "/indexing-status", timeout)
|
||||
.GET()
|
||||
.build(), "poll indexing status", apiKey);
|
||||
String status = indexingStatus(last);
|
||||
assertTrue(INDEXING_STATUSES.contains(status),
|
||||
() -> "Dify indexing-status 返回未知状态:" + safeSnippet(status));
|
||||
if ("completed".equals(status)) {
|
||||
return last;
|
||||
}
|
||||
JsonNode current = last;
|
||||
assertFalse("error".equals(status), () -> "Dify indexing-status 进入 error:"
|
||||
+ redactedJsonSummary(current, apiKey));
|
||||
Thread.sleep(2_000L);
|
||||
}
|
||||
throw new AssertionError("Dify indexing-status 未在等待窗口内 completed,last=" + redactedJsonSummary(last, apiKey));
|
||||
}
|
||||
|
||||
private JsonNode retrieve(HttpClient client, String baseUrl, String apiKey, Duration timeout,
|
||||
String datasetId, String suffix) throws IOException, InterruptedException {
|
||||
Map<String, Object> body = new LinkedHashMap<>();
|
||||
body.put("query", "Dify contract marker " + suffix + " 是什么?");
|
||||
body.put("retrieval_model", Map.of(
|
||||
"search_method", "semantic_search",
|
||||
"reranking_enable", false,
|
||||
"top_k", 3,
|
||||
"score_threshold_enabled", false));
|
||||
return sendJson(client, request(baseUrl, apiKey, "/v1/datasets/" + datasetId + "/retrieve", timeout)
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString(JsonUtils.toJsonString(body)))
|
||||
.build(), "retrieve dataset", apiKey);
|
||||
}
|
||||
|
||||
private HttpRequest.Builder request(String baseUrl, String apiKey, String path, Duration timeout) {
|
||||
return HttpRequest.newBuilder()
|
||||
.uri(endpoint(baseUrl, path))
|
||||
.timeout(timeout)
|
||||
.header("Authorization", "Bearer " + apiKey)
|
||||
.header("Accept", "application/json");
|
||||
}
|
||||
|
||||
private JsonNode sendJson(HttpClient client, HttpRequest request, String operation, String apiKey)
|
||||
throws IOException, InterruptedException {
|
||||
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
|
||||
if (response.statusCode() < 200 || response.statusCode() >= 300) {
|
||||
throw new AssertionError(operation + " HTTP failed: " + httpFailureSummary(response, apiKey));
|
||||
}
|
||||
return assertDoesNotThrow(() -> JsonUtils.parseTree(response.body()),
|
||||
operation + " response 必须是 JSON");
|
||||
}
|
||||
|
||||
private JsonNode firstRecord(JsonNode retrieve) {
|
||||
JsonNode records = retrieve.path("records");
|
||||
assertTrue(records.isArray() && !records.isEmpty(), "retrieve response records 不能为空");
|
||||
return records.get(0);
|
||||
}
|
||||
|
||||
private String indexingStatus(JsonNode response) {
|
||||
JsonNode data = response.path("data");
|
||||
if (data.isArray() && !data.isEmpty()) {
|
||||
return data.get(0).path("indexing_status").asText(data.get(0).path("status").asText(""));
|
||||
}
|
||||
if (data.isObject()) {
|
||||
return data.path("indexing_status").asText(data.path("status").asText(""));
|
||||
}
|
||||
return response.path("indexing_status").asText(response.path("status").asText(""));
|
||||
}
|
||||
|
||||
private String requiredText(JsonNode node, String field, String label) {
|
||||
String value = node.path(field).asText(null);
|
||||
assertTrue(value != null && !value.isBlank(), label + "." + field + " 不能为空");
|
||||
return value;
|
||||
}
|
||||
|
||||
private byte[] liveDocumentContent(String suffix) {
|
||||
String text = """
|
||||
Dify contract marker %s.
|
||||
This document is intentionally small and kept in Dify Datasets as external acceptance evidence.
|
||||
Muse Knowledge Dify contract smoke verifies dataset creation, upload, indexing status and retrieval.
|
||||
""".formatted(suffix);
|
||||
return text.getBytes(StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private MultipartPayload multipartPayload(String dataJson, String fileName, byte[] content) throws IOException {
|
||||
String boundary = "----MuseDify" + UUID.randomUUID().toString().replace("-", "");
|
||||
ByteArrayOutputStream body = new ByteArrayOutputStream();
|
||||
writeAscii(body, "--" + boundary + "\r\n");
|
||||
writeAscii(body, "Content-Disposition: form-data; name=\"data\"\r\n");
|
||||
writeAscii(body, "Content-Type: application/json; charset=UTF-8\r\n\r\n");
|
||||
body.write(dataJson.getBytes(StandardCharsets.UTF_8));
|
||||
writeAscii(body, "\r\n");
|
||||
writeAscii(body, "--" + boundary + "\r\n");
|
||||
writeAscii(body, "Content-Disposition: form-data; name=\"file\"; filename=\"" + fileName + "\"\r\n");
|
||||
writeAscii(body, "Content-Type: text/plain; charset=UTF-8\r\n\r\n");
|
||||
body.write(content);
|
||||
writeAscii(body, "\r\n");
|
||||
writeAscii(body, "--" + boundary + "--\r\n");
|
||||
return new MultipartPayload(body.toByteArray(), "multipart/form-data; boundary=" + boundary);
|
||||
}
|
||||
|
||||
private void writeAscii(ByteArrayOutputStream body, String value) throws IOException {
|
||||
body.write(value.getBytes(StandardCharsets.US_ASCII));
|
||||
}
|
||||
|
||||
private Map<String, Object> redactedEvidence(String baseUrl, String apiKey, String datasetName,
|
||||
String datasetId, String documentId, String batch,
|
||||
JsonNode indexing, String content, double score) {
|
||||
Map<String, Object> evidence = new LinkedHashMap<>();
|
||||
evidence.put("acceptance", "s1-dify-datasets-contract-live");
|
||||
evidence.put("endpoint", endpointSummary(baseUrl, "/v1/datasets"));
|
||||
evidence.put("apiKey", secretFingerprint(apiKey));
|
||||
evidence.put("datasetName", datasetName);
|
||||
evidence.put("datasetId", datasetId);
|
||||
evidence.put("documentId", documentId);
|
||||
evidence.put("batch", batch);
|
||||
evidence.put("indexingStatus", indexingStatus(indexing));
|
||||
evidence.put("retrievedContentLength", content.length());
|
||||
evidence.put("retrievedContentSha256Prefix", sha256Prefix(content, 12));
|
||||
evidence.put("score", score);
|
||||
evidence.put("datasetRetained", true);
|
||||
return evidence;
|
||||
}
|
||||
|
||||
private Map<String, Object> endpointSummary(String baseUrl, String path) {
|
||||
URI uri = endpoint(baseUrl, path);
|
||||
Map<String, Object> endpoint = new LinkedHashMap<>();
|
||||
endpoint.put("scheme", uri.getScheme());
|
||||
endpoint.put("host", uri.getHost());
|
||||
endpoint.put("port", uri.getPort());
|
||||
endpoint.put("path", uri.getPath());
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
private URI endpoint(String baseUrl, String path) {
|
||||
String normalizedBase = trimRight(baseUrl, "/");
|
||||
String normalizedPath = path.startsWith("/") ? path : "/" + path;
|
||||
if (normalizedBase.endsWith("/v1") && normalizedPath.startsWith("/v1/")) {
|
||||
normalizedPath = normalizedPath.substring("/v1".length());
|
||||
}
|
||||
return URI.create(normalizedBase + normalizedPath);
|
||||
}
|
||||
|
||||
private String httpFailureSummary(HttpResponse<String> response, String apiKey) {
|
||||
Map<String, Object> summary = new LinkedHashMap<>();
|
||||
summary.put("status", response.statusCode());
|
||||
JsonNode root = parseTreeQuietly(response.body());
|
||||
if (root != null) {
|
||||
summary.put("code", safeSnippet(firstText(root.path("code"), root.path("error"), root.path("error_code")),
|
||||
apiKey));
|
||||
summary.put("message", safeSnippet(firstText(root.path("message"), root.path("error_description"),
|
||||
root.path("detail")), apiKey));
|
||||
} else {
|
||||
summary.put("code", "");
|
||||
summary.put("message", safeSnippet(response.body(), apiKey));
|
||||
}
|
||||
return JsonUtils.toJsonString(summary);
|
||||
}
|
||||
|
||||
private JsonNode parseTreeQuietly(String body) {
|
||||
if (body == null || body.isBlank()) {
|
||||
return JsonUtils.parseTree("{}");
|
||||
}
|
||||
try {
|
||||
return JsonUtils.parseTree(body);
|
||||
} catch (RuntimeException ex) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private String redactedJsonSummary(JsonNode node, String apiKey) {
|
||||
if (node == null || node.isMissingNode() || node.isNull()) {
|
||||
return "{}";
|
||||
}
|
||||
Map<String, Object> summary = new LinkedHashMap<>();
|
||||
summary.put("status", safeSnippet(indexingStatus(node), apiKey));
|
||||
summary.put("code", safeSnippet(firstText(node.path("code"), node.path("error"), node.path("error_code")),
|
||||
apiKey));
|
||||
summary.put("message", safeSnippet(firstText(node.path("message"), node.path("error_description"),
|
||||
node.path("detail")), apiKey));
|
||||
return JsonUtils.toJsonString(summary);
|
||||
}
|
||||
|
||||
private String firstText(JsonNode... nodes) {
|
||||
for (JsonNode node : nodes) {
|
||||
if (node != null && !node.isMissingNode() && !node.isNull()) {
|
||||
String value = node.asText(null);
|
||||
if (value != null && !value.isBlank()) {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private String safeSnippet(String value, String... secrets) {
|
||||
String normalized = value == null ? "" : value.replaceAll("\\s+", " ").trim();
|
||||
for (String secret : secrets) {
|
||||
if (secret != null && !secret.isBlank()) {
|
||||
normalized = normalized.replace(secret, "<redacted-secret>");
|
||||
}
|
||||
}
|
||||
String lower = normalized.toLowerCase(Locale.ROOT);
|
||||
if (lower.contains("authorization") || lower.contains("bearer ")
|
||||
|| lower.contains("token") || lower.contains("api_key") || lower.contains("apikey")) {
|
||||
return "<redacted>";
|
||||
}
|
||||
return normalized.length() <= 200 ? normalized : normalized.substring(0, 200);
|
||||
}
|
||||
|
||||
private Map<String, Object> secretFingerprint(String secret) {
|
||||
Map<String, Object> fingerprint = new LinkedHashMap<>();
|
||||
fingerprint.put("length", secret == null ? 0 : secret.length());
|
||||
fingerprint.put("sha256Prefix", sha256Prefix(secret == null ? "" : secret, 12));
|
||||
return fingerprint;
|
||||
}
|
||||
|
||||
private boolean externalAcceptanceEnabled() {
|
||||
return "true".equalsIgnoreCase(System.getenv(ACCEPTANCE_ENV));
|
||||
}
|
||||
|
||||
private String requiredEnv(String name) {
|
||||
String value = System.getenv(name);
|
||||
assertTrue(value != null && !value.isBlank(), "缺少外部验收环境变量:" + name);
|
||||
return value;
|
||||
}
|
||||
|
||||
private int intEnv(String name, int defaultValue) {
|
||||
String value = System.getenv(name);
|
||||
if (value == null || value.isBlank()) {
|
||||
return defaultValue;
|
||||
}
|
||||
return Integer.parseInt(value);
|
||||
}
|
||||
|
||||
private String sha256Prefix(String value, int length) {
|
||||
try {
|
||||
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||
String hex = HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
|
||||
return hex.substring(0, Math.min(length, hex.length()));
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new IllegalStateException("JDK 缺少 SHA-256 摘要算法", e);
|
||||
}
|
||||
}
|
||||
|
||||
private String trimRight(String value, String suffix) {
|
||||
String result = value == null ? "" : value.trim();
|
||||
while (result.endsWith(suffix)) {
|
||||
result = result.substring(0, result.length() - suffix.length());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private record MultipartPayload(byte[] body, String contentType) {
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,5 +1,5 @@
|
||||
# P1R 外部真实验收依赖。
|
||||
# 该文件按用户授权明文提交,仅用于内网 P1R-4 New-API 与 P1R-5 RAGFlow 验收。
|
||||
# 该文件按用户授权明文提交,仅用于内网 New-API / RAGFlow / Dify 外部验收。
|
||||
# 使用方式:set -a && . muse-cloud/scripts/dev/p1r-external-acceptance.env && set +a
|
||||
|
||||
# New-API 系统管理令牌,用于需要管理接口或渠道配置核查的场景。
|
||||
@ -27,3 +27,27 @@ MUSE_KNOWLEDGE_RAGFLOW_API_KEY=ragflow-icnZ7zUcIz3oGCrpyFB2aosK-WTbvTQRaNUBPu6Rt
|
||||
MUSE_KNOWLEDGE_RAGFLOW_TIMEOUT_SECONDS=30
|
||||
MUSE_KNOWLEDGE_RAGFLOW_RETRY_BUDGET=0
|
||||
MUSE_KNOWLEDGE_RAGFLOW_GRAPHRAG_ATTRIBUTION_READY=false
|
||||
|
||||
# --- DIFY_S1_BEGIN ---
|
||||
# Dify 1.15.0 外部验收入口。
|
||||
# S1 起 Dify 是 2.0.0 单人版唯一新增外部服务;app key 与 dataset key 不可混用。
|
||||
MUSE_AI_DIFY_ENABLED=true
|
||||
MUSE_AI_DIFY_BASE_URL=http://100.64.0.8:18080/v1
|
||||
MUSE_AI_DIFY_CONSOLE_BASE_URLS=http://100.64.0.8:18080
|
||||
MUSE_AI_DIFY_WRITING_APP_ID=75f105e8-dbca-4aa6-a296-2113a65ad005
|
||||
MUSE_AI_DIFY_WRITING_API_KEY=app-Un5DDoOiK7eQc617ahdn7ku2
|
||||
MUSE_AI_DIFY_WRITING_CREDENTIAL_REF=dify-writing-s1
|
||||
MUSE_AI_DIFY_PARSER_APP_ID=fed4d25c-1289-4090-9771-a811c89aba44
|
||||
MUSE_AI_DIFY_PARSER_API_KEY=app-It4V7WukHohP8rhWV0gOn2H5
|
||||
MUSE_AI_DIFY_PARSER_CREDENTIAL_REF=dify-parser-s1
|
||||
MUSE_AI_DIFY_CONNECT_TIMEOUT_SECONDS=5
|
||||
MUSE_AI_DIFY_FIRST_BYTE_TIMEOUT_SECONDS=15
|
||||
MUSE_AI_DIFY_NON_STREAM_READ_TIMEOUT_SECONDS=90
|
||||
MUSE_AI_DIFY_STREAM_IDLE_TIMEOUT_SECONDS=30
|
||||
MUSE_AI_DIFY_TOTAL_TIMEOUT_SECONDS=180
|
||||
MUSE_KNOWLEDGE_DIFY_BASE_URL=http://100.64.0.8:18080/v1
|
||||
MUSE_KNOWLEDGE_DIFY_DATASET_API_KEY=dataset-Nyf7yWhF2CMFsD5TAqD39WHP
|
||||
MUSE_KNOWLEDGE_DIFY_TIMEOUT_SECONDS=60
|
||||
MUSE_KNOWLEDGE_DIFY_RETRY_BUDGET=0
|
||||
MUSE_P1R_DIFY_INDEX_WAIT_SECONDS=180
|
||||
# --- DIFY_S1_END ---
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# P1R 分层验收入口。
|
||||
#
|
||||
# WHY: 默认 Maven CI 只能证明纯本地层绿,不能证明真实 PostgreSQL、RAGFlow 或 New-API 链路。
|
||||
# WHY: 默认 Maven CI 只能证明纯本地层绿,不能证明真实 PostgreSQL、New-API、RAGFlow 或 Dify 链路。
|
||||
# 本脚本把三类证据显式分层,避免把台账门禁、默认跳过的 live IT 或 mock 绿混成"真验证"。
|
||||
set -euo pipefail
|
||||
|
||||
@ -11,9 +11,9 @@ REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." >/dev/null 2>&1 && pwd)"
|
||||
CLOUD_DIR="${REPO_ROOT}/muse-cloud"
|
||||
|
||||
LOCAL_TESTS="P1rApiCoverageReportTest,P1r*RealApiGateTest,AgentsInfraIntegrityTest,ContractFirstGateTest,BcBoundaryArchTest,AiGrantRuntimeBoundaryArchTest"
|
||||
REAL_PG_TESTS="${P1R_REAL_PG_TESTS:-P1r*IT,!P1rAiRuntimeEndToEndLiveAcceptanceIT,!P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,!P1rMarketKbForkMaterializationIT,!P1rNewApiLiveAcceptanceIT,!P1rRagFlowLiveAcceptanceIT}"
|
||||
REAL_PG_TESTS="${P1R_REAL_PG_TESTS:-P1r*IT,!P1rAiRuntimeEndToEndLiveAcceptanceIT,!P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,!P1rMarketKbForkMaterializationIT,!P1rImportLlmNewApiLiveAcceptanceIT,!P1rNewApiLiveAcceptanceIT,!P1rRagFlowLiveAcceptanceIT,!P1rDifyChatLiveAcceptanceIT,!P1rDifyDatasetsContractLiveIT}"
|
||||
EXTERNAL_LIVE_TESTS="${P1R_EXTERNAL_LIVE_TESTS:-P1rAiRuntimeEndToEndLiveAcceptanceIT,P1rKnowledgeRuntimeEndToEndLiveAcceptanceIT,P1rMarketKbForkMaterializationIT}"
|
||||
MODULE_LIVE_TESTS="${P1R_MODULE_LIVE_TESTS:-P1rNewApiLiveAcceptanceIT,P1rRagFlowLiveAcceptanceIT}"
|
||||
MODULE_LIVE_TESTS="${P1R_MODULE_LIVE_TESTS:-P1rNewApiLiveAcceptanceIT,P1rImportLlmNewApiLiveAcceptanceIT,P1rRagFlowLiveAcceptanceIT,P1rDifyChatLiveAcceptanceIT,P1rDifyDatasetsContractLiveIT}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
@ -26,7 +26,7 @@ usage() {
|
||||
分层:
|
||||
local 纯本地门禁: coverage/RealApiGate/ArchUnit/契约结构。无 DB、无外部服务。
|
||||
real-pg 非 live P1r*IT: 真实 PostgreSQL _test 库,会 flyway.clean()。
|
||||
external-live live P1r*IT + module live IT: 真实 PG + New-API/RAGFlow,必须显式 opt-in。
|
||||
external-live live P1r*IT + module live IT: 真实 PG + New-API/RAGFlow/Dify,必须显式 opt-in。
|
||||
all 依次运行 local、real-pg、external-live。
|
||||
|
||||
real-pg / external-live 必需环境变量:
|
||||
@ -36,7 +36,7 @@ real-pg / external-live 必需环境变量:
|
||||
|
||||
external-live 额外要求:
|
||||
MUSE_P1R_EXTERNAL_ACCEPTANCE=true
|
||||
已加载 muse-cloud/scripts/dev/p1r-external-acceptance.env 或等价 New-API/RAGFlow 环境变量。
|
||||
已加载 muse-cloud/scripts/dev/p1r-external-acceptance.env 或等价 New-API/RAGFlow/Dify 环境变量。
|
||||
|
||||
示例:
|
||||
set -a && . "$HOME/.config/muse-repo/infra.env" && set +a
|
||||
@ -184,7 +184,7 @@ live_arg_line() {
|
||||
|
||||
run_local() {
|
||||
echo "== P1R local gates: ${LOCAL_TESTS}"
|
||||
(cd "${CLOUD_DIR}" && mvn -B -pl muse-server -am test \
|
||||
(cd "${CLOUD_DIR}" && MUSE_P1R_EXTERNAL_ACCEPTANCE=false mvn -B -pl muse-server -am test \
|
||||
-Dtest="${LOCAL_TESTS}" \
|
||||
-DfailIfNoSpecifiedTests=false \
|
||||
-Dsurefire.failIfNoSpecifiedTests=false)
|
||||
@ -220,7 +220,7 @@ run_external_live() {
|
||||
-DargLine="$(live_arg_line)")
|
||||
|
||||
echo "== P1R external-live module IT: ${MODULE_LIVE_TESTS}"
|
||||
(cd "${CLOUD_DIR}" && mvn -B -pl muse-module-ai/muse-module-ai-server,muse-module-knowledge/muse-module-knowledge-server test \
|
||||
(cd "${CLOUD_DIR}" && mvn -B -pl muse-module-ai/muse-module-ai-server,muse-module-knowledge/muse-module-knowledge-server -am test \
|
||||
-Dtest="${MODULE_LIVE_TESTS}" \
|
||||
-DfailIfNoSpecifiedTests=false \
|
||||
-Dsurefire.failIfNoSpecifiedTests=false \
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user