Repository navigation
fix(models): 校验 Embedding 向量维度与配置一致 - #1145
Merged
Merged
Conversation
_extract_embeddings 在 xerrors#1143 建立了「响应与输入逐条对应」的契约,但不校验每条 向量的长度;test_connection 只比对 embeddings[0],且仅用于连接探针。 MilvusKB._create_new_collection 用配置维度固定集合 schema,写入时不再比对; pymilvus 只校验列数,实测 Prepare.prepare_data 接受 [4, 2, 4] 对 schema dim=4 的批次。供应商切换模型版本、截断向量或聚合网关路由到维度不同的后端时,维度 混杂的批次会静默写入集合,而 PostgreSQL 的 chunk 事实已先行写入。 在响应边界新增维度校验:_extract_embeddings 接收 dimension,逐条比对长度; _require_one_vector_per_input 一并校验,供 batch_encode / abatch_encode 每批复用。 dimension 为 None 时保持既有行为。 Fixes xerrors#1144
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
修改内容
Embedding响应里的向量维度必须与模型配置一致,否则整批失败。修改见backend/yuxi/modules/models/embed.py。#1143已在同一解析边界建立「响应与输入逐条对应」的契约(条数、索引唯一性、拒绝null向量),但不校验每条向量的长度。test_connection只比对embeddings[0],且只用于连接探针。MilvusKB._create_new_collection用embedding_info.dimension or 1024固定集合 schema 的向量维度,写入时不再比对。pymilvus只校验列数,不校验每条向量的维度。我用 pymilvus 3.0.2 实测:供应商切换模型版本、对超长输入返回截断向量、或聚合网关路由到维度不同的后端时,维度混杂的批次会静默写入集合。PostgreSQL 的 chunk 事实已先行写入(
_insert_chunks_to_stores注释写明「PG 为权威源」),问题无法从业务侧发现,只能重建集合。决策
_extract_embeddings新增dimension参数,逐条校验len(item["embedding"]) == dimension,不符抛出固定ValueError。OtherEmbedding.encode/aencode传入self.dimension(来自ModelInfo.dimension)。_require_one_vector_per_input一并校验维度,batch_encode/abatch_encode在每批累加前复用,覆盖「首批正常、后续批维度不符」。dimension is None时跳过校验,保持未配置模型的既有行为。不选择在
MilvusKB写入前校验:那里已丢失「模型配置维度」的语义来源,且校验点会分散到每个调用方。详情见
docs/develop-guides/decisions/implemented/2026-11-04-embedding-dimension-validation.md。验证
backend/test/unit/models/test_embed_dimension.py,13 项:同步/异步直接调用的短维度、过长维度、仅第二项维度错误;_extract_embeddings拒绝不符、接受一致、未配置时跳过;批处理后续批次维度不符仍整批失败。test_embed_alignment.py(fix(models): 校验 Embedding 响应与输入逐条对应 #1143 的回归)合并运行,49 项通过。dimension参数尚不存在);修复后全部通过。ruff check、ruff format --check、ruff check --select I在backend/下对改动文件通过。未覆盖
test/unit全量套件受 POSIX 专属路径(os.O_DIRECTORY)阻塞,因此只在隔离环境运行了test/unit/models下的相关用例;容器内全量需由 CI 执行。Fixes #1144