Skip to content

fix(filter): exclude padding from semantic deduplication embeddings - #592

Merged
SunnyHaze merged 1 commit into
OpenDCAI:mainfrom
JustinBinber:codex/fix-semantic-dedup-embeddings
Nov 18, 2026
Merged

SunnyHaze merged 1 commit into
OpenDCAI:mainfrom
JustinBinber:codex/fix-semantic-dedup-embeddings

Conversation

@JustinBinber

Copy link
Copy Markdown
Contributor

Fixes #591

Problem

Padding positions currently contribute to sentence embeddings. Adding a longer row can change the similarity between existing short texts and cause the operator to discard an otherwise retained row.

Change

Use attention-mask-weighted mean pooling in get_text_embedding, retaining existing special tokens, normalization, thresholds and duplicate-selection behavior. No public API or dependency changes.

Add offline CPU regressions using a locally saved tiny BERT and tokenizer. They cover batch invariance for short/empty strings, unchanged unpadded pooling, and public run() decisions through both input-key forms, including duplicate removal and output labels.

Validation

  • On main: the new file reports 5 failed, 1 passed; the failures demonstrate embedding changes and an incorrectly discarded row.
  • After the fix: 6 passed in the new file on both Transformers 4.57.6 and current 5.23.0 (CPU, Python 3.11.7).
  • Explicit offline regression scope (new file, K-center sampling, storage format contract, streaming record contract): 79 passed, 1 skipped (optional openpyxl is not installed).
  • New test file passes Black and Ruff; git diff --check passes.
  • No external model requests or downloaded pretrained weights. GPU and pretrained semantic-quality evaluation were not run.

Related PR #571 also touches the embedding helper for batching, but still averages padding positions. This change only corrects pooling and may need a small rebase if that PR lands first.

@SunnyHaze

Copy link
Copy Markdown
Collaborator

LGTM, thanks!

@SunnyHaze
SunnyHaze merged commit 316322e into OpenDCAI:main Nov 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SemDeduplicateFilter includes padding in sentence embeddings

2 participants