E2e rag manifest and token fixes - #2645
Merged
hanyunfan merged 9 commits intoJul 29, 2026
Merged
Conversation
…en count Manifest verification (Item 1): different implementations chunk and index the corpus at different times, so passage counts, corpus sha256, and per-index sample-embedding cosine legitimately differ across correct submissions. Gate verify_manifest() solely on mean probe-query top-K document-URL overlap (retrieval_accuracy >= 0.999), with URLs normalized to a corpus-stable key and compared as sets. Passage count, sha256, and cosine are now informational metrics that never fail; only embedding_dim mismatch (wrong retriever model) still fails. Widen the probe set to 100 queries so the 99.9% threshold is statistically meaningful. Thread --retrieval_threshold through datasetup_accuracy_eval.py. TEST09 token count (Item 3): the SUT logged the answer's UTF-8 byte length as n_tokens, but run_verification.py decodes response data as int32 token IDs and counts them. Emit an int32 blob whose length equals the real answer-generation token count (LLM usage.completion_tokens, exposed via LLMLogger.get_component_output_tokens), captured before end_query() clears the per-query state; pass that count as n_tokens. The human-readable answer stays in results.json for auditing. Update the TEST09 threshold to the real reference mean OSL of 273.81 (n=810 from logs_result.json) -> [246.43, 301.19]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gate Root cause of cross-system manifest failures is corpus sourcing, not string formatting. Verified against the official download: docs.tar.gz ships a FROZEN Wikipedia snapshot (2515 files) with a fixed url_mapping.json, but download_docs.py re-scrapes live Wikipedia and fetches a newer revision (e.g. 100_metres: 1360491832 vs 1364600010) -- identical filenames, different bytes, different passage counts, and genuinely different retrieval results. Any submission that ingests the official corpus stores the identical original_url per document, so retrieved URLs match verbatim. - download_dataset_and_models.sh: extract docs.tar.gz into doc_html/ and copy url_mapping.json; drop the download_docs.py re-scrape from the next-steps and warn against it. - README: document that the corpus is frozen and download_docs.py is for snapshot regeneration only, not submissions. - db_manifest.py: compare probe-query top-K URLs verbatim as sets (remove _normalize_url and the now-unused re import). With a shared frozen corpus, normalization is unnecessary and its #anchor stripping would mask real section-level retrieval divergence the gate should catch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The docstore sha256 depends on passage ordering, which varies with ingestion thread timing across systems, so it can never match even for identical source documents. It was already informational (never failed the check); remove the comparison entirely so it no longer appears as a confusing MISMATCH diagnostic. Retrieval overlap remains the sole gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DB manifest gate compares canonical Wikipedia URLs across systems, but datasetup passages only carried the filename-form source. Load the frozen corpus url_mapping.json and stamp each passage's metadata with its original_url so cross-system manifest verification can match documents. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
Contributor
|
WG: need to merge this asap, AMD is reviewing this PR. @mrmhodak |
Contributor
Author
|
@rpoornac Hi Rajesh, appreciate your review on this PR. |
rpoornac
reviewed
Jul 28, 2026
| retriever_model=None): | ||
| def evaluate_accuracy(log_dir, output_dir, database_path, retriever_model=None, | ||
| manifest_path=None, cosine_threshold=0.9999, | ||
| top_k_depth=3, retrieval_threshold=0.90): |
Contributor
There was a problem hiding this comment.
For accuracy we use retrieval_threshold=0.90 and performance its retrieval_threshold=0.95 (line # 10)?
Contributor
There was a problem hiding this comment.
it has to be the same 0.95.
rpoornac
reviewed
Jul 28, 2026
rpoornac
left a comment
rpoornac
left a comment
Contributor
There was a problem hiding this comment.
Please clarify on retrieval_threshold=0.90 vs. 0.95.
Contributor
|
@rpoornac Thanks for catching this. Fix was pushed. |
rpoornac
approved these changes
Jul 28, 2026
Contributor
Author
|
@mrmhodak could you please help merge this PR |
pgmpablo157321
approved these changes
Jul 28, 2026
Contributor
Author
|
@pgmpablo157321 would this be auto merged? |
Contributor
|
Due to the time-sensitive nature of the submission deadline, I will merge this PR for now and temporarily waive the autopep8 check above. Please address the formatting issue in a follow-up PR. Error: Not autopep8-formatted - run: autopep8 --in-place -a --max-line-length 79 e2e-rag/reference_SUT.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FIX 1:
Dataset download (root cause): The corpus was under-downloaded (2503 vs. the expected 2515 files); the 12 missing Wikipedia pages degraded retrieval,cascading into empty/short answers, lower OSL, and the TEST09 token-length
failures. Restoring the full 2515-file corpus fixed retrieval and answer generation at the source, raising OSL back into range.
FIX 2:
The previous verification was too rigid: it required a byte-for-byte reproducible DB, effectively forcing a single embedding implementation. It compared top-K retrieval per query by exact rank order, and
it enforced identical embedding numerics — which differ across frameworks even for the same model.
We found that independently rebuilding the same DB would fail this check. Passages with equal (or near-equal) similarity scores frequently swap ranks with their neighbors — especially under the
approximate HNSW algorithm and the non-deterministic passage ordering that multi-threaded ingestion produces. These DBs are functionally identical yet fail an order-sensitive comparison.
We therefore propose verifying functional equivalence instead: