From 7d979c7b325923f80c5bd0d5d76ae67b8ef4b090 Mon Sep 17 00:00:00 2001 From: Yue Sun Date: Thu, 25 Jun 2026 16:36:29 +0800 Subject: [PATCH] docs: publish model accuracy report via mkdocs from main Move the reports/ folder (model_accuracy_report.html, download_report.py, README.md) from the gh-pages branch into docs/reports/ on main so the report is version-controlled and published by the docs workflow instead of being hand-edited on gh-pages. - mkdocs publishes the HTML as a versioned static asset; download_report.py and README.md are excluded from the built site via exclude_docs. - supported-models.md links to the report with a source-relative path so it resolves within each docs version. - Bump the report's displayed version: WinML CLI 0.1.0 -> 0.2.0. - Update download_report.py/README prose for the new PR-based refresh flow. - Add a ruff per-file-ignore for the relocated download script. - check-yaml: add --unsafe so mkdocs.yml custom tags don't block commits. --- .pre-commit-config.yaml | 1 + docs/reference/supported-models.md | 2 +- docs/reports/README.md | 59 ++ docs/reports/download_report.py | 172 ++++++ docs/reports/model_accuracy_report.html | 787 ++++++++++++++++++++++++ mkdocs.yml | 2 + pyproject.toml | 1 + 7 files changed, 1023 insertions(+), 1 deletion(-) create mode 100644 docs/reports/README.md create mode 100644 docs/reports/download_report.py create mode 100644 docs/reports/model_accuracy_report.html diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d189b0585..ade35422b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,6 +6,7 @@ repos: - id: trailing-whitespace args: [--markdown-linebreak-ext=md] - id: check-yaml + args: [--unsafe] - repo: https://github.com/Lucas-C/pre-commit-hooks rev: v1.5.5 diff --git a/docs/reference/supported-models.md b/docs/reference/supported-models.md index 4b2b8f094..0cd536b7e 100644 --- a/docs/reference/supported-models.md +++ b/docs/reference/supported-models.md @@ -2,7 +2,7 @@ Windows ML CLI has validated a set of models for compatibility across all Execution Providers (EPs)—see the full -[model compatibility report](https://microsoft.github.io/winml-cli/reports/model_compatibility_report.html). +[Model Accuracy Report](../reports/model_accuracy_report.html). winml-cli supports a wide range of model architectures and tasks. This page lists what's validated and how to discover model support. diff --git a/docs/reports/README.md b/docs/reports/README.md new file mode 100644 index 000000000..2e5f595a2 --- /dev/null +++ b/docs/reports/README.md @@ -0,0 +1,59 @@ +# Model accuracy report + +This folder hosts the **model accuracy report** published on the winml-cli +documentation site, plus the script used to refresh it. + +| File | Purpose | +| --- | --- | +| `model_accuracy_report.html` | The self-contained report page (all data embedded inline). | +| `download_report.py` | Fetches the latest report from the private artifacts repo. | +| `README.md` | This file. | + +> These files live in `docs/reports/` on `main`. mkdocs publishes +> `model_accuracy_report.html` as a static asset under each docs version +> (e.g. `.../latest/reports/model_accuracy_report.html`); `download_report.py` +> and this README are excluded from the built site via `exclude_docs` in +> `mkdocs.yml`. Refreshing the report is a normal docs change to `main`. + +## Source + +The report is generated in the private `gim-home/ModelKitArtifacts` repo and +lives at `e2e_model_coverage_result/model_accuracy_report.html` on its +`site-src` branch. The page embeds all of its data inline, so only this one +file needs to be published — no JSON or other assets are fetched at runtime. + +## Refreshing the report + +### Prerequisites + +- [GitHub CLI](https://cli.github.com) (`gh`) authenticated with an account that + has access to the `gim-home` org. + +### Fetch the latest report + +Work on a branch off `main`. Run the script from this `docs/reports/` folder. +It uses only the Python standard library — no project dependencies or `uv` +required. By default it overwrites `model_accuracy_report.html` next to the +script: + +```powershell +python docs/reports/download_report.py --account +``` + +This sparse-clones the `site-src` branch of the private artifacts repo and copies +the single report file. Use `--out ` to write elsewhere. The script only +fetches — it does not commit or push. + +### Publish + +Commit the refreshed report on your branch and open a PR to `main`: + +```powershell +git add docs/reports/model_accuracy_report.html +git commit -m "Update model accuracy report" +``` + +Once merged, the **Build & Publish Docs** workflow rebuilds the site and the +report is available at `...//reports/model_accuracy_report.html` +(e.g. `.../latest/reports/...`). To refresh an already-released version, run +that workflow via `workflow_dispatch` with the matching version label. diff --git a/docs/reports/download_report.py b/docs/reports/download_report.py new file mode 100644 index 000000000..8105d2827 --- /dev/null +++ b/docs/reports/download_report.py @@ -0,0 +1,172 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# -------------------------------------------------------------------------- +"""Fetch the model accuracy report from gim-home/ModelKitArtifacts. + +For Microsoft internal use only. Requires gh CLI authenticated with an account +that has access to the gim-home org. + +This script downloads the single, self-contained coverage report HTML page so +it can be published on the winml-cli documentation site. It pulls the file +directly from the `site-src` branch of gim-home/ModelKitArtifacts; it does +not download release assets. + +The report, this script, and the README live together in `docs/reports/` on +`main`. mkdocs publishes the HTML as a static asset under each docs version; +the script and README are excluded from the built site via `exclude_docs` in +`mkdocs.yml`. Refreshing the report is a normal docs change to `main`. + +This script uses only the Python standard library, so it runs with a plain +`python` (no `uv` / project dependencies required). + +Usage: + python download_report.py --account + python download_report.py --account --out + +By default the report is written next to this script, overwriting the +published copy (docs/reports/model_accuracy_report.html on `main`). + +PUBLISHING (done by a maintainer via pull request): + See README.md (co-located in this folder) for full instructions. After + fetching, commit the refreshed report on a branch off `main` and open a PR: + + git add docs/reports/model_accuracy_report.html + git commit -m "Update model accuracy report" + + Once merged, the docs workflow rebuilds and republishes the site. +""" + +import argparse +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + + +SOURCE_REPO = "gim-home/ModelKitArtifacts" +SOURCE_BRANCH = "site-src" +SOURCE_FILE = "e2e_model_coverage_result/model_accuracy_report.html" +REPORT_FILENAME = SOURCE_FILE.rsplit("/", 1)[-1] +DEFAULT_OUT = Path(__file__).resolve().parent / REPORT_FILENAME + + +def _get_clone_url(account: str | None = None) -> str: + """Build clone URL using gh account token.""" + gh_account = account or os.environ.get("GH_ACCOUNT") + if not gh_account: + print( + "ERROR: gh account is required.\n" + "Specify via --account or GH_ACCOUNT env var:\n" + " uv run python docs/reports/download_report.py --account \n" + " GH_ACCOUNT= uv run python docs/reports/download_report.py\n" + "\n" + "This script is for Microsoft internal use (gim-home org access required).", + file=sys.stderr, + ) + sys.exit(1) + + if not shutil.which("gh"): + print( + "ERROR: gh CLI is not installed.\nInstall from https://cli.github.com", + file=sys.stderr, + ) + sys.exit(1) + + result = subprocess.run( + ["gh", "auth", "token", "--user", gh_account], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print( + f"ERROR: Could not get token for account '{gh_account}'.\nRun 'gh auth login' first.", + file=sys.stderr, + ) + sys.exit(1) + + token = result.stdout.strip() + print(f"Using gh account: {gh_account}", flush=True) + return f"https://x-access-token:{token}@github.com/{SOURCE_REPO}.git" + + +def _sparse_clone(clone_url: str, dest: Path) -> bool: + """Sparse-clone only the single report file. Returns True on success.""" + result = subprocess.run( + [ + "git", + "clone", + "--depth", + "1", + "--filter=blob:none", + "--sparse", + "--branch", + SOURCE_BRANCH, + clone_url, + str(dest), + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + return False + result = subprocess.run( + ["git", "sparse-checkout", "set", "--no-cone", SOURCE_FILE], + cwd=dest, + capture_output=True, + text=True, + ) + return result.returncode == 0 + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Fetch the model accuracy report from gim-home/ModelKitArtifacts" + ) + parser.add_argument("--account", type=str, help="gh CLI account with access to gim-home org") + parser.add_argument( + "--out", + type=Path, + default=DEFAULT_OUT, + help=f"Output path for the fetched report (default: {DEFAULT_OUT})", + ) + args = parser.parse_args() + + clone_url = _get_clone_url(args.account) + + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) / "repo" + print(f"Downloading {SOURCE_FILE} from {SOURCE_REPO}@{SOURCE_BRANCH}...", flush=True) + + if not _sparse_clone(clone_url, tmp_path): + print( + f"ERROR: Failed to clone {SOURCE_REPO}@{SOURCE_BRANCH}.\n" + "Make sure the GH_ACCOUNT has access to the gim-home org.", + file=sys.stderr, + ) + sys.exit(1) + + src_file = tmp_path / SOURCE_FILE + if not src_file.is_file(): + print( + f"ERROR: {SOURCE_FILE} not found in {SOURCE_REPO}@{SOURCE_BRANCH}.", + file=sys.stderr, + ) + sys.exit(1) + + out_path = args.out.resolve() + out_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src_file, out_path) + + size_kb = out_path.stat().st_size / 1024 + print(f"Done. Wrote {out_path} ({size_kb:.0f} KB).") + print( + "\nNext: commit the refreshed report on a branch off main and open a PR " + "(see the co-located README.md). The docs workflow republishes the site." + ) + + +if __name__ == "__main__": + main() diff --git a/docs/reports/model_accuracy_report.html b/docs/reports/model_accuracy_report.html new file mode 100644 index 000000000..d5593c98b --- /dev/null +++ b/docs/reports/model_accuracy_report.html @@ -0,0 +1,787 @@ + + + + + +Model Accuracy Report + + + + +
+
+

Model Accuracy Report

+ + +
+
WinML CLI 0.2.0  ·  Windows ML 2.0
+
Total Models
199
Total Model · Task Pairs
226
Compatible on All 10 EPs
69
Accuracy-Tested Models
71
Accuracy Pass All 10 EPs
52
+
+
✓ DML GPU
175/226 (77.4%)
✓ MIGraph GPU
182/226 (80.5%)
✓ MLAS CPU
177/226 (78.3%)
✓ OV NPU
160/226 (70.8%)
✓ OV GPU
131/226 (58.0%)
✓ OV CPU
173/226 (76.5%)
✓ QNN NPU
145/226 (64.2%)
✓ QNN GPU
127/226 (56.2%)
✓ TRTRTX GPU
171/226 (75.7%)
✓ VitisAI NPU
172/226 (76.1%)
+
+ +
+
+
+
+
+
+
+
+ + +
+
Each tile is split by | into Compatibility (left) and Accuracy (right):
  • Compatibility — the WinML CLI built the model for that execution provider and completed the winml perf benchmark without errors or timeout (unquantized model).
  • Accuracy — the model's output matches the PyTorch baseline within the metric threshold. A — accuracy half means that pair was not accuracy-tested.
Hover any tile for the metric, task and per-precision results.
✓ pass✗ fail▼ accuracy regression◷ timed out∅ not applicable— no data / not tested
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelTaskDML GPUMIGraph GPUMLAS CPUOV NPUOV GPUOV CPUQNN NPUQNN GPUTRTRTX GPUVitisAI NPU
Comp|AccComp|AccComp|AccComp|AccComp|AccComp|AccComp|AccComp|AccComp|AccComp|Acc
▼sentence-transformers5 models · 12 pairs12/12|9/912/12|9/912/12|9/912/12|9/912/12|9/912/12|9/911/12|9/912/12|9/912/12|9/912/12|9/9
all-MiniLM-L6-v2feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
all-MiniLM-L6-v2sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
all-mpnet-base-v2feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
all-mpnet-base-v2sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
multi-qa-mpnet-base-dot-v1feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
multi-qa-mpnet-base-dot-v1sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
paraphrase-multilingual-MiniLM-L12-v2feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
paraphrase-multilingual-MiniLM-L12-v2sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
paraphrase-multilingual-mpnet-base-v2sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
all-mpnet-base-v2fill-mask✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
multi-qa-mpnet-base-dot-v1fill-mask✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
paraphrase-multilingual-mpnet-base-v2feature-extraction✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—
▼BAAI6 models · 10 pairs10/10|7/710/10|7/710/10|7/710/10|7/710/10|7/710/10|7/710/10|7/710/10|7/710/10|7/710/10|7/7
bge-base-en-v1.5feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-base-en-v1.5sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-large-en-v1.5sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-m3feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-m3sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-small-en-v1.5feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-small-en-v1.5sentence-similarity✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bge-large-en-v1.5feature-extraction✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
bge-reranker-basetext-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
bge-reranker-v2-m3text-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼google-bert4 models · 6 pairs6/6|5/56/6|5/56/6|5/56/6|5/56/6|5/56/6|5/56/6|5/56/6|5/56/6|5/56/6|5/5
bert-base-multilingual-casedfeature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-base-multilingual-casedfill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-base-multilingual-uncasedfill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-base-uncasedfill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-large-uncased-whole-word-masking-finetuned-squadquestion-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-base-multilingual-casedmasked-lm✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼facebook19 models · 21 pairs17/21|6/618/21|6/618/21|6/616/21|6/69/21|6/618/21|6/615/21|6/69/21|4/617/21|6/618/21|6/6
convnext-tiny-224image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
dino-vitb16image-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
dino-vits16image-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
dinov2-smallimage-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
dinov2-baseimage-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼✓|✓✓|✓
dinov2-largeimage-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼✓|✓✓|✓
bart-large-cnnsummarization✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
bart-large-mnlitext-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
bart-large-mnlizero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
detr-resnet-50feature-extraction✓|—✓|—✓|—✗|—✗|—✓|—✗|—✓|—✓|—✓|—
detr-resnet-50object-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
dinov2-giantimage-feature-extraction✗|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
nllb-200-distilled-600Mtranslation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
nougat-baseimage-to-text✓|—✓|—✓|—✗|—✓|—✓|—✓|—✗|—✓|—✓|—
sam-vit-basemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
sam-vit-hugemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
sam-vit-largemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
sam2-hiera-largemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
sam2.1-hiera-largemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
sam2.1-hiera-smallmask-generation✓|—✓|—✓|—✓|—✗|—✓|—✗|—✗|—✓|—✓|—
sam2.1-hiera-tinymask-generation✓|—✓|—✓|—✓|—✗|—✓|—✗|—✗|—✓|—✓|—
▼distilbert4 models4/4|4/44/4|4/44/4|4/44/4|4/44/4|4/44/4|4/44/4|4/44/4|4/44/4|4/44/4|4/4
distilbert-base-cased-distilled-squadquestion-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
distilbert-base-uncasedfill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
distilbert-base-uncased-distilled-squadquestion-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
distilbert-base-uncased-finetuned-sst-2-englishtext-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼microsoft28 models23/28|10/1023/28|9/1023/28|10/1015/28|10/1020/28|9/1023/28|10/1014/28|10/1011/28|7/1023/28|9/1022/28|10/10
resnet-18image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
resnet-50image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
swin-large-patch4-window7-224image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
beit-base-patch16-224-pt22k-ft22kimage-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
llmlingua-2-xlm-roberta-large-meetingbanktoken-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
rad-dinoimage-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼✓|✓✓|✓
swinv2-tiny-patch4-window16-256image-classification✓|✓✓|▼✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
trocr-base-handwrittenimage-to-text✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓
trocr-base-printedimage-to-text✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓
trocr-large-handwrittenimage-to-text✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓
deberta-xlarge-mnlitext-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—
Phi-4-multimodal-instructvisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
speecht5_ttstext-to-speech✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
table-transformer-detectionobject-detection✓|✓✓|✓✓|✓✓|✓✗|✗✓|✓✓|✓✓|✗✓|✗✓|✓
table-transformer-structure-recognitionobject-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
table-transformer-structure-recognition-v1.1-allobject-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
tapex-basetable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-base-finetuned-wikisqltable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-base-finetuned-wtqtable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-largetable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-large-finetuned-tabfacttable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-large-finetuned-wikisqltable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-large-finetuned-wtqtable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
tapex-large-sql-executiontable-question-answering✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
trocr-large-printedimage-to-text✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✗|▼✓|✓✓|✓
VibeVoice-ASR-HFaudio-text-to-text✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
VibeVoice-Realtime-0.5Btext-to-speech✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
xclip-base-patch32video-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼deepset3 models3/3|3/33/3|3/33/3|3/33/3|3/33/3|3/33/3|3/33/3|3/33/3|3/33/3|3/33/3|3/3
bert-large-uncased-whole-word-masking-squad2question-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
roberta-base-squad2question-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
tinyroberta-squad2question-answering✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼FacebookAI4 models3/4|3/44/4|4/44/4|4/44/4|4/44/4|4/44/4|3/44/4|4/44/4|3/44/4|4/44/4|4/4
roberta-basefill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
roberta-largefill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
xlm-roberta-basefill-mask✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
xlm-roberta-largefill-mask✗|◷✓|✓✓|✓✓|✓✓|✓✓|◷✓|✓✓|▼✓|✓✓|✓
▼Intel4 models · 5 pairs5/5|2/25/5|2/25/5|2/25/5|2/24/5|2/24/5|2/24/5|2/25/5|2/25/5|2/24/5|2/2
bert-base-uncased-mrpcfeature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bert-base-uncased-mrpctext-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
dpt-hybrid-midasdepth-estimation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
dpt-largedepth-estimation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
zoedepth-nyu-kittidepth-estimation✓|—✓|—✓|—✓|—✗|—✗|—✗|—✓|—✓|—✗|—
▼openai5 models · 9 pairs6/9|6/66/9|5/66/9|6/66/9|6/66/9|6/66/9|6/63/9|6/66/9|4/66/9|6/66/9|3/6
clip-vit-base-patch16feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
clip-vit-base-patch32feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
clip-vit-base-patch32zero-shot-image-classification✓|✓✓|—✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼
clip-vit-base-patch16zero-shot-image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓✓|✓
clip-vit-base-patch16zero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
clip-vit-base-patch32zero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
clip-vit-large-patch14zero-shot-image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|▼✓|✓✓|▼
clip-vit-large-patch14-336zero-shot-image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✗|✓✓|▼✓|✓✓|▼
gpt-oss-20btext-generation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼google10 models · 12 pairs7/12|2/29/12|2/27/12|2/27/12|2/25/12|2/27/12|2/27/12|2/22/12|2/25/12|2/28/12|2/2
vit-base-patch16-224image-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
vit-base-patch16-224-in21kimage-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
deplotvisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
flan-t5-base✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
flan-t5-basesummarization✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
flan-t5-basetranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
madlad400-3b-mttranslation✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✓|—
pegasus-xsumsummarization✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✓|—✗|—
pix2struct-ai2d-basevisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
pix2struct-docvqa-basevisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
siglip-base-patch16-224zero-shot-image-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
siglip-so400m-patch14-384zero-shot-image-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
▼laion2 models · 4 pairs3/4|3/33/4|2/33/4|2/33/4|3/33/4|3/33/4|2/33/4|3/33/4|3/33/4|3/33/4|2/3
CLIP-ViT-B-32-laion2B-s34B-b79Kfeature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
CLIP-ViT-B-32-laion2B-s34B-b79Kzero-shot-image-classification✓|✓✓|—✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼
CLIP-ViT-H-14-laion2B-s32B-b79Kzero-shot-image-classification✓|✓✓|✓✓|◷✓|✓✓|✓✓|◷✓|✓✓|✓✓|✓✓|✓
CLIP-ViT-B-32-laion2B-s34B-b79Kzero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼apple2 models1/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/1
mobilevit-smallimage-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
DepthPro-hfdepth-estimation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼Babelscape1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
wikineural-multilingual-nertoken-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼dbmdz1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
bert-large-cased-finetuned-conll03-englishtoken-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼dima8061 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
fairface_age_image_detectionimage-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼dslim1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
bert-base-NERtoken-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼Isotonic1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
distilbert_finetuned_ai4privacy_v2token-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼joeddav2 models1/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/11/2|1/1
xlm-roberta-large-xnlizero-shot-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
bart-large-mnli-yahoo-answerszero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼ProsusAI1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
finberttext-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼rizvandwiki1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
gender-classificationimage-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼w11wo1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
indonesian-roberta-base-posp-taggertoken-classification✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼cross-encoder3 models3/3|—3/3|—3/3|—3/3|—2/3|—3/3|—3/3|—3/3|—3/3|—3/3|—
ms-marco-MiniLM-L4-v2text-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
ms-marco-MiniLM-L6-v2text-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
nli-deberta-v3-smallzero-shot-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼StanfordAIMI2 models2/2|1/12/2|1/12/2|1/12/2|1/12/2|1/12/2|1/12/2|1/12/2|0/12/2|1/12/2|1/1
dinov2-base-xray-224image-feature-extraction✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼✓|✓✓|✓
stanford-deidentifier-basetoken-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼AdamCodd1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
vit-base-nsfw-detectorimage-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼ahotrod1 model1/1|1/11/1|0/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
electra_large_discriminator_squad2_512question-answering✓|✓✓|▼✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼amunchet1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
rorshark-vit-baseimage-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼Falconsai1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
nsfw_image_detectionimage-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼hustvl1 model1/1|0/11/1|0/11/1|0/11/1|0/11/1|0/11/1|0/11/1|0/11/1|0/11/1|0/11/1|0/1
yolos-smallobject-detection✓|▼✓|▼✓|▼✓|▼✓|▼✓|▼✓|▼✓|▼✓|▼✓|▼
▼Jean-Baptiste1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
camembert-ner-with-datestoken-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼kredor1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
punctuate-alltoken-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼lxyuan1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
distilbert-base-multilingual-cased-sentiments-studentzero-shot-classification✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼monologg1 model1/1|1/11/1|0/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
koelectra-small-v2-distilled-korquad-384question-answering✓|✓✓|▼✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼patrickjohncyh1 model1/1|1/11/1|0/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|0/1
fashion-clipzero-shot-image-classification✓|✓✓|—✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼
▼tau1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
splinter-basequestion-answering✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—✓|—
▼valentinafeve1 model1/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|0/11/1|1/11/1|1/11/1|0/1
yolos-fashionpediaobject-detection✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|▼✓|✓✓|✓✓|▼
▼nvidia6 models6/6|3/36/6|3/36/6|3/35/6|3/30/6|3/36/6|3/34/6|3/36/6|3/36/6|3/36/6|3/3
segformer-b1-finetuned-ade-512-512image-segmentation✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓✓|✓✓|✓✓|✓
segformer-b2-finetuned-ade-512-512image-segmentation✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓✓|✓✓|✓✓|✓
segformer-b5-finetuned-ade-640-640image-segmentation✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✗|✓✓|✓✓|✓✓|✓
segformer-b0-finetuned-ade-512-512image-segmentation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
segformer-b0-finetuned-cityscapes-1024-1024image-segmentation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
segformer-b5-finetuned-cityscapes-1024-1024image-segmentation✓|—✓|—✓|—✗|—✗|—✓|—✗|—✓|—✓|—✓|—
▼cardiffnlp1 model1/1|1/11/1|1/11/1|1/10/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
twitter-roberta-base-sentiment-latesttext-classification✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼mattmdjaga1 model1/1|1/11/1|1/11/1|1/11/1|1/10/1|1/11/1|1/11/1|1/11/1|1/11/1|1/11/1|1/1
segformer_b2_clothesimage-segmentation✓|✓✓|✓✓|✓✓|✓✗|✓✓|✓✓|✓✓|✓✓|✓✓|✓
▼ai-forever1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
Real-ESRGAN✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼alibaba-damo1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
mgp-str-baseimage-to-text✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼breezedeus1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
pix2text-mfrimage-to-text✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼buildborderless1 model1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—
CommunityForensics-DeepfakeDet-ViTimage-classification✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—✓|—
▼dandelin1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
vilt-b32-finetuned-vqavisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼depth-anything3 models3/3|—3/3|—3/3|—3/3|—0/3|—3/3|—3/3|—3/3|—3/3|—3/3|—
Depth-Anything-V2-Base-hfdepth-estimation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
Depth-Anything-V2-Large-hfdepth-estimation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
Depth-Anything-V2-Small-hfdepth-estimation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼fashn-ai1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
fashn-human-parserimage-segmentation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼flaviagiammarino1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—0/1|—1/1|—1/1|—
medsam-vit-basemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
▼google-t54 models · 7 pairs5/7|—7/7|—5/7|—5/7|—5/7|—5/7|—5/7|—0/7|—0/7|—7/7|—
t5-3bsummarization✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✓|—
t5-3btranslation✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✓|—
t5-basesummarization✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
t5-basetranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
t5-largesummarization✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
t5-smallsummarization✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
t5-smalltranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—
▼Helsinki-NLP5 models3/5|—3/5|—3/5|—3/5|—3/5|—3/5|—3/5|—0/5|—3/5|—3/5|—
opus-mt-en-rutranslation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
opus-mt-es-entranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—
opus-mt-fr-entranslation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
opus-mt-nl-entranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—
opus-mt-tr-entranslation✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—
▼hi-wesley1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
gemma3-vision-encoderimage-feature-extraction✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼internlm1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
internlm-xcomposer2d5-7bvisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼intfloat1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—
multilingual-e5-largesentence-similarity✓|—✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—
▼jonathandinu1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
face-parsingimage-segmentation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼kha-white1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—0/1|—1/1|—1/1|—
manga-ocr-baseimage-to-text✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—✓|—
▼knkarthick1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
MEETING_SUMMARYsummarization✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼LiheYoung3 models2/3|—2/3|—2/3|—2/3|—0/3|—2/3|—2/3|—2/3|—2/3|—2/3|—
depth-anything-base-hfdepth-estimation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
depth-anything-large-hfdepth-estimation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
depth-anything-small-hfdepth-estimation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼Marqo1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
marqo-fashionSigLIPzero-shot-image-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼mixedbread-ai1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
mxbai-rerank-xsmall-v1text-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼MoritzLaurer4 models4/4|—4/4|—4/4|—4/4|—0/4|—4/4|—4/4|—4/4|—4/4|—2/4|—
DeBERTa-v3-large-mnli-fever-anli-ling-wanlizero-shot-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✗|—
deberta-v3-large-zeroshot-v2.0zero-shot-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✗|—
mDeBERTa-v3-base-mnli-xnlizero-shot-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
mDeBERTa-v3-base-xnli-multilingual-nli-2mil7zero-shot-classification✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼moussaKam1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
mbarthezsummarization✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼naver-clova-ix2 models2/2|—2/2|—2/2|—1/2|—2/2|—2/2|—0/2|—0/2|—2/2|—2/2|—
donut-baseimage-to-text✓|—✓|—✓|—✗|—✓|—✓|—✗|—✗|—✓|—✓|—
donut-base-finetuned-cord-v2image-to-text✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—✓|—
▼nlpconnect1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—0/1|—1/1|—0/1|—
vit-gpt2-image-captioningimage-to-text✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✓|—✗|—
▼obi1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—
deid_roberta_i2b2token-classification✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—
▼oliverguhr1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—
fullstop-punctuation-multilang-largetoken-classification✓|—✓|—✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—
▼openai-community1 model1/1|—1/1|—1/1|—0/1|—0/1|—0/1|—0/1|—0/1|—1/1|—0/1|—
gpt2text-generation✓|—✓|—✓|—✗|—✗|—✗|—✗|—✗|—✓|—✗|—
▼PekingU3 models3/3|—2/3|—3/3|—3/3|—0/3|—3/3|—3/3|—0/3|—3/3|—3/3|—
rtdetr_r101vd_coco_o365object-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
rtdetr_r50vd_coco_o365object-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
rtdetr_v2_r18vdobject-detection✓|—✗|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
▼philschmid1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
bart-large-cnn-samsumsummarization✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼Qwen7 models1/7|—3/7|—1/7|—0/7|—0/7|—0/7|—0/7|—0/7|—2/7|—0/7|—
Qwen2.5-0.5B-Instructtext-generation✓|—✓|—✓|—✗|—✗|—✗|—✗|—✗|—✓|—✗|—
Qwen2.5-1.5B-Instructtext-generation✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✓|—✗|—
Qwen2.5-3B-Instructtext-generation✗|—✓|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
Qwen2.5-7B-Instructtext-generation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
Qwen3-0.6Btext-generation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
Qwen3-1.7Btext-generation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
Qwen3-8Btext-generation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼Salesforce5 models1/5|1/11/5|0/11/5|1/11/5|1/11/5|1/11/5|1/10/5|1/10/5|0/11/5|1/11/5|1/1
blip-image-captioning-baseimage-to-text✓|✓✓|▼✓|✓✓|✓✓|✓✓|✓✗|✓✗|▼✓|✓✓|✓
blip-vqa-basevisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
blip2-flan-t5-xlvisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
blip2-opt-2.7bvisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
blip2-opt-2.7b-cocovisual-question-answering✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼sshleifer1 model1/1|—1/1|—1/1|—1/1|—1/1|—1/1|—0/1|—0/1|—0/1|—1/1|—
distilbart-cnn-12-6summarization✓|—✓|—✓|—✓|—✓|—✓|—✗|—✗|—✗|—✓|—
▼TahaDouaji1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—1/1|—1/1|—1/1|—
detr-doc-table-detectionobject-detection✓|—✓|—✓|—✓|—✗|—✓|—✓|—✓|—✓|—✓|—
▼timm2 models—|——|——|——|——|——|——|——|—2/2|——|—
mobilenetv3_small_100.lamb_in1kimage-classification∅|—∅|—∅|—∅|—∅|—∅|—∅|—∅|—✓|—∅|—
repghostnet_200.in1kimage-classification∅|—∅|—∅|—∅|—∅|—∅|—∅|—∅|—✓|—∅|—
▼timpal0l1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—0/1|—1/1|—1/1|—1/1|—
mdeberta-v3-base-squad2question-answering✓|—✓|—✓|—✓|—✗|—✓|—✗|—✓|—✓|—✓|—
▼trl-internal-testing1 model1/1|—1/1|—1/1|—0/1|—0/1|—0/1|—0/1|—0/1|—1/1|—0/1|—
tiny-Qwen2ForCausalLM-2.5text-generation✓|—✓|—✓|—✗|—✗|—✗|—✗|—✗|—✓|—✗|—
▼valhalla1 model0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—0/1|—
distilbart-mnli-12-3zero-shot-classification✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼wanglab1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—0/1|—1/1|—1/1|—
medsam-vit-basemask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
▼Xenova2 models0/2|—0/2|—0/2|—0/2|—0/2|—0/2|—0/2|—0/2|—0/2|—0/2|—
paraphrase-multilingual-MiniLM-L12-v2feature-extraction✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
segformer-b0-finetuned-ade-512-512image-segmentation✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—✗|—
▼Zigeng1 model1/1|—1/1|—1/1|—1/1|—0/1|—1/1|—1/1|—0/1|—1/1|—1/1|—
SlimSAM-uniform-77mask-generation✓|—✓|—✓|—✓|—✗|—✓|—✓|—✗|—✓|—✓|—
+
+
Compatibility (build & run) across 10 execution providers for 226 model · task pairs; accuracy graded against the PyTorch baseline for 83 of them. · 69/199 models run on every EP.
+ + + diff --git a/mkdocs.yml b/mkdocs.yml index c6c21debe..101c1184d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -11,6 +11,8 @@ exclude_docs: | /design/ /naming-convention.md /pytest-best-practices.md + /reports/download_report.py + /reports/README.md extra: version: diff --git a/pyproject.toml b/pyproject.toml index 3e5d69d1f..14412488a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -298,6 +298,7 @@ lint.per-file-ignores."examples/**" = [ "ANN", "D100", "D103", "S101", "T20" ] lint.per-file-ignores."scripts/e2e_eval/**" = [ "ANN", "D103", "E501", "PERF203", "T20" ] # Download scripts: Allow subprocess, print, missing docstrings lint.per-file-ignores."scripts/download_rules.py" = [ "D103", "E501", "PERF401", "S603", "S607", "SIM108", "T20" ] +lint.per-file-ignores."docs/reports/download_report.py" = [ "D103", "E501", "PERF401", "S603", "S607", "SIM108", "T20" ] # CLI: Allow print statements lint.per-file-ignores."src/winml/modelkit/cli.py" = [ "T20", "T201" ] lint.per-file-ignores."src/winml/modelkit/commands/**" = [ "T20", "T201" ]