Make Paradee model conversion reproducible across CPUs

.github/workflows/eliza-paradee-conversion.yml

@@ -0,0 +1,85 @@
+name: Paradee conversion reproducibility
+on:
+  pull_request:
+    paths:
+      - 'tools/paradee/convert_*.py'
+      - 'tools/paradee/pins.json'
+      - 'gguf-py/**'
+      - '.github/workflows/eliza-paradee-conversion.yml'
+  workflow_dispatch:
+
+jobs:
+  conversion:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-24.04, macos-15]
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install conversion dependencies
+        shell: bash
+        run: |
+          if [[ "$RUNNER_OS" == "Linux" ]]; then
+            python -m pip install torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu
+          else
+            python -m pip install torch==2.14.0
+          fi
+          python -m pip install numpy==2.5.4 onnx==1.23.2 pyyaml==6.0.3 requests==2.34.2 tqdm==4.70.1
+      - name: Convert pinned models twice and check complete file hashes
+        shell: bash
+        run: |
+          python - <<'PY'
+          import hashlib, json, os, platform, subprocess, sys, urllib.request
+          from pathlib import Path
+          pins = json.loads(Path('tools/paradee/pins.json').read_text())
+          out = Path('test-results/paradee-conversion')
+          out.mkdir(parents=True, exist_ok=True)
+          expected_machine = 'arm64' if sys.platform == 'darwin' else 'x86_64'
+          assert platform.machine() == expected_machine, (platform.machine(), expected_machine)
+          report = {'platform': platform.platform(), 'machine': platform.machine(), 'python': sys.version, 'models': []}
+          failures = []
+          models = [('1.0', pins), ('1.1', pins['v1.1']), ('1.2', pins['v1.2']), ('delibes', pins['delibes'])]
+          for version, pin in models:
+              source = out / ('source-' + version)
+              for name, expected in pin['files'].items():
+                  if name.startswith('samples/'):
+                      continue
+                  target = source / name
+                  target.parent.mkdir(parents=True, exist_ok=True)
+                  url = f"https://huggingface.co/{pin['model']['repo']}/resolve/{pin['model']['revision']}/{name}"
+                  data = urllib.request.urlopen(url).read()
+                  assert len(data) == expected['size'] and hashlib.sha256(data).hexdigest() == expected['sha256'], name
+                  target.write_bytes(data)
+              hashes = []
+              for mode in ['native', 'scalar']:
+                  target = out / f'{version}-{mode}.gguf'
+                  env = dict(os.environ, OMP_NUM_THREADS='1', MKL_NUM_THREADS='1')
+                  if mode == 'scalar':
+                      env['ATEN_CPU_CAPABILITY'] = 'default'
+                  if version == 'delibes':
+                      script, selection = 'convert_paradee_onnx_to_gguf.py', ['--family', 'delibes']
+                  else:
+                      script, selection = 'convert_paradee_pt_to_gguf.py', ['--version', version]
+                  subprocess.run([sys.executable, 'tools/paradee/' + script, *selection,
+                                  '--model-dir', str(source), '--output', str(target)], check=True, env=env)
+                  hashes.append(hashlib.sha256(target.read_bytes()).hexdigest())
+              expected = pin['gguf']['f32']
+              row = {'version': version, 'hashes': hashes, 'expected': expected['sha256'],
+                     'bytes': target.stat().st_size, 'expectedBytes': expected['size']}
+              report['models'].append(row)
+              if hashes[0] != hashes[1] or hashes[0] != expected['sha256'] or row['bytes'] != expected['size']:
+                  failures.append(version)
+              (out / 'report.json').write_text(json.dumps(report, indent=2) + '\n')
+          assert not failures, f'Conversion pin or reproducibility failure: {failures}'
+          PY
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: paradee-conversion-${{ matrix.os }}
+          path: test-results/paradee-conversion/report.json
+          if-no-files-found: error

tools/paradee/README.md

@@ -100,6 +100,12 @@ front end.
 - **No F16 pin:** `--outtype f16` halves the file (16 MB) but fails the parity gate (end-to-end SNR
   16–23 dB), so it is not pinned or shipped. See [F16 and v1.1](#f16-and-v11) for the full evaluation.
 
+The converter folds weight normalization with `math.fsum` over binary64 products, then rounds the result
+to F32. PyTorch's F32 reduction changes with CPU dispatch and can produce a different file from the same
+weights. The conversion workflow checks complete file hashes on Linux x86_64 and macOS arm64. The GGUF
+also records the converter's Git blob: even a source edit that does not change tensors needs new output
+pins. Delibes records both converter blobs, so its pin changes when the shared converter changes too.
+
 ## Memory
 
 `paradee_load_model` memory-maps an all-F32 GGUF (the converter's default) read-only and the layers read the

tools/paradee/convert_paradee_pt_to_gguf.py

@@ -18,9 +18,8 @@
 
 Every input is checked against tools/paradee/pins.json (size + sha256) before
 torch.load(weights_only=True) touches it. weight_norm pairs are folded with
-torch._weight_norm, the same function PyTorch uses when it removes the
-parametrisation, so the GGUF weights are bit-identical to the reference
-module's effective weights. Every state-dict key must be consumed (only the
+a fixed binary64 reduction before rounding to float32, so conversion does
+not depend on PyTorch's CPU dispatch or reduction order. Every state-dict key must be consumed (only the
 unused ALBERT pooler is skipped), and every tensor shape is checked against
 the hparams written to the GGUF.
 
@@ -58,6 +57,7 @@
 import argparse
 import hashlib
 import json
+import math
 import os
 import sys
 from pathlib import Path
@@ -205,7 +205,7 @@ def load_vocab(model_dir: Path, version: str = "1.0") -> list[tuple[str, int]]:
 
 
 def fold_weight_norm(sd: dict) -> dict:
-    """Fold weight_g/weight_v into weight with torch._weight_norm(v, g, 0). A state dict without
+    """Fold weight_g/weight_v into weight with a fixed reduction order. A state dict without
     weight_norm pairs (for example one read from ONNX, where the weights are already folded) passes
     through unchanged and does not need torch."""
     out = {}
@@ -217,12 +217,15 @@ def fold_weight_norm(sd: dict) -> dict:
             pairs.setdefault(k[: -len(".weight_g")], {})["g"] = v
         else:
             out[k] = v
-    if pairs:
-        import torch
     for base, p in pairs.items():
         if set(p) != {"v", "g"}:
             raise SystemExit(f"incomplete weight_norm pair for {base}")
-        out[base + ".weight"] = torch._weight_norm(p["v"], p["g"], 0)
+        v = p["v"].detach().cpu().numpy()
+        g = p["g"].detach().cpu().numpy().astype(np.float64)
+        # SIMD and threaded float32 reductions change the pinned GGUF bytes.
+        norms = np.array([math.sqrt(math.fsum(float(x) * float(x) for x in row))
+                          for row in v.reshape(v.shape[0], -1)]).reshape(g.shape)
+        out[base + ".weight"] = (v.astype(np.float64) * (g / norms)).astype(np.float32)
     return out
 
 

tools/paradee/pins.json

@@ -76,8 +76,8 @@
     "f32": {
       "file": "paradee-8m-v1.0-f32.gguf",
       "size": 32072864,
-      "sha256": "ac9983f724c15688ef650cba9e7dfaf2197294c9cc4d4fd4b62122428e374690",
-      "note": "convert_paradee_pt_to_gguf.py --outtype f32 output; byte-reproducible (check with --check-output). Changes whenever the converter changes, because its git blob sha is in the metadata. A v1.0 file written by an earlier converter (for example 357477651abb…, 32072768 bytes, before paradee.version / paradee.decoder.head existed) still loads as v1.0 and gives byte-identical audio. The tensors are those of the previous pin c0d8f2010571…; only the converter's blob sha in the metadata changed (v1.2 support)."
+      "sha256": "5c7a282f577deeeef23272a39e85a835cb58842bd0481c7707fdbb3f5552cae1",
+      "note": "F32 output with deterministic binary64 weight_norm folding (math.fsum), then one F32 rounding. Normalized tensor values differ slightly from the former platform-dependent torch._weight_norm output; native stage and waveform parity remain gated. The converter git blob is also recorded in metadata, so any converter source edit requires a new pin."
     }
   },
   "fixtures": {
@@ -158,8 +158,8 @@
       "f32": {
         "file": "paradee-8m-v1.1-f32.gguf",
         "size": 31275232,
-        "sha256": "01d12ea2751642636ecc8cb5cd48be8d940620c3e1540be0b94cd002a10bfe74",
-        "note": "convert_paradee_pt_to_gguf.py --version 1.1 --outtype f32 output; byte-reproducible (check with --check-output). Changes whenever the converter changes, because its git blob sha is in the metadata. The tensors are those of the previous pin f205ab055f01…; only the converter's blob sha in the metadata changed (v1.2 support)."
+        "sha256": "71f2558ba38eb70a1e59114426c1d5400551e9a24754a8b5bd2ec228305e6e0f",
+        "note": "F32 output with deterministic binary64 weight_norm folding (math.fsum), then one F32 rounding. Normalized tensor values differ slightly from the former platform-dependent torch._weight_norm output; native stage and waveform parity remain gated. The converter git blob is also recorded in metadata, so any converter source edit requires a new pin."
       }
     }
   },
@@ -243,8 +243,8 @@
       "f32": {
         "file": "paradee-8m-v1.2-f32.gguf",
         "size": 31276064,
-        "sha256": "a43118932f7ec01bb3e9f1b92a56b24dd2622011c8fd5c77dca1e353b56e6df6",
-        "note": "convert_paradee_pt_to_gguf.py --version 1.2 --outtype f32 output; byte-reproducible (check with --check-output). Changes whenever the converter changes, because its git blob sha is in the metadata."
+        "sha256": "4b2819180a9e3b096fb71a93d6a719794246ee230f7faa857edcc0bee2a1529e",
+        "note": "F32 output with deterministic binary64 weight_norm folding (math.fsum), then one F32 rounding. Normalized tensor values differ slightly from the former platform-dependent torch._weight_norm output; native stage and waveform parity remain gated. The converter git blob is also recorded in metadata, so any converter source edit requires a new pin."
       }
     }
   },
@@ -364,8 +364,8 @@
       "f32": {
         "file": "delibes-8m-eses-f32.gguf",
         "size": 31275328,
-        "sha256": "a2f5acff3ac8218c7973c0a76a449365faace53ee6af4cf0f3bdc1f61868f0da",
-        "note": "convert_paradee_onnx_to_gguf.py --family delibes output; byte-reproducible (check with --check-output). Changes whenever either converter changes (both git blob shas are in the metadata)."
+        "sha256": "3b635dc24a8952254066daad77ff64aa08368a8b5a689abf4be05bdf2b0afe51",
+        "note": "convert_paradee_onnx_to_gguf.py --family delibes output. All tensor bytes are unchanged; only converter.base_git_blob_sha1 metadata changes for deterministic folding in the shared PyTorch converter. Both converter blobs are recorded, so any source edit requires a new pin."
       }
     }
   }