Backpropagation-free, layer-by-layer training of Differentiable Logic Gate Networks — with immediate discretization, adaptive depth, and incremental logic simplification.
Train one logic layer at a time with a local loss, discretize it, freeze it, and let the next layer learn on real 0/1 bits. Stop adding layers when accuracy plateaus. Simplify the circuit as it grows.
Proof-of-concept. Runs on CPU in a few minutes. A single self-contained script, no dependencies beyond torch and scikit-learn.
Just me playing around, not research. I don't read papers — I bounce ideas off an AI assistant, run the experiments, and enjoy watching the accuracy points move. That's the whole thing. Nothing here is peer-reviewed, and the only "literature search" was asking the AI, so I make no novelty or priority claims. Plenty of these ideas probably already exist under names I don't know. Read it as a reproducible playground log, not a contribution; if it duplicates prior work, that's expected — pointers are welcome via an issue.
Differentiable Logic Gate Networks (LGNs) learn circuits of 2-input logic gates by relaxing gate choice to a softmax over 16 Boolean functions. They achieve extremely fast, DSP-free inference on FPGAs. But training them end-to-end with backpropagation has three known pain points:
- Vanishing gradients in depth. With the standard parameterization, gradient norms fall below machine precision after ~16 logic layers (Light DLGN, 2025). Current fixes (residual initializations, reparameterizations) work within the backprop framework.
- The discretization gap. Networks are trained as soft mixtures of gates and discretized afterwards; the mismatch costs accuracy and is an active research topic (Mind the Gap, 2025).
- Training memory. Every gate holds 16 float logits, and backprop must keep the whole network soft at once.
This repo explores a different route: remove backpropagation across layers entirely.
input bits ──► [train layer 1 (soft, local GroupSum loss)]
│ discretize + freeze
▼ hard 0/1 bits
[train layer 2 on hard bits]
│ discretize + freeze
▼
... grow until validation accuracy plateaus ...
│
▼
[simplify circuit: constant folding, pass-through
removal, duplicate merge, dead-gate elimination]
- Each layer is trained with its own local objective (GroupSum + cross-entropy), in the spirit of greedy layer-wise pretraining, Cascade-Correlation (Fahlman & Lebiere, 1990) and the Forward-Forward algorithm (Hinton, 2022). No gradient ever crosses a frozen layer boundary.
- Because each frozen layer is discretized before the next layer trains, later layers learn on genuine Boolean inputs. The greedy network has zero discretization gap by construction — the reported accuracy is the accuracy of the final hard circuit.
- Only one training window is ever soft: float memory for gate logits is
gates × 16 × windowinstead ofgates × 16 × depth(default window = 1 layer). - Depth is not a hyperparameter: layers are added until the hard-probe validation accuracy stops improving.
- After training, a simplification pass (constant folding → pass-through/NOT reduction → duplicate merge → dead-gate elimination) shrinks the circuit and is verified to be bit-exact against the original.
- Optional extensions, all off by default:
--skip-input(re-expose input bits to every layer's wiring pool),--window W --commit J(train W layers ahead with backprop bounded to the window, freeze J at a time).
Everything on the main track runs at a fixed budget — 500 gates/layer, single network — because the game here is watching which ideas move the accuracy points, not how much compute gets spent. The starting point, on sklearn digits (8×8, thermometer-binarized to 192 bits), CPU:
| greedy (this repo) | end-to-end backprop | |
|---|---|---|
| depth | 4 (chosen automatically) | 4 (copied from greedy) |
| hard-circuit test accuracy | 88.2% | 93.6% |
| discretization gap | 0 (by construction) | 0.0 at this scale |
| float logits held during training | 8,000 (one layer) | 32,000 (×4) |
| circuit after simplification | 2,000 → 1,316 gates (65.8%), bit-identical | — |
Plain local training loses ~5 pt of accuracy to backprop — in exchange for zero discretization gap, ~1/depth training memory, automatic depth, and a simplifiable circuit. From that start, the idea ladder so far (hard-circuit test accuracy, gap still structurally zero throughout):
| idea (500 gates, single net) | digits (3 seeds) | MNIST |
|---|---|---|
| plain greedy (GroupSum + CE) | 88.4% | 74.3% |
+ skip-input (--skip-input) |
89.1% | —¹ |
+ windowed lookahead (--window 2 --commit 2) |
90.4% | 76.6% |
Forward-Forward objective (--objective ff) |
86.0% | 76.8% |
| FF + window | 88.0% | 78.2% |
FF + window + hard-negative mining (--ff-neg) |
89.7%² | 82.0%³ |
¹ skip alone is a depth/width-synergy lever — modest at 500-gate single net, and it actively hurts FF, so it isn't a fixed-budget winner. ² mix negatives. ³ review + 0.5 warm-up.
The winner depends on the judge. On digits (near its ceiling, so barely discriminating) windowed lookahead on GroupSum tops it at 90.4%. On MNIST — the more discriminating dataset, and the one trusted when the two disagree — the Forward-Forward stack wins clearly at 82.0% (+7.7 pt over plain greedy), with FF and windowed lookahead each being the strongest single idea (~+2.5 pt) and compounding.
Wider layers (--gates) and ensembles of independent nets (--ensemble) are a different kind of lever: they spend compute and inference-circuit area, and reliably buy accuracy — but they don't tell you which idea is any good, so they sit outside the fixed-budget arena. For reference, where they take the same pipeline:
| plain greedy (start) | best with scaling | how | |
|---|---|---|---|
| digits | 88.2% | 96.4% | 2,000 gates + --skip-input, ×4 ensemble (majority vote) |
| MNIST | 74.3% | 90.9% | 4,000 gates + --skip-input, ×4 ensemble (soft vote) |
End-to-end backprop at equal training memory averages 91.5% on digits — the scaled stack is above it, at the cost of more inference area. Details: memory-matched width, ensemble voting, MNIST scaling. This track is parked; it gets revisited only when a fixed-budget winner deserves a one-off scale check.
All experiments (details in RESULTS.md)
Main track — fixed-budget ideas (500 gates/layer, single net):
| experiment | headline | details |
|---|---|---|
| Depth stress test | e2e collapses to chance at ~12 layers (vanishing gradients); greedy still learns at layer 40 | → |
Skip connections (--skip-input) |
depth finally pays: peak 88.2%@4 → 90.4%@8. DenseNet-style --skip-all tested, negative. (Synergizes with width — see scaling track) |
→ |
Windowed lookahead (--window) |
training 2 layers ahead closes ~⅔ of the myopia gap: 90.4% vs e2e's 91.5% (3 seeds), +2.4 pt on MNIST; overlap/receding-horizon variant loses to plain blocks | → |
Forward-Forward objective (--objective ff) |
goodness = popcount on binary layers, so the whole FF inference is one logic circuit; 2.4 pt behind supervised local CE on digits (86.0%) but +2.5 pt ahead on MNIST (76.8%) — and the first lever to exploit depth (17 layers) without skip wiring; needs label-bit replication for sparse random wiring | → |
FF × window, FF negative mining (--ff-neg) |
windowed lookahead stacks with FF (digits 88.0%) unlike with skip; mining works once you warm up before mining (--ff-neg review --ff-neg-warmup 0.5): flat on digits but MNIST 78.2% → 82.0% — the repo's best fixed-budget net. Pure hard negatives without warm-up collapse. Structured data×label wiring (--ff-struct) replaces the label-replication hack at equal MNIST accuracy (tie, not a win) with a tiny pool and zero wasted gates |
→ |
Scaling track — reference (width / ensembles / bigger budgets, parked):
| experiment | headline | details |
|---|---|---|
| Memory-matched width | at equal training memory (4× wider layers), greedy beats e2e: 95.0% vs 91.5% mean, 3 seeds | → |
| MNIST first pass | the pattern replicates at 45× the data: memory-matched greedy+skip 84.6% vs e2e 80.1% (absolute numbers far below difflogic-scale budgets, stated honestly) | → |
Ensemble voting (--ensemble) |
parallel hard circuits + vote: stacks with everything (digits 96.4% — repo best); on MNIST 4×500-gate members beat the single 2,000-gate best at half the training memory; not a substitute for direct width | → |
| MNIST scaling | width is the dominant lever (4,000 gates: 89.8% single) and ensembling stacks: 90.9% with 4×4,000+skip; more epochs and window×width confirmed dead (+0.1 pt each); 8,000 gates OOMs at 6 GB (pools, not eval temporaries) | → |
Full run logs (environment, commands, raw output): one GitHub issue per experiment (#1 main run … #10 Forward-Forward), linked from each RESULTS.md section.
pip install torch scikit-learn
# main track (500 gates, single net)
python experiment.py # full run, a few min on CPU
python experiment.py --gates 200 --epochs 30 --max-layers 3 # ~20 s smoke test
python experiment.py --skip-e2e # greedy + simplification only
python experiment.py --device cuda # same experiment on GPU (~10x faster)
python experiment.py --window 2 --commit 2 --win-loss all # 2-layer lookahead blocks (+2 pt)
python experiment.py --objective ff --ff-label-rep 38 --skip-e2e # Forward-Forward objective
python experiment.py --objective ff --ff-label-rep 38 --window 2 --commit 2 --ff-neg review --ff-neg-warmup 0.5 --skip-e2e # best fixed-budget stack
# scaling track (reference)
python experiment.py --ensemble 4 --skip-e2e # 4 independent nets + voting
python experiment.py --device cuda --gates 2000 --skip-input --max-layers 16 --skip-e2e --ensemble 4 # digits best with scaling (96.4%)
python experiment.py --dataset mnist --device cuda --batch 4096 --epochs 30 --gates 2000 --skip-input --max-layers 10 --skip-e2e # MNIST (GPU recommended)- Depth stress test — backprop dies at ~12 layers, greedy survives 40 (details).
- Memory-matched comparison — greedy wins at equal training memory (details).
- Skip connections —
--skip-inputmakes depth useful;--skip-allnegative (details). - MNIST first pass — pattern replicates; absolute accuracy still small-budget (details).
- Windowed lookahead —
--window 2recovers most of the myopia deficit; window > 2 and overlapping commits don't help (details). - Ensemble voting —
--ensemble Mstacks with every other lever; repo best on digits (96.4%) and the training-memory-free path to MNIST scaling (details). - MNIST scaling, first round — width × ensembles reaches 90.9%; epochs and window×width are dead ends (details).
- Forward-Forward objective — popcount goodness works: behind supervised local CE on digits, ahead on MNIST, and exploits depth without skip wiring (details).
- Mono-Forward-style projection losses; better input binarization (fixed-budget friendly).
- (parked, scaling track) MNIST absolute accuracy: 8,000-gate layers (needs the pool-memory fix), FF × width/ensemble, convolutional wiring, CIFAR-10 on difflogic CUDA kernels.
- Simplify between growth steps (currently done once at the end) and rewire the next layer to the simplified circuit.
- Export simplified circuits to Verilog / run through ABC for comparison with proper logic synthesis.
Almost every ingredient here is from prior work — this section is about being explicit, not claiming credit. The whole repo is organized around one simple recipe:
Train one logic layer with a local loss, discretize it immediately, freeze it, and train the next layer on the real 0/1 bits.
I have not surveyed the literature and don't claim this recipe — or any piece of it — is new; it may well exist already. What I can say concretely about each piece, without any novelty claim:
| property | where it comes from |
|---|---|
| No multipliers / DSPs / floats; maps to FPGA LUTs | Inherited from LGNs (difflogic) — not mine, just the platform. |
| Zero discretization gap | Follows directly from the recipe (each layer is discretized before the next trains, so the reported accuracy is the hard circuit's). I haven't seen this exact setup in the few LGN papers I've looked at, but I haven't searched properly — take that as ignorance, not a claim. Not "the first verified-equals-deployed network" either (exact-by-construction routes exist outside LGNs, e.g. LogicNets' truth-table enumeration). |
| Training memory = one layer, not depth | Not special — any greedy layer-wise scheme (Cascade-Correlation, Forward-Forward) has this. |
| Adaptive depth / grow-and-freeze | Cascade-Correlation heritage (1990) — old idea. One reading I liked: since circuit depth = critical-path latency, stopping at the accuracy plateau happens to give a low-latency circuit for that accuracy. Post-deployment growth is likewise possible in principle, but my own depth-stress data shows added depth only pays off with skip wiring, so treat it as hand-waving. |
Windowed lookahead (--window) |
Block-wise greedy training exists (Belilovsky et al., 2019, with auxiliary heads). What I added on top: the blocks are discretized and frozen as they're committed (bit-exact prefix preserved), depth stays adaptive, and I report the overlap ablation (commit < window) — including the negative result that overlap doesn't beat plain blocks. |
- Deep Differentiable Logic Gate Networks (Petersen et al., NeurIPS 2022) and difflogic
- Convolutional Differentiable Logic Gate Networks (NeurIPS 2024) — includes post-training logic synthesis
- Light Differentiable Logic Gate Networks (2025) — depth via reparameterization (the backprop-side answer to the same problem)
- The Forward-Forward Algorithm (Hinton, 2022)
- Cascade-Correlation (Fahlman & Lebiere, 1990) — the original "grow and freeze" network
- Greedy layerwise learning can scale to ImageNet (Belilovsky et al., ICML 2019) — block-wise greedy training with auxiliary heads, the closest relative of
--window - I have not done a proper literature search (just asked an AI), so I make no claims about what is or isn't new. This combination — or any part of it — may already exist under names I don't know; if you know of prior work, please open an issue so I can point to it.
MIT
論理ゲートネットワーク(DLGN)を逆伝播なしで1層ずつ学習する実証実験です。各層をローカルな損失(GroupSum+交差エントロピー)で学習したら即座に離散化して凍結し、次の層は本物の0/1ビットの上で学習します。検証精度が頭打ちになったら層の追加を止めるため、深さは自動決定されます。学習後に回路を簡略化し、出力が完全に同一であることをビット単位で検証します。
論文も読まない素人がAIと壁打ちしながらのお遊びです。 AIとアイデアを出し合って、実験して、精度(ポイント)の変化を楽しんでいるだけです。査読も受けていませんし、文献調査もAIに聞いた程度なので、新規性や優先権は一切主張しません。ここにあるアイデアの多くは、私が知らない名前で既に存在しているはずです。再現できる遊びのログとして読んでください。もし既存研究と重複していたら、それが普通です — issueで教えてもらえると助かります。
本線は500ゲート/層・単発ネットの固定予算です。この遊びの主役は「計算資源を増やさずにアイデアだけでポイントがどれだけ動くか」で、MNISTを審判にした現在の階段は 素74.3% → 先読み窓76.6% → FF76.8% → FF+窓78.2% → +誤答の重点復習 82.0%(+7.7pt)。digits(天井で判定力なし)だと先読み窓のgroupsum(90.4%)が勝ちますが、食い違ったらMNISTを信じる方針です。詳細な表は英語本文の「The arena」を参照。
スケーリングレバー(幅=--gates とアンサンブル=--ensemble)は別トラック(参考)です。計算資源=推論回路面積を突っ込めば確実に精度を買えますが、アイデアの良し悪しは分かりません。参考値: digits 88.2%→96.4%(2,000ゲート+skip+×4多数決)、MNIST 74.3%→90.9%(4,000ゲート+skip+×4 soft vote)。このトラックは休止中で、固定予算で大きな当たりが出たときだけスケール確認を1本やります。
素のgreedyはend-to-end逆伝播に約5pt負けます(88.2% vs 93.6%)が、代わりに離散化ギャップが構造的にゼロ・学習メモリが深さ分の1・深さの自動決定という利点があります。主な観察:
- 深さ耐性: 逆伝播は12層でチャンスレベル(10%)に崩壊、greedyは40層目でも学習が成立。ただしskipなしでは深さが精度に貢献しない
- skip connections(
--skip-input): ゲートを増やさず配線だけで深さ劣化を解消し、初めて深さが精度に貢献(88.2%@4 → 90.4%@8)。ただしFFには逆効果 - メモリ等価比較: 学習時のfloat予算を揃える(greedyを幅4倍にする)とgreedyがe2eに3シード全勝(95.0% vs 91.5%)。コストは推論回路の面積
- 先読み窓(
--window 2): 2層先まで逆伝播で共同学習してからまとめて離散化。近視由来のギャップの約2/3を回収。反証: オーバーラップコミット(再計画)はブロック式に勝てない - Forward-Forward(
--objective ff): goodnessがバイナリ層ではpopcountに退化し、10ラベル試行の推論まで含めて純論理回路のまま。skipなしで深さ17層を活用できる唯一のレバーで、「まず普通に学習→模試→誤答を重点復習」の負例マイニング(--ff-neg review --ff-neg-warmup)まで積むとMNIST 82.0%(固定予算の最高値) - アンサンブル投票(
--ensemble M): 他の全レバーと加算。ただし推論面積を揃えると幅の直接拡大に負ける(正直な限界) - 死にレバー: エポック増、window×幅、window×skip、warmupなしのhard負例(崩壊)
各実験のセットアップ・数値表・反証された仮説(オーバーラップコミットはブロック式に勝てない、skipはパススルーを減らさない、DenseNet式は僅かに劣る、など)は RESULTS.md に、生ログは実験ごとの個別issue(#1〜#7、RESULTS.mdの各セクションからリンク)にあります。
位置づけ: 構成要素のほとんどは先行研究からの借り物です。全体は「各層を学習→即離散化→凍結し、次層を本物のビット上で学習する」という素朴なレシピで組み立てられていますが、これが新しいかどうかは分かりません(ちゃんと調べていないので既出の可能性は高いです)。離散化ギャップゼロはこのレシピの帰結、メモリ効率と適応深さはCascade-Correlation / Forward-Forward由来です。詳細は英語本文の「What this borrows, and what it puts together」を参照してください。