diff --git a/.github/workflows/github-actions-cron-update-yosys.yml b/.github/workflows/github-actions-cron-update-yosys.yml index 3fd4a9e708..8e356bc117 100644 --- a/.github/workflows/github-actions-cron-update-yosys.yml +++ b/.github/workflows/github-actions-cron-update-yosys.yml @@ -1,6 +1,8 @@ name: Create draft PR for updated YOSYS submodule on: push: + branches: + - master schedule: - cron: "0 8 * * MON" # Allows you to run this workflow manually from the Actions tab @@ -9,6 +11,9 @@ on: jobs: update: runs-on: ${{ vars.USE_SELF_HOSTED == 'true' && 'self-hosted' || 'ubuntu-latest' }} + permissions: + contents: write + pull-requests: write steps: - name: Check out repository code recursively uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 diff --git a/.gitignore b/.gitignore index 84d1e39569..8e1825467c 100644 --- a/.gitignore +++ b/.gitignore @@ -108,3 +108,13 @@ MODULE.bazel.lock # python venv venv/ tmp/ + +# Congestion/thermal ML — generated data, model weights, and logs +flow/util/ml/congestion/data/*.npy +flow/util/ml/congestion/data/*.npz +flow/util/ml/congestion/data/*.png +flow/util/ml/congestion/checkpoints/*.pt +flow/util/ml/data/ +flow/thermal_report.html +flow/util/ml/congestion/*.log +flow/util/ml/congestion/pipeline/logs/ diff --git a/.gitmodules b/.gitmodules index 20440a806e..774907c56b 100644 --- a/.gitmodules +++ b/.gitmodules @@ -3,7 +3,7 @@ url = ../../The-OpenROAD-Project/yosys.git [submodule "tools/OpenROAD"] path = tools/OpenROAD - url = ../OpenROAD.git + url = https://github.com/The-OpenROAD-Project/OpenROAD.git [submodule "tools/kepler-formal"] path = tools/kepler-formal url = https://github.com/keplertech/kepler-formal diff --git a/PR_EXTENSION_DEV_LOG.md b/PR_EXTENSION_DEV_LOG.md new file mode 100644 index 0000000000..5d31d88651 --- /dev/null +++ b/PR_EXTENSION_DEV_LOG.md @@ -0,0 +1,903 @@ +# P&R Extension — Development Log + +Branch: `pr-extension` +Started: 2026-08-14 +Author: JayRaj21 + +This file is the canonical dev log for the `pr-extension` branch. +Every decision, change, and next step is recorded here so work can resume +from a cold start without losing context. Update it after every session. + +--- + +## Branch Context + +`pr-extension` carries all the ML prediction work from the earlier `thermal-solver` / +`congestion-ml` branch (see commit history below), plus new P&R stage augmentation work. + +### Inherited commits (ML pipeline — do not re-do) + +| Commit | Date | Summary | +|---|---|---| +| `e1a1f93` | 2026-08-13 | thermal: replace uniform power model with cell-type-weighted model | +| `a875c09` | 2026-08-12 | gitignore: exclude ML training logs and pipeline log directory | +| `fe37faf` | 2026-08-11 | Add thermal prediction pipeline: HotSpot U-Net, dataset builder, variant generator | +| `1261b7e` | 2026-08-06 | Fix extraction scripts: wrong OpenROAD Python API usage | +| `3b569d0` | 2026-08-06 | Add run_pipeline.sh and extract_existing.sh | +| `c6ee9e9` | 2026-08-06 | Add test suite and fix Swin LayerNorm shape bug | +| `a6286ac` | 2026-08-06 | Add Swin, RF/XGBoost, Ensemble, Diffusion congestion models | +| `10cec03` | 2026-08-06 | Add congestion ML pipeline from scratch (U-Net + GNN, 3 output heads) | + +The ML work is fully documented in `flow/util/ml/congestion/DESIGN_RUNS.md`. +Do not duplicate that content here — read it for ML context. + +--- + +## New Direction: P&R Stage Augmentation + +### Decision (2026-08-14) + +Goal: augment the Place & Route flow to demonstrate understanding of how P&R works. +Constraint: do not rewrite core algorithms (placement/routing engines). +Approach: add tooling that sits *around* the existing stages — analysis, feedback, and +post-processing — using OpenROAD's Tcl/Python APIs and ORFS hook points. + +### Options evaluated (2026-08-14) + +Four directions were considered: + +| # | Option | Demonstrates | Complexity | +|---|---|---|---| +| 1 | **Timing-driven post-placement cell perturbation** | placement ↔ timing feedback loop | Medium | +| 2 | **Congestion-feedback floorplan parameter tuner** | ML integration into flow | Medium — needs trained model | +| 3 | **Stage-by-stage quality metric aggregator** | quality trajectory across P&R | Low | +| 4 | **CTS skew analysis and buffer profiler** | CTS internals | Medium | + +**Decision: start with option 1 (timing-driven perturbation), with option 3 as a +supporting diagnostic layer.** + +Rationale: +- Option 1 has a clear success metric (improved WNS/TNS) and directly demonstrates + the most fundamental P&R trade-off: placement quality drives timing closure. +- Option 3 is lightweight and makes the results of option 1 visible — shows before/after + HPWL, WNS, TNS, congestion overflow at each checkpoint. +- Option 2 requires a trained congestion model; blocked until ML data collection is done. +- Option 4 is interesting but CTS is self-contained — less central than timing feedback. + +--- + +## Codebase Map + +``` +flow/ +├── scripts/ +│ ├── global_place.tcl # Stage 3_1: global placement +│ ├── detail_place.tcl # Stage 3_5: detail placement +│ ├── cts.tcl # Stage 4_1: clock tree synthesis +│ ├── global_route.tcl # Stage 5_1: global routing +│ ├── detail_route.tcl # Stage 5_2: detail routing +│ └── final_report.tcl # Stage 6: metrics collection +├── ml/ +│ └── congestion/ # All ML work (see DESIGN_RUNS.md) +│ ├── data_collection/ +│ ├── models/ +│ ├── training/ +│ └── inference/ +└── PR_EXTENSION_DEV_LOG.md # This file +``` + +P&R stage checkpoints written to `results////`: + +| File | Stage | Contents | +|---|---|---| +| `3_1_place.odb` | Global placement | Cell positions (not legalised) | +| `3_5_place.odb` | Detail placement | Legalised, optimised placement | +| `4_1_cts.odb` | Post-CTS | Clock buffers inserted, timing propagated | +| `5_1_grt.odb` | Global routing | Route topology without geometry | +| `5_2_route.odb` | Detail routing | Full geometry | +| `6_final.odb` | Final | Sign-off ready | + +--- + +## Implementation Plan + +### Phase 1 — Stage-by-stage metric aggregator (option 3) ✓ DONE + +**File: `flow/util/pr_metrics.py`** + +Parses existing ORFS `.rpt` and `.log` files (no OpenROAD process needed) and prints +a stage-by-stage table of WNS, TNS, worst slack, Fmax, HPWL, GRT overflow, and power. + +**Status: complete. Tested on nangate45/ibex/base and nangate45/adder4/base.** + +### Phase 2 — Timing-driven post-placement cell perturbation (option 1) ✓ DONE + +**File: `flow/scripts/post_cts_timing_repair.tcl`** + +Tcl hook sourced at the `POST_CTS` point inside `cts.tcl`. Runs inside the live +OpenROAD session so all STA and ODB APIs are available. + +**Algorithm:** +1. Read WNS via `sta::worst_slack -max`. Exit early if no setup violations. +2. Capture `report_timing -path_count 10 -path_delay max` to a string using `redirect -string`. +3. Parse instance/cell pairs from the timing report using regex on the pin lines. +4. Build an upsize map dynamically from loaded libraries: `TYPE_X → TYPE_X<2N>`. +5. For each unique instance on a critical path (excluding DFFs and clock cells): + - Call `$inst swapMaster $new_master` via ODB to replace the master in-place. +6. After all swaps: run `detailed_placement` to re-legalise (widths changed), then + `estimate_parasitics -placement` to update wire models. +7. Report before/after WNS. + +**Hook variable:** `POST_CTS_TCL` (not `POST_CTS` — discovered from `util.tcl:source_step_tcl`). + +**Wiring it in (per design or globally):** +```makefile +# In flow/designs///config.mk: +export POST_CTS_TCL = $(SCRIPTS_DIR)/post_cts_timing_repair.tcl +``` + +**Key design decisions:** +- DFFs and clock cells are excluded — swapping them changes hold/setup arcs and + disturbs the CTS-balanced clock tree. +- Upsize map built from loaded libs at runtime, not hardcoded — works for any PDK + following the `_X` convention. +- Re-legalisation is run once after all swaps, not per-swap, to avoid redundant work. +- `redirect -string` used to capture timing report without writing a temp file. + +**Status: complete and verified end-to-end on nangate45/ibex/base.** + +**First successful run (2026-08-14):** +``` +INFO [pctr] WNS -0.007 ns — starting cell upsizing on critical paths. +INFO [pctr] Upsize map: 84 candidate transitions loaded. +INFO [pctr] swapped _27049_ AND2_X1 -> AND2_X2 +INFO [pctr] 1 cell(s) upsized, 0 skipped. +INFO [pctr] WNS: -0.007 ns -> -0.004 ns (delta +0.003 ns) +``` +ODB SHA changed (161557dd vs 76a0e40c baseline), confirming the hook made real design modifications. + +--- + +## API Notes (OpenROAD version in orfs:latest as of 2026-08-14) + +The following STA/ODB Tcl API calls were discovered during debugging: + +| Call | Status | Notes | +|---|---|---| +| `sta::worst_slack -max` | ✓ works | Returns float | +| `find_timing_paths -path_delay max -sort_by_slack` | ✓ works | Returns list of PathEnd objects | +| `find_timing_paths -path_count N ...` | ✗ not supported | Use `lrange` on result instead | +| `[$path_end path]` | ✓ works | Returns Path object | +| `[$path pin]` | ✓ works | Returns OpenSTA Pin* | +| `[get_full_name $sta_pin]` | ✓ works | Returns "inst_name/port" string | +| `[$path prevPath]` | ✓ works | Returns previous Path* or NULL | +| `redirect -string { ... }` | ✗ not available | Not defined in this build | +| `redirect $file { ... }` | ✗ not available | Not defined in this build | +| `sta::report_path_string` | ✗ not available | Not defined in this build | +| `$block findInst $name` | ✓ works | ODB lookup by instance name | +| `$inst swapMaster $master` | ✓ works | ODB in-place cell swap | +| `detailed_placement` | ✓ works | Re-legalises after width changes | +| `estimate_parasitics -placement` | ✓ works | Wire model update | + +--- + +## Known ORFS Hook Points + +ORFS supports pre/post hooks for each stage via variables: +``` +PRE_GLOBAL_PLACE / POST_GLOBAL_PLACE +PRE_DETAIL_PLACE / POST_DETAIL_PLACE +PRE_CTS / POST_CTS +PRE_GLOBAL_ROUTE / POST_GLOBAL_ROUTE +PRE_DETAIL_ROUTE / POST_DETAIL_ROUTE +``` + +Set in design config or `Makefile`: +```makefile +export POST_CTS = $(SCRIPTS_DIR)/my_post_cts_hook.tcl +``` + +The hook is sourced inside the OpenROAD session that already has the ODB loaded, +so all `odb`, `sta`, `grt`, `dpl` commands are available. + +--- + +## Session Log + +### 2026-08-14 — Session start, direction set + +- Reviewed P&R stage scripts: `global_route.tcl`, `detail_place.tcl`, `cts.tcl`. +- Reviewed existing ML work in `flow/util/ml/congestion/`. +- Evaluated four augmentation directions (documented above). +- Decision: deterministic augmentation only — no ML in the P&R extension. + Rationale: ML earns its place where EDA tools have no answer (thermal, pre-placement + congestion). For post-CTS timing repair, OpenROAD's STA already has exact ground truth; + using ML there would replace a precise answer with an approximation. +- Decision: implement metric aggregator (phase 1) then timing perturbation (phase 2). + +--- + +### 2026-08-14 — Phase 1 complete: stage-by-stage metric aggregator + +**New file: `flow/util/pr_metrics.py`** + +Standalone Python script (no OpenROAD required) that parses existing ORFS report and log +files and prints a stage-by-stage quality trajectory table. + +**Metrics collected per stage:** + +| Metric | Source | Stage(s) | +|---|---|---| +| WNS (worst negative slack, ns) | `.rpt` | all | +| TNS (total negative slack, ns) | `.rpt` | all | +| Worst slack (ns) | `.rpt` | all | +| Fmax (MHz) | `.rpt` | all | +| HPWL (half-perimeter wirelength, µm) | `3_3_place_gp.log` | global place | +| GRT overflow | `3_3_place_gp.log`, `5_1_grt.log` | global place, global route | +| Total power (W) | `.rpt` | last available | + +**Usage:** +```bash +# From repo root: +python3 flow/util/pr_metrics.py --platform nangate45 --design ibex --tag base +python3 flow/util/pr_metrics.py --platform nangate45 --design adder4 --tag base + +# With explicit paths: +python3 flow/util/pr_metrics.py \ + --reports-dir flow/reports/nangate45/ibex/base \ + --logs-dir flow/logs/nangate45/ibex/base +``` + +**Example output (ibex/base — timing-stressed design):** +``` +Stage WNS (ns) TNS (ns) Worst slack Fmax (MHz) HPWL (um) GRT overflow +Global place +0.000 +0.000 +0.020 459.4 331,831,045 1.2875 +Resizer +0.000 +0.000 +0.020 459.4 — — +Detail place -0.030 -1.430 -0.030 448.9 — — +CTS -0.000 -0.000 -0.000 454.5 — — +Global route -0.260 -81.060 -0.260 406.9 — — +Finish +0.000 +0.000 +0.000 455.5 — — +Total power (post-route): 3.1700e-02 W +``` + +This makes the timing degradation at each stage visible — detail placement introduces hold +violations, global route reveals setup violations at real wire parasitics, final sign-off +recovers them. This baseline is needed to measure the impact of the phase 2 cell swapping hook. + +--- + +--- + +### 2026-08-14 — Phase 2 verified end-to-end + +After resolving several API incompatibilities in the orfs:latest Docker image +(no `redirect`, no `-path_count` flag on `find_timing_paths`, no +`sta::report_path_string`), the hook was rewritten to use direct path object +traversal: `[$path_end path]` → `[$path pin]` → `get_full_name` → ODB lookup. + +**Key mechanics confirmed working:** +- `find_timing_paths -path_delay max -sort_by_slack` returns path end objects +- `[$path_end path]` / `[$path prevPath]` traverses path backwards +- `get_full_name [$path pin]` gives "inst_name/port" which we split to get inst name +- `$block findInst $name` converts name to ODB dbInst* +- `$inst swapMaster $new_master` swaps the cell in-place +- `detailed_placement` re-legalises cleanly (0 displacement after upsize) +- `estimate_parasitics -placement` updates wire models + +**Result on ibex/base:** 1 cell swapped (AND2_X1 → AND2_X2), WNS -0.007 → -0.004 ns. +Only 1 cell found because repair_timing had already upsized most candidates; the +remaining violation was in a deep path with limited upsize opportunity. + +**How to run the hook:** +```bash +rm results/nangate45/ibex/base/4_1_cts.odb # force rebuild +util/docker_shell make cts \ + DESIGN_CONFIG=designs/nangate45/ibex/config.mk \ + POST_CTS_TCL=/work/scripts/post_cts_timing_repair.tcl +``` + +--- + +### 2026-08-14 — Controlled before/after comparison + +Ran `util/compare_hook.sh` to compare the full flow from the same `3_place.odb` +checkpoint, with and without the POST_CTS hook. Stage 1–4 numbers were identical +in both runs, confirming a clean controlled comparison. + +**Results (`nangate45/ibex/base`):** + +``` +Stage Baseline WNS Hook WNS Delta +Global place +0.000 +0.000 — +Resizer +0.000 +0.000 — +Detail place +0.000 +0.000 — +CTS -0.010 -0.010 — (report written before hook runs) +Global route -0.020 -0.000 +0.020 ns ← key improvement +Finish +0.000 +0.000 — +``` + +**Global route TNS:** -0.110 ns (baseline) → -0.000 ns (hook) +**Global route Fmax:** 451.4 MHz (baseline) → 454.0 MHz (hook, +2.6 MHz) +**Total power:** identical at 3.17e-02 W — upsize did not measurably increase power. + +**Interpretation:** +The hook's 3 ps improvement at CTS (AND2_X1 → AND2_X2 swap on `_27049_`) translated +into 20 ps of recovered slack at global route, eliminating all setup violations before +detail route ran. The gain amplified because upsizing reduces gate delay across the +cell's entire fanout cone; when real wire parasitics were added at global route, the +baseline was marginal enough to be pushed into violation while the hook version had +just enough headroom to absorb them. Both designs closed timing at finish, but the +hook version arrived at detail route with a cleaner slate. + +**Tooling added:** `util/compare_hook.sh` — runs both flows and prints tables +back-to-back for repeatable before/after comparison. + +--- + +--- + +### 2026-08-14 — Phase 2 upgraded: iterative upsizing + +**Changed: `flow/scripts/post_cts_timing_repair.tcl`** + +The single-pass `run` proc was refactored into an iterative loop: + +**New structure:** + +- `collect_candidates upsize_arr path_count seen_arr` — finds new upsize candidates + on the N worst paths, skipping instances already swapped in prior iterations. +- `apply_swaps candidates` — applies ODB swaps, returns {swap_count skip_count}. +- `run {path_count 10} {max_swaps 30} {max_iters 5}` — outer loop: + 1. Collect candidates (deduped via `seen` array) + 2. Apply swaps + 3. Re-legalise + re-estimate parasitics + 4. Re-run STA; stop if WNS ≥ 0, no candidates, or no swaps applied + 5. Repeat up to `max_iters` times + +**Why iterative matters:** +A single pass swaps cells on the *current* critical paths. After those swaps + +re-legalisation, the critical paths may change — a previously non-critical path may +become the new worst path. Each iteration finds new candidates on the updated critical +paths, so the hook converges rather than leaving residual violations untouched. + +**The `seen` array spans all iterations**, so a cell that was upsized in iteration 1 +(e.g., `AND2_X1 → AND2_X2`) is not considered again in iteration 2 — it is already at +the higher drive level and would need a second upsize (`AND2_X2 → AND2_X4`) to improve +further. This is intentional: one upsize per cell per hook invocation keeps the area +budget predictable. + +**Signature unchanged** — still invoked as `pctr::run` with no arguments for the +standard 10-path / 30-swap / 5-iteration defaults. + +--- + +--- + +### 2026-08-14 — aes comparison revealed post-GRT hook gap; added post_grt_timing_repair.tcl + +**Observation from aes comparison:** +The post-CTS hook correctly skipped on `aes` because CTS timing was met (WNS +0.000). +The violations in aes (-0.020 at global route, -0.010 at finish) only appear once real +wire parasitics are loaded by the GRT step — they are invisible at CTS time. + +**New file: `flow/scripts/post_grt_timing_repair.tcl`** + +Same iterative upsizing algorithm as `post_cts_timing_repair.tcl`, wired to the +`POST_GLOBAL_ROUTE_TCL` hook point. One critical difference in the parasitic +re-estimation step: + +| Hook | Parasitic call after swaps | Why | +|---|---|---| +| post_cts_timing_repair.tcl | `estimate_parasitics -placement` | GRT not yet run | +| post_grt_timing_repair.tcl | `estimate_parasitics -global_routing` | GRT topology available | + +Using `-global_routing` means the hook's STA reflects the actual route topology, so +the WNS reported inside the hook matches the global route report — no artificial +optimism from placement-only estimates. + +Namespace: `pgtr` (vs `pctr` for the CTS hook) to avoid name collisions when both +hooks are active in the same session. + +**Updated: `flow/util/compare_hook.sh`** + +Now accepts both hooks simultaneously by default (CTS + GRT). Flags to disable either: +```bash +# Both hooks (default) +util/compare_hook.sh --platform nangate45 --design aes + +# CTS hook only +util/compare_hook.sh --platform nangate45 --design aes --no-grt-hook + +# GRT hook only +util/compare_hook.sh --platform nangate45 --design aes --no-cts-hook +``` + +--- + +### 2026-08-14 — aes comparison shows post-GRT hook is redundant; architectural insight + +**Result:** aes baseline and hook numbers are identical. The GRT hook does not improve timing. + +**Root cause — reading `flow/scripts/global_route.tcl`:** + +The `POST_GLOBAL_ROUTE_TCL` hook fires at line 151, *after* all of the following have +already run: +1. `global_route` — builds the routing topology +2. `estimate_parasitics -global_routing` — loads real wire RC +3. `repair_design_helper` — fixes max-cap/max-slew violations +4. `repair_timing_helper` — fixes setup/hold with gate sizing, buffer insertion, cell swapping +5. Another `estimate_parasitics -global_routing` +6. `report_metrics 5 "global route"` — writes `5_global_route.rpt` +7. ← **Our hook fires here** + +ORFS's built-in `repair_timing` at step 4 is far more capable than our simple upsizing +(it does buffer insertion, VT swaps, and multi-objective repair). By the time our hook +runs, there are few or no candidates left. + +**Why ibex worked but aes does not:** +The post-CTS hook fires *before* the GRT repair. Our upsize reduces gate delay on the +critical path, which becomes the starting point for GRT repair to refine further. That +compounding effect is what eliminated ibex's violations entirely. + +For aes, CTS timing is met (+0.000 WNS), so the post-CTS hook correctly skips. +The violations at global route (-0.020 WNS) appear when real parasitics are loaded, +but the built-in GRT repair partially addresses them. The residual violations at finish +(-0.010 WNS) are introduced by *detail routing* — the actual wire geometry after DRC- +legal routing differs from the GRT topology estimate. Post-detail-route violations require +an ECO (Engineering Change Order) flow, not simple cell upsizing. + +**Conclusion:** +The post-GRT hook is architecturally redundant with ORFS's built-in repair. The +post-CTS hook is the correct intervention point: it runs before the built-in GRT repair, +so improvements compound rather than compete. + +`post_grt_timing_repair.tcl` is kept for completeness and as a documented dead-end +that explains *why* the post-CTS hook is the right intervention point. + +--- + +--- + +### 2026-08-22 — Phase 3: triage agent + +**New file: `flow/util/triage_agent.py`** + +LLM-powered diagnostic layer that sits on top of the existing toolchain: + +``` +pr_metrics.py → collect() → triage_agent.py → Claude → diagnosis +``` + +**What it does:** +1. Calls `pr_metrics.collect()` to read the stage-by-stage quality trajectory. +2. Computes notable stage-to-stage WNS deltas (threshold: ≥5 ps change). +3. Builds a structured prompt with the trajectory table, deltas, and final metrics. +4. Calls `claude-opus-5` with adaptive thinking and a system prompt encoding + P&R domain knowledge — known failure patterns, what each stage does, and + the ORFS parameters and hooks available on this branch. +5. Prints a structured diagnosis: root cause, evidence, recommended actions, + expected outcome. + +**Model:** `claude-opus-5` with `thinking: {type: "adaptive"}`. + +**Usage:** +```bash +export ANTHROPIC_API_KEY= # or: ant auth login + +python3 flow/util/triage_agent.py --platform nangate45 --design ibex --tag base +python3 flow/util/triage_agent.py --platform nangate45 --design aes --tag base +``` + +**Why this is distinct from ORFS-Agent (ABKGroup):** +ORFS-Agent tunes top-level flow parameters (utilisation, density) across +multiple parallel runs. This triage agent reads the *inside* of a completed +run — the per-stage quality trajectory — and diagnoses which specific stage +caused the failure and why. It operates on a single run and produces a +targeted intervention recommendation rather than a search over parameter space. + +**Branch story (complete):** +``` +observe → pr_metrics.py (what happened at each stage?) +intervene → post_cts_*_tcl (fix it inside the live OpenROAD session) +decide → triage_agent.py (diagnose why, recommend what to try next) +``` + +--- + +--- + +### 2026-08-22 — Phase 4: validate triage diagnosis on aes + +**Goal:** confirm the triage agent's recommended fix actually closes timing on aes. + +**What the triage agent diagnosed (aes/nangate45/base):** +- CTS WNS +0.000 ns, GRT WNS −0.330 ns — classic CTS→GRT parasitic cliff +- Root cause: CTS uses `estimate_parasitics -placement` (optimistic); real wire RC + only known after GRT, causing endpoints to look clean at CTS but violate at GRT +- Recommendation: `SETUP_SLACK_MARGIN=0.03`, `TNS_END_PERCENT=100`, + `POST_CTS_TCL=$(SCRIPTS_DIR)/post_cts_timing_repair.tcl`; re-run CTS then finish + +**Validation run (variables passed on make command line to reach Docker container):** +```bash +util/docker_shell make DESIGN_CONFIG=designs/nangate45/aes/config.mk \ + SETUP_SLACK_MARGIN=0.03 \ + POST_CTS_TCL=/work/scripts/post_cts_timing_repair.tcl \ + cts +util/docker_shell make DESIGN_CONFIG=designs/nangate45/aes/config.mk \ + SETUP_SLACK_MARGIN=0.03 \ + POST_CTS_TCL=/work/scripts/post_cts_timing_repair.tcl \ + finish +``` + +**Key lesson — Docker variable passing:** +The container runs make from `/OpenROAD-flow-scripts/flow/` (image copy of the repo), +not from `/work/` (the mounted workspace). Local `config.mk` changes are NOT seen. +Variables must be passed explicitly as `make VAR=value` arguments on every invocation. +`HOOK_PATHS` in `loop_agent.py` uses `/work/scripts/...` (Docker workspace path); +`CONFIG_HOOK_PATHS` stores `$(SCRIPTS_DIR)/...` (ORFS-canonical) for config.mk write-back. + +**Result:** +| Metric | Before | After | +|--------|--------|-------| +| GRT WNS | −0.330 ns | −0.010 ns | +| Finish WNS | −0.010 ns | 0.000 ns | +| Finish TNS | −0.330 ns | 0.000 ns | +| Fmax | ~1190 MHz | ~1239 MHz (+49 MHz) | + +Triage agent's prediction ("GRT WNS ≥ −0.005 after fix") confirmed. + +**Committed:** `bad2f2bd8` — aes: apply triage-agent recommendations to close timing + +--- + +### 2026-08-22 — Phase 5: closed-loop optimization agent + +**New file: `flow/util/loop_agent.py`** + +Autonomous observe→diagnose→intervene→verify cycle. No human intervention needed. + +**Architecture:** +``` +loop_agent.py + ├── get_metrics → calls pr_metrics.collect(), formats trajectory table + ├── set_config_param → queues param change; translates "enabled" → Docker hook path + ├── run_stage → deletes stale ODB files, runs docker_shell make + └── finish → terminates loop; on success calls write_config_params +``` + +**Four tools exposed to Claude Opus 5:** +1. `get_metrics` — read current WNS/TNS/Fmax/overflow trajectory +2. `set_config_param(param, value)` — allowlisted params only, value checked against + `UNSAFE_VALUE_PATTERNS` (blocks `$(`, `${`, backticks, shell metacharacters) to + prevent Make-injection via config.mk write-back; "enabled" → hook path +3. `run_stage(stage)` — valid stages: `place`, `cts`, `grt`, `finish` +4. `finish(summary, success)` — terminate; if success=True, write params to config.mk + +**PARAM_ALLOWLIST:** +`SETUP_SLACK_MARGIN`, `TNS_END_PERCENT`, `OPT_POST_GRT_WNS`, +`PLACE_DENSITY_LB_ADDON`, `POST_CTS_TCL`, `POST_GLOBAL_ROUTE_TCL` + +**STAGE_STALE_FILES** — files deleted before each stage re-run: +- `place`: `3_3_place_gp.odb` through `3_place.odb` (PLACE_DENSITY_LB_ADDON affects global place) +- `cts`: `4_1_cts.odb`, `4_cts.odb` +- `grt`: `5_1_grt.odb`, `5_1_grt.sdc` +- `finish`: `5_2_route.odb`, `5_route.odb` + +**Write-back (`write_config_params`):** +On success, updates `designs///config.mk` in-place: +- Regex-matches existing `export PARAM = ...` lines and updates them +- Appends new params with `# Written by loop_agent.py` comment +- Translates Docker paths (`/work/scripts/...`) → ORFS-canonical (`$(SCRIPTS_DIR)/...`) + +**End-to-end result on aes/nangate45/base:** +Single iteration, no human intervention. Agent called `set_config_param` 3×, +`run_stage("cts")`, `run_stage("finish")`, verified metrics, called `finish(success=True)`. +Final WNS 0.000, Fmax 1239 MHz. Params written to config.mk. + +**Commits:** +- `7671426da` — loop_agent: add closed-loop optimization agent +- `1afb28fd9` — loop_agent: add write-back and placement-stage support + +--- + +### 2026-08-23 — Phase 6: unit tests + +**New file: `flow/util/test_loop_agent.py`** + +24 unit tests covering all non-Docker, non-API logic. No API key or Docker required. + +**Test classes:** +- `TestAllowlist` — rejects unknown params (including injection attempts); accepts all 6 allowlisted +- `TestHookTranslation` — `"enabled"` → `/work/scripts/...` for both hook params; case-insensitive; + numeric params untouched; explicit paths not double-translated +- `TestStaleFilePaths` — correct files for each stage; `place` list starts at `3_3_place_gp.odb`; + no CTS outputs in place list +- `TestWriteConfigParams` — in-place update, append, Docker→canonical path translation, + no duplication, error on missing file, comment only added for new params + +**Run:** +```bash +cd flow && python3 util/test_loop_agent.py +``` +All 24 pass in ~0.004 s. + +**Commit:** `b84e1d4ac` — loop_agent: add unit test suite (24 tests, no API/Docker required) + +--- + +### 2026-08-26 — Review fixes: value-side injection blocklist, hook dedup, regression tests + +**Problem:** `set_config_param` validated the param *name* against `PARAM_ALLOWLIST` but +not the *value*. Since `write_config_params` writes the value verbatim into `config.mk` +(a GNU Make include), an adversarial or hallucinated value containing `$(shell ...)` — +or its `${shell ...}` equivalent, since Make treats `$(...)` and `${...}` as +interchangeable — would execute arbitrary shell code on the next `make` invocation. + +**Fix (`flow/util/loop_agent.py`):** added `validate_param_value()`, called from +`impl_set_config_param()` before a value is queued. Rejects values containing any of +`UNSAFE_VALUE_PATTERNS` (`$(`, `${`, backtick, `;`, `|`, `&`, newline/CR). + +**Also:** `post_cts_timing_repair.tcl` and `post_grt_timing_repair.tcl` were near +byte-for-byte duplicates (~200 lines each). Factored the shared upsizing logic into +`flow/scripts/timing_repair_common.tcl` (namespace `::trepair`), parameterized by +log-prefix and parasitics mode (`-placement` vs `-global_routing`); both hook files are +now thin wrappers that source the common lib. + +**Tests:** added regression cases in `test_loop_agent.py` covering both the `$(` and +`${` value-injection forms for an allowlisted param, distinct from the existing +name-injection test. Suite is now 28 tests (was 24). + +**Commits:** +- `8c77f24cf` — validate config param values; dedupe timing-repair Tcl hooks into shared lib +- `01cb3c686` — block `${` Make-syntax variant in config param value validation +- `d34d787fd` — add regression tests for config value injection blocklist + +--- + +### 2026-08-23 — PR opened + +**PR #1:** https://github.com/JayRaj21/OpenROAD-flow-scripts/pull/1 + +Title: "pr-extension: LLM-driven P&R triage, closed-loop optimization, and congestion ML pipeline" + +Branch `pr-extension` → `master`. + +--- + +### 2026-08-27 — Regression / benchmark dashboard + +**Goal:** turn the single-run `pr_metrics.py` snapshot into a history that can catch +regressions across runs/commits, usable both interactively and as a CI gate. + +**`flow/util/benchmark_dashboard.py`:** new module, two subcommands, argparse styled +after `pr_metrics.py` (`--platform`/`--design`/`--tag` or `--reports-dir`/`--logs-dir`, +same `--flow-dir` default). Imports `collect()` from `pr_metrics.py` rather than +re-parsing reports/logs — that duplication (pr_metrics.py/triage_agent.py/loop_agent.py/ +compare_hook.sh all independently extracting the same metrics) was already flagged as a +review issue, so this is strictly a history/regression layer on top of the existing +parser. `pr_metrics.py` itself is untouched. + +- `record`: runs `collect()`, appends one JSON object (timestamp, `git rev-parse HEAD`, + platform/design/tag, per-stage metrics dict) as a line to + `flow/util/benchmark_history/____.jsonl`. JSONL + open-append + (`"a"` mode) was chosen specifically so a crash or concurrent writer can never + corrupt or rewrite prior history — each record is independent and the file is safe to + tail/grep. +- `report`: reads the history file for a stage (default `Finish`), prints a table with + per-metric deltas vs. the previous record and `worse-than-best-*` flags vs. the + best-ever value across history. Regression detection compares only the latest record + against its immediate predecessor (not best-ever) against three configurable + thresholds — WNS worsening (`--wns-threshold`, default 0.01 ns), Fmax percentage drop + (`--fmax-threshold-pct`, default 1.0), GRT/GP overflow increase (`--overflow-threshold`, + default 0.001) — and exits 1 if any fire, 0 otherwise, so it drops straight into a CI + pipeline as a gate. Fewer than 2 records just prints the single row and exits 0. + `--html` additionally emits one self-contained HTML file (inline `` line charts + for WNS/Fmax/HPWL, inline ``, and has no +`http(s)://` references (confirms it's genuinely offline-renderable), and +`resolve_dirs()` tag-derivation tests (`--reports-dir` derives the tag from the path's +last component when `--tag` isn't passed; an explicit `--tag` overrides derivation). +Ran together with the existing suite: + +```bash +cd flow/util && python3 -m pytest test_benchmark_dashboard.py test_loop_agent.py -v +``` +62 passed (34 new + 28 existing), confirming no regression to `loop_agent.py`. Formatted +both new files with `black` (26.5.1). + +**Left out of scope:** no Makefile/CI wiring to auto-invoke `record` after every flow +run (roadmap says infra-only for this pass; wiring belongs with whichever CI workflow +task consumes it), no retention/pruning policy for history files (JSONL is cheap and +append-only; pruning can be a follow-up if files get large), no cross-design aggregate +dashboard (each `____` gets its own file/report, matching how +`pr_metrics.py` is already scoped to one run at a time). + +## Planned Next Steps + +1. ~~Implement `pr_metrics.py`~~ ✓ +2. ~~Implement `post_cts_timing_repair.tcl` — single-pass~~ ✓ +3. ~~Controlled before/after comparison on ibex~~ ✓ +4. ~~Make hook iterative~~ ✓ +5. ~~Add post-GRT hook — tested, found redundant with built-in repair~~ ✓ +6. ~~Triage agent — LLM diagnosis of per-stage quality trajectory~~ ✓ +7. ~~Validate triage diagnosis on aes (end-to-end timing closure)~~ ✓ +8. ~~Closed-loop optimization agent (loop_agent.py)~~ ✓ +9. ~~Write-back to config.mk on success~~ ✓ +10. ~~Unit tests (28, no API/Docker required)~~ ✓ +11. ~~Open PR~~ ✓ +12. ~~Value-side injection blocklist for config param write-back~~ ✓ +13. ~~Dedupe post-CTS/post-GRT timing-repair hooks into shared lib~~ ✓ +14. **Integration test**: run loop agent end-to-end on aes baseline with API key to confirm + full cycle (observe → diagnose → intervene → verify → write-back) works live +15. **Placement-stage test**: run a high-utilization aes variant (CORE_UTILIZATION=80) + to exercise the `PLACE_DENSITY_LB_ADDON` / `place` re-run path end-to-end + (currently unit-tested only) +16. **Second design**: run triage + loop on ibex or another design to validate generalization +17. (Blocked on ML data) Congestion-feedback parameter tuner +18. ~~Regression/benchmark dashboard (`benchmark_dashboard.py`)~~ ✓ + +--- + +### 2026-09-11 — Independent validator review: 7 fixes to benchmark_dashboard.py + +An independent validator agent re-ran the CI-gate scenarios end-to-end against +`flow/util/benchmark_dashboard.py` and found seven ways the "gate" could report +green (or crash) on a genuinely broken run. All seven are fixed on this branch, +`benchmark_dashboard.py` only: + +- **HIGH — torn/corrupt newest line silently gated green.** `load_records` now + returns `(records, dropped_last_line)`; if the *most recent* physical line in + the history file was corrupt/malformed, `cmd_report` prints a clear + `stderr` error ("history file has a corrupt/truncated record and cannot be + safely compared") and exits 1, instead of silently comparing record N-2 vs + N-1 and reporting success. +- **HIGH — empty latest-stage metrics gated green.** `detect_regressions` now + flags `{}`/missing metrics on the current record (when the previous record + had non-empty metrics for the same stage) as its own regression + ("stage produced no metrics — design may have failed to reach this stage"), + so `cmd_report` exits 1 instead of reporting a clean run when a design + stopped producing timing numbers for the requested stage. +- **HIGH — `resolve_dirs` accepted a one-level-too-high `--reports-dir`.** + Previously any path with ≥3 components was silently sliced into + platform/design/tag, so pointing `--reports-dir` at a *design* directory + (missing the tag level) produced `platform='reports'` and a garbage history + file. `resolve_dirs` now checks that the component 4 levels above the + presumed tag is literally `"reports"`; if not, it raises a clear + `SystemExit` ("does not look like .../reports///") + instead of proceeding. +- **MEDIUM — non-dict/null JSON lines crashed with a raw traceback.** + `load_records` now validates each parsed line is a JSON object with the + expected shape (top-level dict; `stages`, if present, a dict whose values + are each a dict or `null`) and treats anything else as corrupt using the + same skip+warn+last-line-tracking path as a `JSONDecodeError`. `timestamp` + is now read defensively (`rec.get("timestamp") or "—"`) like `git_sha` + already was, and `build_report_rows`/`best_ever` guard against a `null` + nested stage value (`.get(stage) or {}`) instead of crashing on + `None.get(...)`. +- **MEDIUM — non-numeric metric value crashed formatting.** `fmt`/`fmt_delta` + now render anything that isn't `int`/`float` (not just `None`) as `"—"` + instead of raising `ValueError` out of `str.format`. +- **MEDIUM — `cmd_record` died with an unhandled traceback on a malformed + report.** The `collect()` call in `cmd_record` is now wrapped in + `try`/`except Exception`, printing a clear message naming the reports dir + and the underlying exception to `stderr` and exiting 1, rather than letting + a raw traceback surface. `pr_metrics.py` itself was not touched (shared + file, out of scope for this branch). +- **MEDIUM — reader took no lock.** `load_records` now takes a shared lock + (`fcntl.flock(..., LOCK_SH)`) around the read, matching the exclusive lock + `append_record` already takes, so a reader can no longer observe a + partially-written record from a concurrent `record` invocation. + +**Tests (`flow/util/test_benchmark_dashboard.py`):** extended to 50 (from 43), +covering all seven fixes above, plus updated three pre-existing tests that +exercised the old (buggy) `resolve_dirs`/`load_records` behavior directly — +`test_reports_dir_derives_tag_from_path_when_not_passed` and +`test_reports_dir_explicit_tag_overrides_path_derivation` now use a +`--reports-dir` that actually has `reports/` in the right position, and all +`bd.load_records(...)` call sites were updated to unpack the new +`(records, dropped_last_line)` return. + +```bash +cd flow/util && python3 -m pytest test_benchmark_dashboard.py -v +``` +50 passed. + +### 2026-09-11 — Round-2 independent validator review: 5 remaining fixes to benchmark_dashboard.py + +A second, independent validator agent re-tested the fixes above end-to-end +with real CLI runs and found five more real issues, all now fixed on this +branch, `benchmark_dashboard.py` only: + +- **HIGH — `resolve_dirs`'s "4 levels up must be literally `reports`" check + was too strict, and its own error message's suggested workaround was + impossible.** `--platform` and `--reports-dir` are in a mutually-exclusive, + required argparse group, so telling a user hitting the error to "pass + `--platform`/`--design`/`--tag` explicitly" was a dead end for `--platform`. + Worse, it newly rejected previously-working inputs: a relative + `nangate45/ibex/base` path (no `reports` ancestor) or a bare CI artifact + dir like `/tmp/artifacts/nangate45/ibex/base` (no `reports` component at + all) now hard-failed. Fixed by locating the *last* literal `reports` (or + `logs`, mirroring whichever kind of dir is being resolved) path component + via search instead of a fixed offset. If found, exactly 3 components + (platform/design/tag) must follow it — this still catches the original + "one level too high" bug. If no `reports`/`logs` component exists anywhere + in the path, fall back to the prior permissive behavior (last 3 path + components) instead of hard-erroring. (At the time, there was no + argparse-valid way to explicitly override the check in the + `--reports-dir` case — see the follow-up fix below.) +- **MEDIUM — `compute_delta` still crashed on two non-numeric metric + values.** The `fmt`/`fmt_delta` hardening from round 1 didn't cover the + subtraction in `compute_delta` itself, so a history file with `"wns": + "n/a"` in two consecutive records raised an unhandled `TypeError` — + exiting 1 for the same reason a real regression exits 1, making corruption + indistinguishable from a genuine quality regression. `compute_delta` now + returns `None` unless both operands are real numbers. Audited and fixed + the same exposure in `detect_regressions`'s `prev_fmax > 0` comparison and + `render_html`'s point-series filtering (feeding `y_span = y_max - y_min`). +- **MEDIUM — `dropped_last_line` detection was defeated by a trailing blank + line.** It keyed on `lineno == total_lines` (the last *physical* line), but + blank lines are skipped before that check runs, so a corrupt record + immediately followed by a blank line silently escaped detection — exactly + the gap round-1's fix #1 was meant to close. `load_records` now tracks the + last *non-blank* line number and compares against that instead. +- **MEDIUM-LOW — `dropped_last_line`'s exit-1 check in `cmd_report` never + ran when history had zero valid records left after dropping corrupt + lines**, because the `if not records: ... sys.exit(0)` short-circuit ran + first — an all-garbage history file reported exit 0 ("No history found") + instead of flagging corruption. `cmd_report` now checks + `dropped_last_line` before the empty-records short-circuit. +- **LOW — `fmt`/`fmt_delta` accepted `bool`** (since `bool` is an `int` + subclass in Python), rendering a stray JSON `true`/`false` as `1.000`/ + `0.000` instead of `"—"`. Added a shared `is_number()` helper + (`isinstance(val, (int, float)) and not isinstance(val, bool)`) used by + `fmt`, `fmt_delta`, `compute_delta`, `detect_regressions`, and + `render_html`'s series filter. + +**Tests (`flow/util/test_benchmark_dashboard.py`):** extended to 58 (from +50), adding: a `--reports-dir` with no `reports` component and a relative +3-component path both still resolving correctly (item 1); a two-record +history where both records have non-numeric metrics not crashing +`build_report_rows`/`print_report` (item 2); a corrupt-record-followed-by- +blank-line history correctly flagged as `dropped_last_line` (item 3); an +all-garbage history file exiting non-zero via the CLI (item 4); and a bool +JSON value rendering as `"—"` in both `fmt` and `fmt_delta` (item 5). + +```bash +cd flow/util && python3 -m pytest test_benchmark_dashboard.py -v +``` +58 passed. + +### 2026-09-11 — Follow-up: make the strict-shape override actually reachable + +Round-2's fix still left a real usability gap: when the strict path-shape +check does fire, its own suggested remediation ("pass `--platform`, +`--design`, and `--tag` explicitly") was unreachable via the CLI, since +`--platform` and `--reports-dir` lived in the same mutually-exclusive, +required argparse group. Fixed by dropping that group — `--platform` and +`--reports-dir` can now both be passed. `resolve_dirs` now checks +`args.platform` (not just `args.reports_dir`) first: when `--platform` is +given (with or without `--reports-dir`), it derives the path from +platform/design/tag as before, bypassing path-shape validation entirely. +Passing `--reports-dir` alone still goes through the shape check unchanged. +Error messages were updated to point at this override instead of the +now-fixed advice. + +**Tests:** added +`test_record_cli_reports_dir_with_explicit_overrides_bypasses_shape_check`, +exercising the previously-impossible override end-to-end via the CLI +(not just `resolve_dirs()` in isolation): confirms plain `--reports-dir` +one level too high still fails the shape check, and that adding +`--platform`/`--design`/`--tag` alongside it now succeeds. + +```bash +cd flow/util && python3 -m pytest test_benchmark_dashboard.py -v +``` +59 passed. diff --git a/create_pr.sh b/create_pr.sh new file mode 100755 index 0000000000..42de689acd --- /dev/null +++ b/create_pr.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +BODY_FILE="$SCRIPT_DIR/create_pr_body.md" + +gh pr create \ + --title "pr-extension: LLM-driven P&R triage, closed-loop optimization, and congestion ML pipeline" \ + --body-file "$BODY_FILE" \ + --base master diff --git a/create_pr_body.md b/create_pr_body.md new file mode 100644 index 0000000000..4f2d10c183 --- /dev/null +++ b/create_pr_body.md @@ -0,0 +1,19 @@ +## Summary + +- **Triage agent** (`flow/util/triage_agent.py`): reads stage-by-stage metrics and diagnoses P&R quality failures (CTS→GRT parasitic cliff, congestion, residual violations) using Claude Opus 5 with adaptive thinking +- **Closed-loop agent** (`flow/util/loop_agent.py`): autonomously applies triage recommendations, re-runs affected flow stages via Docker, verifies improvement, and writes successful parameters back to `config.mk` +- **Post-CTS/GRT hooks** (`flow/scripts/post_cts_timing_repair.tcl`, `post_grt_timing_repair.tcl`): iterative cell upsizing using placement parasitics; activated by the loop agent via `POST_CTS_TCL`/`POST_GLOBAL_ROUTE_TCL` +- **Metrics collector** (`flow/util/pr_metrics.py`): parses WNS/TNS/Fmax/overflow from ORFS report and log files across all P&R stages into a single trajectory table +- **aes/nangate45 baseline fix**: triage agent correctly identified the CTS→GRT parasitic underestimation cliff; applying `SETUP_SLACK_MARGIN=0.03` + `POST_CTS_TCL` closed timing from WNS −0.330 ns to 0.000 ns, Fmax +49 MHz +- **Congestion ML pipeline** (`flow/util/ml/congestion/`): U-Net + GNN models for routing congestion prediction, thermal estimation, and variant generation +- **Unit tests** (`flow/util/test_loop_agent.py`): 28 tests covering allowlist enforcement, value-side injection blocklist, hook path translation, stale file sets, and config write-back — no API key or Docker required + +## Test plan + +- [ ] `python3 flow/util/test_loop_agent.py` — all 28 unit tests pass +- [ ] `ANTHROPIC_API_KEY= python3 flow/util/loop_agent.py --platform nangate45 --design aes --tag base` — loop agent closes timing and writes params to config.mk +- [ ] `ANTHROPIC_API_KEY= python3 flow/util/triage_agent.py --platform nangate45 --design aes --tag base` — triage agent diagnoses the CTS→GRT cliff correctly + +🤖 Generated with [Claude Code](https://claude.com/claude-code) + +https://claude.ai/code/session_019kUei3bDhQVmVchT8GsDMo diff --git a/docs/references/OpenROAD_Thermal_and_Overview_Findings.pdf b/docs/references/OpenROAD_Thermal_and_Overview_Findings.pdf new file mode 100644 index 0000000000..58254e7fdc Binary files /dev/null and b/docs/references/OpenROAD_Thermal_and_Overview_Findings.pdf differ diff --git a/flow/designs/nangate45/aes/config.mk b/flow/designs/nangate45/aes/config.mk index 2f7d50ef09..570e34cbd9 100644 --- a/flow/designs/nangate45/aes/config.mk +++ b/flow/designs/nangate45/aes/config.mk @@ -12,5 +12,11 @@ export TNS_END_PERCENT = 100 # workaround for high congestion in post-grt repair export SKIP_INCREMENTAL_REPAIR = 1 +# Triage-agent recommendations: pessimistic margin so CTS exposes violations +# that only appear at GRT under real wire RC; post-CTS hook sizes them while +# placement is still legalisable. +export SETUP_SLACK_MARGIN = 0.03 +export POST_CTS_TCL = $(SCRIPTS_DIR)/post_cts_timing_repair.tcl + export SWAP_ARITH_OPERATORS = 1 export OPENROAD_HIERARCHICAL = 1 diff --git a/flow/scripts/post_cts_timing_repair.tcl b/flow/scripts/post_cts_timing_repair.tcl new file mode 100644 index 0000000000..b75f373209 --- /dev/null +++ b/flow/scripts/post_cts_timing_repair.tcl @@ -0,0 +1,22 @@ +# post_cts_timing_repair.tcl +# +# POST_CTS hook: identify instances on setup-critical paths and upsize them +# to the next drive strength available in the loaded libraries. +# +# After all swaps the placement is re-legalised (cell widths change) and +# parasitics are re-estimated so downstream timing reflects the new sizes. +# +# Shared implementation lives in timing_repair_common.tcl (namespace +# ::trepair) — this file just supplies the post-CTS log prefix and +# parasitics mode. +# +# Usage — add to a design config or Makefile: +# export POST_CTS_TCL = $(SCRIPTS_DIR)/post_cts_timing_repair.tcl +# +# Or source manually inside an OpenROAD session: +# source flow/scripts/post_cts_timing_repair.tcl + +source [file join [file dirname [info script]] timing_repair_common.tcl] + +# Run automatically when sourced as a POST_CTS hook +trepair::run pctr -placement diff --git a/flow/scripts/post_grt_timing_repair.tcl b/flow/scripts/post_grt_timing_repair.tcl new file mode 100644 index 0000000000..ee6cc4d86f --- /dev/null +++ b/flow/scripts/post_grt_timing_repair.tcl @@ -0,0 +1,25 @@ +# post_grt_timing_repair.tcl +# +# POST_GLOBAL_ROUTE hook: identify instances on setup-critical paths and +# upsize them to the next drive strength available in the loaded libraries. +# +# Complements post_cts_timing_repair.tcl. At the post-CTS stage, parasitics +# are estimated from placement; violations that only appear under real wire +# geometry (like aes) are not yet visible. By the time global routing has run, +# actual route topology is known, so this hook catches those late-appearing +# violations before detail routing locks in the geometry. +# +# Shared implementation lives in timing_repair_common.tcl (namespace +# ::trepair) — this file just supplies the post-GRT log prefix and +# parasitics mode (-global_routing instead of -placement). +# +# Usage — add to a design config or Makefile: +# export POST_GLOBAL_ROUTE_TCL = $(SCRIPTS_DIR)/post_grt_timing_repair.tcl +# +# Or source manually inside an OpenROAD session after global_route has run: +# source flow/scripts/post_grt_timing_repair.tcl + +source [file join [file dirname [info script]] timing_repair_common.tcl] + +# Run automatically when sourced as a POST_GLOBAL_ROUTE hook +trepair::run pgtr -global_routing diff --git a/flow/scripts/timing_repair_common.tcl b/flow/scripts/timing_repair_common.tcl new file mode 100644 index 0000000000..b61565ae7c --- /dev/null +++ b/flow/scripts/timing_repair_common.tcl @@ -0,0 +1,248 @@ +# timing_repair_common.tcl +# +# Shared implementation for the post-CTS and post-global-route timing repair +# hooks (post_cts_timing_repair.tcl, post_grt_timing_repair.tcl). Identifies +# instances on setup-critical paths and upsizes them to the next drive +# strength available in the loaded libraries. +# +# Only combinational cells following the TYPE_X naming convention are +# swapped. Flip-flops, clock cells, and cells already at maximum drive are +# left untouched. +# +# Each caller sources this file and then invokes: +# trepair::run ?path_count? ?max_swaps? ?max_iters? +# +# where is the flag passed to estimate_parasitics +# (e.g. -placement or -global_routing). + +namespace eval trepair { +# ----------------------------------------------------------------------- +# Build a map: current_cell_name -> next_drive_cell_name +# Discovered dynamically from whatever libraries are loaded, so this +# works for any PDK that follows the _X convention. +# ----------------------------------------------------------------------- +proc build_upsize_map { } { + array set by_base {} + set db [::ord::get_db] + + foreach lib [$db getLibs] { + foreach master [$lib getMasters] { + set name [$master getName] + if { [regexp {^(.+_X)(\d+)$} $name -> base drive] } { + lappend by_base($base) [list [expr { int($drive) }] $name] + } + } + } + + array set upsize {} + foreach base [array names by_base] { + # Sort by drive strength numerically, build consecutive pairs + set sorted [lsort -integer -index 0 $by_base($base)] + for { set i 0 } { $i < [llength $sorted] - 1 } { incr i } { + set curr [lindex [lindex $sorted $i] 1] + set next [lindex [lindex $sorted [expr { $i + 1 }]] 1] + set upsize($curr) $next + } + } + + return [array get upsize] +} + +# ----------------------------------------------------------------------- +# Search all loaded libs for a master by name. +# ----------------------------------------------------------------------- +proc find_master { name } { + set db [::ord::get_db] + foreach lib [$db getLibs] { + set m [$lib findMaster $name] + if { $m ne "NULL" && $m ne "" } { return $m } + } + return "" +} + +# ----------------------------------------------------------------------- +# Cell types excluded from upsizing. +# Flip-flops: changing drive alters hold/setup arcs non-trivially. +# Clock cells: CTS balanced the tree for a specific drive; don't disturb. +# ----------------------------------------------------------------------- +proc is_excluded { cell_name } { + foreach prefix {DFF SDFF DFFR DFFS DFFRS SDFFR SDFFS SDFFRS DLL DLH CLKBUF CLKGATE CLKGATETST} { + if { [string match "${prefix}*" $cell_name] } { return 1 } + } + return 0 +} + +# ----------------------------------------------------------------------- +# Collect upsize candidates from the N worst setup paths. +# Returns a list of {inst_name curr_cell next_cell}, deduped. +# Already-seen instances (array passed by name) are skipped. +# ----------------------------------------------------------------------- +proc collect_candidates { upsize_arr path_count seen_arr } { + upvar $upsize_arr upsize + upvar $seen_arr seen + + set candidates {} + set db [::ord::get_db] + set block [[$db getChip] getBlock] + + set all_ends [find_timing_paths -path_delay max -sort_by_slack] + set path_ends [lrange $all_ends 0 [expr { $path_count - 1 }]] + + foreach path_end $path_ends { + set cur "" + catch { set cur [$path_end path] } + for { set depth 0 } { $depth < 200 } { incr depth } { + if { $cur eq "" || $cur eq "NULL" } { break } + + set pin_name "" + catch { set pin_name [get_full_name [$cur pin]] } + + if { $pin_name ne "" } { + set slash [string last "/" $pin_name] + if { $slash > 0 } { + set inst_name [string range $pin_name 0 [expr { $slash - 1 }]] + if { ![info exists seen($inst_name)] } { + set odb_inst [$block findInst $inst_name] + if { $odb_inst ne "NULL" && $odb_inst ne "" } { + set cell_name [[$odb_inst getMaster] getName] + if { ![is_excluded $cell_name] && [info exists upsize($cell_name)] } { + set seen($inst_name) 1 + lappend candidates [list $inst_name $cell_name $upsize($cell_name)] + } + } + } + } + } + + set prev "" + catch { set prev [$cur prevPath] } + if { $prev eq "" || $prev eq "NULL" } { break } + set cur $prev + } + } + return $candidates +} + +# ----------------------------------------------------------------------- +# Apply a list of {inst_name curr next} swaps via ODB. +# Returns {swap_count skip_count}. +# ----------------------------------------------------------------------- +proc apply_swaps { candidates log_prefix } { + set db [::ord::get_db] + set block [[$db getChip] getBlock] + set swap_count 0 + set skip_count 0 + + foreach candidate $candidates { + lassign $candidate inst_name curr next + + set inst [$block findInst $inst_name] + if { $inst eq "NULL" || $inst eq "" } { + puts "WARN \[$log_prefix\] instance not found: $inst_name — skipping." + incr skip_count + continue + } + + set new_master [find_master $next] + if { $new_master eq "" } { + puts "WARN \[$log_prefix\] master not found: $next — skipping." + incr skip_count + continue + } + + $inst swapMaster $new_master + puts "INFO \[$log_prefix\] swapped $inst_name $curr -> $next" + incr swap_count + } + return [list $swap_count $skip_count] +} + +# ----------------------------------------------------------------------- +# Main procedure — iterative upsizing. +# +# Each iteration: +# 1. Find the N worst setup paths and collect upsize candidates. +# 2. Apply swaps (skipping instances already swapped in prior iterations). +# 3. Re-legalise placement and re-estimate parasitics. +# 4. Re-run STA; stop if timing closed or no improvement was made. +# +# log_prefix : short tag used in "INFO [tag] ..." log lines +# parasitics_flag : flag passed to estimate_parasitics (-placement or +# -global_routing) +# path_count : paths to inspect per iteration +# max_swaps : hard cap on total swaps across all iterations +# max_iters : iteration limit (guards against non-converging loops) +# ----------------------------------------------------------------------- +proc run { log_prefix parasitics_flag { path_count 10 } { max_swaps 30 } { max_iters 5 } } { + set wns_before [sta::worst_slack -max] + + if { $wns_before >= 0 } { + puts "INFO \[$log_prefix\] WNS [format %+.3f $wns_before] ns — no setup violations, skipping." + return + } + puts "INFO \[$log_prefix\] WNS [format %+.3f $wns_before] ns — starting iterative cell upsizing." + + array set upsize [build_upsize_map] + puts "INFO \[$log_prefix\] Upsize map: [array size upsize] candidate transitions loaded." + + # 'seen' tracks every instance swapped across all iterations so we never + # upsize the same cell twice (it would already be at the next drive level). + array set seen {} + set total_swaps 0 + set wns_current $wns_before + + for { set iter 1 } { $iter <= $max_iters } { incr iter } { + set remaining [expr { $max_swaps - $total_swaps }] + if { $remaining <= 0 } { + puts "INFO \[$log_prefix\] Iter $iter: swap cap ($max_swaps) reached — stopping." + break + } + + puts "INFO \[$log_prefix\] --- Iteration $iter (WNS [format %+.3f $wns_current] ns) ---" + + set candidates [collect_candidates upsize $path_count seen] + + if { [llength $candidates] == 0 } { + puts "INFO \[$log_prefix\] Iter $iter: no new candidates on critical paths — stopping." + break + } + + # Cap this iteration's swaps to what's left in the budget + if { [llength $candidates] > $remaining } { + set candidates [lrange $candidates 0 [expr { $remaining - 1 }]] + } + + lassign [apply_swaps $candidates $log_prefix] swap_count skip_count + incr total_swaps $swap_count + puts "INFO \[$log_prefix\] Iter $iter: $swap_count cell(s) upsized, $skip_count skipped." + + if { $swap_count == 0 } { + puts "INFO \[$log_prefix\] Iter $iter: nothing applied — stopping." + break + } + + # Re-legalise (widths changed) then update wire models + set result [catch { detailed_placement } msg] + if { $result != 0 } { + puts "WARN \[$log_prefix\] detailed_placement failed: $msg" + } + estimate_parasitics $parasitics_flag + + set wns_new [sta::worst_slack -max] + set delta [format %+.3f [expr { $wns_new - $wns_current }]] + set wns_msg "WNS [format %+.3f $wns_current] -> [format %+.3f $wns_new] ns" + puts "INFO \[$log_prefix\] Iter $iter: $wns_msg (delta $delta ns)" + + set wns_current $wns_new + + if { $wns_current >= 0 } { + puts "INFO \[$log_prefix\] Timing closed after iteration $iter." + break + } + } + + set total_delta [format %+.3f [expr { $wns_current - $wns_before }]] + set done_msg "WNS [format %+.3f $wns_before] -> [format %+.3f $wns_current] ns" + puts "INFO \[$log_prefix\] Done: $total_swaps swap(s), $done_msg (total $total_delta ns)" +} +} ;# namespace trepair diff --git a/flow/util/benchmark_dashboard.py b/flow/util/benchmark_dashboard.py new file mode 100644 index 0000000000..5e8d09616e --- /dev/null +++ b/flow/util/benchmark_dashboard.py @@ -0,0 +1,637 @@ +#!/usr/bin/env python3 +""" +Regression / benchmark dashboard for P&R quality history. + +Builds a history/regression layer on top of `pr_metrics.collect()` — it does +not re-parse ORFS reports or logs itself. Each `record` invocation appends one +JSON line to a per-design/tag history file; `report` reads that history back +and flags regressions against thresholds so it can gate a CI pipeline. + +Usage: + python3 flow/util/benchmark_dashboard.py record --platform nangate45 --design ibex --tag base + python3 flow/util/benchmark_dashboard.py report --platform nangate45 --design ibex --tag base + python3 flow/util/benchmark_dashboard.py report --platform nangate45 --design ibex --tag base \ + --stage "Global route" --last 10 --html out.html +""" + +import argparse +import fcntl +import html +import json +import os +import subprocess +import sys +from datetime import datetime, timezone + +from pr_metrics import collect + +HISTORY_DIRNAME = "benchmark_history" + +DEFAULT_WNS_THRESHOLD_NS = 0.01 +DEFAULT_FMAX_THRESHOLD_PCT = 1.0 +DEFAULT_OVERFLOW_THRESHOLD = 0.001 + + +def history_dir(flow_util_dir): + return os.path.join(flow_util_dir, HISTORY_DIRNAME) + + +def history_path(flow_util_dir, platform, design, tag): + fname = f"{platform}__{design}__{tag}.jsonl" + return os.path.join(history_dir(flow_util_dir), fname) + + +def git_sha(repo_dir): + try: + out = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=repo_dir, + capture_output=True, + text=True, + check=True, + ) + return out.stdout.strip() + except (subprocess.CalledProcessError, OSError, FileNotFoundError): + return None + + +def rows_to_stage_dict(rows): + return {name: metrics for name, metrics in rows} + + +def append_record(path, record): + os.makedirs(os.path.dirname(path), exist_ok=True) + line = json.dumps(record) + "\n" + # flock rather than relying on OS atomic-append: a full multi-stage record + # can exceed PIPE_BUF (4096 bytes), so plain O_APPEND no longer guarantees + # writes from concurrent `record` invocations won't interleave. + with open(path, "a") as f: + fcntl.flock(f.fileno(), fcntl.LOCK_EX) + try: + f.write(line) + f.flush() + os.fsync(f.fileno()) + finally: + fcntl.flock(f.fileno(), fcntl.LOCK_UN) + + +def load_records(path): + """Return (records, dropped_last_line). + + dropped_last_line is True if the most recent physical line in the file + was corrupt/malformed and had to be skipped — callers that compare the + latest record against history must treat that as unsafe to report on, + since the "latest" record would silently become a stale one. + """ + records = [] + dropped_last_line = False + if not os.path.isfile(path): + return records, dropped_last_line + + with open(path) as f: + fcntl.flock(f.fileno(), fcntl.LOCK_SH) + try: + lines = f.readlines() + finally: + fcntl.flock(f.fileno(), fcntl.LOCK_UN) + + last_nonblank_lineno = 0 + for lineno, raw_line in enumerate(lines, start=1): + if raw_line.strip(): + last_nonblank_lineno = lineno + + for lineno, raw_line in enumerate(lines, start=1): + line = raw_line.strip() + if not line: + continue + try: + rec = json.loads(line) + except json.JSONDecodeError as e: + print( + f"WARNING: skipping corrupt history line {lineno} in {path}: {e}", + file=sys.stderr, + ) + if lineno == last_nonblank_lineno: + dropped_last_line = True + continue + + stages = rec.get("stages") if isinstance(rec, dict) else None + valid_shape = isinstance(rec, dict) and ( + "stages" not in rec + or ( + isinstance(stages, dict) + and all(v is None or isinstance(v, dict) for v in stages.values()) + ) + ) + if not valid_shape: + print( + f"WARNING: skipping malformed history line {lineno} in {path}: " + "record is not a valid JSON object with the expected shape", + file=sys.stderr, + ) + if lineno == last_nonblank_lineno: + dropped_last_line = True + continue + + records.append(rec) + return records, dropped_last_line + + +def overflow_of(stage_metrics): + if "gp_overflow" in stage_metrics: + return stage_metrics["gp_overflow"] + return stage_metrics.get("grt_overflow") + + +def compute_delta(prev_metrics, cur_metrics, key): + prev = prev_metrics.get(key) if prev_metrics else None + cur = cur_metrics.get(key) + if not is_number(prev) or not is_number(cur): + return None + return cur - prev + + +def detect_regressions( + prev_metrics, cur_metrics, wns_threshold, fmax_threshold_pct, overflow_threshold +): + regressions = [] + + if prev_metrics and not cur_metrics: + regressions.append( + "REGRESSION: stage produced no metrics — design may have failed " + "to reach this stage" + ) + + wns_delta = compute_delta(prev_metrics, cur_metrics, "wns") + if wns_delta is not None and wns_delta < -wns_threshold: + regressions.append( + f"REGRESSION: WNS worsened by {wns_delta:.4f} ns " + f"(threshold {wns_threshold} ns)" + ) + + prev_fmax = prev_metrics.get("fmax_mhz") if prev_metrics else None + cur_fmax = cur_metrics.get("fmax_mhz") + if is_number(prev_fmax) and is_number(cur_fmax) and prev_fmax > 0: + pct_drop = (prev_fmax - cur_fmax) / prev_fmax * 100.0 + if pct_drop > fmax_threshold_pct: + regressions.append( + f"REGRESSION: Fmax dropped {pct_drop:.2f}% " + f"(threshold {fmax_threshold_pct}%)" + ) + + prev_overflow = overflow_of(prev_metrics) if prev_metrics else None + cur_overflow = overflow_of(cur_metrics) + if prev_overflow is not None and cur_overflow is not None: + overflow_delta = cur_overflow - prev_overflow + if overflow_delta > overflow_threshold: + regressions.append( + f"REGRESSION: routing overflow increased by {overflow_delta:.5f} " + f"(threshold {overflow_threshold})" + ) + + return regressions + + +def best_ever(records, stage, key, better="lower"): + values = [] + for r in records: + v = (r.get("stages", {}).get(stage) or {}).get(key) + if v is not None: + values.append(v) + if not values: + return None + return min(values) if better == "lower" else max(values) + + +def is_number(val): + return isinstance(val, (int, float)) and not isinstance(val, bool) + + +def fmt(val, fmt_str, missing="—"): + if not is_number(val): + return missing + return fmt_str.format(val) + + +def fmt_delta(val, fmt_str, missing="—"): + if not is_number(val): + return missing + return fmt_str.format(val) + + +def build_report_rows( + records, stage, wns_threshold, fmax_threshold_pct, overflow_threshold +): + """Return (table_rows, regressions_against_latest).""" + table_rows = [] + prev_metrics = None + + best_wns = best_ever(records, stage, "wns", "higher") + best_fmax = best_ever(records, stage, "fmax_mhz", "higher") + best_hpwl = best_ever(records, stage, "hpwl", "lower") + + for idx, rec in enumerate(records): + metrics = rec.get("stages", {}).get(stage) or {} + + wns_delta = compute_delta(prev_metrics, metrics, "wns") + tns_delta = compute_delta(prev_metrics, metrics, "tns") + fmax_delta = compute_delta(prev_metrics, metrics, "fmax_mhz") + hpwl_delta = compute_delta(prev_metrics, metrics, "hpwl") + + flags = [] + if ( + best_wns is not None + and metrics.get("wns") is not None + and metrics["wns"] < best_wns + ): + flags.append("worse-than-best-WNS") + if ( + best_fmax is not None + and metrics.get("fmax_mhz") is not None + and metrics["fmax_mhz"] < best_fmax + ): + flags.append("worse-than-best-Fmax") + if ( + best_hpwl is not None + and metrics.get("hpwl") is not None + and metrics["hpwl"] > best_hpwl + ): + flags.append("worse-than-best-HPWL") + + regressions = [] + if idx > 0: + regressions = detect_regressions( + prev_metrics, + metrics, + wns_threshold, + fmax_threshold_pct, + overflow_threshold, + ) + + table_rows.append( + { + "record": rec, + "metrics": metrics, + "wns_delta": wns_delta, + "tns_delta": tns_delta, + "fmax_delta": fmax_delta, + "hpwl_delta": hpwl_delta, + "flags": flags, + "regressions": regressions, + } + ) + prev_metrics = metrics + + latest_regressions = table_rows[-1]["regressions"] if table_rows else [] + return table_rows, latest_regressions + + +def print_report(stage, table_rows, label): + print(f"\nBenchmark history — {label} — stage: {stage}") + print("=" * 110) + header = ( + f"{'Timestamp':<21} {'SHA':<9} {'WNS (ns)':>10} {'dWNS':>8} " + f"{'Fmax(MHz)':>10} {'dFmax':>8} {'HPWL':>12} {'dHPWL':>10} {'Flags':<24}" + ) + print(header) + print("-" * 110) + + for row in table_rows: + rec = row["record"] + m = row["metrics"] + ts = (rec.get("timestamp") or "—")[:19] + sha = (rec.get("git_sha") or "—")[:8] + wns = fmt(m.get("wns"), "{:+.3f}") + dwns = fmt_delta(row["wns_delta"], "{:+.3f}") + fmax = fmt(m.get("fmax_mhz"), "{:.1f}") + dfmax = fmt_delta(row["fmax_delta"], "{:+.1f}") + hpwl = fmt(m.get("hpwl"), "{:,.0f}") + dhpwl = fmt_delta(row["hpwl_delta"], "{:+,.0f}") + flags = ",".join(row["flags"]) if row["flags"] else "" + + print( + f"{ts:<21} {sha:<9} {wns:>10} {dwns:>8} {fmax:>10} {dfmax:>8} " + f"{hpwl:>12} {dhpwl:>10} {flags:<24}" + ) + + print("-" * 110) + + latest_regressions = table_rows[-1]["regressions"] if table_rows else [] + if latest_regressions: + print() + for r in latest_regressions: + print(r) + elif len(table_rows) >= 2: + print("\nNo regressions detected against previous record.") + else: + print("\nOnly one record present — nothing to compare against yet.") + print() + + +def render_html(records, stage, table_rows, label, out_path): + width, height = 760, 260 + pad_left, pad_right, pad_top, pad_bottom = 60, 20, 20, 30 + + def series(key): + return [ + (i, row["metrics"].get(key)) + for i, row in enumerate(table_rows) + if is_number(row["metrics"].get(key)) + ] + + def svg_for(key, title, color): + pts = series(key) + if len(pts) < 2: + return f"

Not enough data to chart {title}.

" + xs = [p[0] for p in pts] + ys = [p[1] for p in pts] + x_min, x_max = min(xs), max(xs) + y_min, y_max = min(ys), max(ys) + if y_min == y_max: + y_min -= 1 + y_max += 1 + x_span = max(x_max - x_min, 1) + y_span = y_max - y_min + + def sx(x): + return pad_left + (x - x_min) / x_span * (width - pad_left - pad_right) + + def sy(y): + return ( + height + - pad_bottom + - (y - y_min) / y_span * (height - pad_top - pad_bottom) + ) + + poly = " ".join(f"{sx(x):.1f},{sy(y):.1f}" for x, y in pts) + circles = "".join( + f'' + for x, y in pts + ) + return f""" +
+

{html.escape(title)}

+ + + + + {circles} + max {y_max:.4g} + min {y_min:.4g} + +
+ """ + + charts = ( + svg_for("wns", "WNS (ns)", "#c0392b") + + svg_for("fmax_mhz", "Fmax (MHz)", "#2471a3") + + svg_for("hpwl", "HPWL", "#27ae60") + ) + + rows_html = "" + for row in table_rows: + rec = row["record"] + m = row["metrics"] + flags = ", ".join(row["flags"]) if row["flags"] else "" + regressions = "
".join(html.escape(r) for r in row["regressions"]) + rows_html += ( + "" + f"" + f"" + f"" + f"" + f"" + f"" + f"" + "\n" + ) + + safe_label = html.escape(label) + safe_stage = html.escape(stage) + + doc = f""" + + + +Benchmark dashboard — {safe_label} + + + +

Benchmark dashboard — {safe_label} — stage: {safe_stage}

+{charts} +
{html.escape(str(rec.get('timestamp', ''))[:19])}{html.escape(str(rec.get('git_sha') or '')[:8])}{fmt(m.get('wns'), '{:+.3f}')}{fmt(m.get('fmax_mhz'), '{:.1f}')}{fmt(m.get('hpwl'), '{:,.0f}')}{html.escape(flags)}{regressions}
+ + +{rows_html} + +
TimestampSHAWNSFmaxHPWLFlagsRegressions
+ + +""" + with open(out_path, "w") as f: + f.write(doc) + + +def cmd_record(args, flow_dir, flow_util_dir, reports_dir, logs_dir, label): + if not os.path.isdir(reports_dir): + print(f"ERROR: reports directory not found: {reports_dir}", file=sys.stderr) + sys.exit(1) + + try: + rows = collect(reports_dir, logs_dir) + except Exception as e: + print( + f"ERROR: failed to parse reports in {reports_dir}: {e}", + file=sys.stderr, + ) + sys.exit(1) + + repo_dir = os.path.dirname(flow_dir) + record = { + "timestamp": datetime.now(timezone.utc).isoformat(), + "git_sha": git_sha(repo_dir), + "platform": args.platform, + "design": args.design, + "tag": args.tag, + "stages": rows_to_stage_dict(rows), + } + + path = history_path(flow_util_dir, args.platform, args.design, args.tag) + append_record(path, record) + print(f"Recorded benchmark for {label} -> {path}") + + +def cmd_report(args, flow_dir, flow_util_dir, reports_dir, logs_dir, label): + path = history_path(flow_util_dir, args.platform, args.design, args.tag) + records, dropped_last_line = load_records(path) + + if dropped_last_line: + print( + f"ERROR: history file {path} has a corrupt/truncated record and " + "cannot be safely compared", + file=sys.stderr, + ) + sys.exit(1) + + if not records: + print(f"No history found at {path}") + sys.exit(0) + + stage = args.stage + # best-ever and deltas are computed over the FULL history so that --last + # only narrows what's displayed, never what "worse than all-time best" + # or the regression check against the immediately-previous record means. + table_rows, latest_regressions = build_report_rows( + records, + stage, + args.wns_threshold, + args.fmax_threshold_pct, + args.overflow_threshold, + ) + + display_rows = table_rows[-args.last :] if args.last else table_rows + + print_report(stage, display_rows, label) + + if args.html: + render_html(records, stage, display_rows, label, args.html) + print(f"Wrote HTML dashboard: {args.html}") + + sys.exit(1 if latest_regressions else 0) + + +def add_common_args(parser, flow_dir_default): + parser.add_argument("--platform", help="Platform name (e.g. nangate45)") + parser.add_argument( + "--reports-dir", + help="Direct path to reports directory. May be combined with " + "--platform/--design/--tag to override path-derived values and " + "skip path-shape validation.", + ) + + parser.add_argument("--design", help="Design name (required with --platform)") + parser.add_argument("--tag", help="Tag / variant (default: base)", default=None) + parser.add_argument("--logs-dir", help="Direct path to logs directory") + parser.add_argument( + "--flow-dir", + default=flow_dir_default, + help=f"Path to flow/ directory (default: {flow_dir_default})", + ) + + +def resolve_dirs(args): + if not args.platform and not args.reports_dir: + raise SystemExit("error: one of --platform or --reports-dir is required") + + if args.reports_dir: + reports_dir = args.reports_dir + logs_dir = args.logs_dir or reports_dir.replace("/reports/", "/logs/") + label = reports_dir + if not args.platform or not args.design or not args.tag: + parts = os.path.normpath(reports_dir).split(os.sep) + dir_kind = "reports" if "reports" in parts else "logs" + if dir_kind in parts: + anchor = len(parts) - 1 - parts[::-1].index(dir_kind) + remainder = parts[anchor + 1 :] + if len(remainder) != 3: + raise SystemExit( + "error: --reports-dir " + f"{reports_dir!r} does not look like " + f".../{dir_kind}/// (expected " + "exactly platform/design/tag after the " + f"'{dir_kind}' directory); pass --platform, --design, " + "and --tag explicitly alongside --reports-dir to " + "override path derivation" + ) + args.platform = args.platform or remainder[0] + args.design = args.design or remainder[1] + args.tag = args.tag or remainder[2] + elif len(parts) >= 3: + args.platform = args.platform or parts[-3] + args.design = args.design or parts[-2] + args.tag = args.tag or parts[-1] + + if not args.platform or not args.design: + raise SystemExit( + "error: could not determine --platform/--design from " + f"--reports-dir {reports_dir!r} (need at least " + "// path components); pass " + "--platform and --design explicitly alongside --reports-dir" + ) + args.tag = args.tag or "base" + else: + if not args.design: + raise SystemExit("--design is required when using --platform") + args.tag = args.tag or "base" + reports_dir = os.path.join( + args.flow_dir, "reports", args.platform, args.design, args.tag + ) + logs_dir = os.path.join( + args.flow_dir, "logs", args.platform, args.design, args.tag + ) + label = f"{args.platform}/{args.design}/{args.tag}" + return reports_dir, logs_dir, label + + +def main(): + script_dir = os.path.dirname(os.path.abspath(__file__)) + flow_dir_default = os.path.dirname(script_dir) + + parser = argparse.ArgumentParser(description="Regression / benchmark dashboard") + sub = parser.add_subparsers(dest="command", required=True) + + p_record = sub.add_parser("record", help="Record one run's metrics into history") + add_common_args(p_record, flow_dir_default) + + p_report = sub.add_parser( + "report", help="Print history report and detect regressions" + ) + add_common_args(p_report, flow_dir_default) + p_report.add_argument( + "--stage", default="Finish", help="Stage name to report on (default: Finish)" + ) + p_report.add_argument( + "--last", type=int, default=None, help="Limit to N most recent records" + ) + p_report.add_argument( + "--html", help="Write a self-contained HTML dashboard to this path" + ) + p_report.add_argument( + "--wns-threshold", + type=float, + default=DEFAULT_WNS_THRESHOLD_NS, + help=f"WNS regression threshold in ns (default: {DEFAULT_WNS_THRESHOLD_NS})", + ) + p_report.add_argument( + "--fmax-threshold-pct", + type=float, + default=DEFAULT_FMAX_THRESHOLD_PCT, + help=f"Fmax regression threshold in %% (default: {DEFAULT_FMAX_THRESHOLD_PCT})", + ) + p_report.add_argument( + "--overflow-threshold", + type=float, + default=DEFAULT_OVERFLOW_THRESHOLD, + help=f"Routing overflow regression threshold (default: {DEFAULT_OVERFLOW_THRESHOLD})", + ) + + args = parser.parse_args() + flow_dir = args.flow_dir + flow_util_dir = os.path.dirname(os.path.abspath(__file__)) + reports_dir, logs_dir, label = resolve_dirs(args) + + if args.command == "record": + cmd_record(args, flow_dir, flow_util_dir, reports_dir, logs_dir, label) + elif args.command == "report": + cmd_report(args, flow_dir, flow_util_dir, reports_dir, logs_dir, label) + + +if __name__ == "__main__": + main() diff --git a/flow/util/compare_hook.sh b/flow/util/compare_hook.sh new file mode 100755 index 0000000000..6353b2918b --- /dev/null +++ b/flow/util/compare_hook.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# compare_hook.sh +# +# Runs a design twice from the same placement checkpoint: +# 1. Baseline: CTS through finish, no hook +# 2. Hook: CTS with POST_CTS_TCL, then finish +# Then prints both pr_metrics.py tables side by side for comparison. +# +# Usage (from flow/): +# util/compare_hook.sh --platform nangate45 --design ibex +# util/compare_hook.sh --platform nangate45 --design aes --tag base +# +# All make targets run inside the Docker container via util/docker_shell. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- +PLATFORM=nangate45 +DESIGN=ibex +TAG=base +CTS_HOOK=/work/scripts/post_cts_timing_repair.tcl +GRT_HOOK=/work/scripts/post_grt_timing_repair.tcl + +usage() { + cat < --design [options] + +Options: + --tag Flow tag (default: base) + --cts-hook Container path to POST_CTS hook (default: $CTS_HOOK) + --grt-hook Container path to POST_GRT hook (default: $GRT_HOOK) + --no-cts-hook Disable the POST_CTS hook + --no-grt-hook Disable the POST_GRT hook +EOF + exit 1 +} + +USE_CTS_HOOK=1 +USE_GRT_HOOK=1 + +# --------------------------------------------------------------------------- +# Parse arguments +# --------------------------------------------------------------------------- +while [[ $# -gt 0 ]]; do + case "$1" in + --platform) PLATFORM="$2"; shift 2 ;; + --design) DESIGN="$2"; shift 2 ;; + --tag) TAG="$2"; shift 2 ;; + --cts-hook) CTS_HOOK="$2"; shift 2 ;; + --grt-hook) GRT_HOOK="$2"; shift 2 ;; + --no-cts-hook) USE_CTS_HOOK=0; shift ;; + --no-grt-hook) USE_GRT_HOOK=0; shift ;; + -h|--help) usage ;; + *) echo "Unknown argument: $1"; usage ;; + esac +done + +if [[ -z "$PLATFORM" || -z "$DESIGN" ]]; then + echo "ERROR: --platform and --design are required." + usage +fi + +DESIGN_CONFIG=designs/$PLATFORM/$DESIGN/config.mk +RESULTS=results/$PLATFORM/$DESIGN/$TAG +REPORTS=reports/$PLATFORM/$DESIGN/$TAG +LOGS=logs/$PLATFORM/$DESIGN/$TAG +BASELINE_DIR=/tmp/${PLATFORM}_${DESIGN}_${TAG}_hook_baseline + +clean_downstream() { + rm -f "$RESULTS"/4_* "$RESULTS"/5_* "$RESULTS"/6_* +} + +echo "======================================================" +echo " compare_hook.sh — $PLATFORM/$DESIGN/$TAG before/after" +echo "======================================================" + +# --- Baseline run --- +echo "" +echo "[1/4] Cleaning CTS and downstream..." +clean_downstream + +echo "[2/4] Running baseline (no hook) through finish..." +util/docker_shell make finish DESIGN_CONFIG="$DESIGN_CONFIG" + +echo " Saving baseline reports to $BASELINE_DIR" +rm -rf "$BASELINE_DIR" +cp -r "$REPORTS" "$BASELINE_DIR" + +# --- Hook run --- +echo "" +echo "[3/4] Cleaning CTS and downstream..." +clean_downstream + +echo " Running CTS$([ "$USE_CTS_HOOK" = 1 ] && echo " with POST_CTS hook" || echo " (no CTS hook)")..." +if [[ "$USE_CTS_HOOK" = 1 ]]; then + util/docker_shell make cts \ + DESIGN_CONFIG="$DESIGN_CONFIG" \ + POST_CTS_TCL="$CTS_HOOK" +else + util/docker_shell make cts DESIGN_CONFIG="$DESIGN_CONFIG" +fi + +echo " Running route and finish$([ "$USE_GRT_HOOK" = 1 ] && echo " with POST_GRT hook" || echo "")..." +if [[ "$USE_GRT_HOOK" = 1 ]]; then + util/docker_shell make finish \ + DESIGN_CONFIG="$DESIGN_CONFIG" \ + POST_GLOBAL_ROUTE_TCL="$GRT_HOOK" +else + util/docker_shell make finish DESIGN_CONFIG="$DESIGN_CONFIG" +fi + +# --- Compare --- +echo "" +echo "[4/4] Results" +echo "" +echo "--- BASELINE (no hook) ---" +python3 "$SCRIPT_DIR/pr_metrics.py" \ + --reports-dir "$BASELINE_DIR" \ + --logs-dir "$LOGS" + +echo "--- WITH HOOK ---" +python3 "$SCRIPT_DIR/pr_metrics.py" \ + --platform "$PLATFORM" --design "$DESIGN" --tag "$TAG" diff --git a/flow/util/loop_agent.py b/flow/util/loop_agent.py new file mode 100644 index 0000000000..2079264bc6 --- /dev/null +++ b/flow/util/loop_agent.py @@ -0,0 +1,587 @@ +#!/usr/bin/env python3 +""" +P&R Closed-Loop Optimization Agent + +Observes stage-by-stage metrics, diagnoses timing failures, applies targeted +ORFS parameter changes, re-runs the affected flow stages, and verifies +improvement — without human intervention. + +Builds on pr_metrics.py and the triage-agent system prompt. Tools give the +model direct access to read metrics, queue parameter changes, and trigger +Docker-based make runs. + +Usage: + python3 flow/util/loop_agent.py --platform nangate45 --design aes --tag base + +Requirements: + pip install anthropic + export ANTHROPIC_API_KEY= # or `ant auth login` + Docker available with openroad/orfs:latest image +""" + +import argparse +import json +import os +import subprocess +import sys + +try: + import anthropic +except ImportError: + print( + "ERROR: anthropic package not installed. Run: pip install anthropic", + file=sys.stderr, + ) + sys.exit(1) + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from pr_metrics import collect # noqa: E402 + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +MAX_TOOL_TURNS = 20 # hard cap on total API round-trips + +# Parameters the agent is allowed to change +PARAM_ALLOWLIST = { + "SETUP_SLACK_MARGIN", + "TNS_END_PERCENT", + "OPT_POST_GRT_WNS", + "PLACE_DENSITY_LB_ADDON", + "POST_CTS_TCL", + "POST_GLOBAL_ROUTE_TCL", +} + +# Hook scripts available inside the Docker container (workspace = /work) +HOOK_PATHS = { + "POST_CTS_TCL": "/work/scripts/post_cts_timing_repair.tcl", + "POST_GLOBAL_ROUTE_TCL": "/work/scripts/post_grt_timing_repair.tcl", +} + +# Canonical paths for config.mk write-back (ORFS make-variable style) +CONFIG_HOOK_PATHS = { + "POST_CTS_TCL": "$(SCRIPTS_DIR)/post_cts_timing_repair.tcl", + "POST_GLOBAL_ROUTE_TCL": "$(SCRIPTS_DIR)/post_grt_timing_repair.tcl", +} + +# Characters/sequences that would let a value escape a plain scalar and +# inject Make or shell syntax when written into config.mk or passed as a +# KEY=value argv token to `make`. +UNSAFE_VALUE_PATTERNS = ("$(", "${", "`", ";", "|", "&", "\n", "\r") + +# Stale ODB files to delete when forcing a stage re-run +STAGE_STALE_FILES = { + "place": [ + "results/{p}/{d}/{t}/3_3_place_gp.odb", + "results/{p}/{d}/{t}/3_4_place_resized.odb", + "results/{p}/{d}/{t}/3_5_place_dp.odb", + "results/{p}/{d}/{t}/3_place.odb", + "results/{p}/{d}/{t}/3_place.sdc", + ], + "cts": ["results/{p}/{d}/{t}/4_1_cts.odb", "results/{p}/{d}/{t}/4_cts.odb"], + "grt": ["results/{p}/{d}/{t}/5_1_grt.odb", "results/{p}/{d}/{t}/5_1_grt.sdc"], + "finish": [ + "results/{p}/{d}/{t}/5_2_route.odb", + "results/{p}/{d}/{t}/5_route.odb", + ], +} + +# --------------------------------------------------------------------------- +# System prompt +# --------------------------------------------------------------------------- + +SYSTEM_PROMPT = """\ +You are a closed-loop P&R optimization agent for OpenROAD-flow-scripts (ORFS). \ +You have tools to read metrics, queue parameter changes, re-run flow stages, \ +and declare completion. + +## Workflow + +1. Call get_metrics to read the current stage-by-stage quality trajectory. +2. Diagnose the root cause of any timing failures. +3. Call set_config_param for each targeted fix. +4. Call run_stage for each affected stage, starting from the earliest changed \ +stage (cts, then grt if needed, then finish). +5. Call get_metrics again to verify improvement. +6. Repeat up to 3 iterations. When WNS ≥ 0 ns and TNS ≥ -0.05 ns at finish, \ +or when your budget is exhausted, call finish. + +## Flow context + +- Stage order: global place → resizer → detail place → CTS → global route \ +→ finish (detail route + sign-off). +- At CTS, parasitics are estimated from placement (optimistic, underestimates \ +real wire RC). Real RC is only known after global route. +- A CTS→GRT cliff (CTS shows WNS +0.000, GRT shows WNS < -0.010) is \ +parasitic underestimation. Fix: set SETUP_SLACK_MARGIN = 0.03. +- ORFS runs repair_timing at global route before writing the GRT report, so \ +violations visible in the GRT row survived built-in repair. +- The post-CTS hook does iterative cell upsizing using placement parasitics. \ +Set POST_CTS_TCL = "enabled" when violations will be visible at CTS after \ +applying SETUP_SLACK_MARGIN. + +## Failure patterns and fixes + +**CTS→GRT timing cliff** (CTS WNS ≈ 0, GRT WNS < -0.010): +Parasitic underestimation at CTS. Fix: SETUP_SLACK_MARGIN=0.03, \ +TNS_END_PERCENT=100, POST_CTS_TCL=enabled. Re-run: cts, finish. + +**Routing congestion** (GRT overflow > 0.40 at global place row, or overflow \ +persisting across runs): +Placement too dense. Fix: increase PLACE_DENSITY_LB_ADDON by 0.05 (max 0.50). \ +Re-run: place, cts, finish. This is expensive — only apply if overflow > 0.40. + +**Residual GRT violations after timing fix** (GRT WNS still < -0.005 after \ +cts re-run): +Add OPT_POST_GRT_WNS=1 for a VT-swap pass. Re-run: grt, finish. + +## Allowlisted parameters + +- SETUP_SLACK_MARGIN (float, 0.0–0.10 ns): extra setup margin during repair. \ +Typical: 0.03. +- TNS_END_PERCENT (int, 0–100): % of violating endpoints to repair. Set to 100 \ +whenever TNS > 0. +- OPT_POST_GRT_WNS (0 or 1): VT-swap repair pass after GRT. +- PLACE_DENSITY_LB_ADDON (float, 0.0–0.50): extra placement density margin. \ +Only increase if GRT overflow > 0.40. Each 0.05 step increases HPWL ~2–4%. +- POST_CTS_TCL: set to "enabled" to activate the post-CTS upsizing hook. +- POST_GLOBAL_ROUTE_TCL: set to "enabled" to activate the post-GRT hook. + +## Stage re-run rules (least expensive first) + +- Changed SETUP_SLACK_MARGIN, TNS_END_PERCENT, OPT_POST_GRT_WNS, \ +POST_CTS_TCL, or POST_GLOBAL_ROUTE_TCL → run cts, then finish. +- Changed PLACE_DENSITY_LB_ADDON → run place, then cts, then finish \ +(expensive — placement re-runs global place + resize + detail place). + +Budget: max 3 iterations. Call finish when done regardless of outcome. +""" + +# --------------------------------------------------------------------------- +# Tool definitions +# --------------------------------------------------------------------------- + +TOOLS = [ + { + "name": "get_metrics", + "description": ( + "Read the current stage-by-stage quality trajectory (WNS, TNS, Fmax, " + "GRT overflow). Call at the start and after each run_stage to see " + "the updated results." + ), + "input_schema": { + "type": "object", + "properties": {}, + "required": [], + }, + }, + { + "name": "set_config_param", + "description": ( + "Queue a parameter change that will be passed to the next run_stage " + "call as a make variable. Only allowlisted parameters are accepted. " + "For POST_CTS_TCL and POST_GLOBAL_ROUTE_TCL, pass value='enabled'." + ), + "input_schema": { + "type": "object", + "properties": { + "param": { + "type": "string", + "description": "ORFS make parameter name (must be allowlisted).", + }, + "value": { + "type": "string", + "description": ( + "Value to set. For hook paths use 'enabled'. " + "Numeric values as strings, e.g. '0.03' or '100'." + ), + }, + }, + "required": ["param", "value"], + }, + }, + { + "name": "run_stage", + "description": ( + "Re-run a flow stage inside the Docker container using all queued " + "parameter changes. Stale output files are deleted first to force " + "re-execution. Returns the tail of the make output." + ), + "input_schema": { + "type": "object", + "properties": { + "stage": { + "type": "string", + "enum": ["place", "cts", "grt", "finish"], + "description": ( + "Flow stage to re-run. 'place' re-runs global place " + "+ resize + detail place (expensive, only when " + "PLACE_DENSITY_LB_ADDON changed)." + ), + }, + }, + "required": ["stage"], + }, + }, + { + "name": "finish", + "description": ( + "Terminate the optimization loop. Call when timing has closed " + "(WNS ≥ 0, TNS ≥ -0.05 at finish) or when the iteration budget " + "is exhausted." + ), + "input_schema": { + "type": "object", + "properties": { + "summary": { + "type": "string", + "description": ( + "What was diagnosed, what parameters were changed, " + "and what the final metrics show." + ), + }, + "success": { + "type": "boolean", + "description": "True if timing closed, False if budget exhausted.", + }, + }, + "required": ["summary", "success"], + }, + }, +] + +# --------------------------------------------------------------------------- +# Tool implementations +# --------------------------------------------------------------------------- + + +def impl_get_metrics(platform, design, tag, flow_dir): + reports_dir = os.path.join(flow_dir, "reports", platform, design, tag) + logs_dir = os.path.join(flow_dir, "logs", platform, design, tag) + if not os.path.isdir(reports_dir): + return f"ERROR: reports directory not found: {reports_dir}" + rows = collect(reports_dir, logs_dir) + header = ( + f"{'Stage':<16} {'WNS (ns)':>10} {'TNS (ns)':>10}" + f" {'Fmax (MHz)':>11} {'GRT overflow':>13}" + ) + lines = [header, "-" * 62] + for name, m in rows: + wns = f"{m['wns']:+.3f}" if "wns" in m else "—" + tns = f"{m['tns']:+.3f}" if "tns" in m else "—" + fmax = f"{m['fmax_mhz']:.1f}" if "fmax_mhz" in m else "—" + overflow = ( + f"{m.get('gp_overflow', m.get('grt_overflow')):.4f}" + if "gp_overflow" in m or "grt_overflow" in m + else "—" + ) + lines.append(f"{name:<16} {wns:>10} {tns:>10} {fmax:>11} {overflow:>13}") + for _, m in reversed(rows): + if "wns" in m: + lines.append(f"\nFinal WNS: {m['wns']:+.3f} ns") + break + for _, m in reversed(rows): + if "tns" in m: + lines.append(f"Final TNS: {m['tns']:+.3f} ns") + break + return "\n".join(lines) + + +def validate_param_value(value): + """Reject values that could inject Make/shell syntax via config.mk or argv. + + Returns an error string if the value is unsafe, or None if it is fine. + """ + if not isinstance(value, str) or not value: + return "value must be a non-empty string" + for pattern in UNSAFE_VALUE_PATTERNS: + if pattern in value: + return f"value contains disallowed sequence '{pattern}'" + return None + + +def impl_set_config_param(param, value, pending_params, change_log): + if param not in PARAM_ALLOWLIST: + return ( + f"ERROR: '{param}' is not allowlisted. " + f"Allowed: {sorted(PARAM_ALLOWLIST)}" + ) + error = validate_param_value(value) + if error: + return f"ERROR: invalid value for '{param}': {error}" + if param in HOOK_PATHS and value.lower() == "enabled": + value = HOOK_PATHS[param] + pending_params[param] = value + change_log.append({"action": "set_param", "param": param, "value": value}) + return f"OK: {param} = {value}" + + +def impl_run_stage(stage, platform, design, tag, pending_params, flow_dir, change_log): + # Delete stale output files to force make to re-run + for pattern in STAGE_STALE_FILES.get(stage, []): + path = os.path.join(flow_dir, pattern.format(p=platform, d=design, t=tag)) + if os.path.exists(path): + os.remove(path) + + cmd = ( + [ + "util/docker_shell", + "make", + f"DESIGN_CONFIG=designs/{platform}/{design}/config.mk", + ] + + [f"{k}={v}" for k, v in pending_params.items()] + + [stage] + ) + change_log.append( + {"action": "run_stage", "stage": stage, "params": dict(pending_params)} + ) + + print(f"\n[loop-agent] $ {' '.join(cmd)}", flush=True) + try: + result = subprocess.run( + cmd, cwd=flow_dir, capture_output=True, text=True, timeout=1800 + ) + output = result.stdout + result.stderr + except subprocess.TimeoutExpired: + return "ERROR: make timed out after 30 minutes" + + tail = output[-3000:] if len(output) > 3000 else output + return tail + + +# --------------------------------------------------------------------------- +# Config write-back +# --------------------------------------------------------------------------- + + +def write_config_params(params, platform, design, flow_dir): + """Write successfully applied params back to the design's config.mk. + + Existing 'export PARAM = ...' lines are updated in-place; new params are + appended with a loop-agent comment. Docker-specific hook paths are + translated back to the portable $(SCRIPTS_DIR)/... form. + """ + import re as _re + + config_path = os.path.join(flow_dir, "designs", platform, design, "config.mk") + if not os.path.exists(config_path): + return f"ERROR: {config_path} not found" + + trusted_values = set(HOOK_PATHS.values()) | set(CONFIG_HOOK_PATHS.values()) + for param, value in params.items(): + if value in trusted_values: + continue + error = validate_param_value(value) + if error: + return f"ERROR: refusing to write '{param}': {error}" + + # Translate Docker hook paths → ORFS-canonical paths for config.mk + writeback = {} + for param, value in params.items(): + if param in CONFIG_HOOK_PATHS and value.startswith("/work/scripts/"): + writeback[param] = CONFIG_HOOK_PATHS[param] + else: + writeback[param] = value + + with open(config_path) as f: + lines = f.readlines() + + updated = set() + new_lines = [] + for line in lines: + replaced = False + for param, value in writeback.items(): + if _re.match(rf"^\s*export\s+{_re.escape(param)}\s*[=]", line): + new_lines.append(f"export {param} = {value}\n") + updated.add(param) + replaced = True + break + if not replaced: + new_lines.append(line) + + # Append params not already present in the file + new_params = {p: v for p, v in writeback.items() if p not in updated} + if new_params: + if new_lines and not new_lines[-1].endswith("\n"): + new_lines.append("\n") + new_lines.append("\n# Written by loop_agent.py\n") + for param, value in new_params.items(): + new_lines.append(f"export {param} = {value}\n") + + with open(config_path, "w") as f: + f.writelines(new_lines) + + return ( + f"Updated: {sorted(updated)} | Added: {sorted(new_params)}" + f" | Path: {config_path}" + ) + + +# --------------------------------------------------------------------------- +# Agent loop +# --------------------------------------------------------------------------- + + +def run_loop(platform, design, tag, flow_dir): + client = anthropic.Anthropic() + pending_params = {} + change_log = [] + label = f"{platform}/{design}/{tag}" + + print(f"\nLoop agent — {label}") + print(f"Max tool turns: {MAX_TOOL_TURNS}") + print("=" * 70) + + initial_metrics = impl_get_metrics(platform, design, tag, flow_dir) + print(initial_metrics) + print("=" * 70) + + messages = [ + { + "role": "user", + "content": ( + f"Design run: {label}\n\n" + f"Current quality trajectory:\n{initial_metrics}\n\n" + "Diagnose any timing issues and close them. " + "You have at most 3 iterations." + ), + } + ] + + turn = 0 + finished = False + + while not finished and turn < MAX_TOOL_TURNS: + turn += 1 + + response = client.messages.create( + model="claude-opus-5", + max_tokens=8000, + thinking={"type": "adaptive"}, + system=SYSTEM_PROMPT, + tools=TOOLS, + messages=messages, + ) + + messages.append({"role": "assistant", "content": response.content}) + + # Print any visible text from the agent + for block in response.content: + if block.type == "text" and block.text.strip(): + print(f"\n[agent] {block.text}") + + if response.stop_reason == "end_turn": + break + + if response.stop_reason != "tool_use": + print(f"[loop-agent] Unexpected stop_reason: {response.stop_reason}") + break + + tool_results = [] + for block in response.content: + if block.type != "tool_use": + continue + + inp = block.input + print(f"\n[tool:{block.name}] {json.dumps(inp)}", flush=True) + + if block.name == "get_metrics": + result = impl_get_metrics(platform, design, tag, flow_dir) + + elif block.name == "set_config_param": + result = impl_set_config_param( + inp["param"], inp["value"], pending_params, change_log + ) + + elif block.name == "run_stage": + result = impl_run_stage( + inp["stage"], + platform, + design, + tag, + pending_params, + flow_dir, + change_log, + ) + + elif block.name == "finish": + status = "SUCCESS" if inp.get("success") else "BUDGET EXHAUSTED" + print(f"\n{'='*70}") + print(f"[loop-agent] {status}") + print(inp.get("summary", "")) + print(f"{'='*70}") + change_log.append({"action": "finish", **inp}) + finished = True + # Persist successful parameter changes back to config.mk + if inp.get("success") and pending_params: + wb = write_config_params(pending_params, platform, design, flow_dir) + print(f"[loop-agent] Write-back → {wb}") + result = "Loop terminated." + + else: + result = f"ERROR: unknown tool '{block.name}'" + + short = result[:300] + ("..." if len(result) > 300 else "") + print(f"[result] {short}") + + tool_results.append( + { + "type": "tool_result", + "tool_use_id": block.id, + "content": result, + } + ) + + if tool_results: + messages.append({"role": "user", "content": tool_results}) + + # Persist change log + log_dir = os.path.join(flow_dir, "logs", platform, design, tag) + os.makedirs(log_dir, exist_ok=True) + log_path = os.path.join(log_dir, "loop_agent_changes.json") + with open(log_path, "w") as f: + json.dump(change_log, f, indent=2) + print(f"\n[loop-agent] Change log → {log_path}") + + return change_log + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +def main(): + parser = argparse.ArgumentParser(description="P&R closed-loop optimization agent") + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument("--platform", help="Platform name (e.g. nangate45)") + group.add_argument("--reports-dir", help="Direct path to reports directory") + + parser.add_argument("--design", help="Design name (required with --platform)") + parser.add_argument("--tag", default="base", help="Tag / variant (default: base)") + + script_dir = os.path.dirname(os.path.abspath(__file__)) + flow_dir = os.path.dirname(script_dir) + parser.add_argument( + "--flow-dir", + default=flow_dir, + help=f"Path to flow/ directory (default: {flow_dir})", + ) + + args = parser.parse_args() + + if args.platform: + if not args.design: + parser.error("--design is required when using --platform") + platform, design, tag = args.platform, args.design, args.tag + else: + parts = args.reports_dir.rstrip("/").split("/") + tag, design, platform = parts[-1], parts[-2], parts[-3] + flow_dir = args.flow_dir + + run_loop(platform, design, tag, flow_dir) + + +if __name__ == "__main__": + main() diff --git a/flow/util/ml/Dockerfile.ml b/flow/util/ml/Dockerfile.ml new file mode 100644 index 0000000000..2380055ad6 --- /dev/null +++ b/flow/util/ml/Dockerfile.ml @@ -0,0 +1,49 @@ +# Custom ORFS image with ML dependencies (HotSpot thermal solver + Python packages). +# +# Build: +# docker build -t openroad/orfs-ml:latest -f flow/util/ml/Dockerfile.ml flow/util/ml/ +# +# Use (instead of plain docker_shell): +# OR_IMAGE=openroad/orfs-ml:latest util/docker_shell +# +# The base image already contains OpenROAD, Yosys, KLayout, and all ORFS tooling. +# This layer adds: +# - HotSpot v7.0 (compact thermal solver, RC circuit model) +# - Python packages needed by the ML pipeline + +FROM openroad/orfs:latest + +USER root + +# ── System dependencies ──────────────────────────────────────────────────── +# git/make/gcc: build HotSpot from source (not packaged in any distro) +# libblas-dev: HotSpot links against BLAS for matrix ops in its thermal solver +RUN apt-get update && apt-get install -y --no-install-recommends \ + git \ + gcc \ + make \ + libblas-dev \ + && rm -rf /var/lib/apt/lists/* + +# ── HotSpot v7.0 ────────────────────────────────────────────────────────── +# Clone, build, and install to /usr/local/bin so it's on PATH everywhere. +# --depth 1 fetches only the latest commit — faster, no history needed. +# We remove the source tree after installing to keep the image small. +RUN git clone --depth 1 https://github.com/uvahotspot/HotSpot.git /tmp/hotspot \ + && cd /tmp/hotspot \ + && make \ + && cp hotspot /usr/local/bin/hotspot \ + && rm -rf /tmp/hotspot + +# ── Python ML packages ───────────────────────────────────────────────────── +# These are needed by training/inference scripts inside the container. +# torch-geometric and its deps (torch-scatter etc.) are installed separately +# because they require matching the PyTorch version already in the base image. +RUN pip3 install --no-cache-dir \ + numpy \ + scipy \ + scikit-learn \ + xgboost \ + torch \ + torch-geometric + diff --git a/flow/util/ml/congestion/DESIGN_RUNS.md b/flow/util/ml/congestion/DESIGN_RUNS.md new file mode 100644 index 0000000000..cd914f2106 --- /dev/null +++ b/flow/util/ml/congestion/DESIGN_RUNS.md @@ -0,0 +1,627 @@ +# ML Pipeline — Development Log + +This file is the **canonical development log** for the ML work on the `congestion-ml` branch. +Every significant change, decision, and planned next step is recorded here so the project can +be resumed from a cold start without losing context. Update it after every working session. + +--- + +## Project Overview + +**Goal:** Add ML-based prediction capabilities that OpenROAD currently lacks. + +**Primary track: Thermal prediction** (branch focus as of 2026-08-10) + +| Track | Input | Model | Labels | Status | +|---|---|---|---|---| +| **Thermal prediction** | Post-placement ODB | U-Net (`models/unet.py`) | HotSpot thermal maps | Extractor + pipeline wired; awaiting data | +| Pre-placement congestion | Post-synthesis netlist graph | GNN (`models/gnn.py`) | GRT congestion maps | Deprioritised — code kept, not actively developed | + +**Why thermal is the focus:** +- OpenROAD has no thermal solver at all. HotSpot runs take minutes; a trained U-Net + surrogate runs in milliseconds and can be embedded directly into the ORFS flow. +- Pre-placement congestion prediction (GNN) is a useful capability but not urgent — + it remains in the codebase and the pipeline still extracts netlist graphs for free, + but it is not the primary development target on this branch. + +--- + +## Codebase Map + +``` +flow/util/ml/ +├── Dockerfile # Custom image: ORFS + HotSpot + Python ML packages +├── congestion/ +│ ├── DESIGN_RUNS.md # This file — canonical dev log +│ ├── data/ # Extracted .npz datasets (features + labels) +│ ├── data_collection/ +│ │ ├── extract_features.py # Post-placement features from ODB (4 channels, 64x64) +│ │ ├── extract_labels.py # Congestion labels from GRT ODB +│ │ ├── extract_thermal_labels.py # Thermal labels via HotSpot +│ │ ├── extract_netlist_features.py # Pre-placement netlist graph (Track 1 GNN input) +│ │ ├── extract_existing.sh # Batch extract from pre-existing ORFS result dirs +│ │ └── batch_run.sh # Helper for manual batch runs +│ ├── models/ +│ │ ├── unet.py # U-Net: spatial features → congestion/thermal heatmap +│ │ ├── gnn.py # GNN: netlist graph → congestion heatmap +│ │ └── heads.py # Shared output heads (heatmap, hotspot, score) +│ ├── training/ +│ │ ├── dataset.py # CongestionDataset loader + split_dataset +│ │ ├── metrics.py # heatmap_mae, hotspot_iou, score_pearson, compute_all +│ │ ├── train_unet.py # U-Net training script +│ │ └── train_gnn.py # GNN training script +│ ├── inference/ +│ │ ├── evaluate.py # Evaluate all models on held-out test set +│ │ └── predict.py # Single-design inference +│ ├── pipeline/ +│ │ ├── run_pipeline.py # Automated ORFS data collection pipeline +│ │ ├── designs.json # Design configs for pipeline runs +│ │ └── logs/ # Per-run error logs + summary logs +│ ├── tests/ +│ │ ├── test_models.py # Smoke tests: shapes, ranges, training, checkpoints +│ │ └── generate_synthetic_data.py # Synthetic .npz generator for tests +│ └── checkpoints/ +│ ├── unet_best.pt / unet_last.pt +│ └── gnn_best.pt / gnn_last.pt +└── data/ # Pre-placement GNN data from prior experiments + ├── *_graph.npz # Netlist graphs (Track 1 input features) + ├── *_congestion.npy # Congestion maps (Track 1 labels) + └── *_floorplan.npz # Floorplan data (larger designs) +``` + +--- + +## Changelog + +### 2026-08-11 — Option A: pre-diffused input channel + data expansion plan + +**`training/thermal_dataset.py` — added 5th input channel (pre-diffused cell density):** +- Added `scipy.ndimage.gaussian_filter(cell_density, sigma=3.0)` as channel 4. +- Normalised blurred channel to [0,1] independently before stacking. +- Rationale: U-Net has no knowledge of thermal diffusion (heat spreading laterally). + The blurred channel approximates the Green's function kernel of the steady-state + heat equation, giving the model a "pre-spread" view of the power distribution. + The model then learns the residual between this approximation and the true HotSpot output. +- Input shape: (4, 64, 64) → (5, 64, 64). + +**`training/train_thermal.py` + `inference/visualize_thermal.py`:** `in_channels=4 → 5`. + +**`data_collection/generate_variants.sh` — new script for ORFS flow variant generation:** +- Generates utilization variants (60%, 70%, 90%) for ibex, jpeg, swerv, ariane133, riscv32i. +- Generates aspect-ratio variants (0.5, 1.5, 2.0) for ibex, jpeg, swerv, riscv32i. +- Run with `--dry-run` to preview make commands without executing them. +- Adds ~24 new training samples (from 26 → ~50) once extracted. +- `adder4` and `gcd` intentionally excluded — near-flat thermal maps (ΔT ≈ 0) add noise. + +**Alternative model options noted for future (not yet implemented):** +- **Option B** — Physics-informed Laplacian loss: `L_total = L_mse + λ·||∇²T_pred||²` + Penalises non-smooth gradients without needing PDE solver. ~20 lines in train_thermal.py. +- **Option C** — Fourier Neural Operator (FNO): operates in frequency domain via FFT. + Theoretically most principled for PDE solutions (∇·(k∇T)+Q=0). Needs new models/fno.py. + Recommended once dataset exceeds 60 samples. +- **Option D** — Swin Transformer: global attention = larger effective receptive field. + Already existed in repo (deleted). Better than U-Net for large dies (ariane136 ΔT=54°C). + Needs 50+ samples to outperform U-Net reliably. + +**Next steps:** +1. Retrain with 5-channel input: `python3 ml/congestion/training/train_thermal.py --data-dir ml/congestion/data --checkpoint-dir ml/congestion/checkpoints --epochs 200` +2. Run variant generation (dry-run first to check): `bash ml/congestion/data_collection/generate_variants.sh --dry-run` +3. Run for real (takes several hours): `bash ml/congestion/data_collection/generate_variants.sh` +4. Re-extract thermal labels for new variants: `bash ml/congestion/data_collection/extract_thermal_batch.sh` +5. Retrain again on expanded dataset (~50 samples). + +--- + +### 2026-08-10 — Thermal training pipeline: per-sample normalisation + timeout fix + +**Root cause of "21 designs failed" in batch:** +The original batch script had no per-design timeout. Large designs (ariane133, ariane136, +swerv) take 40+ minutes for ODB loading alone in OpenROAD Python mode. The batch ran the +first 5 small/fast designs successfully (asap7 ×3 + nangate45/adder4/aes), then appeared +to stall on ariane133 (which actually completed after 41 min). The remaining designs were +simply waiting in sequence. + +**Current dataset: 8 complete pairs** (6 nangate45 + 2 asap7, extracted 2026-08-10): +- asap7: aes_base, jpeg_hi_util_75, jpeg_pipeline_85 +- nangate45: adder4_base, aes_base, ariane133_base, gcd_base, ibex_base + +**`data_collection/extract_thermal_batch.sh` — timeout support added:** +- Default 3600s (1 hour) per extractor call via `timeout "$TIMEOUT_S" util/docker_shell ...` +- Exit code 124 = timeout → prints `[TIMEOUT]` message rather than generic `[FAIL]` +- `--timeout N` flag to override from command line +- Skip counter now also incremented for features (was only counting thermal skips) + +**`training/thermal_dataset.py` — switched to per-sample normalisation:** +- Each thermal map is independently normalised to [0,1] using its own min/max. + Reason: HotSpot absolute temperatures vary ~100× across process nodes and die sizes + (asap7 50µm die at 500mW → 2000°C; nangate45 ibex 0.24mm die → 100°C). The ML model + needs to learn spatial hotspot patterns, not cross-process temperature scales. +- `__getitem__` now returns `{"x", "thermal", "t_min", "t_max"}` per sample. +- `denormalize()` signature updated to take explicit `(t_norm, t_min, t_max)`. +- Dataset-level `self.t_min` / `self.t_max` kept as per-sample lists for diagnostics. + +**`training/train_thermal.py` — updated for per-sample norm:** +- Removed `thermal_norm.json` write (no longer a global constant). +- Val metric is now `val_mae (norm)` [0,1] instead of °C (meaningless cross-process). + +**`inference/predict_thermal.py` — updated for per-sample norm:** +- Removed `--norm` argument (no external norm JSON needed). +- Output `.npz` now contains only `thermal_pred_norm` (relative heatmap [0,1]). +- 1.0 = predicted hottest point in that specific design. + +**Smoke test:** 5-epoch training run on 8 samples converged (val MSE 0.061 on 1 val sample). +GPU used (CUDA available). Full training pipeline verified end-to-end. + +--- + +### 2026-08-10 — Thermal inference script + U-Net fix + +**`models/unet.py`** — added `num_heatmap_layers` parameter to `CongestionUNet.__init__` +(default 10 for congestion, backwards-compatible). For thermal, pass `num_heatmap_layers=1` +to get a proper 1-channel output instead of wasting 9 unused channels. + +**`training/train_thermal.py`** — updated to use `num_heatmap_layers=1` and removed the +`pred.heatmap[:, :1, :, :]` channel-slice hack. Now uses `pred.heatmap` directly. + +**`inference/predict_thermal.py`** — inference script for trained thermal model. + +Two usage modes: +- `--features ` (no OpenROAD needed): loads pre-extracted feature file, runs model, + outputs predicted thermal map in normalised [0,1] and °C forms. +- `--odb ` (auto-extracts): calls `extract_features.py` via `docker_shell` internally, + then runs model. Requires `OR_IMAGE=openroad/orfs-ml:latest` or base image with OpenROAD. + +Outputs `thermal_pred_norm` (64×64), `thermal_pred_c` (64×64 in °C), and normalisation +constants to a `.npz`. Run from `flow/`: +```bash +python3 ml/congestion/inference/predict_thermal.py \\ + --features ml/congestion/data/