Conversation
Add six public benchmark designs, three RTL sources each implemented on
asap7 and nangate45:
- xiangshan_coupledl2: the CoupledL2 subsystem of OpenXiangShan
(MinimalConfig, one core) generated with Chisel 7.3.0 / firtool 1.135.0,
with the SRAM arrays as bsg_fakeram macros.
- picorv32: the LogikBench PicoRV32 benchmark.
- tpu: the LogikBench TPU benchmark at its N=8 default.
Each design directory carries a self-contained config.mk, constraint.sdc
and rules-base.json, plus the fakeram.cfg that records how its SRAM views
were generated. The SRAM macro views themselves are shared platform
collateral under platforms/<pdk>/{lef,lib,verilog}/, following the existing
fakeram convention. Each source directory carries the RTL, a SOURCE.md that
records the upstream repositories, commits and local transformations, and
the upstream license texts under LICENSES/.
All design-specific settings are plain variables in the design config.mk
files; no flow hook, dont-use override, ORFS flow script or platform file
is used or modified. asap7 xiangshan_coupledl2 sets SKIP_DETAILED_ROUTE=1
because FlexDR reports DRT-0206 on this design, so its Final report uses
global-route parasitics. nangate45 xiangshan_coupledl2 sets
OPT_POST_GRT_WNS=1 because the nangate45 platform default of 0 wraps
recover_power in an incremental global_route pair that does not receive
-allow_congestion and fails with GRT-0116 on this congested design. The
new designs are listed in the commented DESIGN_CONFIG examples of
flow/Makefile.
All six SDCs use set_max_fanout 16. A full-flow comparison against
set_max_fanout 32 and no fanout constraint kept 16 as the best overall
setup WNS/TNS.
Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates several new designs into the flow, including picorv32, tpu, and xiangshan_coupledl2, along with their respective configurations, SDC constraints, and base rules for the asap7 and nangate45 platforms. It also imports the required RTL source files, provenance metadata, and license files. I have no additional feedback to provide as the review comment was incorrect regarding the file contents.
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 1 without a comparable baseline. |
|
PR-merge fail is not related to this PR. Master CI has the same issue. |
|
@maliberty This PR adds new designs. Jenkin CI should be updated to run the new designs. |
| create_clock -name core_clock -period $clk_period [get_ports clk] | ||
| create_clock -name vclk_core -period $clk_period | ||
| set_max_fanout 16 [current_design] | ||
| set_input_delay $io_delay -clock vclk_core [all_inputs -no_clocks] |
There was a problem hiding this comment.
I think this is wrong and non-idiomatic for a CPU. set_input_delay causes hold cell infestation and a CPU is going to be connected to a system bus and that system bus is going to be connected to a clock crossing bridge, so set_input/output_delay is wrong here. See asap7/constraints.sdc for how OpenROAD today can best create idiomatic constraints for a CPU that sits as a macro in a core clock domain.
|
Careful work — the provenance mapping, license texts, fakeram collateral and waiver table are better documented than much of what's in the tree. But I want to put a prior question: what does the project get from another design? I think the honest answer is less than it looks, and that the same effort spent differently would be worth a lot more. 1. The model that works already exists, and has receiptsOpenROAD has That is what a design is worth when it's aimed at a concern: six named fixes in one tool, from RTL with no architecture in it at all. Note where it lives. I'll be honest about the limits of the example, since I'd rather make the weak version of the argument than the flattering one: mock-array's marginal value today is lower than it was. OpenROAD is essentially solid on that topology now, and a test that stops finding things stops earning its runtime. It's also worth noting that negative results on unmerged code aren't visible anymore, so a test's catches are harder to count than they used to be. The point isn't that mock-array should run forever — it's that a design aimed at a concern produces a list like that one, and a design aimed at being a design doesn't. 2. Big designs are mines for concern tests, not examples to be followedI'm running XiangShan through asap7 in bazel-orfs (The-OpenROAD-Project/bazel-orfs#1055, a draft working record). The reason that's worth doing is not that it produces an example. It produces failures, and each failure, once characterized, becomes a small test that runs in seconds. How that went: global route on a 3.6 mm die was seven hours into maze iteration 1 of 30 at 105 GB. Pin-fitted mocks brought the die to 2.27 mm and CTS to 6 minutes; every knob arm of an overnight matrix then timed out in the same maze iteration. A margin sweep reached the zero-iteration route on every arm and every arm hit the same wall — FastRoute's own 100× capacity guard, GRT-0228, on a GCell edge a few microns from a macro's pin side — or, at the tightest margin, a die too dense to route at all. Weeks, and I still couldn't characterize the wall. What characterized it was a synthetic: one pin-wall block, one flop per pin on the parent side, seconds per point. Fourteen arms, 1,000 to 10,000 pins across 10–100 µm channels. With the logic facing the pins, every arm routes clean at iteration 0 — total overflow at most 31 edges, largest edge 15 over, even at 10,000 pins across a 10 µm channel. Pins facing the logic they talk to never wall. The walls were lateral traffic, not pin density. That's an OpenROAD-level fact about FastRoute, and the big design had been actively misleading about it for four takes. The big design's value was as a source of questions. Its value as an example is zero, and I'd never propose putting it in 3. Which makes the config.mk problem the real oneA design in This PR's own So: the marginal value of making the existing designs idiomatic is higher than the marginal value of six more. That's unglamorous work with no announcement attached to it, and it's the work that makes the next person's copy correct. If the six land as-is, the debt grows by six. And it raises the question underneath: if a design's 4. What is tested where, todayOpenROAD — ~5,700 files under ORFS — 73 design/platform configs (asap7 21, nangate45 17, gf12 13, sky130hd 7, gf180 6, sky130hs 5, gt2n 3, gf55 1) against The division I'd propose. OpenROAD owns any concern a constructed input can exhibit — seconds, deterministic, golden, bisectable on every commit, and the tool has a strong enough mental model of the input that the expected result is unambiguous. ORFS owns composition and the scripts: stage handoffs, metric drift, Corollary: every waiver in ORFS is a missing OpenROAD test. The four here — DRT-0206 terminal connectivity, GRT-0116 on the 5. Two audiences, opposite budgetsORFS is both a post-merge integration test of OpenROAD, run occasionally, where hours are affordable — and the per-PR test of ORFS's own scripts, where a one-line And 6. Don't commit generated RTLMost of these 115k lines are Chisel output elaborated once by hand with Chisel 7.3.0 and firtool 1.135.0, then committed. Nobody can act on those files: you can't change the configuration, can't make a smaller variant to iterate on — which a 2.3 h flow badly needs — can't move to a newer Chisel, firtool, or XiangShan, can't take an upstream fix. And the snapshot is taken at the moment of least knowledge, structurally: the one-off generation happens at the start, before the work teaches you what the configuration should have been, and every later insight lands on the far side of an unrepeatable step. The alternative runs today. In bazel-orfs, XiangShan V3 is built from Scala in the build graph: eleven One lesson from that work applies directly: bazel-orfs's retired gallery pinned a different Chisel than the upstream designs targeted and ended up carrying a compatibility patch per library, one of them 7,700 lines. Matching upstream costs nothing. A one-off pin against a XiangShan that has moved on is the same trap, except permanent — with no regeneration path there's no version to correct to. 7. So what would I do instead
On motivation, and I mean this without edge: everyone cares about what they're working on, and a recognizable CPU name in Which bug list will |
|
The title says "benchmark designs", and I want to take that word seriously, because it decides what the PR is for. What "benchmark" means, borrowing from softwareThe clearest worked example lives outside EDA, so it's worth spelling out. SPEC CPU (from the Standard Performance Evaluation Corporation) is how the software and computer-architecture world compares machines and compilers. What makes it work is almost none of it technical:
The suite is the agreement, not the programs. That's the whole thing. EDA's closest analogues are the contest suites — ISPD and ICCAD placement, routing, sizing, TAU timing — plus EPFL's combinational suite, IWLS 2005, ITC'99, OpenCores. Those are benchmarks in the SPEC sense: a community agreed they're representative of a stated concern, and results on them are comparable across tools and papers. By that standard the ORFS design directory isn't a benchmark suite and shouldn't try to become one. Nobody outside this project runs That's a role, not a deficiency. Minting a new benchmark suite isn't ORFS's job and ORFS has no mechanism to make one stick. If the goal is to demonstrate OpenROAD's QoR, the way to do it is to compete on the existing suites, where the numbers are comparable to something. Designs added here produce numbers comparable to nothing. The literature is blunt about what a suite costs to maintain — and note that the retirement problem is the same one SPEC solves with versioning. ISPD2005 exists because MCNC and ISPD98, having driven a decade of progress, were judged no longer representative. A suite is a maintained artifact with a constituency, not a directory of designs. The diversity critique lands directly on this PR too: the HighTide survey finds 16 of 27 blocks in existing open-source suites are RISC-V-related, and calls that "redundancy rather than architectural variation". Two of the three designs here are RISC-V cores. If the motivation is coverage, that's the axis with the least of it. (HighTide also makes my generated-RTL point from the other direction: it uses git submodules specifically so the suite "incorporates upstream changes rather than serving as a static snapshot". A committed one-off elaboration is the static snapshot that critique names.) What ORFS's checks actually measure
What it structurally cannot express is a trade. A change that shrinks the core 8% and gives up 1% of worst slack fails the slack rule and says nothing about the area rule it just improved. Read literally that's a regression; read honestly it's a point that moved along the front. The two aren't distinguishable from So, directly: ORFS tells you that one metric of one design crossed a padded bound. It tells you nothing about the front. Six more designs is six more tripwires, not more QoR signal. WNS is the wrong thing to record, and it's throwing information awayThis one is worth fixing regardless of what happens to this PR.
It's also the wrong shape of number. WNS is signed and lives near zero, so a fractional change in it is meaningless: a design at +5.8 ps moving to +0.9 ps has "lost 85%" of nothing, while one at −274 ps moving to −280 ps has "lost 2%" of a great deal. Both readings are noise dressed as precision, and a tolerance expressed as a percentage of WNS is correspondingly absurd — tight where WNS is small, loose where it's large, for no reason connected to the design. And the worst part: WNS saturates. Once a design meets its constraint the optimizer stops working, so positive slack isn't headroom — it's the flow having stopped caring. Whatever the design could actually have run at is destroyed by the constraint having been loose. Which means a QoR number needs one of two things: a constraint tight enough that the tool is still working the whole way (overconstrained, so the achieved period means something), or a constraint sweep that finds the period at which the design just closes. That sweep result is the real QoR figure — a property of design-plus-tool rather than of a number somebody typed into an ORFS records neither. It records a signed slack against an unstated datum, and pads it by a percentage of a period it doesn't put beside it. Two small upstream changes would fix all of this, and neither breaks anything:
After that, a front is computable by anyone, and the timing axis is a stable positive quantity of order the clock instead of a signed number hovering around zero. What a front check looks like once you have the measurementsIn bazel-orfs I run one against those values — internal use, not something I'm proposing ORFS adopt wholesale, but the shape is the arguable part:
Why a 6-hour design can't be a QoR instrumentThis bears on the PR regardless of what anyone thinks of the above. A QoR verdict needs a tie band, and a tie band is a noise measurement: repeat the same inputs on the same toolchain and see how much the number moves. You can't repeat a design that takes 2.3 or 6 hours. So its band can only ever be a guessed default, and a guessed band manufactures confident verdicts — tight where it shouldn't be, loose where it matters. And a minimum period, per the section above, needs a constraint sweep — several runs, not one. At 6 hours a sweep isn't an experiment, it's a week. Baseline drift compounds both. Checked-in rules lag the flow by however long since the last regeneration, and that drift lands in the same diff as the change under test. On asap7 I've measured 9 ps of setup-slack drift over six weeks — the same size as the effects being measured. The only comparison that isolates a change is both arms measured on the same toolchain on the same day, which at 6 hours a side is a 12-hour experiment per point. So The domain-knowledge partPicking axes isn't generic. Whether area or period or power is the axis you're allowed to spend is a property of what the design is, and knowing that means knowing its domain problem — the same knowledge that makes a So, the question back to the PR
I don't think one set of designs can be both, and the PR reads as if it's aiming at the second while landing in the first. |
|
The title says "benchmark designs", so the question this PR really asks is which designs deserve that word. I said last time that So: what makes a design benchmark-grade, which one ORFS already has, and which ones are traps. What a benchmark-grade design has to beRecognized outside the project. Results on it have to be comparable to something. A design nobody outside this repo runs produces numbers comparable to nothing, no matter how good the design is. Real RTL somebody shipped, and I mean something specific by shipped: silicon in a product that somebody paid money for. Not a demonstration, not a stunt, not a one-off elaboration checked in as a snapshot, and not a free shuttle tapeout. That is not a line drawn to exclude anything in particular — it is the only line that carries information, because of what selects on each side of it. A part someone bought was selected for doing its job well under real constraints. A free shuttle slot was selected for being small, open and cheap to fabricate. A benchmark inherits whichever pressure produced the design it is made of, and only one of those two is the pressure a flow will meet in practice. The design's job is to be representative; anything optimized to be extreme along a single axis is representative of that axis and nothing else. Cheap enough to repeat. This is the one that gets skipped. A QoR verdict needs a tie band, and a tie band is a noise measurement — you run the same inputs on the same toolchain several times and see how far the number moves. You cannot repeat a design that takes hours, so its band can only ever be a guessed default, and a guessed band manufactures confident verdicts: tight where it shouldn't be, loose where it matters. Size is a cost, never a feature. Sequential, with a real clock. The achieved minimum period only means something if there are registers for it to be a property of. Fetched from a pinned upstream, not committed. A committed elaboration is a static snapshot, and a snapshot stops being the design its author ships the day after you take it. Permissively licensed, obviously, and diverse with respect to what the suite already has. The HighTide survey (arXiv:2606.04126) finds 16 of 27 blocks in existing open-source suites are RISC-V-related and calls that "redundancy rather than architectural variation". Two of the three designs in this PR are RISC-V cores. If the motivation is coverage, that is the axis with the least of it. ORFS already has the best candidate, and it is
|
| asap7 | nangate45 | sky130hd | |
|---|---|---|---|
| std cells at placeopt | 14,321 | 16,773 | 20,353 |
finish__design__instance__area |
1,663 | 21,614 | 142,679 |
finish__timing__setup__ws |
−24.2 | −0.0667 | −0.425 |
detailedroute__route__drc_errors |
0 | 0 | 0 |
| clocks | 2 | 2 | 2 |
Against the list above: openly licensed, real RTL somebody shipped, sequential, two clocks (so it exercises the multi-clock case, which most small test designs do not), DRC-clean everywhere, minutes per run rather than hours — and not a RISC-V core, which is the one diversity axis this PR moves backwards on.
So the constructive version of this PR adds nothing
Take the design you already have on 13 platforms and change what you record about it:
- Record the achieved minimum period (
clock − WNS) per clock, as its own metric.finish__timing__setup__wsis stored without the constraint it is relative to, and it saturates the moment the design closes — so positive slack is not headroom, it is the optimizer having stopped caring. - Record the unpadded measured value next to each padded threshold in the rules file, so a front is computable by anyone rather than only a bound being checkable.
- Measure the tie band by repeating the run, instead of inheriting a percentage. At minutes per run this is affordable; that is the whole reason to prefer a small design.
- Publish the disclosure — tool commits, PDK, the full variable set in force, the constraint datum — so a reader can tell what was actually measured.
That is a change in what ORFS measures, not an addition to what ORFS maintains. It is also the only version of this that survives contact with the question "compared to what?"
I should be straight about the limit: even done this way, the numbers would be ORFS's own. That makes aes a well-instrumented reference design, not a benchmark in the SPEC sense, and nothing living in flow/designs can be the latter. The difference between a reference design and a tripwire is worth having anyway.
Bad choices, and why
Code-golf designs — SERV is the canonical example. SERV is an advocacy success: the world's smallest RISC-V CPU, award-winning, endlessly shared, and on the back of that visibility it has crept into a benchmark set or two — LogikBench carries it at 977 cells on asap7. That visibility is the whole of its claim to be there. It is not a design anyone ships: a bit-serial datapath that retires one instruction over dozens of cycles is a demonstration of a lower bound, not a block that goes into a product. Where it has reached silicon it is through free shuttle programmes — demonstration and coursework tapeouts — which is the distinction drawn at the top of this comment rather than one invented here to exclude it. It is absent from every set with conference provenance: IWLS 2005, ISCAS, ITC'99, the ISPD and ICCAD contest suites. And its ratio of logic to flops is so far from a normal block that whatever the flow does to it generalizes nowhere. Small is not the same as cheap-and-representative, and selecting on size alone lands you exactly here.
Designs that take hours — xiangshan_coupledl2 at 2.3 h and about 6 h. Per the section above, it cannot carry a tie band, so it cannot be a QoR instrument. A constraint sweep on it is not an experiment, it is a week. It can only be a tripwire whose bound gets re-baselined when it trips, at hours a time. That is a legitimate thing to want; it is not what "benchmark" means.
Committed generated RTL. A Chisel elaboration checked in is the static snapshot the HighTide critique names — that suite uses submodules specifically so it "incorporates upstream changes rather than serving as a static snapshot". If a design comes from a generator, the benchmark is the generator at a pinned commit.
More RISC-V. See the redundancy point above.
A config.mk that is knobs tuned until the flow finished. The four waivers in this PR are missing OpenROAD tests wearing a design's clothes. A design whose settings exist to get past a tool failure measures the tool failure.
If you actually want a suite, here is what that costs
Minting one is not ORFS's job and ORFS has no mechanism to make one stick. But if the goal is to demonstrate OpenROAD's QoR against something real, these are the places where numbers are comparable to a community, roughly in order of effort:
- ISPD Initial Detailed Routing, 2018 / 2019. The one place OpenROAD already holds a published, comparable score — TritonRoute took first place in the 2018 contest, and OpenROAD's own documentation reports 0 DRVs on ISPD18 and on all but one ISPD19 case. Tracking it per release would be a genuine regression signal against numbers nobody here chose. Costs: DEF + route-guide ingestion outside the flow, and the official evaluator is not open.
- ISPD Discrete Gate Sizing, 2012 / 2013. Published leakage-and-runtime rankings, and the designs are the open OpenCores/ITC'99 lineage (
pci_bridge32,des_perf,vga_lcd,b19,leon3mp,netcard). OpenROAD has the resizer; the contest libraries are the obstacle. - IWLS 2005. The set
aes_corealready belongs to. Openly licensed throughout, and it brings a dozen recognized designs at once instead of one. No PPA leaderboard, but the best recognition-per-unit-effort of any open set. - ICCAD CAD Contest, annually. Highest visibility, moving target.
- EPFL Combinational Benchmark Suite. The maintained leaderboard is real — best LUT-6 size and depth, with the author and method named against every entry, still updated in 2026. The agreement is LUT-6, so it scores a mapper rather than a flow, but it is the clearest live example of what "published run rules" actually looks like.
- LogikBench and HighTide, both 2026 and both worth watching: they are the only open suites publishing ASIC reference results with a disclosure block and a release cadence. HighTide tags releases and uses submodules rather than snapshots; LogikBench publishes per-PDK results with the tool versions recorded beside every number.
Every one of these is a maintained artifact with a constituency, which is the actual cost — re-run per release, or the baselines drift and the drift lands in the same diff as the change under test.
The ask
Don't add six designs. Take aes, record the achieved period, measure its tie band by repeating it, and publish the disclosure. If that shape turns out to be useful, xiangshan_coupledl2 can come back as what it actually is — a long-running tripwire on a named CI tier — rather than as a benchmark.
jhkim-pii
changed the title
|
@oharboe Hard to read due to the many comments. Replied to a few that caught my eye.
Dont' add six designs. Take
|
3 participants
Summary
config.mk,constraint.sdc,rules-base.json, upstream license texts, and aSOURCE.mdthat records the upstream repositories, commits, and local transformations.xiangshan_coupledl2are shared platform collateral underplatforms/<pdk>/{lef,lib,verilog}/, following the existing fakeram convention; the design keeps thefakeram.cfgthat records how they were generated.LICENSES/; the generated XiangShan RTL is mapped file by file to its Scala sources and licenses inPROVENANCE/selected-rtl.tsv.New designs
xiangshan_coupledl2picorv32tpuN=8,DW=8,ACCW=32)make synthof this branch (Yosys 0.68, slang frontend). Full-flow wall times aremake finishwith the pinned OpenROAD90e29809c3,NUM_CORES=16forxiangshan_coupledl2and 4 for the others, measured while many flows shared a 160-core host, so they are upper bounds for an idle CI machine.Runtime-reduction settings
xiangshan_coupledl2trades timing-repair depth for runtime; picorv32 and tpu run the default flow. All six designs setLEC_CHECK=0.SYNTH_HIERARCHICAL/SYNTH_MINIMUM_KEEP_SIZETNS_END_PERCENTSKIP_CTS_REPAIR_TIMING,SKIP_INCREMENTAL_REPAIRSKIP_ANTENNA_REPAIR,SKIP_ANTENNA_REPAIR_POST_DRTGPL_TIMING_DRIVEN,GPL_ROUTABILITY_DRIVENENABLE_RESISTANCE_AWAREDETAILED_ROUTE_END_ITERATIONGLOBAL_ROUTE_ARGS-congestion_iterations 5 -allow_congestion-congestion_iterations 5 -allow_congestionN=8instead of LogikBenchN=128(ABC runtime)Waivers
xiangshan_coupledl2SKIP_DETAILED_ROUTE=1; Final report on global-route parasitics; no detailed-route rulesxiangshan_coupledl2OPT_POST_GRT_WNS=1global_routepair aroundrecover_powerwithout-allow_congestion, which fails with GRT-0116 on this congested design; the WNS repair it enables is already off viaSKIP_INCREMENTAL_REPAIRxiangshan_coupledl2PWR_NETS_VOLTAGESandGND_NETS_VOLTAGESempty (PDNSim skipped)xiangshan_coupledl2<=rule)