Establish architectural design process for orchestration rework#74
Conversation
Bumps [astro](https://github.com/withastro/astro/tree/HEAD/packages/astro) from 5.18.1 to 6.0.6. - [Release notes](https://github.com/withastro/astro/releases) - [Changelog](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md) - [Commits](https://github.com/withastro/astro/commits/astro@6.0.6/packages/astro) --- updated-dependencies: - dependency-name: astro dependency-version: 6.0.6 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
|
@lythx fix the conflicts |
| 3. Optimized result transport using S3-compatible object storage and efficient binary formats instead of large JSON payloads through the message broker. | ||
| 4. Optimized result merge by replacing pure-Python averaging with high-performance numerical libraries. |
There was a problem hiding this comment.
S3 storage and binary formats are the technical details which would come up after many more interations and discussions of all pros and cons. In this document I will focus more on the goals and user stories.
we also need a lot of estimations for many items:
- typical times needed to connect to HPC cluster (like Ares)
- typical time the job sits in the waiting queue
- number of parallel jobs per simulation users would like to HPC
- number of simulations single user would like to run in parallel
- time needed to dump single binary results to disk
- how many files per simulation
- how many pages per file
I suspect mine input would be needed in many cases here.
You can ask LLM to formulate many (i.e. >20) open questions, you can commit them here. I could help you and provide my input for that.
Take a look at the bottom of https://github.com/APTG/dedx_web/blob/master/docs/04-feature-specs/entity-selection.md - these are leftovers of similar process I had
|
@copilot resolve the merge conflicts in this pull request |
…al-design-process # Conflicts: # package-lock.json # package.json Co-authored-by: lythx <93780882+lythx@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new documentation area to formalize the architectural design process for the simulation orchestration rework (vision, current-state analysis, bottlenecks, and a Phase 1 profiling runbook), and wires it into the docs site navigation/rendering.
Changes:
- Added
rework-orchestration/doc section with vision page, current architecture/bottlenecks context, ADR registry + template, open-questions backlog, and a Phase 1 profiling plan. - Updated Starlight sidebar to include “Orchestration Rework” with autogenerated subsections.
- Upgraded Astro/Starlight dependencies and added a Mermaid rendering plugin for the new diagrams.
Reviewed changes
Copilot reviewed 10 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/content/docs/rework-orchestration/index.mdx | Adds vision, goals, scope, and links into the orchestration rework doc set. |
| src/content/docs/rework-orchestration/context/current-architecture.mdx | Documents as-is orchestration flows; introduces Mermaid sequence diagrams. |
| src/content/docs/rework-orchestration/context/current-bottlenecks.mdx | Captures bottlenecks and links to the profiling runbook. |
| src/content/docs/rework-orchestration/context/open-questions.mdx | Adds structured discovery backlog/questions for Phase 1–3. |
| src/content/docs/rework-orchestration/research/phase-1-profiling-plan.mdx | Adds reproducible profiling protocol and measurement outputs. |
| src/content/docs/rework-orchestration/adr/index.mdx | Adds ADR registry scaffold. |
| src/content/docs/rework-orchestration/adr/_template.mdx | Adds a reusable ADR template (hidden/draft). |
| src/content/docs/rework-orchestration/design/.gitkeep | Keeps the design directory tracked for future documents. |
| astro.config.mjs | Adds Mermaid Starlight plugin + new sidebar section. |
| package.json | Upgrades Astro/Starlight/sitemap/sharp and adds Mermaid plugin dependency. |
| src/content/docs/local-setup/local-celery.mdx | Tweaks the .env snippet fence formatting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| To use S3 (for the full licensed version), create a `.env` file in the project root with the following variables: | ||
|
|
||
| ```env | ||
| ```bash title=".env" |
There was a problem hiding this comment.
This fenced block is for a .env file, but it’s marked as bash. Using an env/dotenv/ini lexer (depending on what Starlight supports) would give more accurate syntax highlighting and reduce confusion that the lines are shell commands.
| ```bash title=".env" | |
| ```env title=".env" |
| "dependencies": { | ||
| "@astrojs/react": "^5.0.1", | ||
| "@astrojs/sitemap": "3.2.1", | ||
| "@astrojs/starlight": "^0.37.6", | ||
| "@astrojs/sitemap": "3.7.1", | ||
| "@astrojs/starlight": "^0.38.2", | ||
| "@pasqal-io/starlight-client-mermaid": "^0.1.0", | ||
| "@types/react": "^19.2.14", | ||
| "@types/react-dom": "^19.2.3", | ||
| "astro": "^5.6.1", | ||
| "astro": "^6.0.8", | ||
| "react": "^19.2.4", | ||
| "react-dom": "^19.2.4", | ||
| "sharp": "^0.34.2", | ||
| "sharp": "^0.34.5", | ||
| "swagger-ui-react": "^5.32.1" |
There was a problem hiding this comment.
These dependency upgrades appear to raise the minimum supported Node.js version (e.g., @astrojs/react@5.0.1 declares engines.node >=22.12.0 in the lockfile). To avoid CI/dev environment drift, consider adding an explicit engines.node constraint (and/or updating any docs/CI that still assume Node 20).
|
|
||
| Pseudo-complexity: | ||
|
|
||
| - O(T _ P _ V), where: |
There was a problem hiding this comment.
The pseudo-complexity line uses underscores (O(T _ P _ V)) which Markdown will typically interpret as emphasis, so it may render incorrectly. Consider formatting this as inline code and using an unambiguous multiplication symbol (e.g., O(T*P*V) or O(T·P·V)).
| - O(T _ P _ V), where: | |
| - `O(T*P*V)`, where: |
Merge conflicts resolved in commit |
This PR adds a docs/rework-orchestration directory which will be used for documenting research, decisions and progress related to the orchestration rework as described in this issue. I used GPT 5.3 Codex to generate the initial pages, focusing on the current architecture, bottlenecks and profiling plan.