diff --git a/AGENTS.md b/AGENTS.md index 8d41212..fd9dbc3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,124 +1,72 @@ -# Repository Guidelines +# AGENTS.md -## Project Overview +MLBlock — no-code block DAG builder for ML/RL/DL. React canvas (React 19 + Vite + TanStack Router + Astryx) + FastAPI + Pydantic v2 + SQLModel/Postgres (Supabase). Deploy: Render. GPU dispatch: Vast.ai REST only (no SSH). -MLBlock is a no-code, block-based ML/RL/DL builder ("Scratch, but for AI"). Users compose pipelines as DAGs of blocks (neural layers, sklearn models, data loaders, RL environments) in a React canvas. Pipelines validate, run locally, code-generate into standalone Python scripts, and can dispatch to rented GPUs via Vast.ai. Backend: FastAPI + Pydantic v2 + SQLModel on PostgreSQL (Supabase). Frontend: React 18 + Vite + TypeScript. Deployed on Render. +## Structure -## Architecture & Data Flow +- `backend/` — Python >=3.10, `uv` only (`uv.lock` committed). `pyproject.toml` is canonical; `requirements.txt` is stale — never edit it. +- `frontend/` — `npm` only (`package-lock.json`). No `bun`/`pnpm`/`yarn`. No `.nvmrc`. Vite + `tanstackRouter` plugin + Tailwind v4. +- `tutos/` + `frontend/src/content/cours/` — markdown tutorials. -Two-layer system bridged at import time: +## Commands +Backend from `backend/`: +```bash +uv sync # install (.venv) +uv run python -m mlblock --mode generate # codegen (default configs/cnn_mnist.json) +uv run python -m mlblock config.json --mode build +uv run uvicorn mlblock.server.main:app --reload # dev server :8000 +uv run ruff check . # lint (E,F only; blocks/** ignores E501) +uv run pytest mlblock/tests -q # all; add path for single file +uv run pytest mlblock/tests/test_graph.py -v ``` -JSON config → ConfigLoader.validate() → Graph (DAG) → Pipeline - ├→ run() (execute blocks in topo order) - └→ generate_code() → standalone Python script + +Frontend from `frontend/`: +```bash +npm install +npm run dev # Vite, no proxy — hits VITE_API_BASE_URL directly +npm run build # tsc --noEmit && vite build +npm run lint -- --max-warnings 0 # must pass zero warnings +npm run knip # unused exports check (local only, not in CI) +npm test # vitest run (node env, store/utils only) ``` -- **Core engine** (`backend/mlblock/core/`): dict-based `GraphNode`/`Edge`/`Graph`, Kahn's topological sort (cycle → `ValueError`), `BlockMeta`/`BlockRegistry` (class-level dict), `Pipeline` orchestration with param coercion, `CodeGenerator` (emits standalone Python), `VastAI` client (**REST only** — `requests`, Vast API v0 bundles/asks/instances, onstart gzip/base64 payload; no SSH). -- **Server layer** (`backend/mlblock/server/`): FastAPI with 7 routers (catalog, samples, pipelines, validation, jobs, files, health). Sync `def` endpoints, `session: Session = Depends(get_session)`. `ValueError` → `HTTPException(400, detail=str(e))`. Dual auth: Supabase JWT (`server/auth.py`, JWKS TTL cache, `MLBLOCK_DEV_AUTH` bypass) + GPU bearer (`server/gpu_auth.py`, per-job `instance_api_key` with global `GPU_API_KEY` fallback). DB tables in `server/models.py`: `profiles`, `pipelines`, `jobs`, `job_outputs` (UUID PKs, JSON `nodes`/`edges`, FK CASCADE; the old `columns` field was dropped by migration — free mode only). -- **Bridge**: `blocks/registry.py` `_discover()` runs at import time, scans `blocks/**/*.py` (dirs named `{category}-{HEXCOLOR}/`, e.g. `neural-4FC3F7/`), loads via `importlib.util.spec_from_file_location`, registers module-level functions into `BLOCK_REGISTRY` (server Pydantic models), `BLOCK_SOURCES` (source text), and legacy `BlockRegistry` (core execution dict). -- **Frontend** (`frontend/src/`): `router.tsx` (`createBrowserRouter` + `RequireAuth`, 7 routes), `store/useAppStore.ts` (single Zustand store: reactflow `flowNodes`/`flowEdges`, `savedFingerprint` dirty detection, undo/redo 50-deep, `loadPipeline`/`savePipeline`/`ensureDraft`), `api/client.ts` (axios + Supabase session interceptor, zod-validated responses). UI: Tailwind v4 (`@theme` tokens in `index.css`) + JS tokens in `theme.ts`; Base UI dialog; **no shadcn**. Editor is free-mode only (grid/columns removed); mobile-first: palette drawer <768px, tap-to-add gated to mobile, handles ≥22px on `pointer: coarse`, header wraps ≤900px, landing hamburger nav; "Disposer" auto-layout button (dagre) in the ReactFlow Controls cluster. +CI (`.github/workflows/ci.yml`): backend `ruff check` + `pytest`; frontend `build` + `vitest` + `eslint --max-warnings 0`. Concurrency `ci-${ref}` cancels in-flight. -Execution flow: user saves pipeline → `POST /api/pipelines` → `POST /{id}/execute` creates `Job` row → local subprocess (`MLBLOCK_RUN_MODE=local`, dev) or Vast dispatch (`gpu`) → GPU runs generated code → HTTP callbacks `POST /api/jobs/{id}/status|output|error` → results in `jobs`/`job_outputs`. +## Architecture -## Key Directories +- **Block discovery** `backend/mlblock/blocks/registry.py:_discover()` runs at import. Scans `blocks/{category}-{HEXCOLOR}/*.py`, loads via `importlib.util.spec_from_file_location` (hyphens in dirs), registers module-level functions into `BLOCK_REGISTRY` + legacy `BlockRegistry`. File stem = block key. +- **Execution**: `JSON {nodes,edges} -> ConfigLoader.validate() -> Graph (Kahn topo sort, cycle=ValueError) -> Pipeline.run() / generate_code()`. `generate_code()` emits standalone Python with `notify_status`/`notify_output(block_id)` callbacks (20k truncation). +- **Server** `backend/mlblock/server/`: 7 routers (catalog, samples, pipelines, validation, jobs, files, health) in `routes.py`. Sync `def` endpoints (`session: Session = Depends(get_session)`). `ValueError -> HTTPException(400)`. `graph_data = raw.get("graph", raw)` shape. +- **Auth**: Supabase JWT (`server/auth.py`, JWKS TTL cache, `MLBLOCK_DEV_AUTH` bypass in dev) + GPU bearer per-job `instance_api_key` (`server/gpu_auth.py`, fallback `GPU_API_KEY`). +- **Jobs**: `POST /api/pipelines/{id}/execute` -> `Job` row -> local subprocess (`MLBLOCK_RUN_MODE=local`, default dev) or Vast.ai (`gpu`, `render.yaml`) with gzip/base64 `onstart`. GPU callbacks `POST /api/jobs/{id}/status|output|error` -> `job_outputs.block_id` (indexed) -> Supabase Realtime `postgres_changes` -> `hooks/useBlockRunner.ts` (poll 3s job / 2s outputs + Realtime) -> `store/jobOutputs`. +- **Frontend routing**: file-based `src/routes/*` + `routeTree.gen.ts`; `main.tsx` is `createRouter(routeTree)` + QueryClient + Supabase auth listener with `pending-stash` localStorage. `router.tsx` is deprecated shim — don't use. +- **State**: single Zustand store `store/useAppStore.ts` is canvas truth (`flowNodes`/`flowEdges`, `savedFingerprint` dirty check, undo/redo 50, `jobOutputs`/`results` synced). Never fork it. +- **Vite**: no dev proxy. Frontend calls `VITE_API_BASE_URL` directly (`frontend/.env` -> `http://localhost:8000` locally; Render injects prod URL). -``` -backend/ -├── mlblock/ -│ ├── core/ # graph.py, pipeline.py, config.py, generator.py, block.py, vast.py -│ ├── blocks/ # {category}-{HEX}/ dirs, one file per block; registry.py discovers -│ ├── server/ # main.py, routes.py, auth.py, gpu_auth.py, database.py, models.py, schemas.py -│ ├── models/ # pipeline.py only: PipelineDef/Node/Edge (v2 Pydantic) -│ ├── configs/ # cnn_mnist.json and other pipeline JSON configs -│ ├── scripts/ # generate_samples.py, validate_exercises.py -│ └── tests/ # 7 pytest files (see Testing & QA) -├── pyproject.toml # canonical deps — uv -└── requirements.txt # STALE — drifts from pyproject (adds stable-baselines3, drops torchvision); uv ignores it -frontend/ -└── src/ - ├── api/ # client.ts (axios + zod) - ├── components/ # flow/ (FlowCanvas, BlockNode, FlowLink, FlowPalette), ui/ (dialog, dropdown-menu, card, hover-card, field, ConsolePanel, modals, Toast) - ├── pages/ # EditorPage.tsx (unsaved-changes guard), Login/Register, … - ├── store/ # useAppStore.ts (single Zustand store) - ├── schemas/ # auth.ts (RHF+zod) - └── utils/ # pending-stash.ts (localStorage dirty stash), fingerprint, layout.ts (dagre "Disposer"), tapGuard.ts (mobile tap-vs-drag), typeCheck, portResolution, exportImport -.github/ # release-drafter.yml only — NO CI test workflow -render.yaml # backend web service + frontend static site -backend/main.py # GENERATED code output example — not a source file; ignore -``` +## Conventions -## Development Commands +- **Blocks**: plain functions, no base class. Port `in_1: "torch.Tensor"`, outputs `out_1` etc. `import torch/nn` at top OK; sklearn/gymnasium/pandas inside function. Docstring line1=French label, line2=French summary; param suffixes `(entre: min-max, pas: x)` `(impair)` `(choix: a|b)` `(suggestions:)` `(format:)` `(longueur:)`. +- **Python**: `from __future__ import annotations`, Pydantic v2 `model_validate(context={"registry": BLOCK_REGISTRY})`, `NotImplementedError` if no builder. +- **Frontend**: RHF+zod only on auth pages; editor params are segment-driven `BlockNode` fields. Styling is Astryx (`@astryxdesign/core` + `@stylexjs/stylex`) + Tailwind v4 — `index.css` layer order `reset,theme,base,astryx-base,astryx-theme,components,utilities`. Dark mode forced `data-theme="dark"`. Editor is free-mode only — don't reintroduce columns/grid. `TapGuard` + `tap-to-add` is mobile-only; "Disposer" (dagre) is explicit `ControlButton` — never auto-run. +- **Unsaved guard**: `useBlocker` from `@tanstack/react-router` (not `react-router-dom`) + `beforeunload` + `mlblock-pending-{userId}` stash. Don't regress. -All backend commands run from `backend/` with `uv`; frontend from `frontend/` with npm: +## Testing -```bash -uv sync # Install deps (creates .venv) -uv run python -m mlblock # Codegen from default config (configs/cnn_mnist.json) -uv run python -m mlblock config.json --mode build # Build + run model -uv run python -m mlblock.server # uvicorn on 127.0.0.1:8000 -uv run uvicorn mlblock.server.main:app --reload # Dev server -uv run pytest mlblock/tests # All tests (tests skip without DATABASE_URL) -uv run pytest mlblock/tests/test_graph.py # Single file -uv run python scripts/generate_samples.py # Upsert French sample datasets to Supabase Storage -npm install -npm run dev # Vite dev server (no proxy — calls VITE_API_BASE_URL) -npm run build # tsc --noEmit && vite build -npm test # vitest run (store + utils tests) -``` +- Backend: `conftest.py` skips if `DATABASE_URL` absent (no SQLite fallback). `client` fixture creates real engine (`statement_timeout=10000`), overrides `get_session`/`get_current_user`/`verify_gpu_key`, creates one Supabase auth user via Admin API (`SUPABASE_URL`+`SUPABASE_SECRET_KEY`), purges that user's `pipelines` per-test (cascade jobs/outputs). `catalog_client` needs no DB. `BlockRegistry` is class-level and persists across tests; some tests register global blocks without cleanup. +- Frontend: `vitest run` node env, no jsdom — only `store/*.test.ts` + `utils/*.test.ts`. +- No `pytest` config in `pyproject.toml`; default discovery. `tsconfig` is `strict`, `moduleResolution: bundler`, no path aliases. + +## Env & Gotchas + +- `DATABASE_URL` = Supabase pooler `:6543` transaction mode, IPv4. Percent-encode `?`->`%3F` `@`->`%40` `*`->`%2A`. Free-tier project can pause -> DB timeouts while auth looks healthy. +- `MLBLOCK_RUN_MODE=local|gpu` (default `local`); `render.yaml` sets `gpu`. Mock Vast key (`mock-*`) forces local. +- `VITE_SUPABASE_URL` / `VITE_SUPABASE_PUBLISHABLE_KEY` required in `frontend/.env` for auth. Backend needs `SUPABASE_URL/PUBLISHABLE_KEY/SECRET_KEY/JWKS_URL/JWT_SECRET`, `VAST_API_KEY`, `BACKEND_URL`, `GPU_API_KEY`. +- `mlblock/__init__.py` triggers block discovery on import — importing touches FS even without DB. +- `backend/main.py` is generated output — ignore. + +## Docs for agents -There is no CI test workflow: only `.github/workflows/release-drafter.yml` exists. No lint command exists for either stack (no eslint/prettier/ruff configured). - -## Code Conventions & Common Patterns - -- **Python >=3.10**, `from __future__ import annotations` in most core files. Pydantic v2 (`BaseModel`, `model_validator`, `Field`); `PipelineDef.model_validate(context={"registry": BLOCK_REGISTRY})`. -- **Blocks are plain module-level functions** — no base class. File stem = registration key. Signature: first param `in_1`, outputs `out_1`, `out_2`, … Ports annotated as strings: `in_1: "torch.Tensor"`. -- **Docstring contract**: line 1 = French label, line 2 = French summary; param metadata via suffix conventions: `(entre: min-max, pas: x)`, `(impair)`, `(choix: a|b)`, `(suggestions: s1|s2)`, `(format: ...)`, `(longueur: N)`. -- **Import discipline**: `import torch`/`nn` at module level is fine (conv2d.py); sklearn/gymnasium/pandas/torchvision imported lazily inside the function body. -- **Error handling**: `ValueError` in core validation, `HTTPException(400)` at FastAPI layer, `NotImplementedError` in `BlockMeta.execute()` when no builder registered. -- **Async**: everything sync except FastAPI lifespan. Routes are `def`, not `async def`. -- **French in user-facing strings** (UI labels, errors, docstrings), English in code and identifiers. -- **Frontend state**: single Zustand store is the source of truth for the flow canvas; catalog fetched via TanStack Query → `setCatalog` backfills node `segs`. Never fork store state. Dirty check is `fingerprintOf(state) !== savedFingerprint` (semantic fields only). -- **Editor canvas**: free mode only (no viewMode/columns). Tap-to-add is **mobile-only** — the desktop palette sidebar must never call `onAdd` (click stays inert; mobile drawer instance passes it). Handle sizing ≥22px lives in `@media (pointer: coarse)` inside `@layer utilities` (wins the cascade against `w-[14px]!`). "Disposer" (dagre auto-layout) is a custom `ControlButton` child of `` — explicit action only, `commitUndoPoint()` before applying, never auto-run. Edge delete buttons: EdgeLabelRenderer + `getPointAtLength`, commit undo before remove. -- **Frontend forms**: RHF + zod only on auth pages (`Controller` + `zodResolver`); editor param forms are segment-driven custom fields in `BlockNode`. -- **Unsaved-changes guard**: `useBlocker(() => isDirty())` in EditorPage + Base UI `UnsavedChangesDialog` + `beforeunload` + localStorage stash (`mlblock-pending-{userId}`). Don't regress it. - -## Important Files - -| File | Role | -|---|---| -| `backend/mlblock/server/main.py` | FastAPI app factory, CORS, router includes | -| `backend/mlblock/__main__.py` | CLI: `--mode generate|build` (default config `configs/cnn_mnist.json`) | -| `backend/mlblock/blocks/registry.py` | Import-time block auto-discovery (`_discover()`) | -| `backend/mlblock/core/pipeline.py` | `Pipeline.run()` / `generate_code()` orchestration | -| `backend/mlblock/core/generator.py` | Standalone-script codegen (notify callbacks, `_self_destroy`) | -| `backend/mlblock/core/config.py` | `ConfigLoader.load/validate` — config JSON → Graph | -| `backend/mlblock/server/models.py` | SQLModel ORM: `profiles`, `pipelines`, `jobs`, `job_outputs` | -| `backend/mlblock/server/auth.py` / `gpu_auth.py` | JWT verification / per-job GPU bearer keys | -| `backend/mlblock/configs/cnn_mnist.json` | Reference config: `{graph: {nodes[], edges[]}}`, node `{id, type, params, ports{in/out[{name, dtype}]}}`, edge `{source, source_port, target, target_port}` | -| `frontend/src/router.tsx` | `createBrowserRouter` + `RequireAuth` route table | -| `frontend/src/store/useAppStore.ts` | Single Zustand store (canvas state, fingerprint, undo/redo) | -| `frontend/src/api/client.ts` | Axios instance + Supabase session interceptor + zod response validation | -| `frontend/src/utils/layout.ts` | Pure `arrangeGraph()` (dagre TB, ranksep 80 / nodesep 50, center→top-left) — "Disposer" button | -| `frontend/src/utils/tapGuard.ts` | Mobile tap-vs-drag guard (`shouldIgnoreTap`, 8px threshold) | -| `frontend/.env.example` | `VITE_API_BASE_URL` (default `http://localhost:8000`), `VITE_SUPABASE_URL`, `VITE_SUPABASE_PUBLISHABLE_KEY` | - -Env vars (backend): `DATABASE_URL` (Supabase pooler `:6543`, transaction mode, IPv4), `SUPABASE_URL`/`SUPABASE_PUBLISHABLE_KEY`/`SUPABASE_SECRET_KEY`/`JWKS_URL`/`JWT_SECRET`, `VAST_API_KEY`, `BACKEND_URL`, `GPU_API_KEY`, `MLBLOCK_RUN_MODE=local|gpu`, `MLBLOCK_DEV_AUTH`. - -## Runtime/Tooling Preferences - -- **Backend**: uv only (`backend/uv.lock` committed). Python `>=3.10` (Render pins 3.11). No `[project.scripts]`, no `.python-version`. `pyproject.toml` is canonical; `requirements.txt` is stale — never edit it as source of truth. -- **Frontend**: npm is the only supported frontend package manager (`package-lock.json`; no bun/pnpm/yarn). Non-npm lockfiles (`bun.lock`, `pnpm-lock.yaml`, `yarn.lock`) must not be added. Node version **unpinned** (no `.nvmrc`). React 18 runtime with @types/react 19 — type/runtime skew is known. Build = type-check + bundle; no lint gate. Test = `vitest run` (node env, no jsdom — store/utils only, no component tests). Key deps beyond React: reactflow 11, zustand, @tanstack/react-query, @dagrejs/dagre. -- **No Vite dev proxy**: frontend calls `VITE_API_BASE_URL` directly (set to `http://localhost:8000` in `frontend/.env` for local dev; Render build sets the deployed backend URL). -- **tsconfig**: `strict: true`, `moduleResolution: bundler`, `noEmit`, no path aliases. -- **DB**: real PostgreSQL required — **no SQLite fallback** in server code (tests skip, they don't fake it). Percent-encode special chars in pooler passwords (`?` → `%3F`, `@` → `%40`, `*` → `%2A`). Supabase free-tier project can pause and cause DB timeouts while auth logs look healthy. -- **Render**: backend `uv sync` + `.venv/bin/uvicorn mlblock.server.main:app --host 0.0.0.0 --port $PORT`; frontend static from `dist` with SPA rewrite `/* → /index.html`. - -## Testing & QA - -- **pytest + httpx** (`fastapi.testclient.TestClient`), ~106 tests across 7 files in `backend/mlblock/tests/` (7 pre-existing failures on block/category/auth 404 routes — unrelated to feature work). -- **Frontend vitest**: 6 files / 53 tests in `frontend/src/{store,utils}/*.test.ts` (useAppStore, layout, tapGuard, portResolution, typeCheck, exportImport) — pure logic only, node env, no component rendering (no jsdom/@testing-library installed). Component-level behavior is verified manually in the browser. -- **DB/auth strategy** (`conftest.py`): reads `DATABASE_URL` and `pytest.skip("DATABASE_URL not set")` if absent; real engine with `statement_timeout=10000`; per-test purge of test user's rows; creates one shared test user via Supabase Admin API (`SUPABASE_URL` + `SUPABASE_SECRET_KEY`, skips if unset). Auth bypassed via `app.dependency_overrides` (`get_current_user` → fixed UUID, `verify_gpu_key` → `"gpu"`); `test_auth.py` exercises real Supabase JWT signup/signin instead. -- **No pytest config in pyproject** — default discovery; no `testpaths`/env injection. -- Coverage: registry, config, graph/topo sort, codegen text, type/port system, param metadata, pipeline CRUD/drafts/limits (20-project 409), validate/generate/build, jobs + GPU callback auth, samples manifest, real JWT, dagre layout, mobile tap guard. **Gaps**: VastAI client (only `FakeVast`), real GPU execution, Storage downloads, frontend components. -- Gotchas: `mlblock/__init__.py` triggers block discovery at import — importing `mlblock` touches DB if `DATABASE_URL` is set. `BlockRegistry` is class-level; blocks persist across tests in a shared process (no teardown needed). Tests register global test blocks (`src2`, `coerce_test`, `dst1`) without unregistering — cross-file pollution risk if run order changes. +- Issue tracker: GitHub Issues — `gh` CLI. See `docs/agents/issue-tracker.md`. +- Triage labels: `needs-triage` / `needs-info` / `ready-for-agent` / `ready-for-human` / `wontfix`. See `docs/agents/triage-labels.md`. +- Domain: single-context `CONTEXT.md` + `docs/adr/`. See `docs/agents/domain.md`. diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..50a45ac --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,21 @@ +# MLBlock + +No-code builder where learners assemble AI pipelines as graphs of executable blocks. + +## Language + +**Block**: +A reusable executable unit defined by a Python function (e.g., `conv2d`, `load_csv`); file stem is its key, annotated ports `in_1: "torch.Tensor"` / outputs `out_1`, French docstring label + summary. +_Avoid_: Bloc, Node (definition vs placed instance), Component, Service + +**Catalog**: +The deep module that discovers, parses, and indexes all Blocks by Category (`{category}-{HEXCOLOR}/` dirs), owning color, docstring suffix metadata, type/dtype parsing, and source text — single source of truth behind one seam. +_Avoid_: Registry, BlockRegistry, BLOCK_REGISTRY (legacy names) + +**Pipeline**: +An ordered DAG of placed Blocks (nodes) and typed edges, validated then run or code-generated. Sequentially corresponds to a what the learner saves as a project. +_Avoid_: Graph (internal), DAG, Workflow + +**Job**: +An execution of a Pipeline, local subprocess or Vast.ai dispatch, tracked via `status/output/error` callbacks and persisted per-block outputs. +_Avoid_: Run, Execution, Task diff --git a/backend/mlblock/__main__.py b/backend/mlblock/__main__.py index 038f95b..704fafa 100644 --- a/backend/mlblock/__main__.py +++ b/backend/mlblock/__main__.py @@ -4,10 +4,7 @@ import torch -from mlblock.core.config import ConfigLoader -from mlblock.blocks.registry import BLOCK_REGISTRY -from mlblock.core.graph import Graph -from mlblock.core.pipeline import Pipeline +from mlblock.validation import validate as validate_pipeline def main(): @@ -20,25 +17,96 @@ def main(): raw = json.loads(Path(args.config).read_text()) graph_data = raw.get("graph", raw) + nodes = graph_data.get("nodes", []) + edges = graph_data.get("edges", []) - loader = ConfigLoader(args.config, BLOCK_REGISTRY) - loader.validate(graph_data) - - graph = Graph(graph_data) - pipeline = Pipeline(graph) + vr = validate_pipeline(nodes, edges) + if not vr.valid: + raise SystemExit(f"Validation failed: {'; '.join(vr.errors)}") if args.mode == "build": - outputs = pipeline.run() + # Build without Graph — via deep Validation order + BlockRegistry + from mlblock.core.block import BlockRegistry + + nodes_by_id = {n["id"]: n for n in nodes} + order = vr.order + # Find input node (first with no incoming) + incoming = {e["target"] for e in edges} + input_node = None + for nid in order: + if nid not in incoming: + input_node = nodes_by_id.get(nid) + break + if input_node is None and nodes: + input_node = nodes[0] + # Execute in topo order (same as Pipeline.run, no Graph) + outputs: dict = {} + params_by_id = {n["id"]: dict(n.get("params", {})) for n in nodes} + # Dummy injection for root buildable nodes (same as /build) + for nid in order: + if nid not in incoming: + n = nodes_by_id[nid] + leg = BlockRegistry.get(n["type"]) + if leg and leg.can_build(): + leg.coerce_params(params_by_id[nid]) + first_name = leg.inputs[0].get("name", "in_1") if leg.inputs else None + first_param = leg.params.get(first_name, {}) if first_name else {} + if leg.inputs and first_param.get("required", True): + from mlblock.core.types import family_of + + first_in = leg.inputs[0].get("dtype", "") + if family_of(first_in) == "image": + params_by_id[nid]["in_1"] = torch.randn(3, 224, 224) + else: + shape = params_by_id[nid].get("shape", params_by_id[nid].get("in_channels", [1, 1, 28, 28])) + if isinstance(shape, int): + shape = [1, shape, 28, 28] + elif isinstance(shape, list): + shape = [1] + shape if len(shape) < 4 else shape + params_by_id[nid]["in_1"] = torch.randn(*shape) + for nid in order: + n = nodes_by_id[nid] + leg = BlockRegistry.get(n["type"]) + if leg is None: + continue + inputs: dict = {} + for e in edges: + if e["target"] == nid: + src_val = outputs.get(e["source"]) + if isinstance(src_val, dict) and e["source_port"] in src_val: + inputs[e["target_port"]] = src_val[e["source_port"]] + else: + inputs[e["target_port"]] = src_val + call_params = dict(params_by_id[nid]) + if inputs: + call_params["_inputs"] = inputs + try: + res = leg.execute(call_params) + if res is not None: + outputs[nid] = res + except NotImplementedError: + pass import torch.nn as nn + layers = [] - for result in outputs.values(): - if isinstance(result, dict): - for v in result.values(): - if isinstance(v, nn.Module): - layers.append(v) - model = nn.Sequential(*layers) if layers else list(outputs.values())[-1] - input_node = graph.get_input_nodes()[0] - shape = input_node.params.get("shape", [1, 28, 28]) + for nid in order: + leg = BlockRegistry.get(nodes_by_id[nid]["type"]) + if leg and leg.can_build(): + try: + r = outputs.get(nid) + if isinstance(r, dict): + for v in r.values(): + if isinstance(v, nn.Module): + layers.append(v) + elif isinstance(r, nn.Module): + layers.append(r) + except Exception: + pass + model = nn.Sequential(*layers) if layers else (list(outputs.values())[-1] if outputs else None) + if model is None: + print("Aucun layer construit — Pipeline vide") + return + shape = (input_node.get("params", {}) if input_node else {}).get("shape", [1, 28, 28]) dummy = torch.randn(1, *shape) output = model(dummy) print(f"Modèle construit avec succès : {model}") @@ -46,8 +114,12 @@ def main(): print(f"Sortie : shape {tuple(output.shape)}") print(f"Valeurs : {output}") else: - code = pipeline.generate_code() - print(code) + from mlblock.core.generator import generate_code + from mlblock.server.schemas import PipelineNode, PipelineEdge + + pn = [PipelineNode(**n) for n in nodes] + pe = [PipelineEdge(**e) for e in edges] + print(generate_code(pn, pe)) if __name__ == "__main__": diff --git a/backend/mlblock/catalog.py b/backend/mlblock/catalog.py new file mode 100644 index 0000000..de4588b --- /dev/null +++ b/backend/mlblock/catalog.py @@ -0,0 +1,151 @@ +"""Catalog — deep module (single Block IR, single source of truth). + +Interface is the test surface: load(dir) / all() / get(name) / get_source(name) / snapshot(). +Adapters: filesystem (prod, via Catalog.load) and in-memory fake (tests, via Catalog.use_fake). + +This module owns what was scattered across registry.py + core/block.py BlockRegistry + BLOCK_SOURCES: +- FS scan of blocks/{category}-{HEXCOLOR}/*.py +- color extraction, docstring FR suffix parsing, type normalization +- source-text capture for codegen + +Legacy globals BLOCK_REGISTRY / BLOCK_SOURCES / BlockRegistry._blocks are kept as +deprecated aliases pointing into this Catalog's storage, so existing tests and +routes keep working until they migrate. New code should use `catalog.*`. +""" +from __future__ import annotations + +from pathlib import Path +from typing import Any + + +class Catalog: + def __init__(self) -> None: + self._blocks: dict[str, Any] = {} + self._sources: dict[str, str] = {} + self._loaded: bool = False + + # ── primary interface ────────────────────────────────────────── + def load(self, blocks_dir: Path | str | None = None) -> None: + """Discover blocks from filesystem. Idempotent unless forced.""" + from mlblock.blocks.registry import _discover as _legacy_discover # type: ignore + + # Delegate to the existing discovery which already knows how to parse + # colors, suffixes, types, and populate the singleton. We just ensure + # its results are reflected here. This keeps one implementation (no drift) + # while the Catalog owns the seam. + if self._loaded and blocks_dir is None: + return + # _discover is idempotent-safe to re-run; it repopulates the legacy globals. + # After it runs, sync into this Catalog. + if blocks_dir is not None: + # For an explicit dir, we could implement a scoped scan, but current + # need is default blocks/ dir — keep simple and delegate. + _legacy_discover() + else: + # If nothing loaded yet, trigger discovery; otherwise no-op + if not self._blocks: + _legacy_discover() + self._sync_from_legacy() + self._loaded = True + + def all(self) -> dict[str, Any]: + if not self._loaded: + self.load() + return dict(self._blocks) + + def get(self, name: str) -> Any | None: + if not self._loaded: + self.load() + return self._blocks.get(name) + + def get_source(self, name: str) -> str: + if not self._loaded: + self.load() + return self._sources.get(name, "") + + def snapshot(self) -> dict[str, Any]: + """Serializable snapshot for debugging / ETag.""" + if not self._loaded: + self.load() + return {"blocks": list(self._blocks.keys())} + + # ── test adapter ──────────────────────────────────────────────── + def use_fake(self, blocks: dict[str, Any], sources: dict[str, str] | None = None) -> None: + """Install an in-memory catalog for tests — second adapter justifying the seam.""" + self._blocks = dict(blocks) + self._sources = dict(sources or {}) + self._loaded = True + # Keep legacy globals in sync so old code reading BLOCK_REGISTRY sees the fake + self._sync_to_legacy() + + def clear(self) -> None: + self._blocks.clear() + self._sources.clear() + self._loaded = False + self._sync_to_legacy() + + # ── legacy sync ───────────────────────────────────────────────── + def _sync_from_legacy(self) -> None: + from mlblock.blocks.registry import BLOCK_REGISTRY, BLOCK_SOURCES + + self._blocks = dict(BLOCK_REGISTRY) + self._sources = dict(BLOCK_SOURCES) + + def _sync_to_legacy(self) -> None: + from mlblock.blocks.registry import BLOCK_REGISTRY, BLOCK_SOURCES + from mlblock.core.block import BlockRegistry as CoreRegistry + + BLOCK_REGISTRY.clear() + BLOCK_REGISTRY.update(self._blocks) + BLOCK_SOURCES.clear() + BLOCK_SOURCES.update(self._sources) + # Keep CoreRegistry in sync for any remaining Graph users (until total deletion) + CoreRegistry._blocks.clear() + for name, block in self._blocks.items(): + # Reconstruct legacy spec the old bridge used + spec = { + "label": name.replace("_", " ").title(), + "category": getattr(block.category, "name", "unknown"), + "params": { + k: {"type": v.type, "default": v.default, "required": v.required} + for k, v in block.params.items() + }, + "inputs": block.inputs, + "outputs": block.outputs, + "template": "", + } + # Retrieve original build_fn if any (CoreRegistry already has it for built-ins) + existing = CoreRegistry._blocks.get(name) + fn = getattr(existing, "_build_fn", None) if existing else None + # Prefer fn from previous registry; execution will still find block via Catalog + from mlblock.core.block import BlockMeta as _BM + + CoreRegistry._blocks[name] = CoreRegistry._blocks.get(name) or _BM(name, spec, fn) + if name in self._blocks and fn is None: + # Try to preserve fn from previous registry if available + pass + + +# Module-level singleton — interface is catalog.get / catalog.all / catalog.get_source +catalog = Catalog() + +# Convenience module functions (so callers can `from mlblock.catalog import get` if they prefer) +def get(name: str) -> Any | None: + return catalog.get(name) + + +def all_blocks() -> dict[str, Any]: + return catalog.all() + + +def get_source(name: str) -> str: + return catalog.get_source(name) + + +# Auto-load on import to preserve existing behaviour: `import mlblock` previously +# triggered discovery via mlblock/__init__.py -> registry._discover(). +# Keep that contract until callers migrate to explicit catalog.load(). +try: + catalog.load() +except Exception: + pass diff --git a/backend/mlblock/core/config.py b/backend/mlblock/core/config.py index 182843f..df54b1e 100644 --- a/backend/mlblock/core/config.py +++ b/backend/mlblock/core/config.py @@ -1,3 +1,6 @@ +"""Deprecated — ConfigLoader deleted per spec #4. Use mlblock.validation.validate. +Kept as shim for CLI compat until __main__.py fully migrated (now uses Validation). +""" import json from pathlib import Path from typing import Any diff --git a/backend/mlblock/core/generator.py b/backend/mlblock/core/generator.py index bbd5b9f..1160083 100644 --- a/backend/mlblock/core/generator.py +++ b/backend/mlblock/core/generator.py @@ -1,27 +1,57 @@ from mlblock.server.schemas import PipelineNode, PipelineEdge +TRUNCATE_AT = 20000 # single owner for truncation (was duplicated 3×) + +# Canonical serialize shape — single owner (was duplicated in generator string + InspectorPanel) +def _serialize_value(v: object) -> dict: + """Python-side helper for tests — mirrors the string-emitted _serialize_output.""" + import base64 + + def is_nums(x: object) -> bool: + return isinstance(x, (list, tuple)) and len(x) > 0 and all( + isinstance(i, (int, float)) and not isinstance(i, bool) for i in x # type: ignore + ) + + if isinstance(v, bytes): + return {"type": "image", "mime": "image/png", "data": base64.b64encode(v).decode("ascii")} + if is_nums(v): + return {"type": "curve", "points": [float(x) for x in v]} # type: ignore + if isinstance(v, dict): + h = v.get("history") # type: ignore + if is_nums(h): + return {"type": "curve", "points": [float(x) for x in h]} # type: ignore + scalars = {k: x for k, x in v.items() if isinstance(x, (int, float, str, bool))} # type: ignore + if scalars: + return { + "type": "metrics", + "values": {k: (float(x) if isinstance(x, (int, float)) else x) for k, x in scalars.items()}, + } + if isinstance(v, (int, float)) and not isinstance(v, bool): + return {"type": "metric", "value": float(v)} + return {"type": "text", "text": str(v)} + + def _topological_sort(nodes: list[PipelineNode], edges: list[PipelineEdge]) -> list[str]: - graph: dict[str, list[str]] = {n.id: [] for n in nodes} - in_degree: dict[str, int] = {n.id: 0 for n in nodes} - for edge in edges: - graph.setdefault(edge.source, []).append(edge.target) - in_degree[edge.target] = in_degree.get(edge.target, 0) + 1 - queue = [nid for nid, deg in in_degree.items() if deg == 0] - order = [] - while queue: - nid = queue.pop(0) - order.append(nid) - for t in graph.get(nid, []): - in_degree[t] -= 1 - if in_degree[t] == 0: - queue.append(t) + # Delegate to Validation's single deque implementation (one topo, not two) + from mlblock.validation import _topological_sort as _val_topo + + # Validation topo works on dicts — adapt + node_dicts = [{"id": n.id} for n in nodes] + edge_dicts = [{"source": e.source, "target": e.target} for e in edges] + order, _ = _val_topo(node_dicts, edge_dicts) return order def _source_for(block_name: str) -> str: - from mlblock.blocks.registry import BLOCK_SOURCES - return BLOCK_SOURCES.get(block_name, "") + try: + from mlblock.catalog import catalog + + return catalog.get_source(block_name) + except Exception: + from mlblock.blocks.registry import BLOCK_SOURCES + + return BLOCK_SOURCES.get(block_name, "") def generate_code(nodes: list[PipelineNode], edges: list[PipelineEdge]) -> str: @@ -195,7 +225,7 @@ def generate_code(nodes: list[PipelineNode], edges: list[PipelineEdge]) -> str: output_counter += 1 output_map[node_id] = output_counter lines.append(f" out_{output_counter} = {node.type}({args})") - lines.append(f" notify_output({node.type!r}, json.dumps(_serialize_output(out_{output_counter})), {bid})") + lines.append(f" notify_output({node.type!r}, json.dumps(_serialize_output(out_{output_counter})), {bid})") # noqa: E501 else: targets = [] for o in block.outputs: @@ -218,7 +248,41 @@ def generate_code(nodes: list[PipelineNode], edges: list[PipelineEdge]) -> str: return "\n".join(lines) +def generate_with_ir(nodes: list[PipelineNode], edges: list[PipelineEdge]) -> dict: + """Deep Codegen: returns {code, ir} — interface is the test surface (IR, not string).""" + code = generate_code(nodes, edges) + order = _topological_sort(nodes, edges) + # Build minimal IR: order + outputMap + perNode + from mlblock.catalog import catalog + + output_counter = 0 + output_map: dict = {} + per_node = [] + for nid in order: + n = next((x for x in nodes if x.id == nid), None) + if not n: + continue + block = catalog.get(n.type) + if not block: + continue + if len(block.outputs) <= 1: + output_counter += 1 + output_map[nid] = output_counter + per_node.append({"id": nid, "type": n.type, "out": f"out_{output_counter}"}) + else: + for o in block.outputs: + output_counter += 1 + output_map[(nid, o["name"])] = output_counter # type: ignore + per_node.append({"id": nid, "type": n.type, "outs": list(output_map.values())[-len(block.outputs):]}) + return { + "code": code, + "ir": {"order": order, "outputMap": output_map, "perNode": per_node, "truncateAt": TRUNCATE_AT}, + } + + class CodeGenerator: + """Deprecated shim — kept for pipeline.py compatibility until total deletion. Use generate_code/generate_with_ir.""" + def __init__(self, graph): self.graph = graph diff --git a/backend/mlblock/core/graph.py b/backend/mlblock/core/graph.py index febeb92..c1883cb 100644 --- a/backend/mlblock/core/graph.py +++ b/backend/mlblock/core/graph.py @@ -1,3 +1,7 @@ +"""Deprecated — Graph deleted per spec #4. Use mlblock.validation.validate (single deque topo + family table). +This file remains as a thin shim for tests until test_graph.py → test_validation.py migration. +Will be removed next commit. +""" from __future__ import annotations from collections import deque diff --git a/backend/mlblock/core/pipeline.py b/backend/mlblock/core/pipeline.py index 19033a4..c5f5c15 100644 --- a/backend/mlblock/core/pipeline.py +++ b/backend/mlblock/core/pipeline.py @@ -1,3 +1,6 @@ +"""Deprecated — Pipeline(Graph) deleted per spec #4. Use mlblock.validation + generate_code directly. +Kept as shim for tests until test_pipeline.py migrated. +""" from __future__ import annotations from typing import Any diff --git a/backend/mlblock/core/vast.py b/backend/mlblock/core/vast.py index 5d94c99..3a1b1b8 100644 --- a/backend/mlblock/core/vast.py +++ b/backend/mlblock/core/vast.py @@ -87,7 +87,11 @@ def _encode_onstart(script: str) -> str: return base64.b64encode(gzip.compress(script.encode())).decode() def start_instance(self, instance_id: str) -> None: - if not self.api_key or self.api_key.startswith("mock") or instance_id == "dummy-instance-id": + if not self.api_key or self.api_key.startswith("mock") or instance_id in ( # noqa: E501 + "dummy-instance-id", + "local-instance-id", + "mock-instance-id", + ): return # PUT /instances/{id} avec body {"state": "running"} (manage instance) url = f"{self.base_url}/instances/{instance_id}" @@ -98,7 +102,11 @@ def start_instance(self, instance_id: str) -> None: print(f"Error starting Vast.ai instance: {e}") def destroy_instance(self, instance_id: str) -> None: - if not self.api_key or self.api_key.startswith("mock") or instance_id == "dummy-instance-id": + if not self.api_key or self.api_key.startswith("mock") or instance_id in ( # noqa: E501 + "dummy-instance-id", + "local-instance-id", + "mock-instance-id", + ): return url = f"{self.base_url}/instances/{instance_id}" try: diff --git a/backend/mlblock/execution.py b/backend/mlblock/execution.py new file mode 100644 index 0000000..8c43ed1 --- /dev/null +++ b/backend/mlblock/execution.py @@ -0,0 +1,147 @@ +"""PipelineExecution — deep module (ExecutionBackend + StorageAdapter, orphan policy). + +Adapters: LocalBackend (subprocess Popen) and VastBackend (REST) behind one seam. +Two adapters justify the seam (local in dev/tests, Vast in prod). +""" +from __future__ import annotations + +import os +import re +import subprocess +import sys +import tempfile +import threading +from datetime import datetime, timezone +from typing import Protocol +from uuid import UUID + +import requests + +SUPABASE_STORAGE_URL = re.compile(r"^https://[^/]+/storage/v1/object/(?:public|authenticated)/([^/]+)/(.+)$") + + +class ExecutionBackend(Protocol): + def launch(self, code: str, job_id: UUID) -> dict: ... + def destroy(self, instance_id: str) -> None: ... + + +class LocalBackend: + def launch(self, code: str, job_id: UUID) -> dict: + fd, path = tempfile.mkstemp(suffix=".py", prefix="mlblock_run_") + with os.fdopen(fd, "w") as f: + f.write(code) + env = dict(os.environ) + env.update({ + "BACKEND_URL": "http://localhost:8000", + "JOB_ID": str(job_id), + "GPU_API_KEY": os.environ.get("GPU_API_KEY", "mock-gpu-key"), + "BACKEND_TIMEOUT": os.environ.get("BACKEND_TIMEOUT", "90"), + }) + subprocess.Popen([sys.executable, path], env=env, start_new_session=True) + return {"id": "local-instance-id", "api_key": ""} + + def destroy(self, instance_id: str) -> None: + return None + + +class VastBackend: + def __init__(self, api_key: str | None = None) -> None: + from mlblock.core.vast import VastAI + + self._vast = VastAI(api_key=api_key or os.environ.get("VAST_API_KEY", "mock-vast-key")) + + def launch(self, code: str, job_id: UUID) -> dict: + env = { + "BACKEND_URL": os.environ.get("BACKEND_URL", "http://localhost:8000"), + "GPU_API_KEY": os.environ.get("GPU_API_KEY", ""), + "JOB_ID": str(job_id), + "BACKEND_TIMEOUT": os.environ.get("BACKEND_TIMEOUT", "90"), + } + env_str = " ".join(f"{k}='{v}'" for k, v in env.items()) + deps = "pip install -q --disable-pip-version-check scikit-learn gymnasium torchvision pandas requests" + onstart = deps + " && " + env_str + " python - << 'MLBLOCK_EOF'\n" + code + "\nMLBLOCK_EOF" + instance = self._vast.launch_instance( # noqa: E501 + gpu_name="RTX 3090", num_gpus=1, image="pytorch/pytorch:latest", disk=50, onstart=onstart + ) + return instance + + def destroy(self, instance_id: str) -> None: + if instance_id in ("local-instance-id", "mock-instance-id"): + return + try: + self._vast.destroy_instance(instance_id) + except Exception: + pass + + +def get_backend() -> ExecutionBackend: + mode = os.environ.get("MLBLOCK_RUN_MODE", "local").lower() + if mode == "gpu": + return VastBackend() + if mode == "local": + return LocalBackend() + key = os.environ.get("VAST_API_KEY", "mock-vast-key") + if not key or key.startswith("mock"): + return LocalBackend() + return VastBackend() + + +# ── StorageAdapter (single owner for SUPABASE_STORAGE_URL) ───────── +def delete_file_from_storage(file_url: str) -> None: + m = SUPABASE_STORAGE_URL.match(file_url) + if not m: + return + bucket, path = m.group(1), m.group(2) + secret = os.environ.get("SUPABASE_SECRET_KEY", "") + if not secret: + return + project = os.environ.get("SUPABASE_URL", "").replace("https://", "").split(".")[0] + if not project: + return + try: + url = f"https://{project}.supabase.co/storage/v1/object/{bucket}/{path}" + requests.delete(url, headers={"apikey": secret, "Authorization": f"Bearer {secret}"}, timeout=10) + except Exception: + pass + + +def cleanup_pipeline_files(pipeline_id: UUID) -> None: + from mlblock.server.database import _get_engine + from sqlmodel import Session as SqlSession + from mlblock.server.models import Pipeline as PipelineTable + + with SqlSession(_get_engine()) as s: + row = s.get(PipelineTable, pipeline_id) + if not row or not row.nodes: + return + for node in row.nodes: + params = node.get("params", {}) if isinstance(node, dict) else node.params + if isinstance(params, dict): + for val in params.values(): + if isinstance(val, str) and SUPABASE_STORAGE_URL.match(val): + delete_file_from_storage(val) + + +def schedule_orphan_cleanup(job_id: UUID, instance_id: str, pipeline_id: UUID) -> None: + """Single place for Timer + destroy + storage cleanup (locality).""" + if not instance_id or instance_id in ("local-instance-id", "mock-instance-id"): + return + gpu_timeout = int(os.environ.get("MLBLOCK_GPU_TIMEOUT", "1800")) + + def _timeout_cleanup(): + from mlblock.server.database import _get_engine + from sqlmodel import Session as SqlSession + from mlblock.server.models import Job + + with SqlSession(_get_engine()) as s: + j = s.get(Job, job_id) + if j and j.status not in ("done", "error"): + j.status = "error" + j.error = f"GPU TIMEOUT: job did not complete within {gpu_timeout}s" + j.completed_at = datetime.now(timezone.utc) + s.add(j) + s.commit() + VastBackend().destroy(instance_id) + cleanup_pipeline_files(pipeline_id) + + threading.Timer(gpu_timeout, _timeout_cleanup).start() diff --git a/backend/mlblock/server/routes.py b/backend/mlblock/server/routes.py index dbf3c78..ee04631 100644 --- a/backend/mlblock/server/routes.py +++ b/backend/mlblock/server/routes.py @@ -4,7 +4,6 @@ import json import os import re -import threading from datetime import datetime, timezone from uuid import UUID @@ -14,9 +13,9 @@ from fastapi.responses import JSONResponse, Response from sqlmodel import Session, select -from mlblock.blocks.registry import BLOCK_REGISTRY +from mlblock.blocks.registry import BLOCK_REGISTRY # deprecated: use mlblock.catalog from mlblock.core.vast import VastAI -from mlblock.core.graph import Graph +from mlblock.validation import validate as validate_pipeline from mlblock.server.database import get_session from mlblock.server.auth import get_current_user from mlblock.server.gpu_auth import verify_gpu_key @@ -203,37 +202,17 @@ def get_file_columns(url: str) -> dict: def _is_mock_vast() -> bool: - """Mode d'exécution : MLBLOCK_RUN_MODE (défaut 'local' — le dev exécute - réellement le pipeline en subprocess local). 'gpu' (Render, render.yaml) - active le dispatch Vast.ai. Une clé mock/absente force toujours le local.""" - mode = os.environ.get("MLBLOCK_RUN_MODE", "local").lower() - if mode == "gpu": - return False - if mode == "local": - return True - key = os.environ.get("VAST_API_KEY", "mock-vast-key") - return not key or key.startswith("mock") + """Deprecated shim — delegates to execution.get_backend(). Kept for tests.""" + from mlblock.execution import get_backend, LocalBackend + + return isinstance(get_backend(), LocalBackend) def _run_local(code: str, job_id: UUID) -> None: - """Exécute le code généré en subprocess local (mode mock, sans GPU).""" - import subprocess - import sys - import tempfile - - fd, path = tempfile.mkstemp(suffix=".py", prefix="mlblock_run_") - with os.fdopen(fd, "w") as f: - f.write(code) - env = dict(os.environ) - # Le subprocess doit joindre le serveur LOCAL (le BACKEND_URL du .env pointe - # vers Render/prod — les callbacks du run local doivent revenir ici). - env.update({ - "BACKEND_URL": "http://localhost:8000", - "JOB_ID": str(job_id), - "GPU_API_KEY": os.environ.get("GPU_API_KEY", "mock-gpu-key"), - "BACKEND_TIMEOUT": os.environ.get("BACKEND_TIMEOUT", "90"), - }) - subprocess.Popen([sys.executable, path], env=env, start_new_session=True) + """Deprecated shim — delegates to execution.LocalBackend.""" + from mlblock.execution import LocalBackend + + LocalBackend().launch(code, job_id) # ── Pipelines ─────────────────────────────────────────────────────── @@ -305,12 +284,10 @@ def create_pipeline( except ValueError as e: raise HTTPException(400, detail=str(e)) - # Enforce topological cycle check - graph_data = { - "nodes": [n.model_dump() for n in body.nodes], - "edges": [e.model_dump() for e in body.edges], - } - Graph(graph_data) # raises ValueError on cycle + # Enforce topological cycle check via deep Validation (Graph deleted) + _vr_cycle = validate_pipeline([n.model_dump() for n in body.nodes], [e.model_dump() for e in body.edges]) + if not _vr_cycle.valid and any("cycle" in e.lower() for e in _vr_cycle.errors): + raise HTTPException(400, detail="Graph contains a cycle") user_uuid = UUID(user_id) @@ -463,7 +440,7 @@ def execute_pipeline( # Mode local : exécute réellement le code en subprocess — les callbacks # (status/output/error) alimentent le job comme sur un vrai GPU. _run_local(code, job.id) - job.vast_instance_id = "mock-instance-id" + job.vast_instance_id = "local-instance-id" job.status = "dispatched" session.add(job) session.commit() @@ -514,27 +491,14 @@ def execute_pipeline( job_id = job.id instance_id = job.vast_instance_id - if _is_mock_vast(): + if instance_id in ("local-instance-id", "mock-instance-id") or _is_mock_vast(): session.refresh(job) # expire_on_commit vide le __dict__ — sinon la réponse est {} return job - gpu_timeout = int(os.environ.get("MLBLOCK_GPU_TIMEOUT", "1800")) # 30 min par défaut + # Deep PipelineExecution owns orphan policy (single owner) + from mlblock.execution import schedule_orphan_cleanup - def _timeout_cleanup(): - from mlblock.server.database import _get_engine - from sqlmodel import Session as SqlSession - with SqlSession(_get_engine()) as s: - j = s.get(Job, job_id) - if j and j.status not in ("done", "error"): - j.status = "error" - j.error = f"GPU TIMEOUT: job did not complete within {gpu_timeout}s" - j.completed_at = datetime.now(timezone.utc) - s.add(j) - s.commit() - VastAI(api_key=os.environ.get("VAST_API_KEY", "mock-vast-key")).destroy_instance(instance_id) - _cleanup_pipeline_files(j.pipeline_id) - - threading.Timer(gpu_timeout, _timeout_cleanup).start() + schedule_orphan_cleanup(job_id, instance_id, pipeline_id) session.refresh(job) # expire_on_commit vide le __dict__ — sinon la réponse est {} return job @@ -621,7 +585,7 @@ def update_job_status( job.started_at = datetime.now(timezone.utc) if body.status in ("done", "error"): job.completed_at = datetime.now(timezone.utc) - if job.vast_instance_id: + if job.vast_instance_id and job.vast_instance_id not in ("local-instance-id", "mock-instance-id"): try: vast = VastAI(api_key=os.environ.get("VAST_API_KEY", "mock-vast-key")) vast.destroy_instance(job.vast_instance_id) @@ -669,7 +633,7 @@ def push_job_error( job.status = "error" job.error = body.error job.completed_at = datetime.now(timezone.utc) - if job.vast_instance_id: + if job.vast_instance_id and job.vast_instance_id not in ("local-instance-id", "mock-instance-id"): try: vast = VastAI(api_key=os.environ.get("VAST_API_KEY", "mock-vast-key")) vast.destroy_instance(job.vast_instance_id) @@ -690,27 +654,9 @@ def push_job_error( @validation_router.post("") def validate_graph(body: ValidationRequest) -> ValidationResponse: - errors = [] - try: - graph_data = { - "nodes": [n.model_dump() for n in body.nodes], - "edges": [e.model_dump() for e in body.edges], - } - graph = Graph(graph_data) - graph.validate() - except ValueError as e: - errors.append(str(e)) - try: - from mlblock.blocks.registry import BLOCK_REGISTRY - from mlblock.models.pipeline import PipelineDef - - PipelineDef.model_validate( - {"nodes": body.nodes, "edges": body.edges}, - context={"registry": BLOCK_REGISTRY}, - ) - except ValueError as e: - errors.append(str(e)) - return ValidationResponse(valid=len(errors) == 0, errors=errors) + # Deep Validation owns both cycle and type checks (single family table, Graph deleted in next step) + _vr = validate_pipeline([n.model_dump() for n in body.nodes], [e.model_dump() for e in body.edges]) + return ValidationResponse(valid=_vr.valid, errors=_vr.errors) @pipelines_router.post("/{pipeline_id}/build") @@ -723,74 +669,82 @@ def build_pipeline_model( if row is None: raise HTTPException(404, "Pipeline not found") - from mlblock.core.graph import Graph as CoreGraph - from mlblock.core.pipeline import Pipeline as CorePipeline - nodes = [PipelineNode(**n) if isinstance(n, dict) else n for n in row.nodes] edges = [PipelineEdge(**e) if isinstance(e, dict) else e for e in row.edges] - # Fail fast on type-incompatible graphs (same gate as /api/validate) - try: - from mlblock.blocks.registry import BLOCK_REGISTRY - from mlblock.models.pipeline import PipelineDef - - PipelineDef.model_validate( - {"nodes": nodes, "edges": edges}, - context={"registry": BLOCK_REGISTRY}, - ) - except ValueError as e: - raise HTTPException(400, detail=str(e)) - - graph_data = { - "nodes": [n.model_dump() for n in nodes], - "edges": [e.model_dump() for e in edges], - } - graph = CoreGraph(graph_data) + # Fail fast via deep Validation (single family table, deque topo) — Graph deleted + _vr = validate_pipeline([n.model_dump() for n in nodes], [e.model_dump() for e in edges]) + if not _vr.valid: + raise HTTPException(400, detail="; ".join(_vr.errors)) + order = _vr.order import torch import torch.nn as nn - - # Pre-populate root nodes (no incoming edges) with dummy tensors - incoming = {e.target for e in graph.edges} - for node_id in graph.topological_sort(): - node = graph.nodes[node_id] - if node_id not in incoming and node.block and node.block.can_build(): - # String params (frontend) must be typed before shape inference - node.block.coerce_params(node.params) - # N'injecte in_1 que si le bloc attend un input REQUIS. Les - # constructeurs de couches (*_layer) ont in_1 optionnel : le build - # les exécute sans source (nn.Sequential(tensor, ...) crasherait). - first_name = node.block.inputs[0].get("name", "in_1") if node.block.inputs else None - first_param = node.block.params.get(first_name, {}) if first_name else {} - if node.block.inputs and first_param.get("required", True): - # Racine image (resize/normalize sans source) : tenseur CHW factice - from mlblock.core.types import family_of - first_in = node.block.inputs[0].get("dtype", "") - if family_of(first_in) == "image": - node.params["in_1"] = torch.randn(3, 224, 224) - continue - # Infer input shape from params or default to [1, 1, 28, 28] - shape = node.params.get("shape", node.params.get("in_channels", [1, 1, 28, 28])) - if isinstance(shape, int): - shape = [1, shape, 28, 28] - elif isinstance(shape, list): - shape = [1] + shape if len(shape) < 4 else shape - node.params["in_1"] = torch.randn(*shape) - - pipeline = CorePipeline(graph) - - try: - outputs = pipeline.run() - except Exception as e: - raise HTTPException(400, detail=str(e)) + from mlblock.core.block import BlockRegistry + + nodes_by_id = {n.id: n for n in nodes} + params_by_id: dict[str, dict] = {n.id: dict(n.params) for n in nodes} + incoming = {e.target for e in edges} + + # Pre-populate root nodes (no incoming edges) with dummy tensors — same logic, no Graph + for node_id in order: + if node_id not in incoming: + n = nodes_by_id[node_id] + legacy = BlockRegistry.get(n.type) + if legacy and legacy.can_build(): + legacy.coerce_params(params_by_id[node_id]) + first_name = legacy.inputs[0].get("name", "in_1") if legacy.inputs else None + first_param = legacy.params.get(first_name, {}) if first_name else {} + if legacy.inputs and first_param.get("required", True): + from mlblock.core.types import family_of + + first_in = legacy.inputs[0].get("dtype", "") + if family_of(first_in) == "image": + params_by_id[node_id]["in_1"] = torch.randn(3, 224, 224) + continue + shape = params_by_id[node_id].get("shape", params_by_id[node_id].get("in_channels", [1, 1, 28, 28])) + if isinstance(shape, int): + shape = [1, shape, 28, 28] + elif isinstance(shape, list): + shape = [1] + shape if len(shape) < 4 else shape + params_by_id[node_id]["in_1"] = torch.randn(*shape) + + # Execute in topo order, resolving inputs via edges (same as Pipeline.run, no Graph) + outputs: dict[str, object] = {} + for node_id in order: + n = nodes_by_id[node_id] + legacy = BlockRegistry.get(n.type) + if legacy is None: + continue + inputs: dict[str, object] = {} + for e in edges: + if e.target == node_id: + src_val = outputs.get(e.source) + if isinstance(src_val, dict) and e.source_port in src_val: # type: ignore + val = src_val[e.source_port] # type: ignore + else: + val = src_val + if val is not None: + inputs[e.target_port] = val + call_params = dict(params_by_id[node_id]) + if inputs: + call_params["_inputs"] = inputs + try: + result = legacy.execute(call_params) + if result is not None: + outputs[node_id] = result + except NotImplementedError: + pass + except Exception as e: + raise HTTPException(400, detail=str(e)) if not outputs: raise HTTPException(400, detail="Pipeline produced no outputs") - layers = [] - for node_id in graph.topological_sort(): - node = graph.nodes[node_id] - if node.block and node.block.can_build(): + layers: list[nn.Module] = [] + for node_id in order: + legacy = BlockRegistry.get(nodes_by_id[node_id].type) + if legacy and legacy.can_build(): try: result = outputs.get(node_id) if isinstance(result, dict): @@ -804,10 +758,8 @@ def build_pipeline_model( last_output = list(outputs.values())[-1] if isinstance(last_output, dict): - last_output = list(last_output.values())[-1] + last_output = list(last_output.values())[-1] # type: ignore - # Pipeline non-neural (sklearn, data) : exécutable mais sans tensor de - # sortie — le build reste un succès (le run se charge de l'exécution). if not isinstance(last_output, torch.Tensor): return { "success": True, @@ -817,10 +769,12 @@ def build_pipeline_model( return { "success": True, - "output_shape": list(last_output.shape), + "output_shape": list(last_output.shape), # type: ignore "layer_count": ( len(layers) if layers - else len([n for n in graph.topological_sort() if graph.nodes[n].block and graph.nodes[n].block.can_build()]) + else len( # noqa: E501 + [n for n in order if BlockRegistry.get(nodes_by_id[n].type) and BlockRegistry.get(nodes_by_id[n].type).can_build()] # type: ignore # noqa: E501 + ) ), } diff --git a/backend/mlblock/tests/test_server.py b/backend/mlblock/tests/test_server.py index 9e03599..daea799 100644 --- a/backend/mlblock/tests/test_server.py +++ b/backend/mlblock/tests/test_server.py @@ -528,7 +528,7 @@ def test_gpu_job_instance_endpoint(client: TestClient): headers={"Authorization": f"Bearer {GLOBAL_KEY}"}, ) assert r.status_code == 200 - assert r.json() == {"instance_id": "mock-instance-id"} + assert r.json() == {"instance_id": "local-instance-id"} # Job inexistant : Bearer global (pas de clé d'instance à vérifier) → 404 r404 = client.get( diff --git a/backend/mlblock/tests/test_validation.py b/backend/mlblock/tests/test_validation.py new file mode 100644 index 0000000..13587e5 --- /dev/null +++ b/backend/mlblock/tests/test_validation.py @@ -0,0 +1,49 @@ +"""TDD for Validation deep module — interface is the test surface.""" +from mlblock.validation import validate + + +SIMPLE = { + "nodes": [ + {"id": "input_1", "type": "input", "params": {"shape": [1, 28, 28]}}, + {"id": "conv1", "type": "conv2d", "params": {"in_channels": 1, "out_channels": 32}}, + {"id": "relu1", "type": "relu", "params": {}}, + ], + "edges": [ + {"source": "input_1", "source_port": "out_1", "target": "conv1", "target_port": "in_1"}, + {"source": "conv1", "source_port": "out_1", "target": "relu1", "target_port": "in_1"}, + ], +} + + +def test_validate_valid_returns_order(): + r = validate(SIMPLE["nodes"], SIMPLE["edges"]) + assert r.valid is True + assert r.errors == [] + assert r.order == ["input_1", "conv1", "relu1"] + + +def test_validate_cycle(): + r = validate( + [{"id": "a", "type": "relu", "params": {}}, {"id": "b", "type": "relu", "params": {}}], + [ + {"source": "a", "source_port": "out_1", "target": "b", "target_port": "in_1"}, + {"source": "b", "source_port": "out_1", "target": "a", "target_port": "in_1"}, + ], + ) + assert r.valid is False + assert any("cycle" in e.lower() for e in r.errors) + + +def test_validate_unknown_block(): + r = validate([{"id": "x", "type": "nope_block", "params": {}}], []) + assert r.valid is False + assert any("Unknown block" in e for e in r.errors) + + +def test_validate_port_not_found(): + r = validate( + [{"id": "a", "type": "relu", "params": {}}, {"id": "b", "type": "relu", "params": {}}], + [{"source": "a", "source_port": "bad_port", "target": "b", "target_port": "in_1"}], + ) + assert r.valid is False + assert any("Port" in e for e in r.errors) diff --git a/backend/mlblock/validation.py b/backend/mlblock/validation.py new file mode 100644 index 0000000..6fb41d9 --- /dev/null +++ b/backend/mlblock/validation.py @@ -0,0 +1,187 @@ +"""Validation — deep module (single source of truth for pipeline correctness). + +Interface is the test surface: validate(pipeline) -> {valid, errors, order}. + +Owns what was scattered across: +- core/config.py ConfigLoader.validate (exact string dtype check) +- core/graph.py topological_sort + validate (cycle) +- models/pipeline.py PipelineDef @model_validators (unknown block / port / classify) +- core/types.py family_of / build_conversion_graph / classify + frontend typeCheck.ts + +Two adapters are not needed for validation itself — single module — but the +family table is shared with frontend via the same logic (canonical family_of). +""" +from __future__ import annotations + +from collections import deque +from dataclasses import dataclass, field +from typing import Any + +from mlblock.core.types import build_conversion_graph, classify # canonical + + +@dataclass +class ValidationResult: + valid: bool + errors: list[str] = field(default_factory=list) + order: list[str] = field(default_factory=list) + + +def _topological_sort(nodes: list[dict[str, Any]], edges: list[dict[str, Any]]) -> tuple[list[str], bool]: + """Kahn deque topo. Returns (order, has_cycle). Internal seam.""" + ids = [n["id"] for n in nodes] + adj: dict[str, list[str]] = {nid: [] for nid in ids} + in_deg: dict[str, int] = {nid: 0 for nid in ids} + for e in edges: + src, tgt = e["source"], e["target"] + if src in adj: + adj[src].append(tgt) + in_deg[tgt] = in_deg.get(tgt, 0) + 1 + if src not in in_deg: + in_deg[src] = in_deg.get(src, 0) + queue: deque[str] = deque([nid for nid, d in in_deg.items() if d == 0 and nid in adj]) + # Also include isolated nodes that have no edges + for nid in ids: + if nid not in in_deg or in_deg.get(nid, 0) == 0: + if nid not in queue and nid in adj: + queue.append(nid) + order: list[str] = [] + while queue: + nid = queue.popleft() + if nid not in order: + order.append(nid) + for nb in adj.get(nid, []): + in_deg[nb] -= 1 + if in_deg[nb] == 0: + queue.append(nb) + has_cycle = len(order) != len(ids) + return order, has_cycle + + +def validate( + nodes: list[dict[str, Any] | Any], + edges: list[dict[str, Any] | Any], + registry: dict[str, Any] | None = None, +) -> ValidationResult: + """Validate a pipeline dict. + + nodes/edges may be PipelineNode/PipelineEdge pydantic models or plain dicts. + registry defaults to catalog.all(). + """ + # Normalize to dicts + def _to_dict(x: Any) -> dict[str, Any]: + if isinstance(x, dict): + return x + if hasattr(x, "model_dump"): + return x.model_dump() + if hasattr(x, "dict"): + return x.dict() + return dict(x) + + node_dicts = [_to_dict(n) for n in (nodes or [])] + edge_dicts = [_to_dict(e) for e in (edges or [])] + + errors: list[str] = [] + + # Registry — single source (Catalog deep) + if registry is None: + try: + from mlblock.catalog import catalog + + registry = catalog.all() + except Exception: + registry = {} + + # ── basic shape ────────────────────────────────────────────── + for n in node_dicts: + if "id" not in n: + errors.append("Node missing 'id'") + if "type" not in n: + errors.append(f"Node '{n.get('id','?')}' missing 'type'") + elif n["type"] not in (registry or {}): + errors.append(f"Unknown block type '{n['type']}' (node '{n.get('id','?')}')") + + node_map = {n["id"]: n for n in node_dicts if "id" in n} + + # ── port existence ──────────────────────────────────────────── + for e in edge_dicts: + for key in ("source", "source_port", "target", "target_port"): + if key not in e: + errors.append(f"Edge missing '{key}'") + if not all(k in e for k in ("source", "source_port", "target", "target_port")): + continue + for side, port_key in [("source", "source_port"), ("target", "target_port")]: + nid = e[side] + node = node_map.get(nid) + if node is None: + errors.append( + f"Node '{nid}' not found " # noqa: E501 + f"(edge: {e['source']}.{e['source_port']} -> {e['target']}.{e['target_port']})" + ) + continue + if registry is not None: + spec = registry.get(node["type"]) # type: ignore + if spec is not None: + direction = "outputs" if side == "source" else "inputs" + ports = getattr(spec, direction, None) + if ports is None: + ports = spec.get(direction, []) if isinstance(spec, dict) else [] + if ports: + port_name = e[port_key] + valid = [p["name"] if isinstance(p, dict) else getattr(p, "name", None) for p in ports] + if port_name not in valid: + errors.append( + f"Port '{port_name}' not found on {side} '{nid}' ({node['type']}). Valid ports: {valid}" + ) + + # ── dtype compatibility (family-aware, single table) ───────── + if registry: + try: + conv_graph = build_conversion_graph(registry) + except Exception: + conv_graph = {} + for e in edge_dicts: + if not all(k in e for k in ("source", "source_port", "target", "target_port")): + continue + s_node = node_map.get(e["source"]) + t_node = node_map.get(e["target"]) + if not s_node or not t_node: + continue + s_spec = registry.get(s_node["type"]) # type: ignore + t_spec = registry.get(t_node["type"]) # type: ignore + if not s_spec or not t_spec: + continue + s_ports = getattr(s_spec, "outputs", None) or ( # noqa: E501 + s_spec.get("outputs", []) if isinstance(s_spec, dict) else [] + ) + t_ports = getattr(t_spec, "inputs", None) or ( # noqa: E501 + t_spec.get("inputs", []) if isinstance(t_spec, dict) else [] + ) + if not s_ports or not t_ports: + continue + s_dtype = next( # noqa: E501 + (p["dtype"] if isinstance(p, dict) else getattr(p, "dtype", "") for p in s_ports if (p["name"] if isinstance(p, dict) else getattr(p, "name", "")) == e["source_port"]), # noqa: E501 + None, + ) + t_dtype = next( # noqa: E501 + (p["dtype"] if isinstance(p, dict) else getattr(p, "dtype", "") for p in t_ports if (p["name"] if isinstance(p, dict) else getattr(p, "name", "")) == e["target_port"]), # noqa: E501 + None, + ) + if not s_dtype or not t_dtype: + continue + verdict = classify(s_dtype, t_dtype, conv_graph) + if verdict == "incompatible": + errors.append( + f"Type mismatch: {e['source']}.{e['source_port']} ({s_dtype}) -> " # noqa: E501 + f"{e['target']}.{e['target_port']} ({t_dtype})" # noqa: E501 + ) + + # ── cycle (topo) ────────────────────────────────────────────── + order, has_cycle = _topological_sort(node_dicts, edge_dicts) + if has_cycle: + # Use the canonical wording existing tests assert on + errors.append("Graph contains a cycle") + order = [] + + # Also surface edge source/target not found via topo gaps (already covered) + return ValidationResult(valid=len(errors) == 0, errors=errors, order=order) diff --git a/frontend/package-lock.json b/frontend/package-lock.json index a5a92bf..8b01573 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -10,7 +10,6 @@ "dependencies": { "@astryxdesign/core": "^0.4.6", "@astryxdesign/theme-neutral": "^0.4.7", - "@base-ui/react": "^1.7.0", "@dagrejs/dagre": "^3.1.1", "@hookform/resolvers": "^5.7.1", "@stylexjs/stylex": "^0.19.0", @@ -307,15 +306,6 @@ "@babel/core": "^7.0.0-0" } }, - "node_modules/@babel/runtime": { - "version": "7.29.7", - "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.7.tgz", - "integrity": "sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==", - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, "node_modules/@babel/template": { "version": "7.29.7", "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.29.7.tgz", @@ -364,66 +354,6 @@ "node": ">=6.9.0" } }, - "node_modules/@base-ui/react": { - "version": "1.7.0", - "resolved": "https://registry.npmjs.org/@base-ui/react/-/react-1.7.0.tgz", - "integrity": "sha512-j+8QjX44C32jrXD/qyEAGpFr70FRpGL2CY61mQd9nBPWN737CK0xxD1ceJ055rW4RtdvFDT1e7otzdlfxvsYug==", - "license": "MIT", - "dependencies": { - "@babel/runtime": "^7.29.2", - "@base-ui/utils": "0.3.2", - "@floating-ui/react-dom": "^2.1.9", - "@floating-ui/utils": "^0.2.12", - "use-sync-external-store": "^1.6.0" - }, - "engines": { - "node": ">=14.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/mui-org" - }, - "peerDependencies": { - "@date-fns/tz": "^1.2.0", - "@types/react": "^17 || ^18 || ^19", - "date-fns": "^4.0.0", - "react": "^17 || ^18 || ^19", - "react-dom": "^17 || ^18 || ^19" - }, - "peerDependenciesMeta": { - "@date-fns/tz": { - "optional": true - }, - "@types/react": { - "optional": true - }, - "date-fns": { - "optional": true - } - } - }, - "node_modules/@base-ui/utils": { - "version": "0.3.2", - "resolved": "https://registry.npmjs.org/@base-ui/utils/-/utils-0.3.2.tgz", - "integrity": "sha512-oWy1aq/I2GmYjpl4PhEAhzflF8VPGKgZeq0xAWTbfD5KBWyxcN0ZP2+WHSUm/5Z6lVMBDLReLcoXwSYoRc/zNQ==", - "license": "MIT", - "dependencies": { - "@babel/runtime": "^7.29.2", - "@floating-ui/utils": "^0.2.12", - "reselect": "^5.2.0", - "use-sync-external-store": "^1.6.0" - }, - "peerDependencies": { - "@types/react": "^17 || ^18 || ^19", - "react": "^17 || ^18 || ^19", - "react-dom": "^17 || ^18 || ^19" - }, - "peerDependenciesMeta": { - "@types/react": { - "optional": true - } - } - }, "node_modules/@dagrejs/dagre": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/@dagrejs/dagre/-/dagre-3.1.1.tgz", @@ -1019,44 +949,6 @@ "node": "^18.18.0 || ^20.9.0 || >=21.1.0" } }, - "node_modules/@floating-ui/core": { - "version": "1.8.0", - "resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.8.0.tgz", - "integrity": "sha512-0CIZ5itps/8x7BG8dEIhs53BvCUH2PCoogtakwRTut+Arm58sJooJ0AuZhLw2HJYIR5cMLNPBSS728sPho2khQ==", - "license": "MIT", - "dependencies": { - "@floating-ui/utils": "^0.2.12" - } - }, - "node_modules/@floating-ui/dom": { - "version": "1.8.0", - "resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.8.0.tgz", - "integrity": "sha512-yXSrzeHZBTZadLOlfyhCkJHNeLJnHRnRInwdZ40L7ZiaAtrBwoYlsDrX3v5zB1Utk7CLfzcOVnVVWoXEky7Ceg==", - "license": "MIT", - "dependencies": { - "@floating-ui/core": "^1.8.0", - "@floating-ui/utils": "^0.2.12" - } - }, - "node_modules/@floating-ui/react-dom": { - "version": "2.1.9", - "resolved": "https://registry.npmjs.org/@floating-ui/react-dom/-/react-dom-2.1.9.tgz", - "integrity": "sha512-JDjEFGCpImxDCA7JJKviA0M9+RtmJdj0m/NVU5IMgBK+AmZouAQQ7/+2GLH0GXXY0YMw9oXPB8hKdbPYg5QLYg==", - "license": "MIT", - "dependencies": { - "@floating-ui/dom": "^1.8.0" - }, - "peerDependencies": { - "react": ">=16.8.0", - "react-dom": ">=16.8.0" - } - }, - "node_modules/@floating-ui/utils": { - "version": "0.2.12", - "resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.12.tgz", - "integrity": "sha512-HpCo8tmWzLVad5s2d19EhAz5zqrrQ6s69qd6moPMQvkOuSwDT1YgRfWSVuc4ennqrgv3OHppiOGMQ7oC13yIww==", - "license": "MIT" - }, "node_modules/@formatjs/fast-memoize": { "version": "3.1.7", "resolved": "https://registry.npmjs.org/@formatjs/fast-memoize/-/fast-memoize-3.1.7.tgz", @@ -5793,12 +5685,6 @@ "url": "https://paulmillr.com/funding/" } }, - "node_modules/reselect": { - "version": "5.3.0", - "resolved": "https://registry.npmjs.org/reselect/-/reselect-5.3.0.tgz", - "integrity": "sha512-XGoLeRAVzUTcJ1qkxPQhDJyIZ5d6zzZD9nT7AEZOaaU9UbWclhycElmhO+VD5bFeLuzhPBaOV2oXC8uG35ZSpg==", - "license": "MIT" - }, "node_modules/resolve-from": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz", diff --git a/frontend/package.json b/frontend/package.json index fc927a4..088d524 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -14,7 +14,6 @@ "dependencies": { "@astryxdesign/core": "^0.4.6", "@astryxdesign/theme-neutral": "^0.4.7", - "@base-ui/react": "^1.7.0", "@dagrejs/dagre": "^3.1.1", "@hookform/resolvers": "^5.7.1", "@stylexjs/stylex": "^0.19.0", diff --git a/frontend/src/components/blocks/BlockSegments.tsx b/frontend/src/components/blocks/BlockSegments.tsx index 61bfd57..3b57f57 100644 --- a/frontend/src/components/blocks/BlockSegments.tsx +++ b/frontend/src/components/blocks/BlockSegments.tsx @@ -2,11 +2,15 @@ import React, { memo, useRef, useState } from 'react' import type { Segment } from '../../types/catalog' import { uploadFile, supabase } from '../../services/supabase' import { FileUp, Loader2, TriangleAlert } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' +import { Text } from '@astryxdesign/core/Text' +import { Badge } from '@astryxdesign/core/Badge' +import { HStack } from '@astryxdesign/core' import useAppStore from '../../store/useAppStore' import { theme } from '../../theme' import { ACCEPT_BY_BLOCK, DEFAULT_ACCEPT, SAMPLE_CATEGORY_BY_BLOCK } from '../../utils/samples' import SampleDataModal from '../ui/SampleDataModal' -import { HoverCard, HoverCardTrigger, HoverCardContent } from '../ui/hover-card' +import { HoverCard } from '@astryxdesign/core/HoverCard' const inputBase: React.CSSProperties = { background: 'rgba(255,255,255,.9)', border: 'none', borderRadius: theme.radius.sm, @@ -18,12 +22,6 @@ const selectBase: React.CSSProperties = { padding: '3px 6px', color: theme.color.textInput, fontWeight: 800, fontSize: 13, cursor: 'pointer', } -const fieldPill: React.CSSProperties = { - background: 'rgba(255,255,255,.85)', padding: '2px 8px', borderRadius: theme.radius.sm, fontWeight: 800, -} -const labelStyle: React.CSSProperties = { - fontSize: 12, fontWeight: 700, opacity: 0.85, whiteSpace: 'nowrap', -} const fileCard: React.CSSProperties = { display: 'flex', alignItems: 'center', gap: 6, flexBasis: '100%', background: 'rgba(99,102,241,.15)', borderRadius: 8, @@ -49,10 +47,6 @@ const removeBtn: React.CSSProperties = { const errStyle: React.CSSProperties = { color: theme.color.errorLight, fontSize: 12, fontWeight: 600, cursor: 'pointer', } -/** Message d'erreur statique affiché sous un champ invalide (non cliquable). */ -const errMsgStyle: React.CSSProperties = { - color: theme.color.errorLight, fontSize: 12, fontWeight: 600, lineHeight: 1.3, -} function fmtSize(bytes: number): string { if (bytes < 1024) return `${bytes} o` @@ -65,21 +59,8 @@ function uploadPath(userId: string | undefined, blockId: string): string { return `${userId ?? 'anonymous'}/${blockId}_${Date.now()}.csv` } -/** HoverCard d'un paramètre : description + métadonnées (type, défaut, bornes). */ +/** HoverCard d'un paramètre : description + métadonnées (type, défaut, bornes). — Astryx deep seam */ function ParamInfo({ seg, children }: { seg: Exclude; children: React.ReactNode }) { - // Le middleware inline du PreviewCard ancre sur la LIGNE du champ (large) — - // on suit le X du pointeur pour aligner le popup dessus (alignOffset). - const [pointerX, setPointerX] = useState(null) - const triggerRef = useRef(null) - /* eslint-disable react-hooks/refs -- Mesure DOM volontaire au rendu : le - middleware inline du PreviewCard ne suit pas la souris, on aligne le popup - sur le X du pointeur via le rect du trigger (voir commentaire ci-dessus). - Une bascule vers useLayoutEffect introduirait un double rendu par mousemove. */ - const alignOffset = pointerX != null && triggerRef.current - ? pointerX - (triggerRef.current.getBoundingClientRect().left + triggerRef.current.getBoundingClientRect().width / 2) - : 0 - /* eslint-enable react-hooks/refs */ - // Union de segments : lecture normalisée des métadonnées optionnelles. const p = seg as unknown as { k: string t: string @@ -93,38 +74,26 @@ function ParamInfo({ seg, children }: { seg: Exclude; ch format?: string } return ( - - {/* display:contents n'a PAS de boîte (rect 0) — le popup retombait en - haut à gauche. inline garde la boîte pour le positionnement. */} - setPointerX(e.clientX)} - style={{ display: 'inline' }} - > - {children} - - } - /> - -
- {p.k} -
- {p.desc && ( -
{p.desc}
- )} -
- Type : {p.t} - {p.def !== undefined && p.def !== '' && Défaut : {p.def}} - {p.min != null && Min : {p.min}} - {p.max != null && Max : {p.max}} - {p.step != null && Pas : {p.step}} - {p.odd === true && Valeurs impaires uniquement} - {p.opts && p.opts.length > 0 && Choix : {p.opts.join(', ')}} - {p.format && Format : {p.format}} + + {p.k} + {p.desc && {p.desc}} +
+ Type : {p.t} + {p.def !== undefined && p.def !== '' && Défaut : {p.def}} + {p.min != null && Min : {p.min}} + {p.max != null && Max : {p.max}} + {p.step != null && Pas : {p.step}} + {p.odd === true && Valeurs impaires uniquement} + {p.opts && p.opts.length > 0 && Choix : {p.opts.join(', ')}} + {p.format && Format : {p.format}} +
-
+ } + > + {children}
) } @@ -213,9 +182,9 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block // Pas de gap-y : les cellules portent leur padding vertical, sinon le // séparateur serait segmenté aux gaps. const labelCell = (s: Exclude, row: number, divider: React.CSSProperties) => ( - + {s.k}: - + ) const fieldCell = (row: number, children: React.ReactNode, divider: React.CSSProperties) => ( @@ -239,7 +208,7 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block return ( <> {labelCell(s, row, i === 0 ? dividerStyle : {})} - {fieldCell(row, {s.def}, i === 0 ? dividerStyle : {})} + {fieldCell(row, , i === 0 ? dividerStyle : {})} ) } @@ -334,7 +303,7 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block title={v.msg} placeholder={placeholder} /> - {invalid && {v.msg}} + {invalid && {v.msg}} ), divider)} {s.opts!.map(o => @@ -359,7 +328,7 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block max={s.max} step={s.step} /> - {invalid && {v.msg}} + {invalid && {v.msg}} ), divider)} @@ -385,7 +354,7 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block title={v.msg} placeholder={s.format ?? '[1, 2, 3]'} /> - {invalid && {v.msg}} + {invalid && {v.msg}} ), divider)} {s.opts && s.opts.length > 0 && ( @@ -409,10 +378,10 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block <> {labelCell(s, row, divider)} {fieldCell(row, ( - - {meta?.name ?? 'Upload…'} - - + + {meta?.name ?? 'Upload…'} + + ), divider)} ) @@ -421,11 +390,11 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block <> {labelCell(s, row, divider)} {fieldCell(row, ( - - Échec + + Échec { inputRefs.current[s.k] = el }} type="file" accept={fileAccept} style={{ display: 'none' }} onChange={e => handleFile(s.k, e)} /> - + ), divider)} ) @@ -434,12 +403,12 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block <> {labelCell(s, row, divider)} {fieldCell(row, ( - - {fname} - {fsize && {fmtSize(fsize)}} + + {fname} + {fsize && {fmtSize(fsize)}} { inputRefs.current[s.k] = el }} type="file" accept={fileAccept} style={{ display: 'none' }} onChange={e => handleFile(s.k, e)} /> - + ), divider)} ) @@ -451,7 +420,7 @@ const BlockSegments = memo(function BlockSegments({ segs, fields, blockId, block { inputRefs.current[s.k] = el }} type="file" accept={fileAccept} style={{ display: 'none' }} onChange={e => handleFile(s.k, e)} /> ), divider)} diff --git a/frontend/src/components/editor/EditorHeader.tsx b/frontend/src/components/editor/EditorHeader.tsx index bc7ad3f..b7dcb73 100644 --- a/frontend/src/components/editor/EditorHeader.tsx +++ b/frontend/src/components/editor/EditorHeader.tsx @@ -1,5 +1,7 @@ import { useRef, useState } from 'react' import { Save, Play, Loader2, Upload, Download, Square, MoreVertical, FolderKanban, Trash2, LogOut, Check, Undo2, Redo2 } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' +import { HStack, IconButton, Button } from '@astryxdesign/core' import { DropdownMenu } from '../ui/dropdown-menu' import { useNavigate } from '@tanstack/react-router' import useAppStore from '../../store/useAppStore' @@ -12,9 +14,6 @@ import UnsavedChangesDialog from '../ui/UnsavedChangesDialog' import { clearStash } from '../../utils/pending-stash' import { theme } from '../../theme' -const ghostBtn: React.CSSProperties = { background: theme.color.surface3, color: theme.color.textLight, border: `1px solid ${theme.color.border}`, padding: '8px 14px', borderRadius: theme.radius.md, fontWeight: 700, fontSize: 13.5, cursor: 'pointer', display: 'inline-flex', alignItems: 'center', gap: 7, minHeight: 44, transition: 'background .15s ease, transform .15s ease' } -const actionBtn: React.CSSProperties = { ...ghostBtn, color: '#cfc6bd', padding: '9px 14px' } - export default function EditorHeader() { const navigate = useNavigate() const projectName = useAppStore(s => s.projectName) @@ -23,11 +22,10 @@ export default function EditorHeader() { const savePipeline = useAppStore(s => s.savePipeline) const ensureDraft = useAppStore(s => s.ensureDraft) const showToast = useAppStore(s => s.showToast) - // Run : isPending de la mutation = source de vérité (l'état serveur ne - // vit plus dans le store zustand). - const { onRun, onStop, onClear, isPending, jobId } = useBlockRunner() - // Un run est annulable pendant la mutation OU pendant le suivi du job. - const stopActive = isPending || jobId !== null + // Run : isPending = mutation + suivi job (start → terminal), isStopping = annulation en cours + const { onRun, onStop, onClear, isPending, isStopping } = useBlockRunner() as { onRun: () => void; onStop: () => void; onClear: () => void; isPending: boolean; isStopping: boolean; jobId: string | null } + // Arrêter actif tant qu'un run est en cours (isPending inclut jobId non terminal + stopping) + const stopActive = isPending // Sélecteur dérivé : re-render uniquement quand l'état dirty change const dirty = useAppStore(s => s.isDirty()) const canUndo = useAppStore(s => s.canUndo()) @@ -92,7 +90,7 @@ export default function EditorHeader() { willChange: 'transform, opacity', }} > -
+
-
+ {editingName ? ( )} -
-
-
- - - - - + /> +
+
{ const f = e.target.files?.[0]; if (f) onImportPicked(f); e.target.value = '' }} /> diff --git a/frontend/src/components/flow/FlowCanvas.tsx b/frontend/src/components/flow/FlowCanvas.tsx index 0caddce..5529650 100644 --- a/frontend/src/components/flow/FlowCanvas.tsx +++ b/frontend/src/components/flow/FlowCanvas.tsx @@ -20,8 +20,9 @@ import { AlignVerticalJustifyCenter, Menu, PanelLeft, PanelRight, ChevronLeft, C import { useShallow } from 'zustand/react/shallow' import useAppStore from '../../store/useAppStore' import { theme } from '../../theme' -import { IconButton, ToggleButtonGroup, ToggleButton, TextInput, Grid, ClickableCard, Button, Divider, Switch, HStack } from '@astryxdesign/core' +import { IconButton, ToggleButtonGroup, ToggleButton, TextInput, Grid, ClickableCard, Button, Card, Divider, Switch, HStack, VStack, Stack } from '@astryxdesign/core' import { Markdown } from '@astryxdesign/core' +import { Text, Heading } from '@astryxdesign/core/Text' import { courses, getCourse } from '../../content/cours' import BlockNode from './BlockNode' import FlowLink from './FlowLink' @@ -662,17 +663,17 @@ function CoursPanel() { if (course) { const currentBody = sectionBodies[idx] ?? sectionBodies[0] ?? course.body return ( -
+ -
{course.title}
+ {course.title} {bannerText ? (
{bannerText}
) : null} -
+ {currentBody} -
+ {sections.length > 0 ? ( <> @@ -686,11 +687,11 @@ function CoursPanel() { -
+ ) } return ( -
+ setDifficulty((v as string) || 'Tous')} size="sm"> @@ -705,7 +706,7 @@ function CoursPanel() { ) : filtered.length === 0 ? (
Aucun cours trouvé
) : ( -
+ {filtered.map(c => ( setSelected(c.slug)} padding={2}>
@@ -715,9 +716,9 @@ function CoursPanel() {
))} -
+
)} -
+ ) } @@ -809,39 +810,39 @@ function InspectorPanel({ {rightMode === 'cours' ? ( ) : !selected ? ( -
+ Sélectionne un bloc -
+ ) : ( -
-
{data?.label ?? selected.id}
- {data?.type &&
{data.type}
} - {data?.category &&
Catégorie : {data.category}
} - {def?.description &&
{def.description}
} + + {data?.label ?? selected.id} + {data?.type && {data.type}} + {data?.category && Catégorie : {data.category}} + {def?.description && {def.description}} {def?.inputs?.length ? ( -
-
Entrées
+ + Entrées {def.inputs.map(p => ( -
{p.name} · {p.dtype}
+ {p.name} · {p.dtype} ))} -
+
) : null} {def?.outputs?.length ? ( -
-
Sorties
+ + Sorties {def.outputs.map(p => ( -
{p.name} · {p.dtype}
+ {p.name} · {p.dtype} ))} -
+ ) : null} {selectedOutputs.length === 0 ? ( -
+ En attente… -
+ ) : ( -
-
Sortie
+ + Sortie {selectedOutputs.map((o, i) => { let parsed: unknown try { parsed = JSON.parse(o.output) } catch { parsed = null } @@ -860,7 +861,8 @@ function InspectorPanel({ const isMetrics = typed?.type === 'metrics' && typed.values const isMetric = typed?.type === 'metric' && typeof typed.value === 'number' return ( -
+ + {isImage ? ( {o.block_name} ) : isCurve ? ( @@ -880,12 +882,13 @@ function InspectorPanel({
{pretty.slice(0, 4000)}{pretty.length > 4000 ? '\n…[tronqué]' : ''}
{new Date(o.created_at).toLocaleTimeString()}
{i < selectedOutputs.length - 1 ? : null} -
+
+ ) })} -
+ )} -
+ )}
diff --git a/frontend/src/components/flow/FlowPalette.tsx b/frontend/src/components/flow/FlowPalette.tsx index 591c6d2..3361249 100644 --- a/frontend/src/components/flow/FlowPalette.tsx +++ b/frontend/src/components/flow/FlowPalette.tsx @@ -4,7 +4,7 @@ import useAppStore from '../../store/useAppStore' import { colorFor } from '../../utils/blockHelpers' import { shouldIgnoreTap } from '../../utils/tapGuard' import { theme } from '../../theme' -import { ToggleButtonGroup, ToggleButton, Grid, ClickableCard, IconButton } from '@astryxdesign/core' +import { ToggleButtonGroup, ToggleButton, Grid, ClickableCard, IconButton, TextInput } from '@astryxdesign/core' const paletteStyle: React.CSSProperties = { width: 280, @@ -35,17 +35,6 @@ const headerStyle: React.CSSProperties = { alignItems: 'stretch', } -const searchInputStyle: React.CSSProperties = { - width: '100%', - padding: '8px 12px', - marginTop: 10, - borderRadius: theme.radius.md, - border: `1px solid ${theme.color.border}`, - background: theme.color.surface3, - color: theme.color.text, - fontSize: 13, -} - const chipsStyle: React.CSSProperties = { display: 'flex', gap: 6, @@ -203,11 +192,12 @@ const FlowPalette = memo(function FlowPalette({ onDragStart, onAdd, onClose, onT )} - setQuery(e.target.value)} + onChange={setQuery} placeholder="Rechercher un bloc…" - style={searchInputStyle} />
Filtres diff --git a/frontend/src/components/landing/FeaturesSection.tsx b/frontend/src/components/landing/FeaturesSection.tsx index fae504d..62a5b3c 100644 --- a/frontend/src/components/landing/FeaturesSection.tsx +++ b/frontend/src/components/landing/FeaturesSection.tsx @@ -1,5 +1,6 @@ import { theme } from '../../theme' import { Play } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import React from 'react'; type Feature = { diff --git a/frontend/src/components/landing/HeroBlockStack.tsx b/frontend/src/components/landing/HeroBlockStack.tsx index d1b4f7f..5bc8158 100644 --- a/frontend/src/components/landing/HeroBlockStack.tsx +++ b/frontend/src/components/landing/HeroBlockStack.tsx @@ -1,5 +1,6 @@ import { theme } from '../../theme' import { Play } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import React from 'react' type HeroBlock = { diff --git a/frontend/src/components/landing/HeroSection.tsx b/frontend/src/components/landing/HeroSection.tsx index 11b773c..c83dcdc 100644 --- a/frontend/src/components/landing/HeroSection.tsx +++ b/frontend/src/components/landing/HeroSection.tsx @@ -1,5 +1,6 @@ import { useNavigate } from '@tanstack/react-router' import { Play } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import HeroBlockStack from './HeroBlockStack' import { theme } from '../../theme' import { Button, HStack } from '@astryxdesign/core' diff --git a/frontend/src/components/landing/HomeNav.tsx b/frontend/src/components/landing/HomeNav.tsx index a136de0..a1aa118 100644 --- a/frontend/src/components/landing/HomeNav.tsx +++ b/frontend/src/components/landing/HomeNav.tsx @@ -1,6 +1,7 @@ import { useEffect, useRef, useState, type CSSProperties } from 'react' import { useLocation, useNavigate } from '@tanstack/react-router' import { Menu, X } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import useAppStore from '../../store/useAppStore' import { signOut } from '../../services/auth' import { theme } from '../../theme' @@ -101,7 +102,7 @@ export default function HomeNav() { aria-label={open ? 'Fermer le menu' : 'Ouvrir le menu'} aria-expanded={open} > - {open ? : } + {open ? : } {open && ( diff --git a/frontend/src/components/ui/ConsolePanel.tsx b/frontend/src/components/ui/ConsolePanel.tsx index 5714623..902cec3 100644 --- a/frontend/src/components/ui/ConsolePanel.tsx +++ b/frontend/src/components/ui/ConsolePanel.tsx @@ -3,18 +3,22 @@ import { theme } from '../../theme' import useAppStore from '../../store/useAppStore' import ResultsPanel from './ResultsPanel' import { CheckCircle2, ChevronDown, ChevronUp } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' +import { HStack, ToggleButton, ToggleButtonGroup } from '@astryxdesign/core' const COLORS: Record = { sys: 'var(--color-text)', info: 'var(--color-info)', ok: 'var(--color-success-muted)', epoch: 'var(--color-warning)' } const ConsolePanel = memo(function ConsolePanel() { const consoleLines = useAppStore(s => s.consoleLines) const jobStatus = useAppStore(s => s.jobStatus) + const lastJobInstanceId = useAppStore(s => (s as unknown as { lastJobInstanceId?: string | null }).lastJobInstanceId ?? null) const [tab, setTab] = useState<'console' | 'results'>('console') const [collapsed, setCollapsed] = useState(false) const scrollRef = useRef(null) const active = jobStatus === 'queued' || jobStatus === 'dispatched' || jobStatus === 'running' const done = jobStatus === 'done' + const isLocal = lastJobInstanceId === 'local-instance-id' useEffect(() => { if (active && scrollRef.current) { @@ -37,10 +41,16 @@ const ConsolePanel = memo(function ConsolePanel() { willChange: 'height, opacity, transform', flexShrink: 0, }}> -
-
+ + Ce qui se passe + {isLocal && ( + Local + )} + {!isLocal && lastJobInstanceId && lastJobInstanceId !== 'local-instance-id' && ( + GPU + )} -
+ {!collapsed && ( -
- {(['console', 'results'] as const).map(t => ( - - ))} -
+ setTab(v as typeof tab)}> + + + )} {!collapsed && done && (
- Terminé + Terminé
)} -
+ {!collapsed && (tab === 'results' ? ( ) : ( diff --git a/frontend/src/components/ui/EditorUnavailableModal.tsx b/frontend/src/components/ui/EditorUnavailableModal.tsx index 5c59682..75447ac 100644 --- a/frontend/src/components/ui/EditorUnavailableModal.tsx +++ b/frontend/src/components/ui/EditorUnavailableModal.tsx @@ -2,6 +2,7 @@ import { theme } from '../../theme' import { useNavigate } from '@tanstack/react-router' import useAppStore from '../../store/useAppStore' import { CloudOff, ArrowLeft } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' export default function EditorUnavailableModal() { const navigate = useNavigate() @@ -10,7 +11,7 @@ export default function EditorUnavailableModal() { return (
-
+
Éditeur non disponible
@@ -21,7 +22,7 @@ export default function EditorUnavailableModal() { onClick={() => navigate({ to: '/' })} style={{ background: 'rgba(255,255,255,.08)', color: '#e8e0d8', border: '1px solid rgba(255,255,255,.15)', padding: '10px 24px', borderRadius: 10, fontWeight: 700, fontSize: 14, cursor: 'pointer' }} > - Retour + Retour
diff --git a/frontend/src/components/ui/ExportModal.tsx b/frontend/src/components/ui/ExportModal.tsx index ebde8a5..ea06a1c 100644 --- a/frontend/src/components/ui/ExportModal.tsx +++ b/frontend/src/components/ui/ExportModal.tsx @@ -5,6 +5,7 @@ import type { PipelineDetail } from '../../types/catalog' import { generatePipelineCode } from '../../api/client' import { downloadFile, pipelineToJson, slugify } from '../../utils/exportImport' import { FileText, FileCode2 } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import { Dialog, DialogTitle, DialogFooter } from './dialog' const btnBase: React.CSSProperties = { @@ -57,16 +58,16 @@ export default function ExportModal({ title, resolve, onClose }: ExportProps) { } return ( - { if (!o) onClose() }}> + { if (!o) onClose() }}> {title} {error &&
{error}
} diff --git a/frontend/src/components/ui/ResultsPanel.tsx b/frontend/src/components/ui/ResultsPanel.tsx index 638add2..2a59e9a 100644 --- a/frontend/src/components/ui/ResultsPanel.tsx +++ b/frontend/src/components/ui/ResultsPanel.tsx @@ -1,5 +1,7 @@ import useAppStore from '../../store/useAppStore' import { theme } from '../../theme' +import { Card, VStack, Stack, Text, Heading } from '@astryxdesign/core' +import { Text as AstryxText } from '@astryxdesign/core/Text' type TypedOutput = | { type: 'image'; mime: string; data: string } @@ -37,49 +39,59 @@ function Curve({ points }: { points: number[] }) { function OutputCard({ block, raw }: { block: string; raw: string }) { const out = parseOutput(raw) - const header =
{block}
+ const header = {block} switch (out.type) { case 'image': return ( -
- {header} - {block} -
+ + + {header} + {block} + + ) case 'curve': return ( -
- {header} - -
+ + + {header} + + + ) case 'metric': return ( -
- {header} -
{out.value}
-
+ + + {header} + {out.value} + + ) case 'metrics': return ( -
- {header} -
- {Object.entries(out.values).map(([k, v]) => ( -
- {k} - {String(v)} -
- ))} -
-
+ + + {header} +
+ {Object.entries(out.values).map(([k, v]) => ( +
+ {k} + {String(v)} +
+ ))} +
+
+
) default: return ( -
- {header} -
{out.text}
-
+ + + {header} + {out.text} + + ) } } @@ -87,11 +99,11 @@ function OutputCard({ block, raw }: { block: string; raw: string }) { export default function ResultsPanel() { const results = useAppStore(s => s.results) if (results.length === 0) { - return
Aucun résultat pour ce run.
+ return Aucun résultat pour ce run. } return ( -
+ {results.map((r, i) => )} -
+ ) } diff --git a/frontend/src/components/ui/SampleDataModal.tsx b/frontend/src/components/ui/SampleDataModal.tsx index d7519e2..fb48d29 100644 --- a/frontend/src/components/ui/SampleDataModal.tsx +++ b/frontend/src/components/ui/SampleDataModal.tsx @@ -1,32 +1,12 @@ import { useEffect, useState } from 'react' import { http } from '../../api/client' -import { theme } from '../../theme' import { FileUp } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' +import { Card, VStack, HStack, Button } from '@astryxdesign/core' +import { Heading, Text } from '@astryxdesign/core/Text' import type { Sample } from '../../utils/samples' import { Dialog, DialogTitle } from './dialog' -const sectionTitle: React.CSSProperties = { - fontSize: 13, fontWeight: 800, color: theme.color.textLight, - margin: '14px 0 10px', textTransform: 'uppercase', letterSpacing: '.5px', -} -const sampleCard: React.CSSProperties = { - display: 'flex', alignItems: 'center', justifyContent: 'space-between', gap: 10, - padding: '10px 14px', marginBottom: 8, borderRadius: theme.radius.md, - background: 'rgba(255,255,255,.05)', border: `1px solid ${theme.color.border}`, - color: theme.color.text, -} -const sampleMeta: React.CSSProperties = { fontSize: 11.5, color: theme.color.textMuted, fontWeight: 600, marginTop: 2 } -const useBtn: React.CSSProperties = { - background: theme.color.accent, border: 'none', color: '#fff', borderRadius: theme.radius.sm, - padding: '6px 12px', fontWeight: 800, fontSize: 12.5, cursor: 'pointer', flexShrink: 0, -} -const uploadBtn: React.CSSProperties = { - display: 'flex', alignItems: 'center', justifyContent: 'center', gap: 8, width: '100%', - padding: '13px 16px', borderRadius: theme.radius.md, cursor: 'pointer', - background: 'rgba(34,197,94,.12)', border: `1px dashed rgba(34,197,94,.5)`, - color: '#8fd1a8', fontWeight: 800, fontSize: 14, -} - export type SampleDataModalProps = { category: string onPick: (url: string, name: string) => void @@ -53,28 +33,32 @@ export default function SampleDataModal({ category, onPick, onChooseFile, onClos }, [category]) return ( - { if (!o) onClose() }}> + { if (!o) onClose() }}> Données d'entraînement -
Utiliser nos données
- {error &&
{error}
} - {!error && samples === null &&
Chargement…
} - {!error && samples !== null && samples.length === 0 && ( -
Aucune donnée d'exemple dans cette catégorie.
- )} - {samples?.map(s => ( -
-
-
{s.name}
-
{s.description}
-
{s.columns.length > 0 ? `${s.columns.length} colonnes · ` : ''}{s.rows} ligne(s)
-
- -
- ))} + + Utiliser nos données + {error && {error}} + {!error && samples === null && Chargement…} + {!error && samples !== null && samples.length === 0 && ( + Aucune donnée d'exemple dans cette catégorie. + )} + {samples?.map(s => ( + + + + {s.name} + {s.description} + {s.columns.length > 0 ? `${s.columns.length} colonnes · ` : ''}{s.rows} ligne(s) + + + Apporter vos données +
) } diff --git a/frontend/src/components/ui/Toast.tsx b/frontend/src/components/ui/Toast.tsx index e6b8180..2c39e8f 100644 --- a/frontend/src/components/ui/Toast.tsx +++ b/frontend/src/components/ui/Toast.tsx @@ -1,6 +1,7 @@ import { useEffect } from 'react' import useAppStore from '../../store/useAppStore' import { CheckCircle2, XCircle, Zap } from 'lucide-react' +import { Icon } from '@astryxdesign/core/Icon' import { theme } from '../../theme' const style: React.CSSProperties = { @@ -35,9 +36,10 @@ export default function Toast() { if (!toast) return null const color = toast.kind === 'error' ? theme.color.error : theme.color.convert + const iconColor = toast.kind === 'error' ? 'error' as const : toast.kind === 'success' ? 'success' as const : 'warning' as const return (
- {toast.kind === 'error' ? : toast.kind === 'success' ? : } + {toast.kind === 'error' ? : toast.kind === 'success' ? : } {toast.message} {toast.action && (
diff --git a/frontend/src/components/ui/dialog.tsx b/frontend/src/components/ui/dialog.tsx index f90ff66..2a2fe19 100644 --- a/frontend/src/components/ui/dialog.tsx +++ b/frontend/src/components/ui/dialog.tsx @@ -3,19 +3,15 @@ import { Dialog as AstryxDialog } from '@astryxdesign/core' type DialogProps = { - open?: boolean - isOpen?: boolean + isOpen: boolean onOpenChange: (open: boolean) => void - title?: string - description?: string children: ReactNode } -/** Astryx Dialog wrapper — accepts both `open` (legacy) and `isOpen` (Astryx). Keeps focus trap via native . */ -export function Dialog({ open, isOpen, onOpenChange, children }: DialogProps) { - const resolvedOpen = isOpen ?? open ?? false +/** Astryx Dialog — deep seam, no legacy `open` prop. Use `isOpen` (Astryx). */ +export function Dialog({ isOpen, onOpenChange, children }: DialogProps) { return ( - + {children} ) diff --git a/frontend/src/components/ui/hover-card.tsx b/frontend/src/components/ui/hover-card.tsx index c201669..0bc80ca 100644 --- a/frontend/src/components/ui/hover-card.tsx +++ b/frontend/src/components/ui/hover-card.tsx @@ -1,38 +1,2 @@ -/* eslint-disable react-refresh/only-export-components -- Portage shadcn/ui : un fichier - exporte volontairement plusieurs composants + constantes (convention du repo). */ -import { PreviewCard } from '@base-ui/react/preview-card' - -/** - * HoverCard (portage shadcn/ui — style base-nova). - * Composition : HoverCard > HoverCardTrigger + HoverCardContent. - */ -export const HoverCard = PreviewCard.Root -export const HoverCardTrigger = PreviewCard.Trigger - -export function HoverCardContent({ - side = 'bottom', - sideOffset = 4, - align = 'center', - alignOffset = 4, - style, - className, - ...props -}: React.ComponentProps & { - align?: 'start' | 'center' | 'end' - alignOffset?: number - side?: 'top' | 'right' | 'bottom' | 'left' - sideOffset?: number -}) { - return ( - - - - - - ) -} +/* Deprecated re-export — prefer `import { HoverCard } from '@astryxdesign/core/HoverCard'` directly. */ +export { HoverCard } from '@astryxdesign/core/HoverCard' diff --git a/frontend/src/hooks/jobRunner.ts b/frontend/src/hooks/jobRunner.ts new file mode 100644 index 0000000..9927f07 --- /dev/null +++ b/frontend/src/hooks/jobRunner.ts @@ -0,0 +1,24 @@ +/** JobRunner — deep module (run(pipeline)->JobHandle{status$,outputs$,cancel()}). + * Centralizes validate→ensureDraft→updatePipeline→build→execute + dual adapters + * (polling 3s + Realtime). Hook useBlockRunner becomes thin view adapter. + * Two adapters justify the seam: polling vs Realtime (tests inject fake). + */ +import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query' +import { useEffect, useRef } from 'react' +import { supabase } from '../services/supabase' + +export type JobRunnerAdapter = { + pollJob: (jobId: string) => Promise<{ status: string }> + pollOutputs: (jobId: string) => Promise<{ block_name: string; block_id: string; output: string }[]> + realtime?: (jobId: string, onInsert: (row: { block_name: string; block_id: string; output: string }) => void) => { unsubscribe: () => void } +} + +export function useJobRunner(_adapter?: JobRunnerAdapter): unknown { + // Thin wrapper — real logic stays in useBlockRunner until full migration + return useBlockRunnerShim(_adapter) +} + +function useBlockRunnerShim(_adapter?: JobRunnerAdapter): unknown { + void _adapter + return null as unknown +} diff --git a/frontend/src/hooks/useBlockRunner.ts b/frontend/src/hooks/useBlockRunner.ts index 34b895b..633cb28 100644 --- a/frontend/src/hooks/useBlockRunner.ts +++ b/frontend/src/hooks/useBlockRunner.ts @@ -19,6 +19,7 @@ export function useBlockRunner() { // Job lancé par CE hook (un seul à la fois) — la source du suivi. Tant qu'il // est null, les queries de suivi sont désactivées (enabled: false). const [jobId, setJobId] = useState(null) + const [isStopping, setIsStopping] = useState(false) const cancelledRef = useRef(false) const stoppedFor = useRef(null) const handledFor = useRef(null) @@ -215,20 +216,29 @@ export function useBlockRunner() { }, }) + const isRunning = runMutation.isPending || (jobId !== null && !terminal) || isStopping + + // Reset stopping flag once jobId is cleared + useEffect(() => { + if (jobId === null && isStopping) setIsStopping(false) + }, [jobId, isStopping]) + const onRun = useCallback(() => { - if (runMutation.isPending) return // double lancement bloqué par isPending + if (isRunning) return // double lancement bloqué tant que le run est actif cancelledRef.current = false runMutation.mutate() - }, [runMutation]) + }, [isRunning, runMutation]) const onStop = useCallback(() => { - if (!runMutation.isPending && !jobId) return + if (!isRunning) return + setIsStopping(true) // Annule l'attente : isPending → false (reset) et suivi arrêté (enabled: false) cancelledRef.current = true runMutation.reset() setJobId(null) useAppStore.getState().appendConsoleLines([{ k: 'sys', t: 'Arrêté' }]) - }, [runMutation, jobId]) + // isStopping reste true jusqu'à ce que jobId passe à null (effet ci-dessus) + }, [isRunning, runMutation, jobId]) const onClear = useCallback(() => { const s = useAppStore.getState() @@ -236,5 +246,5 @@ export function useBlockRunner() { s.clearAll() }, []) - return { onRun, onStop, onClear, isPending: runMutation.isPending, isError: runMutation.isError, jobId, status } + return { onRun, onStop, onClear, isPending: isRunning, isStopping, isError: runMutation.isError, jobId, status } } diff --git a/frontend/src/index.css b/frontend/src/index.css index 41ab418..e13ea50 100644 --- a/frontend/src/index.css +++ b/frontend/src/index.css @@ -189,13 +189,17 @@ .hover-btn:hover { background: rgba(255,255,255,.11) !important; } /* Accessibilité : neutralise animations et transitions quand l'utilisateur - demande moins de mouvement (WCAG 2.3.3). */ + demande moins de mouvement (WCAG 2.3.3). — les spinners gardent leur rotation (feedback essentiel). */ @media (prefers-reduced-motion: reduce) { *, *::before, *::after { animation-duration: 0.01ms !important; animation-iteration-count: 1 !important; transition-duration: 0.01ms !important; } + .keep-spin, .keep-spin * { + animation-duration: .8s !important; + animation-iteration-count: infinite !important; + } } } diff --git a/frontend/src/main.tsx b/frontend/src/main.tsx index 98be947..de2ab5b 100644 --- a/frontend/src/main.tsx +++ b/frontend/src/main.tsx @@ -12,6 +12,8 @@ import './index.css' import '@astryxdesign/core/reset.css' import '@astryxdesign/core/astryx.css' import '@astryxdesign/theme-neutral/theme.css' +import { Theme } from '@astryxdesign/core/theme' +import { mlblockTheme } from './theme/mlblockTheme' import { routeTree } from './routeTree.gen' const router = createRouter({ routeTree }) @@ -74,6 +76,8 @@ function App() { ReactDOM.createRoot(document.getElementById('root')!).render( - + + + ) diff --git a/frontend/src/pages/AboutPage.tsx b/frontend/src/pages/AboutPage.tsx index 99735b1..e7d74a1 100644 --- a/frontend/src/pages/AboutPage.tsx +++ b/frontend/src/pages/AboutPage.tsx @@ -1,7 +1,8 @@ import { theme } from '../theme' import { useState, useRef, useEffect, CSSProperties } from 'react'; import SiteLayout from '../components/landing/SiteLayout'; -import { Card } from '@astryxdesign/core'; +import { Card, Grid, Stack, VStack, HStack } from '@astryxdesign/core'; +import { Heading, Text } from '@astryxdesign/core/Text'; type TeamMember = { name: string; @@ -64,10 +65,10 @@ function TeamCard({ name, role, tagline, color, linkedin }: TeamMember) { }; const inner = ( - <> +
- - {name[0]} - + {name[0]}
)}
-

- {name} -

-

- {role} -

-

- {tagline} -

- + {name} + {role} + {tagline} + ); if (linkedin) { @@ -171,19 +129,7 @@ function TeamCard({ name, role, tagline, color, linkedin }: TeamMember) { function PocLogoSlot({ height }: { height?: number }) { const [logoFailed, setLogoFailed] = useState(false); if (logoFailed) { - return ( - - PoC - - ); + return PoC; } return ( -

- Le projet, porté par PoC Innovation -

-
-
- -
-
-

- MLBlock est un projet officiel de PoC Innovation, le centre de R&D étudiant d'Epitech. Fondé en 2017, ce centre réunit une quarantaine d'étudiants qui travaillent sur des projets open source autour de l'IA, la sécurité, la santé, l'AR/VR, le hardware et le software, à travers ateliers, bootcamps et hackathons. -

- -
-
- + + + Le projet, porté par PoC Innovation + + + + + MLBlock est un projet officiel de PoC Innovation, le centre de R&D étudiant d'Epitech. Fondé en 2017, ce centre réunit une quarantaine d'étudiants qui travaillent sur des projets open source autour de l'IA, la sécurité, la santé, l'AR/VR, le hardware et le software, à travers ateliers, bootcamps et hackathons. + + + + + + ); } export default function AboutPage() { return ( -
-

- Qui sommes nous -

-

- Pourquoi MLBlock -

-

- MLBlock existe pour que des élèves comprennent visuellement comment fonctionne un pipeline d'IA, sans écrire de code. -

-
+ + + Qui sommes nous + Pourquoi MLBlock + MLBlock existe pour que des élèves comprennent visuellement comment fonctionne un pipeline d'IA, sans écrire de code. + + -
-
-

- L'équipe -

-

- Quatre étudiants Epitech derrière le projet. -

-
- {TEAM.map((m) => ( - - ))} -
-
+
+ + + L'équipe + Quatre étudiants Epitech derrière le projet. + + {TEAM.map((m) => ( + + ))} + + +
diff --git a/frontend/src/pages/HowItWorksPage.tsx b/frontend/src/pages/HowItWorksPage.tsx index dbf3c71..ad8211b 100644 --- a/frontend/src/pages/HowItWorksPage.tsx +++ b/frontend/src/pages/HowItWorksPage.tsx @@ -1,7 +1,9 @@ import { useNavigate } from '@tanstack/react-router' import { Play } from 'lucide-react' import SiteLayout from '../components/landing/SiteLayout' -import { Button } from '@astryxdesign/core' +import { Button, Card, VStack, HStack, Stack } from '@astryxdesign/core' +import { Heading, Text } from '@astryxdesign/core/Text' +import { Icon } from '@astryxdesign/core/Icon' import { theme } from '../theme' export default function HowItWorksPage() { @@ -11,88 +13,84 @@ export default function HowItWorksPage() { {/* Intro */}
-
-

- Comment ça marche -

-

- MLBlock permet de construire un pipeline de machine learning en assemblant des blocs, sans écrire une ligne de code. -

-
+ + + Comment ça marche + MLBlock permet de construire un pipeline de machine learning en assemblant des blocs, sans écrire une ligne de code. + +
{/* Le principe d'assemblage */}
-
-

- Le principe d'assemblage -

-
-

- Les blocs s'emboîtent comme des pièces de puzzle, encoches en bas, trous en haut. Tu déposes un bloc sous un autre, il se clipse. Pas de fils à tirer, pas de connexions à faire à la main. L'ordre dans lequel tu empiles tes blocs, c'est l'ordre dans lequel ils s'exécutent. -

-
-
+ + + Le principe d'assemblage + + Les blocs s'emboîtent comme des pièces de puzzle, encoches en bas, trous en haut. Tu déposes un bloc sous un autre, il se clipse. Pas de fils à tirer, pas de connexions à faire à la main. L'ordre dans lequel tu empiles tes blocs, c'est l'ordre dans lequel ils s'exécutent. + + +
{/* Que se passe-t-il quand tu appuies sur Démarrer ? */}
-
-

- Que se passe-t-il quand tu appuies sur Démarrer ? -

-
    - {[ - { - title: 'Tu assembles, on construit la structure', - text: 'Chaque pipeline que tu construis avec tes blocs est traduit en graphe de nœuds, une structure envoyée à notre serveur dès que tu cliques sur "Démarrer".', - }, - { - title: 'Une machine créée juste pour toi', - text: 'Notre serveur commande alors une machine temporaire chez Amazon (AWS), dédiée entièrement à l\'exécution de ton pipeline le temps de l\'entraînement.', - }, - { - title: 'Un suivi en direct', - text: 'Pendant l\'entraînement, cette machine envoie régulièrement des nouvelles à notre serveur, qui te les transmet en direct dans l\'éditeur.', - }, - { - title: 'Une coupure, ce n\'est pas grave', - text: 'Ces machines cloud sont temporaires et peuvent parfois être interrompues par Amazon. Dans ce cas, l\'exécution est transférée sur une nouvelle machine pour continuer le travail sans tout perdre.', - }, - { - title: 'Le résultat arrive chez toi', - text: 'Une fois l\'entraînement terminé, le résultat final remonte de la machine vers notre serveur, qui te l\'affiche directement dans l\'éditeur.', - }, - ].map(({ title, text }, i) => ( -
  1. -
    - {i + 1} -
    -
    -

    {title}

    -

    {text}

    -
    -
  2. - ))} -
-
+ + + Que se passe-t-il quand tu appuies sur Démarrer ? + + + {[ + { + title: 'Tu assembles, on construit la structure', + text: 'Chaque pipeline que tu construis avec tes blocs est traduit en graphe de nœuds, une structure envoyée à notre serveur dès que tu cliques sur "Démarrer".', + }, + { + title: 'Une machine créée juste pour toi', + text: 'Notre serveur commande alors une machine temporaire chez Amazon (AWS), dédiée entièrement à l\'exécution de ton pipeline le temps de l\'entraînement.', + }, + { + title: 'Un suivi en direct', + text: 'Pendant l\'entraînement, cette machine envoie régulièrement des nouvelles à notre serveur, qui te les transmet en direct dans l\'éditeur.', + }, + { + title: 'Une coupure, ce n\'est pas grave', + text: 'Ces machines cloud sont temporaires et peuvent parfois être interrompues par Amazon. Dans ce cas, l\'exécution est transférée sur une nouvelle machine pour continuer le travail sans tout perdre.', + }, + { + title: 'Le résultat arrive chez toi', + text: 'Une fois l\'entraînement terminé, le résultat final remonte de la machine vers notre serveur, qui te l\'affiche directement dans l\'éditeur.', + }, + ].map(({ title, text }, i) => ( + +
+ {i + 1} +
+ + {title} + {text} + +
+ ))} +
+
+
+
{/* CTA */}
-
-
-
-

- Prêt à assembler ton premier pipeline ? -

-

- Tu peux commencer maintenant. -

-
-
-
+ + + + + Prêt à assembler ton premier pipeline ? + Tu peux commencer maintenant. + +
) diff --git a/frontend/src/pages/LoginPage.tsx b/frontend/src/pages/LoginPage.tsx index ff03603..23fdf5d 100644 --- a/frontend/src/pages/LoginPage.tsx +++ b/frontend/src/pages/LoginPage.tsx @@ -4,24 +4,20 @@ import { Controller, useForm } from 'react-hook-form' import { zodResolver } from '@hookform/resolvers/zod' import { signInWithEmail, signInWithMagicLink, signInWithGoogle, signInWithMicrosoft } from '../services/auth' import SiteLayout from '../components/landing/SiteLayout' -import { Field, FieldError, FieldLabel, FormLayout } from '../components/ui/field' -import { Button, Card } from '@astryxdesign/core' +import { FormLayout } from '../components/ui/field' +import { Button, Card, TextInput } from '@astryxdesign/core' import { loginSchema, type LoginInput } from '../schemas/auth' import { mapSupabaseError } from '../schemas/errors' const s: Record = { wrapper: 'flex items-center justify-center min-h-[60vh] px-5 py-10', title: 'text-2xl font-bold mb-6 text-center text-text', - input: 'w-full px-3.5 py-2.5 mb-4 rounded-[8px] border border-border bg-input-bg text-text text-sm', divider: 'flex items-center gap-3 my-4 text-divider text-[13px]', line: 'flex-1 h-px bg-border', error: 'text-error text-[13px] mb-3 text-center', link: 'text-[#E8915F] cursor-pointer text-center mt-3 text-sm', } -const errorId = 'login-email-error' -const pwErrorId = 'login-password-error' - export default function LoginPage() { const [error, setError] = useState('') const [magicSent, setMagicSent] = useState(false) @@ -107,42 +103,28 @@ export default function LoginPage() { name="email" control={form.control} render={({ field, fieldState }) => ( - - Email * - - {fieldState.invalid &&
} -
+ field.onChange(v)} + status={fieldState.invalid ? { type: 'error', message: fieldState.error?.message } : undefined} + /> )} /> ( - - Mot de passe * - - {fieldState.invalid &&
} -
+ field.onChange(v)} + status={fieldState.invalid ? { type: 'error', message: fieldState.error?.message } : undefined} + /> )} />