native single-worker kubeflow integration

[bghira]

Continued from #2874

[Copilot]

Pull request overview

This PR extends the existing “Server + Worker” abstraction to support native, single-worker Kubeflow execution for local training jobs by provisioning one-shot GPU Workers as Kubeflow Trainer TrainJobs (admitted via Kueue), while preserving legacy local scheduling and non-Kubeflow Worker behavior.

Changes:

• Add Kubeflow provisioning + lifecycle reconciliation (TrainJob/Secret create, phase mapping, log retrieval/archival, finalize/cancel/reconcile).
• Introduce job-bound Worker semantics (fixed worker_id ↔ job_id binding, specialized dispatch/heartbeat/registration, and WorkerManager health-skip for bound Workers).
• Make publishing + storage compatible with centralized “single PUT + custom header” artifact uploads (ETag header, upload-token recording, final artifact publish trigger).

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/test_worker_manager.py	Adds regression tests for bound Worker health handling + bound-job dispatch behavior.
tests/test_worker_agent.py	Verifies Worker config stays Kubeflow-agnostic; tests JSON/YAML dataloader materialization and asyncio subprocess path.
tests/test_trainer.py	Adds coverage for final artifact directory publishing.
tests/test_s3_upload.py	Updates tests for ETag header + JSONResponse body; adds Kubeflow upload registration test.
tests/test_s3_put_async.py	Aligns async S3 PUT tests with new ETag header and response payload.
tests/test_publishing_providers.py	Tests S3 provider custom header injection + forced single-part PUT behavior.
tests/test_kubeflow_worker_routes.py	Adds route-level tests for bound Worker registration, SSE dispatch, heartbeat constraints, and completion gating on artifacts.
tests/test_kubeflow_scheduler_isolation.py	Ensures Kubeflow jobs are excluded from local GPU scheduler queries.
tests/test_kubeflow_queue_route.py	Tests queue submission path for target=kubeflow bypassing local allocators.
tests/test_kubeflow_queue_cancel_route.py	Tests provider-transparent cancellation behavior via generic queue cancellation endpoint.
tests/test_kubeflow_provisioner.py	Covers TrainJob/Secret manifests, rollback on failure, phase mapping, and log decoding behavior.
tests/test_kubeflow_job_service.py	Covers Kubeflow job submit/cancel/finalize/reconcile flows and publishing config injection.
tests/test_kubeflow_cli.py	Verifies CLI flags and environment export contract for Kubeflow integration.
tests/test_kubeflow_cancel_route.py	Ensures cloud cancel route delegates Kubeflow cancellations to lifecycle owner (not local process/GPU allocator).
simpletuner/worker_agent.py	Adds Kubeflow-aware launch path (async subprocess), dataloader JSON materialization, and final artifact publish trigger env var.
simpletuner/simpletuner_sdk/server/services/worker_manager.py	Adds bound-job dispatch API, preserves binding on failed dispatch, and skips generic health recovery for job-bound Workers.
simpletuner/simpletuner_sdk/server/services/worker_credentials.py	Introduces shared Worker credential/token utilities for manual + provisioned workers.
simpletuner/simpletuner_sdk/server/services/kubeflow.py	Adds Kubeflow settings/env parsing, manifest builders, in-cluster client loading, phase/log handling, and cleanup helpers.
simpletuner/simpletuner_sdk/server/services/kubeflow_job_service.py	Implements Kubeflow job lifecycle orchestration, reconciliation loop, artifact gating, and log archival.
simpletuner/simpletuner_sdk/server/services/cloud/storage/job_repository.py	Excludes Kubeflow jobs from local GPU scheduler job queries/counts.
simpletuner/simpletuner_sdk/server/services/cloud/job_logs.py	Routes log fetches for Kubeflow jobs through the Kubeflow job service.
simpletuner/simpletuner_sdk/server/routes/workers.py	Adds bound Worker validation, bound dispatch on SSE connect, heartbeat constraints, and completion gating/finalization.
simpletuner/simpletuner_sdk/server/routes/queue.py	Adds target=kubeflow submission and provider-aware cancellation behavior.
simpletuner/simpletuner_sdk/server/routes/cloud/storage.py	Adds Kubeflow upload registration, returns ETag header via JSONResponse, and uses stable MD5 ETag computation.
simpletuner/simpletuner_sdk/server/routes/cloud/jobs.py	Extends cancellation to include “starting” and delegates Kubeflow local-job cancellation to the Kubeflow service.
simpletuner/simpletuner_sdk/server/models/worker.py	Adds Worker.is_job_bound helper for Kubeflow-bound one-shot Workers.
simpletuner/simpletuner_sdk/server/app.py	Initializes/shuts down Kubeflow service alongside WorkerManager; enforces Kubeflow requires WorkerManager.
simpletuner/helpers/training/trainer.py	Adds _publish_final_artifacts and gates final artifact publish via env var.
simpletuner/helpers/publishing/providers/s3.py	Adds request header injection + force_single_part PUT support for server-side artifact uploads.
simpletuner/cli/server.py	Adds validated Kubeflow CLI → environment export logic and startup rejection on invalid config.
simpletuner/cli/init.py	Exposes Kubeflow CLI flags for server and jobs submit --target kubeflow.
setup.py	Adds optional extra simpletuner[kubernetes] dependency group.
documentation/OPTIONS.zh.md	Documents Kubeflow single-GPU server options (Chinese).
documentation/OPTIONS.pt-BR.md	Documents Kubeflow single-GPU server options (Portuguese - Brazil).
documentation/OPTIONS.md	Documents Kubeflow single-GPU server options (English).
documentation/OPTIONS.ja.md	Documents Kubeflow single-GPU server options (Japanese).
documentation/OPTIONS.hi.md	Documents Kubeflow single-GPU server options (Hindi).
documentation/OPTIONS.es.md	Documents Kubeflow single-GPU server options (Spanish).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.