Skip to content

feat: add monitor task infrastructure for shadow replayer comparison#32

Merged
bdchatham merged 2 commits intomainfrom
claude/dreamy-williamson
Mar 27, 2026
Merged

feat: add monitor task infrastructure for shadow replayer comparison#32
bdchatham merged 2 commits intomainfrom
claude/dreamy-williamson

Conversation

@bdchatham
Copy link
Copy Markdown
Collaborator

Summary

  • Adds monitor tasks — a third task category alongside plan tasks (sequential init) and scheduled tasks (fire-and-forget cron). Monitor tasks are submitted once in Running phase, polled each reconcile, and trigger controller reactions (Condition + Event) on completion.
  • First monitor task: result-export comparison mode — the sidecar compares local block results against a canonical RPC and completes on app-hash divergence.
  • Sidecar-lost-task detection: ErrNotFound after successful submission immediately marks Failed (sidecar Submit is synchronous, so not-found means the process restarted).
  • ResultExportComplete Condition with reasons DivergenceDetected, TaskFailed, TaskLost for kubectl visibility and alerting.
  • Removes dead scheduled-mode result-export code (CanonicalRPC is now required).
  • Renames PlannedTaskStatusTaskStatus for cross-context clarity.
  • Bumps seictl to v0.0.24 (sei-protocol/seictl#44).

Test plan

  • ensureMonitorTask: submit once, idempotent skip, submit error
  • pollMonitorTasks: completed, failed, still running, skip completed, sidecar lost task, invalid UUID, transient error, unknown error message, empty map
  • reconcileRuntimeTasks: monitor mode submission, replayer without ResultExport, poll requeue, conflict early return
  • Planner: ResultExportMonitorTask returns/nil for all node types
  • Conditions verified on all terminal paths
  • make manifests generate — CRDs regenerated
  • golangci-lint run — 0 issues
  • Full test suite passes

Add a third task category — monitor tasks — that the controller submits
once in Running phase and polls each reconcile for completion. When a
monitor task reaches a terminal state, the controller sets a Condition
and emits a Kubernetes Event.

The first (and currently only) monitor task is result-export in
comparison mode: the sidecar compares local block execution results
against a canonical RPC endpoint and completes when app-hash divergence
is detected.

Key changes:
- MonitorTask struct on SeiNodeStatus with SubmittedAt/CompletedAt
  timestamps for observability
- Rename PlannedTaskStatus → TaskStatus for cross-context clarity
- ResultExportComplete Condition with reasons: DivergenceDetected,
  TaskFailed, TaskLost
- Sidecar-lost-task detection: ErrNotFound after successful submission
  immediately marks Failed (sidecar Submit is synchronous)
- Early return on optimistic lock conflicts to avoid wasted API calls
- Immediate requeue on terminal state for fast downstream reaction
- Remove dead scheduled-mode result-export code (CanonicalRPC is now
  required on ResultExportConfig)
- Bump seictl to v0.0.24 (adds CanonicalRPC to ResultExportTask)
Use planner.ResultRequeueImmediate (RequeueAfter: 1ms) instead of the
deprecated ctrl.Result{Requeue: true} to satisfy staticcheck SA1019.
@bdchatham bdchatham marked this pull request as ready for review March 27, 2026 21:10
@bdchatham bdchatham merged commit dd47a3f into main Mar 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant