Skip to content

Expose Action Scheduler dispatcher starvation in worker health #2889

Description

@chubes4

Summary

Data Machine correctly enqueues workspace retention tasks, but worker health only reports a count of due actions. It does not identify that the Action Scheduler dispatcher trigger is stale, leaving queued jobs without an actionable diagnosis.

Live Reproduction

Job 2128 was created at 2026-07-12 01:45:52 by the Data Machine Code workspace cleanup CLI. Its engine data identifies workspace_retention_cleanup and task_params.source = workspace_cleanup_cli.

The core path is not missing a caller-side scheduling step:

Action Scheduler contains the expected action:

{
  "id": 16173,
  "hook": "datamachine_execute_step",
  "status": "pending",
  "args": { "job_id": 2128, "flow_step_id": "ephemeral_step_0" },
  "scheduled_date": "2026-07-12 01:45:52 +0000"
}

The actual failure is dispatcher starvation:

  • wp datamachine worker status --format=json reports due_actions: 36 and an unlocked worker lock.
  • Recurring action 16082 (datamachine_recurring_workspace_retention_cleanup) has been pending since 2026-06-22 09:40:34 UTC, with only an action created log entry.
  • wp cron event list reports action_scheduler_run_queue due since 2026-06-22 23:06:18 UTC.
  • wp datamachine jobs liveness --limit=100 --format=json identifies seven older pending execute-step jobs as scheduler_starved. The default liveness limit inspects only the 50 oldest processing jobs, so it does not include newer job 2128.

Fix Shape

  1. Extend datamachine worker status with Action Scheduler dispatch health: queue-trigger state, oldest due Data Machine action, and an explicit scheduler_dispatcher_starved condition.
  2. Include a safe supported remediation recommendation without mutating job rows.
  3. Add coverage for an overdue execute-step action plus stale Action Scheduler queue trigger.

AI assistance

  • AI assistance: Yes
  • Tool(s): Claude (Opus 4.5) via Kimaki/OpenCode subagent
  • Used for: Queue diagnosis and issue drafting from live job-table evidence

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions