fix: update patroni etcd hosts on host removal by tsivaprasad · Pull Request #501 · pgEdge/control-plane

tsivaprasad · 2026-11-02T12:38:15Z

Summary

This PR fixes stale Patroni Etcd configuration after removing an Etcd server host from the cluster.

Previously, surviving database instances could continue referencing the removed Etcd server in their Patroni configuration. The host removal workflow now detects and rewrites stale Etcd configuration and updates all databases after the host has left the Etcd cluster.

Changes

PatroniConfig.Refresh compares the Etcd hosts in the existing
patroni.yaml with the current Etcd member list. On a mismatch it
sets etcd_hosts_outdated. The desired state never sets this flag, so
the planner sees a diff and runs Update, which rewrites the config
and reloads Patroni with SIGHUP. Because this is in
orchestrator/common, it covers both swarm and systemd, and any
database update now repairs a stale config.
The RemoveHost workflow now runs in this order:
1. Remove the host from Etcd (new RemoveHostFromEtcd activity).
2. Update the databases.
3. Delete the host record (the RemoveHost activity now only does
  this).
The remove-host handler now includes every modifiable database:
- Databases with instances on the removed host have it taken out of
  their spec, as before.
- Other databases are updated with their spec unchanged.
- Databases that can't be modified (for example, still creating)
  are skipped with a warning. Their next update fixes the config.
Database updates started by host removal now pass
db.Variables(), which was previously left out.

Testing

Verification:

Create db-a with instances on host-1, host-2 and host-3, and
db-b with instances on host-1 and host-2 only.

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp1-req create-database '{
  "id": "db-a",
  "spec": {
    "database_name": "app",
    "database_users": [{"username":"admin","password":"password","db_owner":true,"attributes":["SUPERUSER","LOGIN"]}],
    "nodes": [
      {"name":"n1","port":0,"host_ids":["host-1"]},
      {"name":"n2","port":0,"host_ids":["host-2"]},
      {"name":"n3","port":0,"host_ids":["host-3"]}
    ]
  }
}' 

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp1-req create-database '{
  "id": "db-b",
  "spec": {
    "database_name": "app",
    "database_users": [{"username":"admin","password":"password","db_owner":true,"attributes":["SUPERUSER","LOGIN"]}],
    "nodes": [
      {"name":"n1","port":0,"host_ids":["host-1"]},
      {"name":"n2","port":0,"host_ids":["host-2"]}
    ]
  }
}'

Confirmed all hosts state:

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp-etcdctl member list -w table   # 3 members: host-1, host-2, host-3

for f in docker/control-plane-dev/data/host-*/instances/*/configs/patroni.yaml; do
  echo "== $f"; grep -A4 'hosts:' "$f" | grep -E ':[0-9]{4}'
done
+------------------+---------+--------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------+---------------------------+---------------------------+------------+
| 51b16e19b497b37b | started | host-2 | https://192.168.65.3:2480 | https://192.168.65.3:2479 |      false |
| 7a8e2bd9cb7ab2b7 | started | host-3 | https://192.168.65.3:2580 | https://192.168.65.3:2579 |      false |
| b71f75320dc06a6c | started | host-1 | https://192.168.65.3:2380 | https://192.168.65.3:2379 |      false |
+------------------+---------+--------+---------------------------+---------------------------+------------+

== docker/control-plane-dev/data/host-1/instances/db-a-n1-689qacsi/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
  - 192.168.65.3:2579
== docker/control-plane-dev/data/host-1/instances/db-b-n1-689qacsi/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
  - 192.168.65.3:2579
== docker/control-plane-dev/data/host-2/instances/db-a-n2-9ptayhma/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
  - 192.168.65.3:2579
== docker/control-plane-dev/data/host-2/instances/db-b-n2-9ptayhma/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
  - 192.168.65.3:2579
== docker/control-plane-dev/data/host-3/instances/db-a-n3-ant97dj4/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
  - 192.168.65.3:2579

Stopped and removed host-3

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp-docker-compose stop host-3

cp1-req remove-host host-3 --force | tee /tmp/remove-host.json | cp-follow-task
[+] stop 1/1
 ✔ Container control-plane-dev-host-3-1 Stopped                                                                                                                                                                                            2.1s
{
  "task": {
    "created_at": "2026-11-02T11:42:58Z",
    "entity_id": "host-3",
    "host_id": "host-3",
    "scope": "host",
    "status": "pending",
    "task_id": "01a19c11-dc33-719c-82f3-885fed3b4d00",
    "type": "remove_host"
  },
  "update_database_tasks": [
    {
      "created_at": "2026-11-02T11:42:58Z",
      "database_id": "db-a",
      "entity_id": "db-a",
      "parent_id": "01a19c11-dc33-719c-82f3-885fed3b4d00",
      "scope": "database",
      "status": "pending",
      "task_id": "01a19c11-dc33-7883-9530-98d317a8f0a2",
      "type": "update"
    },
    {
      "created_at": "2026-11-02T11:42:58Z",
      "database_id": "db-b",
      "entity_id": "db-b",
      "parent_id": "01a19c11-dc33-719c-82f3-885fed3b4d00",
      "scope": "database",
      "status": "pending",
      "task_id": "01a19c11-dc33-7f2a-a249-22ced5750dc1",
      "type": "update"
    }
  ]
}
[2026-11-02T11:42:59Z] removing host 'host-3' from etcd
[2026-11-02T11:43:00Z] updating database 'db-a'
[2026-11-02T11:43:00Z] updating database 'db-b'
[2026-11-02T11:44:03Z] successfully updated database 'db-a'
[2026-11-02T11:44:03Z] successfully updated database 'db-b'
[2026-11-02T11:44:04Z] removing host 'host-3'
[2026-11-02T11:44:04Z] successfully removed host 'host-3'

host entity host-3 task 01a19c11-dc33-719c-82f3-885fed3b4d00 completed

Check etcd and the rewritten configs

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp-etcdctl member list -w table   

for f in docker/control-plane-dev/data/host-{1,2}/instances/*/configs/patroni.yaml; do
  echo "== $f"; grep -A4 'hosts:' "$f" | grep -E ':[0-9]{4}'
done

+------------------+---------+--------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------+---------------------------+---------------------------+------------+
| 51b16e19b497b37b | started | host-2 | https://192.168.65.3:2480 | https://192.168.65.3:2479 |      false |
| b71f75320dc06a6c | started | host-1 | https://192.168.65.3:2380 | https://192.168.65.3:2379 |      false |
+------------------+---------+--------+---------------------------+---------------------------+------------+
== docker/control-plane-dev/data/host-1/instances/db-a-n1-689qacsi/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
== docker/control-plane-dev/data/host-1/instances/db-b-n1-689qacsi/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
== docker/control-plane-dev/data/host-2/instances/db-a-n2-9ptayhma/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479
== docker/control-plane-dev/data/host-2/instances/db-b-n2-9ptayhma/configs/patroni.yaml
  - 192.168.65.3:2379
  - 192.168.65.3:2479

Check Patroni reloaded and the databases are healthy

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp1-req list-hosts     - host-3 is gone
➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp1-req list-databases    -  `db-a` and `db-b` are available
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 02 Nov 2026 11:46:07 GMT

{
  databases: [
    {
      created_at: "2026-11-02T11:36:44Z"
      id: "db-a"
      instances: [
        {
          connection_info: {
            addresses: ["127.0.0.1"]
            port: 6617
          }
          created_at: "2026-11-02T11:36:48Z"
          host_id: "host-1"
          id: "db-a-n1-689qacsi"
          node_name: "n1"
          postgres: {
            patroni_state: "running"
            role: "primary"
            version: "18.6"
          }
          spock: {
            read_only: "off"
            subscriptions: [
              {
                name: "sub_n2_n1"
                provider_node: "n2"
                status: "replicating"
              }
            ]
            version: "5.0.13"
          }
          state: "available"
          status_updated_at: "2026-11-02T11:46:05Z"
          updated_at: "2026-11-02T11:43:34Z"
        }
        {
          connection_info: {
            addresses: ["127.0.0.1"]
            port: 13486
          }
          created_at: "2026-11-02T11:36:48Z"
          host_id: "host-2"
          id: "db-a-n2-9ptayhma"
          node_name: "n2"
          postgres: {
            patroni_state: "running"
            role: "primary"
            version: "18.6"
          }
          spock: {
            read_only: "off"
            subscriptions: [
              {
                name: "sub_n1_n2"
                provider_node: "n1"
                status: "replicating"
              }
            ]
            version: "5.0.13"
          }
          state: "available"
          status_updated_at: "2026-11-02T11:46:03Z"
          updated_at: "2026-11-02T11:43:53Z"
        }
      ]
      state: "available"
      updated_at: "2026-11-02T11:42:58Z"
    }
    {
      created_at: "2026-11-02T11:40:42Z"
      id: "db-b"
      instances: [
        {
          connection_info: {
            addresses: ["127.0.0.1"]
            port: 10169
          }
          created_at: "2026-11-02T11:40:45Z"
          host_id: "host-1"
          id: "db-b-n1-689qacsi"
          node_name: "n1"
          postgres: {
            patroni_state: "running"
            role: "primary"
            version: "18.6"
          }
          spock: {
            read_only: "off"
            subscriptions: [
              {
                name: "sub_n2_n1"
                provider_node: "n2"
                status: "replicating"
              }
            ]
            version: "5.0.13"
          }
          state: "available"
          status_updated_at: "2026-11-02T11:46:04Z"
          updated_at: "2026-11-02T11:43:34Z"
        }
        {
          connection_info: {
            addresses: ["127.0.0.1"]
            port: 8771
          }
          created_at: "2026-11-02T11:40:45Z"
          host_id: "host-2"
          id: "db-b-n2-9ptayhma"
          node_name: "n2"
          postgres: {
            patroni_state: "running"
            role: "primary"
            version: "18.6"
          }
          spock: {
            read_only: "off"
            subscriptions: [
              {
                name: "sub_n1_n2"
                provider_node: "n1"
                status: "replicating"
              }
            ]
            version: "5.0.13"
          }
          state: "available"
          status_updated_at: "2026-11-02T11:46:04Z"
          updated_at: "2026-11-02T11:43:59Z"
        }
      ]
      state: "available"
      updated_at: "2026-11-02T11:42:58Z"
    }
  ]
}

➜  control-plane git:(fix/PLAT-639/update-patroni-etcd-config-on-host-removal) ✗ cp1-req get-database db-a | jq '.instances[].host_id'    
"host-1"
"host-2"

Checklist

Tests added or updated

PLAT-639

coderabbitai · 2026-11-02T12:38:29Z

Warning

Review limit reached

Run on-demand review

_{This review includes 2 billable files and costs up to $0.50.}

Or wait 4 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 6b5f25ee-14da-4042-8275-2551fc7d6d5c

Commits

Reviewing files that changed from the base of the PR and between 0764228 and 3963bc3.

Files selected for processing: 2

changes/unreleased/Fixed-20261102-000000.yaml
server/internal/workflows/remove_host.go

Recent review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: e75c5043-69a6-463d-9e31-891ad4d46b6a

Commits

Reviewing files that changed from the base of the PR and between 1e1a441 and 0764228.

Files selected for processing: 9

changes/unreleased/Fixed-20261102-000000.yaml
server/internal/api/apiv1/post_init_handlers.go
server/internal/orchestrator/common/patroni_config.go
server/internal/orchestrator/common/patroni_config_internal_test.go
server/internal/orchestrator/swarm/patroni_config_test.go
server/internal/workflows/activities/activities.go
server/internal/workflows/activities/remove_host.go
server/internal/workflows/remove_host.go
server/internal/workflows/service.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

Host removal now removes Etcd membership before database updates and host-service removal. The API prepares database records for the workflow. Patroni configuration refresh detects missing or differing Etcd host data and marks it as outdated.

Changes: Host Removal

Layer / File(s)	Summary
Prepare database updates `server/internal/api/apiv1/post_init_handlers.go`, `server/internal/workflows/service.go`	The API now processes all databases and prepares records for the workflow. The workflow service uses each database’s ID, spec, and variables when creating update tasks.
Remove Etcd membership before host removal `server/internal/workflows/activities/*`, `server/internal/workflows/remove_host.go`	A registered activity removes Etcd membership before database updates. The host-removal activity now removes only the host-service record. Database task events describe database updates.
Track outdated Etcd hosts in Patroni configuration `server/internal/orchestrator/common/patroni_config.go`, `server/internal/orchestrator/common/patroni_config_internal_test.go`, `server/internal/orchestrator/swarm/patroni_config_test.go`, `changes/unreleased/Fixed-20261102-000000.yaml`	Patroni configuration refresh marks missing or differing Etcd host data as outdated and clears the flag after generating configuration. Tests cover host-set comparisons and desired-state diffs. The changelog records the configuration update on host removal.

Merge Risk: Moderate · previous review 07642

If removing a host from Etcd fails, the affected databases can stay in the modifying state with pending tasks, which blocks further updates until someone repairs them. Handle that failure path before merging. The rest of the change looks sound, apart from a changelog sentence that overstates which databases are updated.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "fix: update patroni etcd hosts on host removal" matches the main change. The PR fixes stale Patroni Etcd host entries after a host is removed. It follows the Conventional Commits style that…
Description check		The description follows the template. It has a Summary that states the fix and why it is needed. The Changes section lists the PatroniConfig refresh logic, the reordered RemoveHost workflow, the handl…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title "fix: update patroni etcd hosts on host removal" matches the main change. The PR fixes stale Patroni Etcd host entries after a host is removed. It follows the Conventional Commits style that the template requests.

Full details: Description check

Explanation

The description follows the template. It has a Summary that states the fix and why it is needed. The Changes section lists the PatroniConfig refresh logic, the reordered RemoveHost workflow, the handler changes, and the variables fix. The Testing section gives concrete manual verification steps with outputs. The Checklist marks tests added or updated. Gaps: the Checklist does not tick the changelog entry box, even though the diff adds a changelog file. The Notes for Reviewers section is absent, and the description does not link an issue in the body, although a PLAT-639 link appears at the end. Documentation status is not stated. These gaps are minor.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 8 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate docstrings

Commit to this branch
Create a new PR

Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit checks the Etcd list,
Then hops to clear the host.
The databases receive their updates,
Patroni refreshes what it knows.
The config drops its outdated flag,
And carrots celebrate the flow.}

codacy-production · 2026-11-02T12:39:22Z

Up to standards ✅

🟢 Issues 1 medium

Results:
1 new issue

Category Results

Complexity 1 medium

View in Codacy

🟢 Metrics 14 complexity · 3 duplication

Metric Results

Complexity 14

Duplication 3

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

coderabbitai

Fix comments: Fix CodeRabbit comments on this PR

fix: update patroni etcd hosts on host removal

0764228

coderabbitai Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread changes/unreleased/Fixed-20261102-000000.yaml Resolved

Comment thread server/internal/workflows/remove_host.go Resolved

addressing AI review comments

3963bc3

jason-lynch approved these changes Nov 4, 2026

View reviewed changes

tsivaprasad merged commit 55a984a into main Nov 4, 2026
5 checks passed

tsivaprasad deleted the fix/PLAT-639/update-patroni-etcd-config-on-host-removal branch November 4, 2026 14:32

Navigation Menu

Repository navigation

fix: update patroni etcd hosts on host removal - #501

fix: update patroni etcd hosts on host removal#501
tsivaprasad merged 2 commits into
mainfrom
fix/PLAT-639/update-patroni-etcd-config-on-host-removal

tsivaprasad commented Nov 2, 2026 •

edited by atlassian Bot

Loading

Uh oh!

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading

Review limit reached

Uh oh!

codacy-production Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tsivaprasad commented Nov 2, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Checklist

Uh oh!

coderabbitai Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

codacy-production Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

coderabbitai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tsivaprasad commented Nov 2, 2026 •

edited by atlassian Bot

Loading

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading

codacy-production Bot commented Nov 2, 2026 •

edited

Loading

coderabbitai Bot left a comment •

edited

Loading