[OSDOCS-11088]: etcd selectable DB size

etcd/etcd-performance.adoc

@@ -71,6 +71,20 @@ include::modules/etcd-increase-db.adoc[leveloffset=+1]
 
 include::modules/etcd-increase-db-troubleshooting.adoc[leveloffset=+1]
 
+//Selectable etcd DB size
+include::modules/etcd-selectable-db.adoc[leveloffset=+1]
+
+include::modules/etcd-selectable-db-changing.adoc[leveloffset=+2]
+
+include::modules/etcd-crashloop-after-quota-increase.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../etcd/etcd-backup-restore/etcd-backup.adoc#etcd-backup[Backing up and restoring etcd data]
+
+include::modules/etcd-alerts-after-quota-increase.adoc[leveloffset=+2]
+
 // Measuring network jitter between control plane nodes
 include::modules/etcd-network-latency-jitter.adoc[leveloffset=+1]
 

modules/etcd-alerts-after-quota-increase.adoc

@@ -0,0 +1,38 @@
+:_mod-docs-content-type: PROCEDURE
+[id="etcd-alerts-after-quota-increase_{context}"]
+= Troubleshooting etcd quota alerts after quota change
+
+[role="_abstract"]
+After you change the `backendQuotaGiB` setting, if you see `etcdDatabaseQuotaLowSpace` alerts, verify database use and determine mitigation steps.
+
+.Prerequisites
+
+* You configured the `backendQuotaGiB` setting.
+* You have access to cluster metrics or the **Observe** → **Metrics** view in the {product-title} web console.
+
+.Procedure
+
+. Confirm current usage by entering the following command:
++
+[source,promql]
+----
+etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes
+----
+
+. Compare absolute database size to the alert thresholds in the `etcdDatabaseQuotaLowSpace` alert thresholds.
+
+. Mitigate elevated usage by using one or more of the following approaches:
++
+--
+* Run or wait for automatic defragmentation.
+* Delete unnecessary Kubernetes objects to reduce churn and size.
+* Increase `backendQuotaGiB` within the 16 GiB maximum if sustained growth is expected.
+* Silence alerts temporarily when growth is expected and documented, following your change management process.
+--
+
+. If critical alerts persist near 85% usage, increase the quota or perform object cleanup before write failures occur.
+
+.Verification
+
+* The usage ratio is stable or decreasing.
+* Alerts are cleared or are acknowledged according to the operations policy.

modules/etcd-crashloop-after-quota-increase.adoc

@@ -0,0 +1,69 @@
+:_mod-docs-content-type: PROCEDURE
+[id="etcd-crashloop-after-quota-increase_{context}"]
+= Recovering etcd from CrashLoopBackOff after increasing backend quota
+
+[role="_abstract"]
+If you increase the `backendQuotaGiB` setting and the etcd pods enter a `CrashLoopBackOff` state, verify whether the failure is memory-related, confirm control plane sizing, and follow supported member recovery steps.
+
+.Prerequisites
+
+* You have cluster administrator access.
+* You have a current etcd backup.
+* Control plane node memory meets sizing guidance for the configured `backendQuotaGiB` setting, or you can increase node memory before recovery.
+
+.Procedure
+
+. Verify whether etcd containers were out-of-memory by entering the following command:
++
+[source,terminal]
+----
+$ oc get pods -n openshift-etcd -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.name}{": "}{.lastState.terminated.reason}{" "}{end}{"\n"}{end}'
+----
++
+If the reason is `OOMKilled`, insufficient memory is the likely cause.
+
+. Check the node memory and the etcd resident memory by entering the following command:
++
+[source,terminal]
+----
+$ oc adm top nodes --selector=node-role.kubernetes.io/control-plane=
+----
++
+In the {product-title} web console, in the **Observe** → **Metrics** view, query `process_resident_memory_bytes{job="etcd"}` and compare results to the sizing table in "Selectable etcd database size for large clusters".
+
+. If control plane memory is below the recommended range for your configured quota, increase memory on all control plane nodes before proceeding. Recovery attempts fail if nodes remain undersized.
+
+. Take a restorable etcd snapshot before member-level recovery by running `cluster-backup.sh` on a control plane node. For more information, see "Backing up and restoring etcd data". Save the snapshot and static pod resources to a secure location outside the cluster.
++
+Optionally, collect supplemental diagnostics by entering the following command:
++
+[source,terminal]
+----
+$ oc adm inspect -n openshift-etcd
+----
+
+. For a single crash-looping etcd member with sufficient memory and no data corruption, complete the following steps:
+
+.. Follow the steps in "Replacing an unhealthy etcd member whose etcd pod is crashlooping" to replace the unhealthy member.
+
+.. Verify etcd cluster health by entering the following commands: 
++
+[source,terminal]
+----
+$ oc get etcd cluster -o yaml
+----
++
+[source,terminal]
+----
+$ oc get pods -n openshift-etcd
+----
+
+. If recovery fails after memory correction and member replacement, collect must-gather logs and contact Red{nbsp}Hat Support. Do not attempt unsupported quota decreases.
+
+. If you must run at a lower `backendQuotaGiB`, restore the cluster from an etcd backup taken before the quota increase.
+
+.Verification
+
+* All etcd pods in `openshift-etcd` are `Running` and ready.
+* The `oc get etcd cluster -o jsonpath='{.items[0].status.conditions[?(@.type=="EtcdMembersAvailable")].status}'` command returns `True`.
+* No recurring `OOMKilled` terminations occur on etcd containers.

modules/etcd-selectable-db-changing.adoc

@@ -0,0 +1,57 @@
+:_mod-docs-content-type: PROCEDURE
+[id="etcd-selectable-db-changing_{context}"]
+= Changing the etcd database size
+
+[role="_abstract"]
+To increase etcd backend quota, set the `backendQuotaGiB` value.
+
+[IMPORTANT]
+====
+Do not use this feature unless you are sure you do not need to eventually decrease the database quota. You cannot decrease the `backendQuotaGiB` setting. The API rejects any attempt to set a value lower than the current configuration with the following error: `etcd backendQuotaGiB may not be decreased`.
+
+The only way to run at a lower quota after an increase is to restore the cluster from an etcd backup taken before the change. Be aware that any changes made since the backup was taken might be lost.
+====
+
+.Prerequisites
+
+* You have cluster administrator access.
+* You have a current etcd backup.
+* The control plane memory and disk meet the requirements for the target quota. For more information, see "Selectable etcd database size for large clusters".
+
+.Procedure
+
+. Patch the `backendQuotaGiB` setting with an integer from `8` through `16`:
++
+[source,terminal]
+----
+$ oc patch/etcd cluster --type=merge -p '{"spec":{"backendQuotaGiB": <quota_gib>}}'
+----
++
+Replace `<quota_gib>` with your target value.
++
+.Example output
+[source,text]
+----
+etcd.operator.openshift.io/cluster patched
+----
+
+. Monitor etcd pod rollout and quota metrics by entering the following command:
++
+[source,terminal]
+----
+$ oc adm wait-for-stable-cluster
+----
+
+.Verification
+
+* Ensure that the `etcd_server_quota_backend_bytes` value reflects the new quota by entering the following commands:
++
+[source,terminal]
+----
+$ oc describe etcd/cluster | grep "Backend Quota"
+----
++
+[source,terminal]
+----
+$ oc describe -n openshift-etcd pod/<etcd_pod> | grep ETCD_QUOTA_BACKEND_BYTES
+----

modules/etcd-selectable-db.adoc

@@ -0,0 +1,62 @@
+:_mod-docs-content-type: CONCEPT
+[id="etcd-selectable-db_{context}"]
+= Selectable etcd database size for large clusters
+
+[role="_abstract"]
+As a cluster administrator, you can increase the etcd backend database quota when large or high-churn workloads approach the default 8 GiB limit. Large and high-churn {product-title} clusters can exhaust the default 8 GiB etcd backend quota, causing write failures and control plane instability.
+
+Starting in {product-title} 5.0, you can increase the supported quota through the `backendQuotaGiB` field on the cluster `etcd` custom resource without enabling a feature gate.
+
+[IMPORTANT]
+====
+Do not use this feature unless you are sure you do not need to eventually decrease the database quota. You cannot decrease the `backendQuotaGiB` setting. The API rejects any attempt to set a value lower than the current configuration with the following error: `etcd backendQuotaGiB may not be decreased`.
+
+The only way to run at a lower quota after an increase is to restore the cluster from an etcd backup taken before the change. Be aware that any changes made since the backup was taken might be lost.
+====
+
+The `backendQuotaGiB` field accepts integer values from 8 GiB through 16 GiB. The default value is 8 GiB, which matches the historical etcd backend quota for {product-title}.
+
+[id="etcd-selectable-db-behavior_{context}"]
+== Disk and growth behavior
+
+Increasing the `backendQuotaGiB` value raises the limit etcd enforces; it does not immediately assign or reserve disk space. The database grows toward the new limit as cluster state changes. Dedicated etcd disk capacity on control plane nodes must exceed the configured quota.
+
+If the database reaches the configured quota, etcd enters a `NOSPACE` state. Write operations fail, including patches that would further increase quota, until you delete Kubernetes objects and reclaim space.
+
+[id="etcd-selectable-db-memory-sizing_{context}"]
+== Control plane memory sizing
+
+etcd memory-maps the database file. Larger quotas allow higher resident memory during normal operation, startup, write-ahead log (WAL) replay, snapshot recovery, and defragmentation.
+
+Minimum:: Allocate control plane node memory at least 4 times the configured database size.
+Realistic workloads at 16 GiB quota:: Plan for up to 10 times the database size, which is about 64–160 GiB per control plane node depending on churn and object count.
+
+.Control plane memory planning
+[cols="1,1,1,2", options="header"]
+|===
+|Configured `backendQuotaGiB` value |Minimum memory |Recommended range |Notes
+
+|8 GiB (default)
+|32 GiB
+|Up to 80 GiB
+|Baseline for most clusters
+
+|12 GiB
+|48 GiB
+|Up to 120 GiB
+|Common intermediate setting for large clusters
+
+|16 GiB (maximum)
+|64 GiB
+|Up to 160 GiB
+|Validated GA maximum; highest memory demand
+|===
+
+[id="etcd-selectable-db-defrag_{context}"]
+== Defragmentation and operational impact
+
+The `cluster-etcd-operator` defragmentation controller triggers when on-disk fragmentation reaches about 45%, which is the gap between `etcd_mvcc_db_total_size_in_bytes` and in-use size after compaction. This fragmentation threshold is not derived from the `backendQuotaGiB` value. Defragmentation timing depends on compaction and object churn, not quota use.
+
+Defragmentation is not considered when on-disk size is below 100 MiB. Defragmentation is rolling, blocking, and performed one etcd member at a time. Duration and transient memory pressure scale with database size. Larger databases also extend write-blocking during defragmentation and increase snapshot sync times during member recovery. Brief write latency during defragmentation is expected.
+
+