From 9050fb1d0b36419554ed1f3ff633802b9f85ca58 Mon Sep 17 00:00:00 2001 From: Ben Scott Date: Wed, 4 Feb 2026 14:40:46 -0500 Subject: [PATCH 1/2] Update infrastructure machine set creation documentation Update and reorganize the infrastructure machine set documentation per OSDOCS-17042. Key changes: - Add new module infra-machine-sets-moving.adoc for resource movement guidance - Add creating-infra-machine-sets-prod.adoc for production considerations - Update creating-infrastructure-machinesets.adoc assembly structure - Refine procedures in creating-infra-machines.adoc - Update infrastructure component movement modules (monitoring, registry, router) - Improve taints and tolerations documentation - Remove elasticsearch references per 103344 - Fix xrefs and definition list rendering Co-Authored-By: Claude Sonnet 4.5 --- .../creating-infrastructure-machinesets.adoc | 99 +++++-------------- ...de-workloads-using-taints-tolerations.adoc | 24 +++-- modules/creating-an-infra-node.adoc | 4 +- modules/creating-infra-machine-sets-prod.adoc | 10 ++ modules/creating-infra-machines.adoc | 68 +++++-------- modules/infra-machine-sets-moving.adoc | 34 +++++++ modules/infrastructure-components.adoc | 3 - modules/infrastructure-moving-monitoring.adoc | 15 +-- modules/infrastructure-moving-registry.adoc | 8 +- modules/infrastructure-moving-router.adoc | 13 ++- 10 files changed, 127 insertions(+), 151 deletions(-) create mode 100644 modules/creating-infra-machine-sets-prod.adoc create mode 100644 modules/infra-machine-sets-moving.adoc diff --git a/machine_management/creating-infrastructure-machinesets.adoc b/machine_management/creating-infrastructure-machinesets.adoc index 880a6cffe337..9bea8cfc509d 100644 --- a/machine_management/creating-infrastructure-machinesets.adoc +++ b/machine_management/creating-infrastructure-machinesets.adoc @@ -6,114 +6,61 @@ include::_attributes/common-attributes.adoc[] toc::[] -include::snippets/machine-user-provisioned-limitations.adoc[leveloffset=+1] - +[role="_abstract"] You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment. -include::modules/infrastructure-components.adoc[leveloffset=+1] - -For information about infrastructure nodes and which components can run on infrastructure nodes, see the "Red Hat OpenShift control plane and infrastructure nodes" section in the link:https://www.redhat.com/en/resources/openshift-subscription-sizing-guide[OpenShift sizing and subscription guide for enterprise Kubernetes] document. - -To create an infrastructure node, you can xref:../machine_management/creating-infrastructure-machinesets.adoc#machineset-creating_creating-infrastructure-machinesets[use a machine set], xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-an-infra-node_creating-infrastructure-machinesets[label the node], or xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infra-machines_creating-infrastructure-machinesets[use a machine config pool]. - -[id="creating-infrastructure-machinesets-clouds"] -=== Creating infrastructure machine sets for different clouds - -Use the sample compute machine set for your cloud. +include::snippets/machine-user-provisioned-limitations.adoc[leveloffset=+1] -include::modules/machineset-yaml-aws.adoc[leveloffset=+3] +include::modules/infrastructure-components.adoc[leveloffset=+1] -Machine sets running on AWS support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-aws.adoc#machineset-non-guaranteed-instance_creating-machineset-aws[Spot Instances]. You can save on costs by using Spot Instances at a lower price compared to -On-Demand Instances on AWS. xref:../machine_management/creating_machinesets/creating-machineset-aws.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-aws[Configure Spot Instances] by adding `spotMarketOptions` to the `MachineSet` YAML file. +include::modules/creating-infra-machine-sets-prod.adoc[leveloffset=+1] -include::modules/machineset-yaml-azure.adoc[leveloffset=+3] +include::modules/machineset-yaml-aws.adoc[leveloffset=+2] -Machine sets running on Azure support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#machineset-non-guaranteed-instance_creating-machineset-azure[Spot VMs]. You can save on costs by using Spot VMs at a lower price compared to standard VMs on Azure. You can xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-azure[configure Spot VMs] by adding `spotVMOptions` to the `MachineSet` YAML file. +include::modules/machineset-yaml-azure.adoc[leveloffset=+2] [role="_additional-resources"] .Additional resources * xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#installation-azure-marketplace-subscribe_creating-machineset-azure[Using the Azure Marketplace offering] -include::modules/machineset-yaml-azure-stack-hub.adoc[leveloffset=+3] - -[NOTE] -==== -Machine sets running on Azure Stack Hub do not support non-guaranteed Spot VMs. -==== +include::modules/machineset-yaml-azure-stack-hub.adoc[leveloffset=+2] -include::modules/machineset-yaml-ibm-cloud.adoc[leveloffset=+3] +include::modules/machineset-yaml-ibm-cloud.adoc[leveloffset=+2] -include::modules/machineset-yaml-gcp.adoc[leveloffset=+3] +include::modules/machineset-yaml-gcp.adoc[leveloffset=+2] -Machine sets running on {gcp-short} support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-gcp.adoc#machineset-non-guaranteed-instance_creating-machineset-gcp[preemptible VM instances]. You can save on costs by using preemptible VM instances at a lower price -compared to normal instances on {gcp-short}. You can xref:../machine_management/creating_machinesets/creating-machineset-gcp.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-gcp[configure preemptible VM instances] by adding `preemptible` to the `MachineSet` YAML file. +include::modules/machineset-yaml-nutanix.adoc[leveloffset=+2] -include::modules/machineset-yaml-nutanix.adoc[leveloffset=+3] +include::modules/machineset-yaml-osp.adoc[leveloffset=+2] -include::modules/machineset-yaml-osp.adoc[leveloffset=+3] +include::modules/machineset-yaml-vsphere.adoc[leveloffset=+2] -include::modules/machineset-yaml-vsphere.adoc[leveloffset=+3] +include::modules/machineset-creating.adoc[leveloffset=+1] -include::modules/machineset-creating.adoc[leveloffset=+2] - -include::modules/creating-an-infra-node.adoc[leveloffset=+2] +include::modules/creating-an-infra-node.adoc[leveloffset=+1] [role="_additional-resources"] .Additional resources -* xref:moving-resources-to-infrastructure-machinesets[Moving resources to infrastructure machine sets] +* xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets_creating-infrastructure-machinesets[Moving resources to infrastructure machine sets] -include::modules/creating-infra-machines.adoc[leveloffset=+2] +include::modules/creating-infra-machines.adoc[leveloffset=+1] [role="_additional-resources"] .Additional resources -* xref:../architecture/control-plane.adoc#architecture-machine-config-pools_control-plane[Node configuration management with machine config pools]. - -[id="assigning-machineset-resources-to-infra-nodes"] -== Assigning machine set resources to infrastructure nodes +* xref:../architecture/control-plane.adoc#architecture-machine-config-pools_control-plane[Node configuration management with machine config pools] -After creating an infrastructure machine set, the `worker` and `infra` roles are applied to new infra nodes. Nodes with the `infra` role applied are not counted toward the total number of subscriptions that are required to run the environment, even when the `worker` role is also applied. - -However, with an infra node being assigned as a worker, there is a chance user workloads could get inadvertently assigned to an infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods you want to control. - -include::modules/binding-infra-node-workloads-using-taints-tolerations.adoc[leveloffset=+2] +include::modules/binding-infra-node-workloads-using-taints-tolerations.adoc[leveloffset=+1] [role="_additional-resources"] .Additional resources -* xref:../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler]. -* xref:moving-resources-to-infrastructure-machinesets[Moving resources to infrastructure machine sets]. -* xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations-about_nodes-scheduler-taints-tolerations[Understanding taints and tolerations]. - -[id="moving-resources-to-infrastructure-machinesets"] -== Moving resources to infrastructure machine sets - -Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created by adding the infrastructure node selector, as shown: - -[source,yaml] ----- -apiVersion: imageregistry.operator.openshift.io/v1 -kind: Config -metadata: - name: cluster -# ... -spec: - nodePlacement: <1> - nodeSelector: - matchLabels: - node-role.kubernetes.io/infra: "" - tolerations: - - effect: NoSchedule - key: node-role.kubernetes.io/infra - value: reserved - - effect: NoExecute - key: node-role.kubernetes.io/infra - value: reserved ----- -<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `: ` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration. - -Applying a specific node selector to all infrastructure components causes {product-title} to xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets[schedule those workloads on nodes with that label]. +* xref:../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler] +* xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets_creating-infrastructure-machinesets[Moving resources to infrastructure machine sets] +* xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations-about_nodes-scheduler-taints-tolerations[Understanding taints and tolerations] + +include::modules/infra-machine-sets-moving.adoc[leveloffset=+1] include::modules/infrastructure-moving-router.adoc[leveloffset=+2] diff --git a/modules/binding-infra-node-workloads-using-taints-tolerations.adoc b/modules/binding-infra-node-workloads-using-taints-tolerations.adoc index 99beddac3f3e..3370469ce328 100644 --- a/modules/binding-infra-node-workloads-using-taints-tolerations.adoc +++ b/modules/binding-infra-node-workloads-using-taints-tolerations.adoc @@ -7,6 +7,9 @@ [id="binding-infra-node-workloads-using-taints-tolerations_{context}"] = Binding infrastructure node workloads using taints and tolerations +[role="_abstract"] +You can apply a taint to the infra node and tolerations for the pods you want to control, to avoid user workloads being inadvertently assigned to an infra node. After creating an infrastructure machine set, the `worker` and `infra` roles are applied to new infra nodes. Nodes with the `infra` role applied are not counted toward the total number of subscriptions that are required to run the environment, even when the `worker` role is also applied. + If you have an infrastructure node that has the `infra` and `worker` roles assigned, you must configure the node so that user workloads are not assigned to it. [IMPORTANT] @@ -76,7 +79,7 @@ spec: ---- ==== + -These examples place a taint on `node1` that has the `node-role.kubernetes.io/infra` key and the `NoSchedule` taint effect. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node. +These examples place a taint on `node1` that has the `node-role.kubernetes.io/infra` key and the `NoSchedule` taint effect. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node. + If you added a `NoSchedule` taint to the infrastructure node, any pods that are controlled by a daemon set on that node are marked as `misscheduled`. You must either delete the pods or add a toleration to the pods as shown in the Red Hat Knowledgebase solution link:https://access.redhat.com/solutions/6592171[add toleration on `misscheduled` DNS pods]. Note that you cannot add a toleration to a daemon set object that is managed by an operator. + @@ -98,15 +101,16 @@ metadata: spec: # ... tolerations: - - key: node-role.kubernetes.io/infra <1> - value: reserved <2> - effect: NoSchedule <3> - operator: Equal <4> + - key: + value: + effect: + operator: Equal ---- -<1> Specify the key that you added to the node. -<2> Specify the value of the key-value pair taint that you added to the node. -<3> Specify the effect that you added to the node. -<4> Specify the `Equal` Operator to require a taint with the key `node-role.kubernetes.io/infra` to be present on the node. +where: + +`node_taint_key`:: Specifies the key that you added to the taint on the node. +`node_taint_value`:: Specifies the value of the key-value pair taint that you added to the node. +`taint_effect`:: Specifies the effect that you added to the node. + This toleration matches the taint created by the `oc adm taint` command. A pod with this toleration can be scheduled onto the infrastructure node. + @@ -117,4 +121,4 @@ Moving pods for an Operator installed via OLM to an infrastructure node is not a . Schedule the pod to the infrastructure node by using a scheduler. See the documentation for "Controlling pod placement using the scheduler" for details. -. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components". +. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components". diff --git a/modules/creating-an-infra-node.adoc b/modules/creating-an-infra-node.adoc index ccfc6faf005e..b8634f6e07a6 100644 --- a/modules/creating-an-infra-node.adoc +++ b/modules/creating-an-infra-node.adoc @@ -14,7 +14,7 @@ See "Creating infrastructure machine sets" for installer-provisioned infrastruct ==== [role="_abstract"] -You can use labels to configure worker nodes as infrastructure nodes, where you can move infrastructure resources. +You can use labels to configure worker nodes as infrastructure nodes, where you can move infrastructure resources. After you create the infrastructure nodes, you can move appropriate workloads to those nodes by using taints and tolerations. @@ -22,6 +22,8 @@ You can optionally create a default cluster-wide node selector. The default node [IMPORTANT] ==== +See Creating infrastructure machine sets for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the machine API. + If the default node selector key conflicts with the key of a pod's label, then the default node selector is not applied. However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as `node-role.kubernetes.io/infra=""`, when a pod's label is set to a different node role, such as `node-role.kubernetes.io/master=""`, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles. diff --git a/modules/creating-infra-machine-sets-prod.adoc b/modules/creating-infra-machine-sets-prod.adoc new file mode 100644 index 000000000000..f3ecb1dc4614 --- /dev/null +++ b/modules/creating-infra-machine-sets-prod.adoc @@ -0,0 +1,10 @@ +// Module included in the following assemblies: +// +// * machine_management/creating-infrastructure-machinesets.adoc + +:_mod-docs-content-type: CONCEPT +[id="creating-infrastructure-machinesets-production_{context}"] += Creating infrastructure machine sets for production environments + +[role="_abstract"] +Use the sample compute machine set for your cloud to deploy an infrastructure machine set. Modify the sample configuration file with the details of your environment. \ No newline at end of file diff --git a/modules/creating-infra-machines.adoc b/modules/creating-infra-machines.adoc index ac03ef41e969..8fd1efe16312 100644 --- a/modules/creating-infra-machines.adoc +++ b/modules/creating-infra-machines.adoc @@ -7,7 +7,8 @@ [id="creating-infra-machines_{context}"] = Creating a machine config pool for infrastructure machines -If you need infrastructure machines to have dedicated configurations, you must create an infra pool. +[role="_abstract"] +You can create a machine configuration pool for infrastructure machines to apply dedicated configuration to infra machines. You may want to apply dedicated configuration to infra machines because they run distinct workloads from other nodes in the cluster. [IMPORTANT] ==== @@ -16,26 +17,19 @@ Creating a custom machine configuration pool overrides default worker pool confi .Procedure -. Add a label to the node you want to assign as the infra node with a specific label: +. Add a label to the node you want to assign as the infra node by running the following command: + [source,terminal] ---- -$ oc label node