Skip to content

OSDOCS-17042 [CQA] Update infrastructure machine set creation documentation #105997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bscott-rh wants to merge 2 commits into
base: main
Choose a base branch
from
Open

OSDOCS-17042 [CQA] Update infrastructure machine set creation documentation #105997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9050fb1
Update infrastructure machine set creation documentation
bscott-rh Feb 4, 2026
eb1506e
Merge review feedback
bscott-rh Mar 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 33 additions & 73 deletions machine_management/creating-infrastructure-machinesets.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,114 +6,74 @@ include::_attributes/common-attributes.adoc[]

toc::[]

include::snippets/machine-user-provisioned-limitations.adoc[leveloffset=+1]

You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.

include::modules/infrastructure-components.adoc[leveloffset=+1]
[role="_abstract"]
To reduce subscription costs, you can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, cluster metrics, and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.

For information about infrastructure nodes and which components can run on infrastructure nodes, see the "Red Hat OpenShift control plane and infrastructure nodes" section in the link:https://www.redhat.com/en/resources/openshift-subscription-sizing-guide[OpenShift sizing and subscription guide for enterprise Kubernetes] document.

To create an infrastructure node, you can xref:../machine_management/creating-infrastructure-machinesets.adoc#machineset-creating_creating-infrastructure-machinesets[use a machine set], xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-an-infra-node_creating-infrastructure-machinesets[label the node], or xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infra-machines_creating-infrastructure-machinesets[use a machine config pool].
Comment thread
bscott-rh marked this conversation as resolved.
To create an infrastructure node, you can xref:../machine_management/creating-infrastructure-machinesets.adoc#machineset-creating_creating-infrastructure-machinesets[use a machine set], xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-an-infra-node_creating-infrastructure-machinesets[label the node], or xref:../machine_management/creating-infrastructure-machinesets.adoc#creating-infra-machines_creating-infrastructure-machinesets[use a machine config pool]. Use the sample compute machine set for your cloud to deploy an infrastructure machine set. Modify the sample configuration file with the details of your environment.

[id="creating-infrastructure-machinesets-clouds"]
=== Creating infrastructure machine sets for different clouds

Use the sample compute machine set for your cloud.
include::snippets/machine-user-provisioned-limitations.adoc[leveloffset=+1]

include::modules/machineset-yaml-aws.adoc[leveloffset=+3]
include::modules/infrastructure-components.adoc[leveloffset=+1]

Machine sets running on AWS support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-aws.adoc#machineset-non-guaranteed-instance_creating-machineset-aws[Spot Instances]. You can save on costs by using Spot Instances at a lower price compared to
On-Demand Instances on AWS. xref:../machine_management/creating_machinesets/creating-machineset-aws.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-aws[Configure Spot Instances] by adding `spotMarketOptions` to the `MachineSet` YAML file.
Comment thread
bscott-rh marked this conversation as resolved.
include::modules/machineset-yaml-aws.adoc[leveloffset=+1]

include::modules/machineset-yaml-azure.adoc[leveloffset=+3]
[role="_additional-resources"]
.Additional resources
* xref:../machine_management/creating_machinesets/creating-machineset-aws.adoc#machineset-non-guaranteed-instance_creating-machineset-aws[Machine sets that deploy machines as Spot Instances]

Machine sets running on Azure support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#machineset-non-guaranteed-instance_creating-machineset-azure[Spot VMs]. You can save on costs by using Spot VMs at a lower price compared to standard VMs on Azure. You can xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-azure[configure Spot VMs] by adding `spotVMOptions` to the `MachineSet` YAML file.
include::modules/machineset-yaml-azure.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources
* xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#machineset-non-guaranteed-instance_creating-machineset-azure[Machine sets that deploy machines as Spot VMs]
* xref:../machine_management/creating_machinesets/creating-machineset-azure.adoc#installation-azure-marketplace-subscribe_creating-machineset-azure[Using the Azure Marketplace offering]

include::modules/machineset-yaml-azure-stack-hub.adoc[leveloffset=+3]

[NOTE]
====
Machine sets running on Azure Stack Hub do not support non-guaranteed Spot VMs.
====
include::modules/machineset-yaml-azure-stack-hub.adoc[leveloffset=+1]

include::modules/machineset-yaml-ibm-cloud.adoc[leveloffset=+3]
include::modules/machineset-yaml-ibm-cloud.adoc[leveloffset=+1]

include::modules/machineset-yaml-gcp.adoc[leveloffset=+3]
include::modules/machineset-yaml-gcp.adoc[leveloffset=+1]

Machine sets running on {gcp-short} support non-guaranteed xref:../machine_management/creating_machinesets/creating-machineset-gcp.adoc#machineset-non-guaranteed-instance_creating-machineset-gcp[preemptible VM instances]. You can save on costs by using preemptible VM instances at a lower price
compared to normal instances on {gcp-short}. You can xref:../machine_management/creating_machinesets/creating-machineset-gcp.adoc#machineset-creating-non-guaranteed-instance_creating-machineset-gcp[configure preemptible VM instances] by adding `preemptible` to the `MachineSet` YAML file.
[role="_additional-resources"]
.Additional resources
* xref:../machine_management/creating_machinesets/creating-machineset-gcp.adoc#machineset-non-guaranteed-instance_legacy-preempt[Machine sets that deploy machines as preemptible VM instances]

include::modules/machineset-yaml-nutanix.adoc[leveloffset=+3]
include::modules/machineset-yaml-nutanix.adoc[leveloffset=+1]

include::modules/machineset-yaml-osp.adoc[leveloffset=+3]
include::modules/machineset-yaml-osp.adoc[leveloffset=+1]

include::modules/machineset-yaml-vsphere.adoc[leveloffset=+3]
include::modules/machineset-yaml-vsphere.adoc[leveloffset=+1]

include::modules/machineset-creating.adoc[leveloffset=+2]
include::modules/machineset-creating.adoc[leveloffset=+1]

include::modules/creating-an-infra-node.adoc[leveloffset=+2]
include::modules/creating-an-infra-node.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources

* xref:moving-resources-to-infrastructure-machinesets[Moving resources to infrastructure machine sets]
* xref:../machine_management/creating-infrastructure-machinesets.adoc#infrastructure-components_creating-infrastructure-machinesets[{product-title} infrastructure components]
* xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets_creating-infrastructure-machinesets[Moving resources to infrastructure machine sets]

include::modules/creating-infra-machines.adoc[leveloffset=+2]
include::modules/creating-infra-machines.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources

* xref:../architecture/control-plane.adoc#architecture-machine-config-pools_control-plane[Node configuration management with machine config pools].
* xref:../architecture/control-plane.adoc#architecture-machine-config-pools_control-plane[Node configuration management with machine config pools]

[id="assigning-machineset-resources-to-infra-nodes"]
== Assigning machine set resources to infrastructure nodes

After creating an infrastructure machine set, the `worker` and `infra` roles are applied to new infra nodes. Nodes with the `infra` role applied are not counted toward the total number of subscriptions that are required to run the environment, even when the `worker` role is also applied.

However, with an infra node being assigned as a worker, there is a chance user workloads could get inadvertently assigned to an infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods you want to control.

include::modules/binding-infra-node-workloads-using-taints-tolerations.adoc[leveloffset=+2]
include::modules/binding-infra-node-workloads-using-taints-tolerations.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources

* xref:../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler].
* xref:moving-resources-to-infrastructure-machinesets[Moving resources to infrastructure machine sets].
* xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations-about_nodes-scheduler-taints-tolerations[Understanding taints and tolerations].

[id="moving-resources-to-infrastructure-machinesets"]
== Moving resources to infrastructure machine sets

Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created by adding the infrastructure node selector, as shown:

[source,yaml]
----
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
name: cluster
# ...
spec:
nodePlacement: <1>
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: ""
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/infra
value: reserved
- effect: NoExecute
key: node-role.kubernetes.io/infra
value: reserved
----
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

Applying a specific node selector to all infrastructure components causes {product-title} to xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets[schedule those workloads on nodes with that label].
* xref:../machine_management/creating-infrastructure-machinesets.adoc#infrastructure-components_creating-infrastructure-machinesets[{product-title} infrastructure components]
* xref:../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler]
* xref:../machine_management/creating-infrastructure-machinesets.adoc#moving-resources-to-infrastructure-machinesets_creating-infrastructure-machinesets[Moving resources to infrastructure machine sets]
* xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations-about_nodes-scheduler-taints-tolerations[Understanding taints and tolerations]

include::modules/infra-machine-sets-moving.adoc[leveloffset=+1]

include::modules/infrastructure-moving-router.adoc[leveloffset=+2]

Expand Down
34 changes: 19 additions & 15 deletions modules/binding-infra-node-workloads-using-taints-tolerations.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,27 @@
[id="binding-infra-node-workloads-using-taints-tolerations_{context}"]
= Binding infrastructure node workloads using taints and tolerations

[role="_abstract"]
To avoid user workloads being inadvertently assigned to an infra node, you can apply a taint to the infra node and tolerations for the pods you want to control. After creating an infrastructure machine set, the `worker` and `infra` roles are applied to new infra nodes. Nodes with the `infra` role applied are not counted toward the total number of subscriptions that are required to run the environment, even when the `worker` role is also applied.

If you have an infrastructure node that has the `infra` and `worker` roles assigned, you must configure the node so that user workloads are not assigned to it.

[IMPORTANT]
====
It is recommended that you preserve the dual `infra,worker` label that is created for infrastructure nodes and use taints and tolerations to manage nodes that user workloads are scheduled on. If you remove the `worker` label from the node, you must create a custom pool to manage it. A node with a label other than `master` or `worker` is not recognized by the MCO without a custom pool. Maintaining the `worker` label allows the node to be managed by the default worker machine config pool, if no custom pools that select the custom label exists. The `infra` label communicates to the cluster that it does not count toward the total number of subscriptions.
====

If you added a `NoSchedule` taint to the infrastructure node, any pods that are controlled by a daemon set on that node are marked as `misscheduled`. You must either delete the pods or add a toleration to the pods as shown in the Red Hat Knowledgebase solution link:https://access.redhat.com/solutions/6592171[add toleration on `misscheduled` DNS pods]. Note that you cannot add a toleration to a daemon set object that is managed by an operator.

.Prerequisites

* Configure additional `MachineSet` objects in your {product-title} cluster.

.Procedure

. Add a taint to the infrastructure node to prevent scheduling user workloads on it:
. Add a taint to the infrastructure node to prevent scheduling user workloads on it.

.. Determine if the node has the taint:
.. Determine if the node has the taint by running the following command:
+
[source,terminal]
----
Expand All @@ -42,7 +47,7 @@ Taints: node-role.kubernetes.io/infra=reserved:NoSchedule
+
This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

.. If you have not configured a taint to prevent scheduling user workloads on it:
.. If you have not configured a taint to prevent scheduling user workloads on it, configure a taint by running the following command:
+
[source,terminal]
----
Expand Down Expand Up @@ -76,9 +81,7 @@ spec:
----
====
+
These examples place a taint on `node1` that has the `node-role.kubernetes.io/infra` key and the `NoSchedule` taint effect. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
+
If you added a `NoSchedule` taint to the infrastructure node, any pods that are controlled by a daemon set on that node are marked as `misscheduled`. You must either delete the pods or add a toleration to the pods as shown in the Red Hat Knowledgebase solution link:https://access.redhat.com/solutions/6592171[add toleration on `misscheduled` DNS pods]. Note that you cannot add a toleration to a daemon set object that is managed by an operator.
These examples place a taint on `node1` that has the `node-role.kubernetes.io/infra` key and the `NoSchedule` taint effect. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
+
[NOTE]
====
Expand All @@ -98,15 +101,16 @@ metadata:
spec:
# ...
tolerations:
- key: node-role.kubernetes.io/infra <1>
value: reserved <2>
effect: NoSchedule <3>
operator: Equal <4>
- key: <node_taint_key>
value: <node_taint_value>
effect: <taint_effect>
operator: Equal
----
<1> Specify the key that you added to the node.
<2> Specify the value of the key-value pair taint that you added to the node.
<3> Specify the effect that you added to the node.
<4> Specify the `Equal` Operator to require a taint with the key `node-role.kubernetes.io/infra` to be present on the node.
where:

`<node_taint_key>`:: Specifies the key that you added to the taint on the node.
`<node_taint_value>`:: Specifies the value of the key-value pair taint that you added to the node.
`<taint_effect>`:: Specifies the effect that you added to the node.
+
This toleration matches the taint created by the `oc adm taint` command. A pod with this toleration can be scheduled onto the infrastructure node.
+
Expand All @@ -117,4 +121,4 @@ Moving pods for an Operator installed via OLM to an infrastructure node is not a

. Schedule the pod to the infrastructure node by using a scheduler. See the documentation for "Controlling pod placement using the scheduler" for details.

. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components".
. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node.
21 changes: 9 additions & 12 deletions modules/creating-an-infra-node.adoc
View file
Open in desktop
Comment thread
bscott-rh marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,17 @@
[id="creating-an-infra-node_{context}"]
= Creating an infrastructure node

[IMPORTANT]
====
See "Creating infrastructure machine sets" for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the machine API.
====

[role="_abstract"]
You can use labels to configure worker nodes as infrastructure nodes, where you can move infrastructure resources.
To reduce subscription costs, you can use labels to configure worker nodes as infrastructure nodes, where you can move infrastructure resources.

After you create the infrastructure nodes, you can move appropriate workloads to those nodes by using taints and tolerations.

You can optionally create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces and creates an intersection with any existing node selectors on a pod, which additionally constrains the pod's selector.

[IMPORTANT]
====
See "Creating infrastructure machine sets" for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the Machine API.

If the default node selector key conflicts with the key of a pod's label, then the default node selector is not applied.

However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as `node-role.kubernetes.io/infra=""`, when a pod's label is set to a different node role, such as `node-role.kubernetes.io/master=""`, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.
Expand All @@ -31,31 +28,31 @@ You can alternatively use a project node selector to avoid cluster-wide node sel

.Procedure

. Add a label to the worker nodes that you want to act as infrastructure nodes:
. Add a label to the worker nodes that you want to act as infrastructure nodes by running the following command:
+
[source,terminal]
----
$ oc label node <node-name> node-role.kubernetes.io/infra=""
----

. Check to see if applicable nodes now have the `infra` role:
. Check to see if applicable nodes now have the `infra` role by running the following command:
+
[source,terminal]
----
$ oc get nodes
----

. Optional: Create a default cluster-wide node selector:
. Optional: Create a default cluster-wide node selector.
+
--
.. Edit the `Scheduler` object:
.. Edit the `Scheduler` object by running the following command:
+
[source,terminal]
----
$ oc edit scheduler cluster
----

.. Add the `defaultNodeSelector` field with the appropriate node selector:
.. Add the `defaultNodeSelector` field with the appropriate node selector by running the following command:
+
[source,yaml]
----
Expand All @@ -73,4 +70,4 @@ This example node selector deploys pods on infrastructure nodes by default.
.. Save the file to apply the changes.
--
+
You can now move infrastructure resources to the new infrastructure nodes. Also, remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components".
You can now move infrastructure resources to the new infrastructure nodes and remove any workloads that you do not want, or that do not belong, on the new infrastructure node.
Loading