Skip to content

OSDOCS Nodes Additional Artifact Store : GA 5.0 - #118494

Merged
mburke5678 merged 1 commit into
openshift:mainfrom
mburke5678:nodes-additional-storage-ga
Sep 9, 2026
Merged

OSDOCS Nodes Additional Artifact Store : GA 5.0#118494
mburke5678 merged 1 commit into
openshift:mainfrom
mburke5678:nodes-additional-storage-ga

Conversation

@mburke5678

@mburke5678 mburke5678 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

https://redhat.atlassian.net/browse/OSDOCS-21685

Preview --
Nodes -> Working with Nodes -> Additional CRI-O storage locations for faster container startup -- Remove TP snippet. Minor edits.
Nodes -> Working with Nodes -> About additional CRI-O storage locations -- Edits.
Nodes -> Working with Nodes -> Configuring additional CRI-O storage locations -- Removed TP prereq and edits.
Nodes -> Working with Nodes -> Limitations and known issues with additional CRI-O storage locations -- New module
Nodes -> Working with Nodes -> Troubleshoot additional CRI-O storage locations -- New module

QE review:

  • QE has approved this change.

@mburke5678 mburke5678 added this to the OCP 5.0 GA milestone Aug 20, 2026
@openshift-ci openshift-ci Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Aug 20, 2026
@ocpdocs-previewbot

ocpdocs-previewbot commented Aug 20, 2026

Copy link
Copy Markdown

@openshift-ci openshift-ci Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 24, 2026
Comment thread modules/nodes-nodes-additional-crio-storage-troubleshooting.adoc Outdated
@mburke5678
mburke5678 force-pushed the nodes-additional-storage-ga branch 2 times, most recently from 9b3bfdb to 9c1de7b Compare August 24, 2026 20:13
@mburke5678

Copy link
Copy Markdown
Contributor Author

@saschagrunert PTAL

@saschagrunert saschagrunert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical review based on OCPNODE-4527 requirements and CRI-O internals. Six inline comments below.


[role="_abstract"]
To reduce application startup time and make your applications run more efficiently, you can configure additional storage locations for the CRI-O container engine to store OCI objects by using the `ContainerRuntimeConfig` custom resource (CR).
To reduce application startup time and make your applications run more efficiently, you can configure additional CRI-O storage locations for the CRI-O container engine by using the `ContainerRuntimeConfig` custom resource (CR).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"CRI-O" appears twice: "additional CRI-O storage locations for the CRI-O container engine." The original phrasing ("additional storage locations for the CRI-O container engine to store OCI objects") was cleaner and included the useful "to store OCI objects" context.

Suggestion:

To reduce application startup time and make your applications run more efficiently, you can configure additional storage locations for the CRI-O container engine by using the `ContainerRuntimeConfig` custom resource (CR).

= Supported storage backends for lazy pulling

[role="_abstract"]
When using the `additionalLayerStores` field to configure lazy pulling with additional CRI-O storage, you must install one of the supported storage backends.

@saschagrunert saschagrunert Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This intro ties the table to additionalLayerStores and says you "must install one of the supported storage backends." But the first row in the table lists zstd:chunked as "Native (built-in)" with GA support. You don't install something that's built-in.

zstd:chunked partial pulling works through CRI-O's built-in pull path (containers/storage uses the embedded TOC to do partial pulls via HTTP Range requests). It does not use additionalLayerStores at all. That field configures paths to externally-managed, read-only layer stores for BYOS plugins.

The intro contradicts the table. Consider either:

  • Rewriting the intro as a general reference for pulling backends, noting which require additionalLayerStores + plugin installation (BYOS) vs. which work natively
  • Or scoping the intro to BYOS plugins only and listing the native backends separately

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this in the docs:

With the additionalLayerStores field, you could enable lazy pulling through a third-party storage plugin, such as stargz-store

If you want to use stargz or nydus to enable lazy pulling, you need to configure additionalLayerStores to indicate where to store the pulled layers? But, if you use zstd:chunked or composefs, you don't need to/cannot use additionalLayerStores?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. additionalLayerStores configures FUSE mount paths for external (BYOS) plugins only. stargz-snapshotter and nydus-storage-plugin serve layers on-demand through a FUSE filesystem, so CRI-O needs to know the mount path (the MCO appends :ref to each path when writing storage.conf).

zstd:chunked works through CRI-O's built-in pull path: the containers/image library uses the embedded TOC to do partial pulls via HTTP Range requests. No external plugin, no additionalLayerStores. composefs is also a native overlay driver option (use_composefs in storage.conf), orthogonal to additionalLayerStores.

One more nuance worth noting: zstd:chunked does "partial pulling" (all missing content is fetched before container start, just deduplicated), not true "lazy pulling" (content fetched on-demand after container start). Only stargz/nydus do true lazy pulling. So the module title "Supported storage backends for lazy pulling" and the intro tying all backends to additionalLayerStores are both inaccurate for the native backends.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, within the Additional CRI-O storage locations for faster container startup topic, we really don't need to mention zstd:chunked and composefs, as they do not require additionalLayerStores.

Additional container image layers for lazy pulling
Use the additionalLayerStores field to enable lazy pulling through a third-party storage plugin, such as stargz-snapshotter or nydus-storage-plugin. These plugins serve image layers on-demand through a FUSE filesystem, so CRI-O needs to know the mount path.

If you are using zstd:chunked or composefs, you do not need to configure a `ContainerRuntimeConfig` object. 

And remove the whole section with this table?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, scoping this topic to BYOS plugins is the right call. zstd:chunked and composefs are native to containers/storage and don't use additionalLayerStores at all (confirmed: zero references to additionalLayerStores in pkg/chunked/, and composefs is an overlay driver option via use_composefs).

The proposed intro text is accurate. One small refinement: "lazy pulling" in this context specifically means on-demand layer serving at runtime via FUSE, which is what stargz-snapshotter and nydus-storage-plugin do. zstd:chunked does "partial pulling" (all needed chunks fetched at pull time via HTTP Range requests). Keeping that distinction helps avoid confusion.

Removing the backends table makes sense since it mixed two architecturally separate things: native pull-time optimizations (zstd:chunked, composefs) and BYOS FUSE plugins (stargz, nydus). Only the latter use additionalLayerStores.

+
To recover from this condition, restart the `DaemonSet` plugin. If CRI-O is stalled, also drain and reboot the node.
+
You can avoid this condition by setting a non-zero `PullProgressTimeout` value in CRI-O when additional layer stores are configured.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This recommends setting PullProgressTimeout but doesn't explain how. This option (pull_progress_timeout in CRI-O TOML config) likely can't be set via ContainerRuntimeConfig, since MCO only exposes a subset of CRI-O options through that CR. Users would need a machine config drop-in under /etc/crio/crio.conf.d/. Either add brief guidance on how to set it or link to the relevant configuration reference.

(Ref: OCPNODE-4527 P2-2)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a link to this Kbase article work?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KBase link is a reasonable starting point since it covers the same CRI-O option. But the docs should still clarify that pull_progress_timeout is not exposed through ContainerRuntimeConfig. The CR only exposes pidsLimit, logLevel, logSizeMax, overlaySize, defaultRuntime, and the three additional storage fields. Users would need a machine config drop-in under /etc/crio/crio.conf.d/ to set it.

So a link to the KBase article plus a brief note that this requires a machine config (not ContainerRuntimeConfig) would cover it.

Comment thread modules/nodes-nodes-additional-crio-storage-troubleshooting.adoc

* Lazy image pulling relies on HTTP Range requests to pull in chunks. As such, your registry must support HTTP range requests. If not supported, CRI-O falls back to standard image pulls.

* For lazy image pulling, you must convert the pulled container images from the standard OCI format to a lazy-pull-compatible format, such as eStargz or Nydus.

@saschagrunert saschagrunert Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lists "eStargz or Nydus" as lazy-pull-compatible formats but omits zstd:chunked, which is the only GA format in the backends compatibility matrix. Users following the GA path would use zstd:chunked, not eStargz or Nydus (both TP).

Note: zstd:chunked does partial pulling (all content fetched before container start, deduplicated), not lazy pulling (on-demand at runtime). Consider updating the terminology to cover both partial and lazy pulling formats.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users following the GA path would use zstd:chunked, not eStargz or Nydus (both TP).

What is the GA path? The user just needs to convert the container image to zstd:chunked format. When you do the pull, CRI-O knows to do the partial pull.
You suggested that zstd:chunked doesn't use additionalLayerStores. Does that mean additionalLayerStores is still TP?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GA path is zstd:chunked partial pulling through CRI-O's native pull path in containers/storage. But it's not just "convert and pull": enable_partial_images in storage.conf defaults to "false" in OpenShift (the MCO template sets it explicitly), and there's no ContainerRuntimeConfig field to toggle it. Users need a MachineConfig to set enable_partial_images = "true" in storage.conf.

additionalLayerStores as a ContainerRuntimeConfig field is GA in 5.0 (the AdditionalStorageConfig feature gate is in inDefault()). But zstd:chunked doesn't use additionalLayerStores at all; those are separate code paths. additionalLayerStores configures FUSE mount paths for BYOS plugins (stargz-snapshotter, nydus-storage-plugin), which are TP.

The limitation bullet should mention zstd:chunked alongside eStargz/Nydus (it also requires format conversion), or clarify that it only applies to BYOS lazy pulling. And the terminology should distinguish partial pulling (zstd:chunked, all chunks fetched at pull time) from lazy pulling (BYOS/FUSE, layers served on-demand at runtime).


include::modules/nodes-nodes-additional-crio-storage-about.adoc[leveloffset=+1]
include::modules/nodes-nodes-additional-crio-storage-backends.adoc[leveloffset=+2]
include::modules/nodes-nodes-additional-crio-storage-limitations.adoc[leveloffset=+2]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OCPNODE-4527 scope includes "Document upgrade path from TP to GA." This isn't covered anywhere in the PR. The Jira discussion provided details for three scenarios:

  • 4.21 to 4.22: new optional fields ignored by older MCO, opt-in via feature gate
  • 4.22 to 4.21 downgrade: delete ContainerRuntimeConfig resources with new fields, nodes reboot
  • GA in 5.0: remove TP caveats, document any API changes

Consider adding an upgrade path module here alongside backends and limitations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"4.21 to 4.22: new optional fields ignored by older MCO, opt-in via feature gate" -- We wouldn't typically document previous versions in the current docs in this way.
"4.22 to 4.21 downgrade: delete ContainerRuntimeConfig resources with new fields, nodes reboot" -- We typically don't document downgrading in the docs. If there is a pressing reason, we could add a note somewhere: "If you need to downgrade your cluster, before the downgrade, delete ContainerRuntimeConfig resources with new fields."
GA in 5.0: remove TP caveats, document any API changes -- Are there any API changes?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on all three points:

  1. 4.21 to 4.22: Makes sense, no need to document previous version behavior in current docs.
  2. Downgrade note: That compromise works. A short note like "before downgrading, delete ContainerRuntimeConfig resources with the new fields" covers it.
  3. API changes: No breaking API changes. The three fields (additionalArtifactStores, additionalImageStores, additionalLayerStores) are unchanged since their TP introduction. The only GA change is that the AdditionalStorageConfig feature gate was promoted to Default, so users no longer need TechPreviewNoUpgrade. There was also a minor validation tightening (blocking .. path traversal, OCPNODE-4526), but that's not a breaking change.


* This feature is not supported for {ms}, which does not use the Machine Config Operator.

* In the event you need to downgrade your cluster to a {product-title} to a version 4.21 or earlier, before the downgrade, delete any `ContainerRuntimeConfig` resource that includes the `additionalArtifactStores`, `additionalImageStores`, or `additionalLayerStores` fields.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [error] RedHat.TermsErrors: Use 'in case', 'if', or 'when' rather than 'In the event'. For more information, see RedHat.TermsErrors.

@mburke5678
mburke5678 force-pushed the nodes-additional-storage-ga branch from e1e0202 to d9f5d6e Compare August 26, 2026 15:05

@saschagrunert saschagrunert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second technical review. Five inline comments: one broken cross-reference from the backends module removal, two grammar fixes, one incomplete zstd:chunked NOTE, and one terminology nit.

* If you are configuring the `additionalLayerStores` field, you must meet the following additional prerequisites:

** A supported storage plugin binary must be installed on each node, such as Stargz Store or Nydus Storage Plugin. See "Stargz Store plugin" or "Nydus Storage Plugin" for more information. You must have installed the plugin by using one of the following methods:
** You must install a supported storage plugin binary on each node, such as Stargz Store or Nydus Storage Plugin. See "Supported storage backends for lazy pulling" for more information. You must install the plugin by using one of the following methods:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "Supported storage backends for lazy pulling" module was deleted in the latest commit, so this is now a dangling reference.

The original line referenced the Stargz Store and Nydus Storage Plugin links directly, which still exist in the Additional Resources section. Consider reverting to that approach or dropping the "See ... for more information" sentence, since the prereq already names both plugins.

Suggested change
** You must install a supported storage plugin binary on each node, such as Stargz Store or Nydus Storage Plugin. See "Supported storage backends for lazy pulling" for more information. You must install the plugin by using one of the following methods:
** You must install a supported storage plugin binary on each node, such as Stargz Store or Nydus Storage Plugin. You must install the plugin by using one of the following methods:

If you are using the Linux native zstd:chunked format for partial pulling, you do not need to configure a `ContainerRuntimeConfig` object. With partial pulling, CRI-O retrieves the chunk metadata, determines which chunks are needed, and fetches only those chunks using HTTP range requests rather than downloading the entire compressed layer.
====
+
To use lazy pulling, you must install a BYOS plugin on your nodes. Then, create a container runtime config to configures the FUSE mount paths.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: "to configures" should be "to configure".

Suggested change
To use lazy pulling, you must install a BYOS plugin on your nodes. Then, create a container runtime config to configures the FUSE mount paths.
To use lazy pulling, you must install a BYOS plugin on your nodes. Then, create a container runtime config to configure the FUSE mount paths.


* This feature is not supported for {ms}, which does not use the Machine Config Operator.

* If you need to downgrade your cluster to a {product-title} to a version 4.21 or earlier, before the downgrade, delete any `ContainerRuntimeConfig` resource that includes the `additionalArtifactStores`, `additionalImageStores`, or `additionalLayerStores` fields.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"to a {product-title} to a version" has a doubled phrase.

Suggested change
* If you need to downgrade your cluster to a {product-title} to a version 4.21 or earlier, before the downgrade, delete any `ContainerRuntimeConfig` resource that includes the `additionalArtifactStores`, `additionalImageStores`, or `additionalLayerStores` fields.
* If you need to downgrade your cluster to {product-title} version 4.21 or earlier, before the downgrade, delete any `ContainerRuntimeConfig` resource that includes the `additionalArtifactStores`, `additionalImageStores`, or `additionalLayerStores` fields.

+
[NOTE]
====
If you are using the Linux native zstd:chunked format for partial pulling, you do not need to configure a `ContainerRuntimeConfig` object. With partial pulling, CRI-O retrieves the chunk metadata, determines which chunks are needed, and fetches only those chunks using HTTP range requests rather than downloading the entire compressed layer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This NOTE tells users they do not need a ContainerRuntimeConfig, but does not mention that enable_partial_images in storage.conf defaults to "false" in the MCO template, and there is no ContainerRuntimeConfig field to toggle it. Users would need a MachineConfig drop-in to set enable_partial_images = "true".

As written, users will reasonably conclude zstd:chunked works with no configuration at all. Consider either mentioning the MachineConfig requirement or narrowing the statement to: "you do not need to configure additionalLayerStores" instead of "you do not need to configure a ContainerRuntimeConfig object."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the update here.
But, is there a Jira that tracks the process for setting enable_partial_images to configure zstd:chunked, showing it was fully tested? The process really needs its own topic in the docs.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there seems to be no dedicated Jira for documentation on enable_partial_images. The broader zstd:chunked support is tracked under OCPNODE-3137, but that epic is still "New".

Agreed it needs its own topic, since the configuration involves a MachineConfig drop-in, not a ContainerRuntimeConfig. That's out of scope for this PR. For now, I'd narrow this NOTE to say users don't need to configure additionalLayerStores for zstd:chunked, and I can file a story under OCPNODE-3137 to track the dedicated zstd:chunked docs.


The following limitations and known issues have been identified for additional CRI-O storage:

* Lazy image pulling relies on HTTP range requests to pull in chunks. As such, your registry must support HTTP range requests. If not supported, CRI-O falls back to standard image pulls.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the rest of this PR carefully distinguishes "lazy pulling" (BYOS, on-demand at runtime via FUSE) from "partial pulling" (zstd:chunked, all chunks before container start). Using "Lazy image pulling" here as a catch-all re-blurs that line. Both modes require registry Range request support, but for different reasons.

Consider: "Lazy pulling and partial pulling rely on HTTP range requests. As such, your registry must support HTTP range requests."

@saschagrunert saschagrunert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few other comments, the rest LGTM

Use the `additionalLayerStores` field to enable lazy pulling through a third-party Bring Your Own Storage (BYOS) plugin. With lazy pulling, you can start a container without waiting for the entire image to be downloaded. Instead, the necessary parts of the image are fetched on-demand during runtime using FUSE.
+
Note that CRI-O falls back to a standard image pull in the following cases:
External BYOS plugins, such as stargz-snapshotter and nydus-storage-plugin, serve container image layers on-demand through a FUSE file system. The `additionalLayerStores` parameter configures the FUSE mount paths for CRI-O.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additionalLayerStores is called "field" on line 71, "parameter" here, and "object" on line 79. Three different terms in 8 lines. In OpenShift docs, "object" typically means a Kubernetes resource (a CR, a Pod), so using it for a CR field is confusing. Suggest "field" consistently.

Suggested change
External BYOS plugins, such as stargz-snapshotter and nydus-storage-plugin, serve container image layers on-demand through a FUSE file system. The `additionalLayerStores` parameter configures the FUSE mount paths for CRI-O.
External BYOS plugins, such as stargz-snapshotter and nydus-storage-plugin, serve container image layers on-demand through a FUSE file system. The `additionalLayerStores` field configures the FUSE mount paths for CRI-O.

+
[NOTE]
====
If you are using the Linux native zstd:chunked format for partial pulling, you do not need to configure the `additionalLayerStores` object. With partial pulling, CRI-O retrieves the chunk metadata, determines which chunks are needed, and fetches only those chunks using HTTP range requests rather than downloading the entire compressed layer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same terminology issue: "object" → "field".

Suggested change
If you are using the Linux native zstd:chunked format for partial pulling, you do not need to configure the `additionalLayerStores` object. With partial pulling, CRI-O retrieves the chunk metadata, determines which chunks are needed, and fetches only those chunks using HTTP range requests rather than downloading the entire compressed layer.
If you are using the Linux native zstd:chunked format for partial pulling, you do not need to configure the `additionalLayerStores` field. With partial pulling, CRI-O retrieves the chunk metadata, determines which chunks are needed, and fetches only those chunks using HTTP range requests rather than downloading the entire compressed layer.

* link:https://github.com/containers/nydus-storage-plugin[Nydus Storage Plugin]
* link:https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md[eStargz format]
* link:https://nydus.dev/[Nydus format]
* xref:../../nodes/nodes/nodes-nodes-additional-crio-storage.adoc#nodes-nodes-additional-crio-storage-troubleshooting_nodes-nodes-additional-crio-storage[Troubleshoot additional storage locations for CRI-O]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module title is "Troubleshoot additional CRI-O storage locations" (troubleshooting.adoc:7). This xref text uses the old word order that the PR changed everywhere else. Same mismatch in limitations.adoc:20.

Suggested change
* xref:../../nodes/nodes/nodes-nodes-additional-crio-storage.adoc#nodes-nodes-additional-crio-storage-troubleshooting_nodes-nodes-additional-crio-storage[Troubleshoot additional storage locations for CRI-O]
* xref:../../nodes/nodes/nodes-nodes-additional-crio-storage.adoc#nodes-nodes-additional-crio-storage-troubleshooting_nodes-nodes-additional-crio-storage[Troubleshoot additional CRI-O storage locations]


* After you configure additional CRI-O storage, the Machine Config Operator (MCO) reboots the affected nodes with the new configuration.

* Storage plugin crashes or hangs can impact container creation. For more information, see "Troubleshoot additional storage locations for CRI-O".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same title mismatch as the assembly xref. The troubleshooting module title is "Troubleshoot additional CRI-O storage locations".

Suggested change
* Storage plugin crashes or hangs can impact container creation. For more information, see "Troubleshoot additional storage locations for CRI-O".
* Storage plugin crashes or hangs can impact container creation. For more information, see "Troubleshoot additional CRI-O storage locations".


* Lazy pulling and partial pulling rely on HTTP range requests. As such, your registry must support HTTP range requests. If not supported, CRI-O falls back to standard image pulls.

* When using an additional layer store for Bring Your Own Storage (BYOS) lazy pulling, you must convert the pulled container images from the standard OCI format to a compatible format, such as eStargz or Nydus. Note that the zstd:chunked format performs _partial pulling_, where all content is fetched before the container starts, then is deduplicated.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first sentence scopes to BYOS. The appended zstd:chunked note describes something that doesn't use additional layer stores at all. Readers could conclude zstd:chunked is a BYOS option. Splitting into two bullets keeps each pulling mode in its own context.

Suggested change
* When using an additional layer store for Bring Your Own Storage (BYOS) lazy pulling, you must convert the pulled container images from the standard OCI format to a compatible format, such as eStargz or Nydus. Note that the zstd:chunked format performs _partial pulling_, where all content is fetched before the container starts, then is deduplicated.
* When using an additional layer store for Bring Your Own Storage (BYOS) lazy pulling, you must convert the pulled container images from the standard OCI format to a compatible format, such as eStargz or Nydus.
* The zstd:chunked format performs _partial pulling_, where all content is fetched before the container starts, then is deduplicated. Partial pulling does not use the `additionalLayerStores` field.


* Storage plugin crashes or hangs can impact container creation. For more information, see "Troubleshoot additional storage locations for CRI-O".

* The additional layer store API is experimental upstream. Breaking changes are possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "upstream" is vague. Naming the project helps users understand where the instability lives.

Suggested change
* The additional layer store API is experimental upstream. Breaking changes are possible.
* The additional layer store API is experimental in the upstream containers/storage project. Breaking changes are possible.

@saschagrunert saschagrunert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing nit (not in the diff, so no inline comment): about.adoc line 12 has a subject-verb mismatch: "By using storage locations ... gives you control." A gerund phrase ("By using...") cannot be the subject of "gives." Should be "Using ... gives" or "By using ..., you gain control."

Other than these three comments, LGTM.


* When using an additional layer store for Bring Your Own Storage (BYOS) lazy pulling, you must convert the pulled container images from the standard OCI format to a compatible format, such as eStargz or Nydus.

* The zstd:chunked format performs _partial pulling_, where all content is fetched before the container starts, then is deduplicated. Partial pulling does not use the `additionalLayerStores` field.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"where all content is fetched before the container starts, then is deduplicated" is imprecise. What zstd:chunked actually does: CRI-O reads the TOC, checks which chunks are already on disk from prior pulls, and fetches only the missing ones via HTTP Range requests. The deduplication happens at pull time (skip what's already present), not as a post-fetch step.

Suggested change
* The zstd:chunked format performs _partial pulling_, where all content is fetched before the container starts, then is deduplicated. Partial pulling does not use the `additionalLayerStores` field.
* The zstd:chunked format performs _partial pulling_, where CRI-O fetches only the missing chunks before the container starts, skipping content already present from prior pulls. Partial pulling does not use the `additionalLayerStores` field.

* The image is in standard OCI format, not a lazy-pull-compatible format such as eStargz or Nydus.
* The storage plugin is not running.
--
When CRI-O needs an image, it accesses the plugin's FUSE filesystem, triggering metadata download and lazy pulling. The container starts after downloading only required chunks.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"filesystem" (one word) here vs "file system" (two words) on line 73. Red Hat style prefers "file system" (two words).

Suggested change
When CRI-O needs an image, it accesses the plugin's FUSE filesystem, triggering metadata download and lazy pulling. The container starts after downloading only required chunks.
When CRI-O needs an image, it accesses the plugin's FUSE file system, triggering metadata download and lazy pulling. The container starts after downloading only required chunks.

@mburke5678

Copy link
Copy Markdown
Contributor Author

@BhargaviGudi PTAL

@BhargaviGudi

Copy link
Copy Markdown

/assign @BhargaviGudi

@BhargaviGudi

Copy link
Copy Markdown

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 2, 2026
@mburke5678
mburke5678 force-pushed the nodes-additional-storage-ga branch from 3d9f017 to 702dd53 Compare September 8, 2026 19:32
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 8, 2026
@openshift-ci

openshift-ci Bot commented Sep 8, 2026

Copy link
Copy Markdown

New changes are detected. LGTM label has been removed.

@mburke5678 mburke5678 added the merge-review-needed Signifies that the merge review team needs to review this PR label Sep 8, 2026
@skopacz1 skopacz1 added the merge-review-in-progress Signifies that the merge review team is reviewing this PR label Sep 9, 2026

@skopacz1 skopacz1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, otherwise will be good to go!

Comment thread modules/nodes-nodes-additional-crio-storage-limitations.adoc Outdated
Comment thread modules/nodes-nodes-additional-crio-storage-troubleshooting.adoc Outdated
Comment thread nodes/nodes/nodes-nodes-additional-crio-storage.adoc Outdated
Comment thread nodes/nodes/nodes-nodes-additional-crio-storage.adoc
@skopacz1 skopacz1 removed merge-review-in-progress Signifies that the merge review team is reviewing this PR merge-review-needed Signifies that the merge review team needs to review this PR labels Sep 9, 2026
@openshift-ci

openshift-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown

@mburke5678: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@mburke5678
mburke5678 merged commit bbc6999 into openshift:main Sep 9, 2026
2 checks passed
@mburke5678
mburke5678 deleted the nodes-additional-storage-ga branch September 9, 2026 20:02
@mburke5678

Copy link
Copy Markdown
Contributor Author

/cherrypick enterprise-5.0

@openshift-cherrypick-robot

Copy link
Copy Markdown

@mburke5678: new pull request created: #119648

Details

In response to this:

/cherrypick enterprise-5.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

branch/enterprise-5.0 ok-to-merge size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants