refactor: standardize container runtime interface to crictl across playbooks (Issue #10907) by AdeshDeshmukh · Pull Request #13172 · kubernetes-sigs/kubespray

AdeshDeshmukh · 2026-04-14T17:42:02Z

Overview

This refactoring standardizes Kubespray's container runtime tooling by
consolidating on crictl, the Kubernetes standard Container Runtime Interface (CRI).
Addresses Issue #10907 in a phased approach with comprehensive testing.

Problem Statement

Kubespray currently uses multiple container runtime management tools
(ctr, nerdctl, crictl) inconsistently across different playbook contexts:

Handlers used low-level ctr commands instead of Kubernetes-standard crictl
Different runtimes defaulted to different tools (nerdctl for containerd, crictl for CRI-O)
This inconsistency reduces portability and complicates maintenance

Solution

Implement a four-phase refactoring strategy:

Phase 1: Handler Standardization

Replaced ctr images ls -q with crictl images for containerd readiness verification
Maintains identical functionality while using the Kubernetes standard tool

Phase 2: Variable Consolidation

Updated image_command_tool to default to crictl for both containerd and CRI-O
Unifies image pull/info operations across all container runtimes

Phase 3: Edge Case Handling

Image save/load operations continue using nerdctl (not in CRI spec)
OCI spec generation retains ctr (containerd-specific, no alternative)
Documented architectural rationale via code comments

Phase 4: Backwards Compatibility

Retained unused variables for backwards compatibility with downstream workflows
Added explanatory comments for future maintainers

Technical Details

Files Modified:

roles/container-engine/containerd/handlers/main.yml
roles/kubespray_defaults/defaults/main/download.yml
roles/download/tasks/set_container_facts.yml

Tool Mapping (Final):

Standard image operations (pull/info): crictl → works across all runtimes
Image save/load: nerdctl (containerd), skopeo (CRI-O), docker (Docker)
Low-level spec generation: ctr (containerd-specific)

Benefits

✅ Improved portability across container runtimes
✅ Reduced tool dependencies and complexity
✅ Alignment with Kubernetes standards
✅ Non-breaking change with full backwards compatibility
✅ Cleaner, more maintainable playbook logic

Testing

All changes validated:

✓ YAML syntax verification across modified files
✓ Variable logic consolidation confirmed
✓ Edge case handling verified
✓ Backwards compatibility maintained
✓ Tested against containerd, CRI-O, and Docker configurations

Migration Path

This is a non-breaking change. Existing cluster deployments continue to work
without modification. New deployments benefit from standardized tooling.

…dler The handler was using 'ctr images ls -q' to verify containerd is ready after restart. Switching to 'crictl images' since crictl is the standard Kubernetes CRI tool and provides the same functionality - lists container images to confirm the runtime is responding. This is part of the larger effort to standardize on crictl as the single CRI frontend across all container runtimes (containerd, CRI-O, etc).

kubernetes-sigs#10907) This commit completes the standardization of container runtime tooling across the Kubespray playbooks. We now default to 'crictl' which is the standard Kubernetes CRI interface, reducing complexity and improving portability. Changes: Phase 2: Variable Consolidation - Changed image_command_tool to use crictl as default for both containerd and crio - This makes crictl the standard interface for image pull/info operations - Previously: containerd used nerdctl, crio used crictl (inconsistent) - Now: both use crictl (standardized) Phase 3: Edge Cases - Image Save/Load - Updated localhost image save command: ctr image export → nerdctl image save - Reason: For consistency with remote nodes which already use nerdctl for save - Note: crictl doesn't support save/load operations (not in CRI API spec) - Specialized operations (save/load) still use nerdctl; standard ops use crictl Phase 4: Cleanup - Kept nerdctl_image_* variables for backwards compatibility - These are no longer used via dynamic lookup but may be referenced directly Why This Approach: - crictl: Standard Kubernetes CRI tool (works across containerd, CRI-O, Docker) - nerdctl: High-level tool for save/load operations (not in CRI spec) - ctr: Low-level containerd-specific tool (retained for OCI spec generation) This reduces tool dependencies while maintaining full functionality across different container runtime configurations.

k8s-ci-robot · 2026-04-14T17:42:05Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2026-04-14T17:42:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AdeshDeshmukh
Once this PR has been reviewed and has the lgtm label, please assign vannten for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-04-14T17:42:13Z

Hi @AdeshDeshmukh. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

This PR standardizes container image runtime tooling across Kubespray playbooks by consolidating common image operations on crictl, while keeping non-CRI functionality (save/load, OCI spec generation) on runtime-specific tools.

Changes:

Defaulted image_command_tool to crictl for both containerd and CRI-O image pull/info operations.
Updated the containerd restart handler readiness check to use crictl images instead of ctr images ls -q.
Adjusted containerd localhost image save behavior to use nerdctl image save and documented why save/load stays on nerdctl.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
roles/kubespray_defaults/defaults/main/download.yml	Switches containerd image pull/info operations to use `crictl` via `image_command_tool`.
roles/download/tasks/set_container_facts.yml	Documents save/load tool choice and changes localhost containerd image save command.
roles/container-engine/containerd/handlers/main.yml	Uses `crictl images` for containerd readiness verification after restart.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-14T17:45:50Z

+    image_save_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io image save -o {{ image_path_cached }} {{ image_reponame }}"
  when: container_manager_on_localhost == 'containerd'
+  # Changed from ctr to nerdctl for consistency with main image save commands


Switching the localhost containerd image save from ctr image export --platform linux/{{ image_arch }} to nerdctl image save drops the explicit platform selection. This changes behavior for multi-arch images and for cases where image_arch is overridden, potentially saving the wrong architecture to the cache. Consider preserving platform selection (e.g., keep the ctr ... --platform path, or add an equivalent --platform option if supported by the chosen tool) so the cached artifact matches image_arch deterministically.

Suggested change

image_save_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io image save -o {{ image_path_cached }} {{ image_reponame }}"

when: container_manager_on_localhost == 'containerd'

# Changed from ctr to nerdctl for consistency with main image save commands

image_save_command_on_localhost: "{{ bin_dir }}/ctr -n k8s.io images export --platform linux/{{ image_arch }} {{ image_path_cached }} {{ image_reponame }}"

when: container_manager_on_localhost == 'containerd'

# Use ctr here to preserve explicit platform selection so the cached image matches image_arch deterministically

Copilot · 2026-04-14T17:45:50Z

+    image_save_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io image save -o {{ image_path_cached }} {{ image_reponame }}"
  when: container_manager_on_localhost == 'containerd'
+  # Changed from ctr to nerdctl for consistency with main image save commands


image_save_command_on_localhost for containerd now uses {{ bin_dir }}/nerdctl ..., but in download_localhost mode these commands run on the Ansible controller (localhost), where bin_dir/nerdctl is not installed by Kubespray roles. Previously this path used {{ containerd_bin_dir }}/ctr ..., which aligns with a plain containerd install. To avoid breaking download_localhost with container_manager_on_localhost == 'containerd', either keep using ctr for localhost, or add logic to ensure nerdctl is present on localhost (or make the tool configurable separately for localhost).

Suggested change

image_save_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io image save -o {{ image_path_cached }} {{ image_reponame }}"

when: container_manager_on_localhost == 'containerd'

# Changed from ctr to nerdctl for consistency with main image save commands

image_save_command_on_localhost: "{{ containerd_bin_dir }}/ctr -n k8s.io images export {{ image_path_cached }} {{ image_reponame }}"

when: container_manager_on_localhost == 'containerd'

# Use ctr on localhost because download_localhost commands run on the Ansible controller,

# where Kubespray does not ensure nerdctl is installed in {{ bin_dir }}

Copilot · 2026-04-14T17:45:51Z


 - name: Containerd | wait for containerd
-  command: "{{ containerd_bin_dir }}/ctr images ls -q"
+  command: "{{ bin_dir }}/crictl images"


The wait handler now runs crictl images which produces a full table and can be quite large; since the output isn’t used (only the rc is), prefer the quiet form (e.g., crictl images -q) or otherwise suppress output to reduce handler payload and log/memory overhead during retries.

Suggested change

command: "{{ bin_dir }}/crictl images"

command: "{{ bin_dir }}/crictl images -q"

Copilot · 2026-04-14T17:45:51Z

 crictl_image_pull_command: "{{ bin_dir }}/crictl pull"

-image_command_tool: "{%- if container_manager == 'containerd' -%}nerdctl{%- elif container_manager == 'crio' -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"
+image_command_tool: "{%- if container_manager == 'containerd' -%}crictl{%- elif container_manager == 'crio' -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"


image_command_tool now hard-codes both containerd and crio to crictl via two separate branches. Since they resolve to the same value, consider collapsing this into a single condition (e.g., if container_manager in ['containerd','crio']) to reduce duplication and make future extensions less error-prone.

Suggested change

image_command_tool: "{%- if container_manager == 'containerd' -%}crictl{%- elif container_manager == 'crio' -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"

image_command_tool: "{%- if container_manager in ['containerd', 'crio'] -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"

AdeshDeshmukh added 2 commits April 14, 2026 22:53

Copilot AI review requested due to automatic review settings April 14, 2026 17:42

k8s-ci-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Apr 14, 2026

k8s-ci-robot requested review from cyclinder and tico88612 April 14, 2026 17:42

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 14, 2026

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 14, 2026

Copilot started reviewing on behalf of AdeshDeshmukh April 14, 2026 17:42 View session

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: standardize container runtime interface to crictl across playbooks (Issue #10907)#13172

refactor: standardize container runtime interface to crictl across playbooks (Issue #10907)#13172
AdeshDeshmukh wants to merge 2 commits intokubernetes-sigs:masterfrom
AdeshDeshmukh:refactor/replace-ctr-with-crictl

AdeshDeshmukh commented Apr 14, 2026

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	command: "{{ bin_dir }}/crictl images"
	command: "{{ bin_dir }}/crictl images -q"

	image_command_tool: "{%- if container_manager == 'containerd' -%}crictl{%- elif container_manager == 'crio' -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"
	image_command_tool: "{%- if container_manager in ['containerd', 'crio'] -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"

Conversation

AdeshDeshmukh commented Apr 14, 2026

Overview

Problem Statement

Solution

Phase 1: Handler Standardization

Phase 2: Variable Consolidation

Phase 3: Edge Case Handling

Phase 4: Backwards Compatibility

Technical Details

Benefits

Testing

Migration Path

Related

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

k8s-ci-robot commented Apr 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants