Feature: pull user defined images for warm pool instances

[dezmodue]

This PR adds the ability to define extra images that can be loaded on the nodes in the warm pool.

In some cases it can be convenient to download large container images during the warming phase to reduce the startup time of workloads after a node has been requested and joined the cluster. Extra images can be specified via the pullExtraImages field and the timeout for the lifecycle hook can be tuned accordingly to allow for the operation to complete.

Tested by building an initial cluster with an IG:

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: test.labs.com
  name: nodes-us-east-1a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20251212
  machineType: t3.large
  maxSize: 3
  minSize: 2
  role: Node
  subnets:
  - us-east-1a
---

Validated that the nodes complete the kops-configuration without errors.

Then adding the warm pool in the IG spec:

spec:
  warmPool:
    minSize: 1
    maxSize: 1
    enableLifecycleHook: true
    lifecycleHookTimeout: 1200
    pullExtraImages:
      - nvcr.io/nvidia/tritonserver:25.12-py3-min

Rolled the IG via kOps, then validated that the kops-configuration completed without errors in both the warm pool and non warm pool instances. The image specified in the spec is pulled during the warming phase and the timeout is correctly set to 1200.

kops get assets includes the image specified in the warm pool spec.

[k8s-ci-robot]

Hi @dezmodue. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dezmodue]

/retest

[k8s-ci-robot]

@dezmodue: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dezmodue]

Verified that both containerRegistry and containerProxy settings work as expected with the additional images:

With warmPool definition

spec:
  warmPool:
    minSize: 1
    maxSize: 1
    enableLifecycleHook: true
    lifecycleHookTimeout: 1200
    pullExtraImages:
      - nvcr.io/nvidia/tritonserver:25.12-py3-min

With container registry:

  assets:
    containerRegistry: example.com/registry

Result of kops get assets:

CANONICAL							DOWNLOAD
nvcr.io/nvidia/tritonserver:25.12-py3-min			example.com/registry/nvcr.io-nvidia-tritonserver:25.12-py3-min
public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.22.0	example.com/registry/public.ecr.aws-aws-ec2-aws-node-termination-handler:v1.22.0
quay.io/cilium/cilium:v1.18.2					example.com/registry/quay.io-cilium-cilium:v1.18.2
...

With container proxy:

  assets:
    containerProxy: proxy.example.com

Result of kops get assets:

CANONICAL							DOWNLOAD
nvcr.io/nvidia/tritonserver:25.12-py3-min			proxy.example.com/nvidia/tritonserver:25.12-py3-min
public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.22.0	proxy.example.com/aws-ec2/aws-node-termination-handler:v1.22.0
quay.io/cilium/cilium:v1.18.2					proxy.example.com/cilium/cilium:v1.18.2
...

Will change from pullExtraImages to additionalImages as requested

[dezmodue]

Pushed change to use additionalImages

Verified again:

spec:
  warmPool:
    minSize: 1
    maxSize: 1
    enableLifecycleHook: true
    lifecycleHookTimeout: 1200
    additionalImages:
      - nvcr.io/nvidia/tritonserver:25.12-py3-min

Result:

.....
Will create resources:
  AutoscalingLifecycleHook/kops-warmpool-nodes-us-east-1a
        ID                      kops-warmpool-nodes-us-east-1a
        HookName                kops-warmpool
        AutoscalingGroup        name:nodes-us-east-1a.test.labs.com id:nodes-us-east-1a.test.labs.com
        DefaultResult           ABANDON
        HeartbeatTimeout        1200
        LifecycleTransition     autoscaling:EC2_INSTANCE_LAUNCHING
        Enabled                 true
.....

  ManagedFile/nodeupconfig-nodes-us-east-1a
        Contents            
                                ...
                                  CAs: {}
                                  ClusterName: test.labs.com
                                + EnableLifecycleHook: true
                                  Hooks:
                                  - null
                                ...
                                  usesLegacyGossip: false
                                  usesNoneDNS: true
                                + warmPoolImages:
                                + - nvcr.io/nvidia/tritonserver:25.12-py3-min
                                + - quay.io/cilium/cilium:v1.18.2
                                + - quay.io/cilium/operator:v1.18.2
                                + - registry.k8s.io/kube-proxy:v1.34.1
                                + - registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.47.0
                                + - registry.k8s.io/provider-aws/cloud-controller-manager:v1.34.0
                                + - registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0
                                + - registry.k8s.io/sig-storage/livenessprobe:v2.16.0

[hakman]

/ok-to-test

[dezmodue]

/retest

[dezmodue]

/test pull-kops-e2e-k8s-gce-ipalias

[dezmodue]

Hi, please let me know if there is anything I can do to help move this forward, thanks!

[hakman]

@dezmodue Finally got to the review, sorry for the delay. The feature works as expected, but some of the choices have unintended side effects. I added some suggestions, please let me know if you have any questions.

[hakman]

Looks great. Thanks for adding this @dezmodue!
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dezmodue]

@hakman thanks a lot for the help, any chance this could land also in 1.33?

[hakman]

@hakman thanks a lot for the help, any chance this could land also in 1.33?

Sorry, but I don't think there's any plan for any 1.33 releases anymore.