helm/v2-alpha: No support for configurable volumes/volumeMounts in generated charts

[asergeant01]

What do you want to happen?

I want to provide and load secrets to the container through volumes and volumeMounts which is a widely used approach.

This would be achieved by using the already configurable args to provide the file path to the controller.

manager:

  args:
    - --leader-elect
    - --health-probe-bind-address=:8081
    - --super-secret-configuration-file=/etc/my-secrets

but also allowing the following amendments:

# ... existing config (replicas, image, env, etc.) ...

  ## Additional volume mounts for the manager container
  ## These are in addition to webhook/metrics volumes which are handled automatically
  ##
  additionalVolumeMounts: []
    # - name: custom-volume
    #   mountPath: /etc/my-secrets
    #   readOnly: true

  ## Additional volumes for the manager pod
  ## These are in addition to webhook/metrics volumes which are handled automatically
  ##
  additionalVolumes: []
    # - name: custom-volume
    #    secret:
    #      name: super-secrets

Extra Labels

No response

[camilamacedo86]

Keeping in mind that:

• Helm charts should be used to package the generated solution, not become another project that diverges from the the scaffolding.
• Also, values.yaml should stay limited to options that make sense for end users and have sensible, working defaults.

On the specific request to expose volumes/volumeMounts via values: does it really make sense to let end users change those? In what concrete scenarios would an operator user need to override volumes?

I don’t think this is something we should expose by default for scaffolded projects. If someone needs custom mounts, they can customize their own Helm chart to match their requirements since:

• User chart customization is not overwritten when re-running kubebuilder edit --plugins=helm/chart.
• Custom changes should not be lost when the project is updated via the AutoUpdate plugin (GitHub Actions) or when running kubebuilder alpha update.

Given that, I suggest we close this as “deferred”, unless there’s a clear, common use case for the default projects produced by Kubebuilder where end users genuinely need to configure volumes via chart values.
What specific use cases do you have in mind?

[isumitsolanki]

I want to work on this

[isumitsolanki]

/assign

[camilamacedo86]

HI @isumitsolanki

Thank you for looking on that. But I do not think we should move with this one. Please, see: #5485 (comment)

[asergeant01]

@camilamacedo86 I agree "Helm charts should package the generated solution" - but packaging for distribution requires flexibility for consumers. I don't want to diverge from scaffolding but it's about making the chart usable in production environments.

Does it really make sense to let end users change those?
In what concrete scenarios would an operator user need to override volumes?

Secret Management is a pretty universal deployment requirement - When organizations deploy operators via Helm charts, they frequently need to mount secrets from external sources such as Credentials managed outside the chart (Vault, external-secrets-operator, cloud secret managers)

In my specific use case, I have secrets that contain access credentials to physical hardware that the operator needs to connect to. Yes I could provide that through the already extracted Environment variables, but that is much more cumbersome and some might argue not as secure as volume mounting.

What I am purposing I don't believe conflicts with the scaffolding philosophy:

• The base scaffold (kustomize) stays clean - no arbitrary volumes
• The Helm chart (distribution artifact) provides flexibility for consumers [additionalVolumes: []] with empty default - doesn't add clutter
• Only users who need it configure it

For me it's exactly the same as is already done for args, env and resources. The scaffold includes [resources: {}] even though most operators work fine without resource limits. Why? Because production deployments need to configure it. Same applies to volumes for secrets in my opinion.

User chart customization is not overwritten when re-running kubebuilder edit --plugins=helm/chart

I assume you meant kubebuilder edit --plugins=helm/v2-alpha. Unfortunately I have found if the resource is part of the Kustomize output, it completely re-templates the file. Files that are not part of the ouput are left untouched.

[camilamacedo86]

Hi @asergeant01

Thanks for raising this and sharing your thoughts.

For your use case, you can update the generated Helm values and add additionalVolumes: [] (and any related fields you need). This change should not be lost when you run kubebuilder edit --plugins=helm, since it lives in the values you manage.

That said, I don’t think we should add this to the default scaffold. The default scaffold should stay clean and minimal, and we should avoid adding options that are only needed for custom setups. We also can’t assume everyone will use PKI (or any specific secret approach), or that all users will want to configure this by default.

If a project needs extra volumes/secrets, the recommended approach is to customize the Helm chart for that project.

Unfortunately I have found if the resource is part of the Kustomize output, it completely re-templates the file. Files that are not part of the ouput are left untouched.

I don’t think that’s true anymore — are you using the latest release?

In the current behavior, any unknown resources are placed under the extras directory, so they should still be preserved there.
See: https://book.kubebuilder.io/plugins/available/helm-v2-alpha

Does it address your need?
Could we close this one? WDYT?

[asergeant01]

Hi @camilamacedo86

Appreciate the discussion.

For your use case, you can update the generated Helm values and add additionalVolumes: [] (and any related fields you need). This change should not be lost when you run kubebuilder edit --plugins=helm, since it lives in the values you manage.

Yes, I do update the values, and you are correct, they persist through a run with kubebuilder edit --plugins=helm/v2-alpha however what does not persist is the manual change I make in the manager.yaml which templates the values defined.

values.yaml

manager:
  args:
    - --my-secrets=/etc/my-secrets

  additionalVolumeMounts:
    - name: custom-volume
      mountPath: /etc/my-secrets
      readOnly: true

  additionalVolumes:
    - name: custom-volume
       secret:
         name: my-secret

values.yaml persists, however when I update manager.yaml manually to add:

volumes:
                {{ - with .Values.manager. additionalVolumes}}
                {{- toYaml . | nindent 16 }}
                {{- end }}

and rerun kubebuilder edit --plugins=helm/v2-alpha the whole file gets regenerated.

I believe the example you are showing works if the both resource and the values have been added manually as the plugin just ignores.

[camilamacedo86]

I think we might be able to:

• Check if in the YAML we have extra/additional values
• If so, then we add to the values as requested

[asergeant01]

@camilamacedo86 That sounds like a nice generic approach. I'll look into this one and see what it might look like.

[porridge]

Could I please add a request for additionalEnvVars in a simlar spirit? 🙏🏻
A bit more context:

• kubebuilder generates the values.yaml with a bunch of env variables that are needed for the operator to work correctly, all good. These are in the "list of key+value objects" format that the container spec expects.
• But now if a user at install time wants to define an additional variable to tweak a feature, they cannot just --set manager.env.SOMEVAR=true, they need to set the whole manager.env, duplicating the whole list from stock values.yaml