Create a global option to disable a sync with `Replace`

[andrii-korotkov-verkada]

Summary

Have a way to disable a sync with replace, the one that users check in the UI. It shouldn't affect replace for large CRDs, which isn't sometimes done if regular sync fails.

Motivation

Allow to configure sync with replace to not happen, e.g. relevant for resources which are owning pods like Deployment, Rollout. Replacing those can cause replica count to drop to 1 and cause issues. In practice, people ignore the warnings about Replace being dangerous too much. And sync with force can solve most problems that sync with replace might be used for.

Proposal

Have an option in argocd-cmd-params-cm like server.sync.replace.allowed, which if false, fails attempts to sync with replace. The users would still see the replace checkbox and can click it, however they'd get an error saying the option was disabled by configuration. This way they can know it exists and might be able to negotiate with administrators if they really need it. If there's an annotation of resources or some big CRD needs to be replaced, that would still happen, i.e. the option would only not allow using the checkbox for Replace.

Historically, multiple proposals have been considered, including RBAC-based one and annotations/options to disable sync with replace in a more granular way, but they were found to be have disadvantages and were ultimately rejected.

[leoluz]

Please enhance your proposal describing how users would be providing this configuration with examples for all the possibilities.

[andrii-korotkov-verkada]

Updated the description more.

[andrii-korotkov-verkada]

Updated the motivation and proposal sections a bit

[leoluz]

I don't think that we should introduce the always and never in the existing config. Instead we should investigate the possibility to use the replace=false configuration to address this use-case.
Also, we still need to decide how the Application config and resource annotations(argocd.argoproj.io/sync-options: Force=true,Replace) will behave when conflicting with what is specified in the sync panel in the UI.
Maybe another direction that we could take to simplify this feature is just introducing a new global configuration to completely disable the replace in Argo CD which would disable the "Replace" checkbox in the UI as well.

[andrii-korotkov-verkada]

Replace=false may be repurposed for this, but as brought up in the contributors meeting, false doesn't really mean the directive, it's more like the option is not checked/enabled, i.e. default behavior.

I think the priority should be:
Resource annotation > Resource GK config > Application annotation > Global config > What's selected in the UI.

So that any option that says "never" would override the UI selection.

Maybe another direction that we could take to simplify this feature is just introducing a new global configuration to completely disable the replace in Argo CD which would disable the "Replace" checkbox in the UI as well.

I think that's too strict, since sometimes you want to do a replace/force, e.g. when resource state got messed up, or you need to update some immutable fields. Having an ability to update annotations without redeploying ArgoCD would be very helpful there.

[andrii-korotkov-verkada]

What's your concern with adding new directives? I think they are pretty clear about what they do and preserve backwards compatibility. I understand there's extra complexity, but IMO it's manageable and allowing for a more granular control of replace/force is a great perk.

[andrii-korotkov-verkada]

Some problems with priority would still be there even if we repurpose Replace=false, e.g. what if application has Replace=true, but resource has Replace=false (this case is quite simple though).

[leoluz]

What's your concern with adding new directives? I think they are pretty clear about what they do

I don't think this is clear and will generate confusion as ppl may think what is the difference between false and never. As per the docs the only existing replace configuration is replace=true. My suggestion is to only implement one additional configuration such as replace=false

Some problems with priority would still be there even if we repurpose Replace=false

Correct. The proposal needs to document a solution to that problem as well.