Source Hydrator not respecting namespace

[Elyytscha]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

if source hydrator is enabled, it somehow does not respect the namespace or better said used the argocd namespace

To Reproduce

1. create a repo
2. add this files

# apps/rancher-ui/chart.yaml

apiVersion: v2
name: rancher
version: 0.1.0

dependencies:
  - name: rancher
    version: 2.12.1
    repository: "https://releases.rancher.com/server-charts/stable"
    values:
      - values.yaml

# apps/rancher-ui/values.yaml
# Default values for rancher-ui.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
rancher:
  # # Override names to ensure RBAC resources are named correctly
  # # The Rancher chart expects the service account to be named 'rancher'
  # fullnameOverride: rancher
  
  # # Override namespace for RBAC subjects - required when using ArgoCD sourceHydrator
  # # because .Release.Namespace picks up 'argocd' instead of the destination namespace
  # namespaceOverride: cattle-system
  
  hostname: rancher.example.com

3. use app like this

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: rancher-ui
  namespace: argocd
spec:
  destination:
    namespace: cattle-system
    server: https://kubernetes.default.svc
  project: default
  sourceHydrator:
    drySource:
      path: apps/rancher-ui
      repoURL: https://github.com/<your-repo>.git
      targetRevision: main
    syncSource:
      path: apps/rancher-ui
      targetBranch: env/prod
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
  syncOptions:
    createNamespace: true

4. This will result in the following

branch: env/prod
file: apps/rancher-ui/manifest.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: rancher-ui
    chart: rancher-2.12.1
    heritage: Helm
    release: rancher-ui
  name: rancher-ui
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: rancher-ui
    namespace: argocd # <--------- expected cattle-system
---
...
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rancher-ui
    chart: rancher-2.12.1
    heritage: Helm
    release: rancher-ui
  name: rancher-ui
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rancher-ui
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: rancher-ui
        release: rancher-ui
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: NotIn
                    values:
                      - windows
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - rancher-ui
                topologyKey: kubernetes.io/hostname
              weight: 100
      containers:
        - args:
            - --http-listen-port=80
            - --https-listen-port=443
            - --add-local=true
          env:
            - name: CATTLE_NAMESPACE
              value: argocd # <--------- expected cattle-system
...
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    helm.sh/hook: pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
    helm.sh/hook-weight: "-1"
  labels:
    app: rancher-ui
    chart: rancher-2.12.1
    heritage: Helm
    release: rancher-ui
  name: rancher-ui-pre-upgrade
  namespace: argocd  # <--------- expected cattle-system

https://github.com/devops-magic/argo-cd-hydration-test/blob/bugreport/source-hydrator-not-respecting-namespace/apps/rancher-ui/manifest.yaml#L53

Expected behavior

namespace=cattle-system

Screenshots

Version

v3.3.0+0e68458.dirty

[crenshaw-dev]

I think this might be a duplicate of #24371

[Elyytscha]

It may be, but this one sound more like its all about the annotations and an enhancement proposal.

The namespace thing more sound like a regression compared to normal applications to me..:

works perfectly fine (namespace=cattle-system)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: rancher-ui
  namespace: argocd
spec:
  destination:
    namespace: cattle-system
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps/rancher-ui
    repoURL: https://github.com/<your-repo>.git
    targetRevision: main

does not work (namespace=argocd)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: rancher-ui
  namespace: argocd
spec:
  destination:
    namespace: cattle-system
    server: https://kubernetes.default.svc
  project: default
  sourceHydrator:
    drySource:
      path: apps/rancher-ui
      repoURL: https://github.com/<your-repo>.git
      targetRevision: main
    syncSource:
      path: apps/rancher-ui
      targetBranch: env/prod

I traced it down to this function

argo-cd/controller/state.go

Lines 257 to 259 in 3c12c01

	if !sendRuntimeState {
	appNamespace = ""
	}

if I remove this, namespace is set to cattle-system instead of argocd

[crenshaw-dev]

Can you update the original issue description output manifest to highlight exactly where you expected cattle-system to be set?

One subtlety to keep in mind here is that there is the manifest rendered (stored in git) and the manifest applied, which may be subtly different. Specifically, it's possible for the namespace to not be set on namespaced resources in git but for that namespace to be applied last-minute before the manifests are sent to k8s.

[Elyytscha]

updated

One subtlety to keep in mind here is that there is the manifest rendered (stored in git) and the manifest applied, which may be subtly different. Specifically, it's possible for the namespace to not be set on namespaced resources in git but for that namespace to be applied last-minute before the manifests are sent to k8s.

which is the case for example the serviceAccount,

but for example the clusterRoleBinding/deployment[env var] it does not work.

For us this is somehow a blocker, because we basically can't use applications with sourceHydrator utilizing the same DRY manifests as their old applications did without sourceHydrator

[crenshaw-dev]

What's in your hydrator.metadata for that app? I'm curious what params are being send to helm template.

[Elyytscha]

You can have a look over here

https://github.com/devops-magic/argo-cd-hydration-test/blob/c13533cb4a61035586c13da20bee035ba069721e/apps/rancher-ui/hydrator.metadata#L1-L7

  "commands": [
    "helm template . --name-template rancher-ui --include-crds"
  ]

[Mattes83]

I am in the same boat. In some cases I can circumvent the issue by not specifying a namespace in my manifests, then it works fine. If I use .Release.Namespace it is replaced with the argo namespace.

[seankhliao]

I also expect the Application spec.destination.namespace to be passed to helm template and be available to the chart as Release.Namespace

For a non hydrator sync, it does seem to be passed in

argo-cd/reposerver/repository/repository.go

Line 1201 in a439c6c

Namespace: q.ApplicationSource.GetNamespaceOrDefault(q.Namespace),