Can't initialize DSS latest/edge, is MLFlow not deployed with --trust?

[motjuste]

Bug Description

There seems to be some issue with the latest/edge update version 0.1-2237f5211c19... release on 2-Dec-2024 where DSS cannot be initialized. This was seen during the Checkbox validation run for DSS and all validations failed for latest/edge with the same error, for example here.

To Reproduce

1. Install microk8s 1.28/stable or 1.31/stable, classic snap, and enable hostpath-storage dns rbac.
2. Install data-science-stack snap from channel latest/edge.
3. Run dss initialize --kubeconfig="$(sudo microk8s config)"

Environment

• microk8s, either 1.28/stable or 1.31/stable, classic snaps
• data-science-stack, latest/edge

Relevant Log Output

Executing initialize command
  Storing provided kubeconfig to /home/ubuntu/snap/data-science-stack/39/.dss/config
  Error:  Failed to initialize dss: Cannot apply required resources. Charm may be missing `--trust`.

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6635.

This message was autogenerated

[motjuste]

Checking the dss status I get:

~$ dss status
[ERROR] Failed to retrieve status: deployments.apps "mlflow" is forbidden: User "system:anonymous" cannot get resource "deployments" in API group "apps" in the namespace "dss".
~$ sudo dss status
Traceback (most recent call last):
  File "/snap/data-science-stack/39/bin/dss", line 5, in <module>
    from dss.main import main
  File "/snap/data-science-stack/39/lib/python3.10/site-packages/dss/main.py", line 4, in <module>
    from dss.create_notebook import create_notebook
  File "/snap/data-science-stack/39/lib/python3.10/site-packages/dss/create_notebook.py", line 19, in <module>
    from dss.remove_notebook import remove_notebook
  File "/snap/data-science-stack/39/lib/python3.10/site-packages/dss/remove_notebook.py", line 8, in <module>
    from dss.utils import does_notebook_exist
  File "/snap/data-science-stack/39/lib/python3.10/site-packages/dss/utils.py", line 21, in <module>
    logger = setup_logger("logs/dss.log")
  File "/snap/data-science-stack/39/lib/python3.10/site-packages/dss/logger.py", line 52, in setup_logger
    os.makedirs(os.path.dirname(log_file_path))
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'logs'

[kimwnasptd]

Looks like it's an underlying issue with gtsystem/lightkube#78 and we can also see it in

• RBAC addon only returns User "system:anonymous" microk8s#4765
• canon-run: Bump juju (to 3.6), charmcraft and requirements files in charm repos  bundle-kubeflow#1156 (comment)

EDIT: For the solution we'll need to bump lightkube to 0.15.6

[orfeas-k]

@motjuste Is this still relevant? This week, we were able to run tests on Canonical k8s.

[motjuste]

I just verified with the newest latest/edge version 0.1-e495bacef97a1b6bf8bdb63dca29, and this issue does not occur, either with microk8s 1.28/stable or 1.31/stable. Closing.