Skip to content

docs: Add an ADR for using Backstage to support OEP-55 #343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feanil merged 1 commit into from
Jun 23, 2022
Merged

docs: Add an ADR for using Backstage to support OEP-55 #343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions oeps/processes/oep-0055-proc-project-maintainers.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,10 +359,12 @@ community to find a new maintainer.
Documenting Maintainers
-----------------------

As it is essential that maintainers are discoverable and reachable,
the person or persons responsible for maintaining a component should
be listed in the repository README file along with the mechanism for
contacting a maintainer.
As it is essential that maintainers are discoverable and reachable, the person
or persons responsible for maintaining a component should be listed in the
repository. Per :doc:`the backstage ADR
<oep-0055/decisions/0001-use-backstage-to-support-maintainers>` we will use a
``catalog-info.yaml`` file to capture the github handles for the maintainers of
a repository.

Jobs of the Maintainer
----------------------
Expand Down Expand Up @@ -392,7 +394,7 @@ categories
* **Technical Vision**: Maintainers should have a documented technical
vision for their components. This vision should minimally answer the
following questions:

* What is the component for?
* How it fits into the overall platform?
* What are its limitations and key areas of debt?
Expand Down Expand Up @@ -444,7 +446,15 @@ Questions about the Program can be directed to the tCRIL administrator(s) via
the ``#ask-tcril`` Slack room in the `Open edX Slack
<https://openedx.slack.com/>`_. Questions or concerns of a more sensitive nature
may be directed to ``maintainer-program-admins@tcril.org``.


Related Decisions
=================

.. toctree::
:glob:
:maxdepth: 1

oep-0055/decisions/*

Change History
==============
Expand All @@ -459,7 +469,7 @@ Source Material
* `Collective Code Construction Contract <https://rfc.zeromq.org/spec/42/>`_
* `Best Practices for Maintainers <https://opensource.guide/best-practices/>`_
* `Producing OSS <https://producingoss.com/en/producingoss-letter.pdf>`_

Notes
=====

Expand Down
139 changes: 139 additions & 0 deletions oeps/processes/oep-0055/decisions/0001-use-backstage-to-support-maintainers.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
0001 Use Backstage to Support Maintainers
#########################################

Status
******

**Provisional**

Context
*******

* :doc:`OEP-55 <../../oep-0055-proc-project-maintainers>` laid out that we should document the maintainers of a component.

* As a maintainer it is hard to know what components you are maintaining.

* As a community member it is currently difficult to quickly look up who the
maintainers are for multiple components quickly.

* Given the number of repos that make up the Open edX platform and the
complexity of the job of maintaining them, we need a system for cataloging,
and making transparent the critical details of our repositories.

Decision
********

To solve the above issues and to enable the potential to ease other
discoverability and maintainership burdens, the Open edX community shall
maintain a `backstage`_ instance and related metadata.

Consequences
Copy link
Copy Markdown
Contributor

@robrap robrap Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 2U/edX were maintaining much of this information in a spreadsheet before the tCRIL-split. This ADR could acknowledge the need to update this existing tooling. Even if not mentioned in the ADR, this work needs to be communicated and planned for.
  2. The existing spreadsheet also has exceptional maintenance requirements for the edx-platform monolith. Does tCRIL plan on supporting that level of detail, or is it planning on having edx-platform maintained by 2U/edX, with the split ownership being decided/documented/maintained by 2U/edX as we see fit?

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

2U/edX were maintaining much of this information in a spreadsheet before the tCRIL-split. This ADR could acknowledge the need to update this existing tooling. Even if not mentioned in the ADR, this work needs to be communicated and planned for.

That feels like a consequence internal to 2U and not one for the open source project. I agree the work needs to be done either way, but I'm not sure mentioning it here is valuable from the open source project perspective.

The existing spreadsheet also has exceptional maintenance requirements for the edx-platform monolith. Does tCRIL plan on supporting that level of detail, or is it planning on having edx-platform maintained by 2U/edX, with the split ownership being decided/documented/maintained by 2U/edX as we see fit?

I think edx-platform maintainership is being avoided for the pilot but we will eventually need to make a plan for it. Given its complexity, I expect there will be a future ADR that addresses its ownership specifically.

Copy link
Copy Markdown
Contributor

@hurtstotouchfire hurtstotouchfire Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@robrap maybe we can touch base over slack about the nature of the maintenance requirements? I'm curious how much of it would be relevant to any organization maintaining edx-platform vs 2U specifically.

************

* To support the backstage instance ownership data will have to be stored in a
structured way. We will follow Backstage best practices and store this data
in a ``catalog-info.yaml`` file in each component repository.

* Once a backstage instance exists we have the opportunity to further leverage
it as a centralized access point for maintainers as well as community members
to quickly and easily assess different components and find relevant links to
code, documentation, owners, etc.

* Backstage will also give us the ability to expose other health metrics to
maintainers so they can more easily assess the components they own.

* As a result of adding the ``catalog-info.yaml`` file we end with two top-level
files that have very similar data. If this decision is not reversed after the
initial pilot, it may make sense to revisit this consequence and see if we can
consolidate this metadata into a single file. `tcril-engineering#331
<https://github.com/openedx/tcril-engineering/issues/331>`_ has been created
to follow up on this work.


Rejected Alternatives
*********************

Copy link
Copy Markdown
Contributor

@hurtstotouchfire hurtstotouchfire Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were there no alternatives? If there are no comparable OOTB options we could perhaps note briefly why we don't want to build it ourselves out of scripts and spreadsheets, for instance

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hurtstotouchfire do you mean were there no alternatives services or out of box tools? If so, there are a few commercial products but their use case tends to be different. They are oriented towards protecting the content rather than making it visible to everyone so often their pricing models and workflows don't make sense for an opensource project. This felt a bit outside of the requirements of what we were looking for so I did not initially mention it since they were not viable alternatives for the basic requirement of everyone needs to be able to see it.

Spreadsheet
===========

Pros
----

* Easy to build and get a high level view of the system.

Cons
----

* Not a beautiful UI for maintainers to work with.

* Data overload as we put more and more data into it with little to no control
over the presentation of the data.

Owners in the Readme
====================

Pros
----

* Easy to see while in the repo.

Cons
----

* Harder to aggregate the data from the unstructured files in a way that would
allow us to easily build a `single pane of glass`_ where maintainers can see
all the things they own.
Copy link
Copy Markdown
Contributor

@hurtstotouchfire hurtstotouchfire Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason I have in my memory something about using github repo tags for this? I think it was @ormsbee's idea. In any case, would we want to integrate that with Backstage somehow or are they just separate concerns?

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't fully understand what you're saying here but feel free to suggest text that you'd like here or another rejected alternative that you would want to document.

Copy link
Copy Markdown
Member

@kdmccormick kdmccormick Jun 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your memory serves you right @hurtstotouchfire -- on the OEP-57 draft, Dave mentioned using GH topics to group repositories into tiers like Core, Supporting, etc. Repo topics are an interesting way of storing metadata in a GH-native way that you might consider employing as you iterate on this @feanil .

Correct me if I'm wrong, but I get the sense that you are hoping to get this ADR over the line so that we can experiment with actually using backstage for the maintainer pilot. After we use backstage for a while it'll probably be easier to tell which GH features we do and don't want to tie in with, as well as what exactly the schema should look like. That is to say: I'm fine if you're not explicitly rejecting any alternatives at the moment, since we're not exactly sure what this thing will look like and how everyone will use it.


References
**********

* `Backstage`_ project site.

* `catalog-info.yaml`_ spec definition

* `Backstage system model`_ has detail about the different kinds of components
and how they connect to each other.

* Sample ``catalog-info.yaml`` file:

.. code:: yaml

apiVersion: backstage.io/v1alpha1
# (Required) Acceptable Values: Component, Resource, System
kind: Component
metadata:
name: 'docs.openedx.org'
description: "This repository contains the root documentation site for https://docs.openedx.org"
links:
- url: "https://docs.openedx.org"
title: "Deployed Site"
icon: "Web"
annotations:
# Annotation keys and values can be whatever you want.
# We use it in open edx to have a comma separated list of github user
# names that might be interested in changes to the architecture of this
# component.
openedx.org/arch-interest-groups: "feanil"
Copy link
Copy Markdown
Contributor

@nedbat nedbat Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we find the explanation of what "annotations" means? Same for "spec", etc. Is this defined by Backstage?

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can find it here, I linked to it from just above this code-block in the references section. Let me know if you think I need more specific stuff within the code.

Copy link
Copy Markdown
Contributor

@nedbat nedbat Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I see the backstage docs now, but I have a feeling that the pure generality of their docs could be narrowed with some specifics for our use of it.

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add another comment here, let's see if that will help. Happy to add a lot more text here to help people understand what's up.

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nedbat does this address your concerns and provide sufficient context or would you like more words?

spec:

# (Required) This can be a group(`group:<group_name>` or a user(`user:<github_username>`)
owner: group:docs.edx.org-maintainers

# (Required) Acceptable Type Values: service, website, library
type: 'website'
Copy link
Copy Markdown
Contributor

@nedbat nedbat Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does "type" relate to "kind"?

Copy link
Copy Markdown
Contributor Author

@feanil feanil Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More details on the system model are here: https://backstage.io/docs/features/software-catalog/system-model

I'll add it as a reference link.


# (Required) Acceptable Lifecycle Values: experimental, production, deprecated
lifecycle: 'production'

# (Optional) The value can be the name of any known component.
subcomponentOf: '<name_of_a_component>'

# (Optional) An array of different components or resources.
dependsOn:
- '<component_or_resource>'
- '<another_component_or_resource>'

.. _Backstage: https://backstage.io
.. _Backstage system model: https://backstage.io/docs/features/software-catalog/system-model
.. _catalog-info.yaml: https://backstage.io/docs/features/software-catalog/descriptor-format
.. _single pane of glass: https://www.webopedia.com/definitions/single-pane-of-glass/