Let's talk about OEP-45 (Configuring & Operating Open edX)

[kdmccormick]

Four years ago we adopted Open edX Proposal (OEP) #45 after significant discussion. I think it's time to take a look about how the proposal matches our reality today. Without evaluating their merits, let me lay the major decisions of the OEP out next to the current state of Open edX operations in Redwood. Please let me know if you disagree with my assessments.

Containerization

The OEP said that we would stop distributing Ansible playbooks to provision Open edX VMs (a.k.a. the "native installation"), and that we would start distributing production-ready Docker images from Dockerfiles in all deployable Open edX repositories.

Towards this goal, many repositories now have a Dockerfile which is build and pushed to the unsupported openedx DockerHub org. As far as I know, however, these files are only used for the deprecated Devstack tool and, in some cases, for 2U production deployments. As a whole, I would say that the project has not adopted these upstream Dockerfiles.

Completely separately, Tutor replaced the Ansible playbooks as the community-supported distribution in Maple, and more recently replaced Devstack as the supported development tool. Tutor maintains its own Jinja2-templated Dockerfiles, which Edly builds and pushes to the overhangio DockerHub org.

Related:

• [DEPR]: Dockerfiles and Docker images public-engineering#263

Micro-Services (IDAs)

Reflecting the operational strategy of edX.org at the time, the OEP emphasizes the need for ease of configuring micro-services, also called "IDAs" (independently deployable applications). Transitioning towards a micro-service architecture is not an explicit goal of the OEP, but it is the unstated basis for so many other decisions that I think we must consider it as a decision in itself.

As of Redwood, I know of six Open edX micro-services deployed outside of edX.org:

• Two are proposed for deprecation: ecommerce and xqueue.
• Two are planned to be reimplemented as edx-platform plugins: forums and edx-notes-api.
• One is supported yet heavily tailored towards edX.org use cases: course-discovery.
• This leaves two which are supported with Open edX community use in mind: credentials and the codejail sandbox.

The vast majority of core Open edX features are still exposed by the LMS (learning management system) and CMS (content management studio), both defined in edx-platform. I am not aware of any plans to extract any portions of edx-platform into micro-services, nor any plans to develop new core features as micro-services.

Django Settings Loading

The OEP declares that sites will configure each IDA's Django settings with an <APPNAME>_CFG yaml file, parsed and loaded by a single special DJANGO_SETTING_MODULE. This contrasts with the Django convention, which is that sites override Django settings using a their own DJANGO_SETTINGS_MODULE. The rationale is that all Open edX customization can be reasonably specified in yaml; so, it is operationally safer to avoid custom DJANGO_SETTINGS_MODULEs, and it is operationally desirable for all operation modes to execute the same Python module for configuration. This was briefly discussed in the OEP-45 review.

For example, production LMS config would look like:

• lms/settings/__init__.py
  • import lms/settings/required.py
    • declare settings that must be overriden
  • import lms/settings/defaults.py
    • set default settings
  • load /openedx/config/lms.yml
  • apply some minimal logic for special lms.yml overrides

As of Redwood, we do not know of any site other than edX.org that successfully uses only yaml files for configuration. The DJANGO_SETTINGS_MODULE that loads these yaml files is not simple, either; it declares defaults, imports from other Django settings modules, sets more defaults, handles dozens of special cases, and has a special Open-edX-specific "derived settings" mechanism to handle settings that depend on other settings.

Tutor does provide yaml files, but it also has custom production and development settings files. The result is multiple layers of indirection between the community-supported Open edX distribution and actual Django settings. So, production LMS configuration actually looks like:

• lms/envs/tutor/production.py
  • is generated by Tutor:
    • from template tutor/templates/apps/openedx/settings/lms/production.py
      • deriving tutor/templates/apps/openedx/settings/partials/common_lms.py
        • deriving tutor/templates/apps/openedx/settings/partials/common_all.py
    • with templates vars from Tutor configuration and plugins (config.yml)
  • import lms/envs/production.py
    • import lms/envs/common.py
      • set defaults
    • set more defaults
    • load /openedx/config/lms.yml
      • which is generated by Tutor:
        • from template tutor/templates/apps/openedx/config/lms.env.yml
          • deriving tutor/templates/apps/openedx/config/partials/auth.yml
    • set more defaults
    • lots of logic for special handling of lms.yml overrides

Django Settings Documentation

The OEP declares that Django settings with and without without reasonable defaults should be declared in separate Python modules, default.py and required.py, respectively. It declares that all of these settings are to be documented using Sphinx autodoc.

As of Redwood, Open edX does not distinguish between default-provided and required settings. Many "default" settings are inappropriate for any environment, and many "required" settings can be easily missed. Tutor, however, follows the spirit of this recommendation by splitting its own configuration options between CONFIG_DEFAULTS and CONFIG_UNIQUE; the former is similar to default.py, and the latter safely generates something similar to required.py.

Furthermore, Open edX does not use Sphinx autodoc; however, we do use code-annotations to generate rST, which matches the OEP's autodoc decision in spirit. Unfortunately, many settings remain undocumented, and we do not lint to enforce that new settings are documented with code-annotations.

Operations Manuals

The OEP recommends that every IDA have a site operator reference manual defined at ./docs/operations. I do not know of any Open edX IDAs that have such a documentation folder.

edx-platform does have a references folder, which is published here. These docs are not a complete operations guide, but they at least contain a list of feature toggles. Some IDAs also have basic operator instructions in their README.

Otherwise, the most complete guide for Open edX operators in Redwood is probably Tutor's documentation.

Sub-decisions

It's worth noting that there are two ADRs nested under OEP-45. I didn't review these in detail, but I believe that they are both significantly closer to reality than the OEP itself:

• https://docs.openedx.org/projects/openedx-proposals/en/latest/architectural-decisions/oep-0045/decisions/0001-tutor-as-replacement-for-edx-configuration.html
• https://docs.openedx.org/projects/openedx-proposals/en/latest/architectural-decisions/oep-0045/decisions/0002-helm-for-multi-instance-k8s.html

[robrap]

Thanks. I hope to be able to dig into this at a later time. One quick question is whether the replacement to ecommerce is not a separate service?