To work in the framework itself you will need Python >= 3.8. Linting, testing,
and docs automation is performed using
tox, which you should install.
For improved performance on the tests, ensure that you have PyYAML
installed with the correct extensions:
apt-get install libyaml-dev
pip install --force-reinstall --no-cache-dir pyyamlThe following are likely to be useful during development:
# Run linting and unit tests
tox
# Run tests, specifying whole suite or specific files
tox -e unit
tox -e unit test/test_charm.py
# Format the code using isort and autopep8
tox -e fmt
# Compile the requirements.txt file for docs
tox -e docs-deps
# Generate a local copy of the Sphinx docs in docs/_build
tox -e docs
# run only tests matching a certain pattern
tox -e unit -- -k <pattern>For more in depth debugging, you can enter any of tox's created virtualenvs
provided they have been run at least once and do fun things - e.g. run
pytest directly:
# Enter the linting virtualenv
source .tox/lint/bin/activate
...
# Enter the unit testing virtualenv and run tests
source .tox/unit/bin/activate
pytest
...
The framework has some tests that interact with a real/live Pebble server. To
run these tests, you must have pebble
installed and available in your path. If you have the Go toolchain installed,
you can run go install github.com/canonical/pebble/cmd/pebble@master. This will
install pebble to $GOBIN if it is set or $HOME/go/bin otherwise. Add
$GOBIN to your path (e.g. export PATH=$PATH:$GOBIN or export PATH=$PATH:$HOME/go/bin in your .bashrc) and you are ready to run the real
Pebble tests:
tox -e pebbleTo do this even more manually, you could start the Pebble server yourself:
export PEBBLE=$HOME/pebble
export RUN_REAL_PEBBLE_TESTS=1
pebble run --create-dirs --http=:4000 &>pebble.log &
# Then
tox -e unit -- test/test_real_pebble.py
# or
source .tox/unit/bin/activate
pytest -v test/test_real_pebble.pyWhen making changes to ops, you'll commonly want to try those changes out in
a charm.
If your changes are in a Git branch, you can simply replace your ops version
in requirements.txt (or pyproject.toml) with a reference to the branch, like:
#ops ~= 2.9
git+https://github.com/{your-username}/operator@{your-branch-name}
git is not normally available when charmcraft is packing the charm, so you'll
need to also tell charmcraft that it's required for the build, by adding
something like this to your charmcraft.yaml:
parts:
charm:
build-packages:
- gitIf your changes are only on your local device, you can inject your local ops
into the charm after it has packed, and before you deploy it, by unzipping the
.charm file and replacing the ops folder in the virtualenv. This small
script will handle that for you:
#!/usr/bin/env bash
if [ "$#" -lt 2 ]
then
echo "Inject local copy of Python Operator Framework source into charm"
echo
echo "usage: inject-ops.sh file.charm /path/to/ops/dir" >&2
exit 1
fi
if [ ! -f "$2/framework.py" ]; then
echo "$2/framework.py not found; arg 2 should be path to 'ops' directory"
exit 1
fi
set -ex
mkdir inject-ops-tmp
unzip -q $1 -d inject-ops-tmp
rm -rf inject-ops-tmp/venv/ops
cp -r $2 inject-ops-tmp/venv/ops
cd inject-ops-tmp
zip -q -r ../inject-ops-new.charm .
cd ..
rm -rf inject-ops-tmp
rm $1
mv inject-ops-new.charm $1If your ops change relies on a change in a Juju branch, you'll need to deploy
your charm to a controller using that version of Juju. For example, with microk8s:
- Build Juju and its dependencies
- Run
make microk8s-operator-update - Run
GOBIN=/path/to/your/juju/_build/linux_amd64/bin:$GOBIN /path/to/your/juju bootstrap - Add a model and deploy your charm as normal
We rely on automation to update charm pins of a bunch of charms that use the operator framework. The script can be run locally too.
In general, new functionality should always be accompanied by user-focused documentation that is posted to https://juju.is/docs/sdk. The content for this site is written and hosted on https://discourse.charmhub.io/c/doc. New documentation should get a new topic/post on this Discourse forum and then should be linked into the main docs navigation page(s) as appropriate. The ops library's SDK page content is pulled from the corresponding Discourse topic. Each page on juju.is has a link at the bottom that takes you to the corresponding Discourse page where docs can be commented on and edited (if you have earned those privileges).
Currently we don't publish separate versions of documentation for separate releases. Instead, new features should be sign-posted (for example, as done for File and directory existence in 1.4) with Markdown like this:
[note status="version"]1.4[/note]next to the relevant content (e.g. headings, etc.).
The ops library's API reference is automatically built and published to ops.readthedocs.io. Please be complete with docstrings and keep them informative for users. The published docs are always for the in-development (main branch) of ops, and do not include any notes indicating changes or additions across versions - we encourage all charmers to promptly upgrade to the latest version of ops, and to refer to the release notes and changelog for learning about changes.
During the release process, changes also get a new entry in CHANGES.md. These are grouped into the same groupings as commit messages (feature, fix, documentation, performance, etc). The only exceptions are changes that are not visible to the built releases, such as CI workflow changes, or are implicit, such as bumping the ops version number. Each entry should be a short, single line, bullet point, and should reference the GitHub PR that introduced the change (as plain text, not a link).
As noted above, you can generate a local copy of the API reference docs with tox:
tox -e docs
open docs/_build/html/index.htmlIf dependencies are updated in pyproject.toml, you can run the following command
before generating docs to recompile the requirements.txt file used for docs:
tox -e docs-depsThe documentation uses Canonical styling which is customised on top of the Furo Sphinx theme. The easiest way to pull in Canonical style changes is by using the Canonical documentation starter pack, see docs and repository.
TL;DR:
- Clone the starter pack repository to a local directory:
git clone git@github.com:canonical/sphinx-docs-starter-pack. - Copy the folder
.sphinxunder the starter pack repo to the operator repodocs/.sphinx.
There are two configuration files: docs/conf.py and docs/custom_conf.py, copied and customised from the starter pack repo.
To customise, change the file docs/custom_conf.py only, and theoretically, we should not change docs/conf.py (however, some changes are made to docs/conf.py, such as adding autodoc, PATH, fixing issues, etc.)
The Canonical documentation starter pack uses Make to build the documentation, which will run the script docs/.sphinx/build_requirements.py and generate a requirement file requirements.txt under docs/.sphinx/.
To pull in new dependency changes from the starter pack, change to the starter pack repository directory, and build with the following command. This will create a virtual environment, generate a dependency file, install the software dependencies, and build the documentation:
make htmlThen, compare the generated file .sphinx/requirements.txtand the project.optional-dependencies.docs section of pyproject.toml and adjust the pyproject.toml file accordingly.
The Python dependencies of ops are kept as minimal as possible, to avoid
bloat and to minimise conflict with the charm's dependencies. The dependencies
are listed in pyproject.toml in the project.dependencies section.
Test environments are managed with tox and executed with pytest, with coverage measured by coverage. Static type checking is done using pyright, and extends the Python 3.8 type hinting support through the typing_extensions package.
Formatting uses isort and autopep8, with linting also using flake8, including the docstrings, builtins and pep8-naming extensions.
All tool configuration is kept in project.toml. The list of
dependencies can be found in the relevant tox.ini environment deps field.
The build backend is setuptools, and the build frontend is build.
To make a release of the ops library, do the following:
- Visit the releases page on GitHub.
- Click "Draft a new release"
- The "Release Title" is simply the full version number, in the form .. and a brief summary of the main changes in the release E.g. 2.3.12 Bug fixes for the Juju foobar feature when using Python 3.12
- Use the "Generate Release Notes" button to get a copy of the changes into the notes field.
- Group the changes by the commit type (feat, fix, etc.) and use full names (e.g., "Features", not "feat") for group headings. Strip the commit type prefix from the bullet point. Strip the username (who did each commit) if the author is a member of the Charm Tech team.
- Where appropriate, collapse multiple tightly related bullet points into a single point that refers to multiple commits.
- Create a new branch, and copy this text to the CHANGES.md file, stripping out links, who did each commit, the new contributor list, and the link to the full changelog.
- Change [version.py][ops/version.py]'s
versionto the appropriate string. - Add, commit, and push, and open a PR to get the changelog and version bump into main (and get it merged).
- Back in the GitHub releases page, tweak the release notes - for example, you might want to have a short paragraph at the intro on particularly noteworthy changes.
- Have someone else in the Charm-Tech team proofread the release notes.
- When you are ready, click "Publish". (If you are not ready, click "Save as Draft".)
This will trigger an automatic build for the Python package and publish it to PyPI) (authorisation is handled via a Trusted Publisher relationship). Note that it sometimes take a bit of time for the new release to show up.
See .github/workflows/publish.yml for details. (Note that the versions in publish.yml refer to versions of the GitHub actions, not the versions of the ops library.)
You can troubleshoot errors on the Actions Tab.