Update app creation docs

Contents (#551)

Documentation

• Move app.py documentation to new page and update it with monitor message and analysis finalisation information

Dependencies

• Update lock file

Add periodic monitor messages and support poetry-based apps on Cloud Run

Summary

Improve the creation experience of Octue services.

Contents (#548)

New features

• Add ability to set up periodic monitor messages from Analysis
• Add poetry installation option to Cloud Run Dockerfile

Enhancements

• Allow revision tag to be set in octue deploy commands
• Tell user SRUID of service being deployed by octue deploy
• Change default internal service name to local/local:local
• Allow Cloud Run execution environment to be set by deployer

Fixes

• Ensure service revision tag is set as environment variable in deployers

Dependencies

• Update flake8 used for pre-commit checks
• Update lockfile to address certificate removal in certifi
• Use new poetry format for dev dependencies in pyproject.toml

Testing

• Update tox.ini to new format

Fix manifest upload in crash diagnostics

Contents (#545)

Fixes

• Avoid double-JSON-encoding input manifests in Service.ask

Dependencies

• Update lock file

Fix and improve crash diagnostics

Summary

This release fixes and adds new features to service crash diagnostics, making it much easier to locally reconstruct errors in a service. This especially true if it has children - crash diagnostics now include a child emulator for every question asked to a child by the service before it failed.

Contents (#541)

IMPORTANT: There are 3 breaking changes.

New features

• Add ability to load crash diagnostics data as a test fixture
• Record messages from each of a service's children when it crashes

Enhancements

• 💥 BREAKING CHANGE: Avoid downloading datasets by default in get-crash-diagnostics
• 💥 BREAKING CHANGE: Save recorded messages to attribute instead of file
• Log a warning if trying to download files from empty directory
• Add Serialisable.from_file method
• Add Manifest.update_dataset_paths method
• Add representation for Runner
• Avoid uploading null strings when values or manifests not present
• Add option to include .octue files when instantiating datasets
• Add ability to filter the files downloaded in GoogleCloudStorageClient.download_all_files

Fixes

• Save original unmutated input and configuration data for crash diagnostics
• Update manifests with local paths if datasets downloaded locally
• Ensure crash diagnostics upload works without manifests
• Ensure empty values or manifests are uploaded for crash diagnostics
• Fix splitting bucket name from signed cloud storage URLs

Refactoring

• 💥 BREAKING CHANGE: Rename filename parameter to path in Serialisable.to_file
• Factor out child instantiation in Runner into a method
• Move AnalysisLogHandlerSwitcher into log_handlers module
• Move AppFrom into new app_loading module
• Remove unused Service._sent_messages attribute
• Rename recorded_messages attributes to received_messages properties
• Factor out manifest/dataset and values uploading for crash diagnostics
• Reduce nesting in Dataset._instantiate_from_cloud
• Factor out app loading and running in Runner

Dependencies

• Update poetry.lock to avoid usage of insecure py package by pytest

Testing

• Make testing downloading crash diagnostics more efficient
• Test crash diagnostics with service that has its own children
• Use test fixture to speed up octue get-crash-diagnostics tests
• Test that dataset/file metadata is uploaded with crash diagnostics
• Test metadata is preserved with octue get-crash-diagnostics

Other

• Clarify service troubleshooting doc

---

Upgrade instructions

💥 Avoid downloading datasets by default in `get-crash-diagnostics`

To keep the previous behaviour, add the --download-datasets flag when using the octue get-crash-diagnostics CLI command

💥 Save recorded messages to attribute instead of file

Instead of using the record_messages_to parameter, set the record_messages parameter to True when using Child.ask or Service.wait_for_answer. Retrieve messages from the received_messages property of either class instance.

💥 Rename filename parameter to path in Serialisable.to_file

If using the positional argument filename in the to_file method of an instance of a Serialisable mixed-in class as a keyword argument, replace it with path.

Fix manifest upload and analysis error logging

Contents (#536)

Fixes

• Always use OctueJSONEncoder when encoding JSON
• Log analysis errors before attempting to save crash diagnostics in Runner

Operations

• Update author email address

Refactoring

• Use Manifest.serialise instead of Manifest.to_primitive in Manifest.to_cloud

Remove `mkver.conf` file and update troubleshooting docs

Contents (#533)

Operations

• Use new version of octue/check-semantic-version action and remove mkver.conf file
• Use latest octue/generate-pull-request-description action
• Add prettier to pre-commit checks

Other

• Fix octue.yaml code block in docs
• Improve troubleshooting services doc
• Update service ID usage in docs

Fully support service namespacing and revision tagging

Summary

This release add support and requirement for service namespaces and service revision tags. This allows services to be versioned and provides a more familiar format for service IDs that's, for example, similar to Docker image names (e.g. octue/example-service:latest). Full validation of service IDs comes as part of this.

The new octue deploy create-push-subscription has also been added as requested by users and will be accompanied by a GitHub action in another repository.

Contents (#529)

IMPORTANT: There are 4 breaking changes.

New features

• 💥 BREAKING CHANGE: Add support for and require revision tags for services
• 💥 BREAKING CHANGE: Validate service IDs before creating or questioning services
• Add ability to set service namespace, name, and revision tag by environment variable
• Add octue deploy create-push-subscription CLI command

Enhancements

• 💥 BREAKING CHANGE: Require services to have a namespace and replace use of organisation with namespace in service configurations
• 💥 BREAKING CHANGE: Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths
• Remove application of octue.services Pub/Sub namespace from Service
• Use a coolname revision tag to allow a service revision with a non-unique SRUID to start as a unique service revision when using the octue start CLI command
• Set the Service.id attribute to the user-friendly SRUID instead of the Pub/Sub-friendly SRUID
• Return SRUID from DataflowDeployer.deploy

Fixes

• Handle inability to find/load version compatibility data file
• Use latest Apache Beam base image in Dataflow Dockerfile
• Fix some log and error messages

Operations

• Speed up and simplify release workflow
• Use new Octue GitHub actions in workflows
• Improve installation process in main Dockerfile
• Use latest commit message checker

Dependencies

• Update Dataflow setup.py file

Refactoring

• Rename OCTUE_NAMESPACE to OCTUE_SERVICES_NAMESPACE
• Remove cool-naming of services when instantiated without id and simplify the name argument

---

Upgrade instructions

💥 Add support for and require revision tags for services

The ServiceConfiguration.service_id attribute and the SERVICE_ID environment variable have been removed.

• ServiceConfiguration: Use octue.cloud.service_id.create_service_id to create IDs from ServiceConfiguration.namespace and ServiceConfiguration.name

• SERVICE_ID environment variable: Use the new OCTUE_SERVICE_NAMESPACE, OCTUE_SERVICE_NAME, and OCTUE_SERVICE_REVISION_TAG environment variables.

💥 Validate service IDs before creating or questioning services

Update your services' namespaces, names, and revision tags to follow the requirements set out in the Creating services doc.

💥 Require services to have a namespace and replace use of organisation with namespace in service configurations

Provide the namespace key in the service configuration (the octue.yaml file). If you were using the organisation key before, the namespace key now replaces it. Examples of a namespace are your organisation's name or your GitHub username.

💥 Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths

The octue.services namespace is now mandatory and applied automatically in topic and subscription paths. Please stop providing the namespace argument if you were providing it before.

Add dynamic children override

Summary

Allow parents to override the default children its child uses (i.e. the parent's grandchildren). This allows services that produce the same type of results but use, for example, different versions of a model to be swapped in and out for each other without modifying the child.

Note that the child has to be running version 0.39.0 to be able to accept the override (otherwise it will just use its default children).

Contents (#524)

New features

• Add ability to dynamically override a child's children when asking it a question

Enhancements

• Use raw service IDs in logs and errors

Fixes

• Use full service ID in MockService when answering question

Refactoring

• Refactor OrderedMessageHandler methods into smaller methods

Testing

• Remove middle-man ask_question_and_wait_for_answer method in TestService

Fix serialisation of datasets instantiated from files

Summary

Ensure that datasets constructed using the files argument are successfully serialised/deserialised without losing file information.

Contents (#525)

Fixes

• Fix serialisation/deserialisation of datasets in Manifest when the datasets have been constructed using the files argument
• Use datafile cloud path in files field if available in Dataset.to_primitive
• Use Manifest.serialise instead of Manifest.to_primitive in Analysis.finalise

Dependencies

• Use twined=^0.5.1

Make Topic and Subscription classes standalone

Summary

Make the Topic and Subscription classes standalone instead of relying on a Service instance and a related subscriber.

Contents (#523)

IMPORTANT: There are 2 breaking changes.

Enhancements

• 💥 BREAKING CHANGE: Remove service argument from Topic constructor
• 💥 BREAKING CHANGE: Remove subscriber argument from Subscription constructor
• Implicitly get Service credentials from environment and remove credentials attribute

Fixes

• Move finally clause to correct nesting level in OrderedMessageHandler.handle_messages

---

Upgrade instructions

💥 Remove `service` argument from `Topic` constructor

Provide the project_name argument instead of the old service argument.

💥 Remove `subscriber` argument from `Subscription` constructor

Remove the old subscriber argument as it's no longer needed.