Improve async event retrieval workflow

Contents (#647)

IMPORTANT: There are 2 breaking changes.

Enhancements

• 💥 BREAKING CHANGE: Return question UUID from Child.ask
• Deserialise manifests from events in get_event
• Raise error if no events found when calling get_events

Fixes

• Use correct base image for python3.11 dockerfile
• Return schema-compliant events and attributes from get_events

Operations

• Import missing APIs into terraform config
• Deploy version 0.5.0 of event handler cloud function and update event store schema
• Update actions/setup-python to version 5

Dependencies

• 💥 BREAKING CHANGE: Make db-dtypes and google-cloud-bigquery optional
• Upgrade gunicorn to avoid vulnerability
• Loosen numpy dependency

Testing

• Test retrieving results from real asynchronous question
• Run tests with python3.10 (python3.9 isn't available on macos-latest for arm64)

Other

• Add DOI badge to readme

---

Upgrade instructions

💥 Return question UUID from `Child.ask`

Instead of writing answer = Child.ask(...), write answer, question_uuid = Child.ask(...) (and the same for ChildEmulator)

💥 Make `db-dtypes` and `google-cloud-bigquery` optional

To keep using the get_events function, add the bigquery optional extra to your installation command e.g. poetry install -E bigquery.

Switch to event-driven infrastructure and improve support for asynchronous questions

Summary

This pull request:

• Makes the SDK fully event-driven by using a single topic to emit/consume events
• Majorly refactors the event handler to facilitate asynchronous event retrieval
• Adds the ability to get and replay events from a BigQuery store

Contents (#632)

IMPORTANT: There are 4 breaking changes.

New features

• 💥 BREAKING CHANGE: Use single topic per workspace (#639)
• Add get_events function for retrieving events asynchronously from BigQuery
• Add EventReplayer class to replay asynchronously-retrieved events
• Add Manifest.download method

Enhancements

• Get subscription project name from topic by default
• Improve asking of asynchronous questions via Child.ask
• Return download path from Dataset.download
• Include question UUID in delivery acknowledgement log message
• Improve handling of invalid events
• Add datetime and uuid attributes to all events

Fixes

• Await successful publishing of question messages
• Fix api_access_endpoint usage in mock_generate_signed_url

Operations

• Add test BigQuery dataset, cloud function, and IAM roles to terraform config
• Switch to reusable workflows where possible

Dependencies

• Add google-cloud-bigquery
• Upgrade coolname
• Add db-dtypes for converting bigquery rows to dataframes

Refactoring

• 💥 BREAKING CHANGE: Rename x.received_messages to x.received_events
• 💥 BREAKING CHANGE: Rename record_messages parameters to record_events
• 💥 BREAKING CHANGE: Update ChildEmulator to use event* instead of message*
• Factor out making minimal dictionary
• Factor out creating push subscription
• Factor out emitting question event in Service.ask
• Factor out event handlers and related logic from OrderedMessageHandler into new AbstractEventHandler
• Move validation module into octue.cloud.events subpackage
• Rename OrderedMessageHandler to GoogleCloudPubSubEventHandler
• Rename "message" to "event" in event handler classes
• Rename GooglePubSubHandler to GoogleCloudPubSubHandler

Chores

• Update licence year to 2024

Testing

• Simplify various tests

Upgrade instructions

• Update all services in your services network to this version of octue or later (0.53.0+)
• Replace any usages of the received_messages methods with received_events
• Replace any usages of the record_messages parameters with record_events
• Replace the word message with event in usages of ChildEmulator methods (apart from in the case of monitor_message)

Warn about messages with duplicate message numbers

Contents (#627)

Enhancements

• Warn about messages with duplicate message numbers

Make event handling faster and resilient to missing events

Contents (#625)

IMPORTANT: There is 1 breaking change.

Enhancements

• Allow setting of maximum number of workers for parallel questions in Child.ask_multiple
• Pull up to 50 messages from answer subscriptions at once instead of 1
• Allow skipping of any missing message after a 10s delay in OrderedMessageHandler
• Suppress name/namespace override warning if the value is the same in the environment and service configuration file
• Speed up event validation by caching service communication JSON schema
• 💥 BREAKING CHANGE: Extract SRUID for child logs context from subscription in message handler

Fixes

• Exit early from message pulling if heartbeat check fails
• Make Manifest.update_dataset_paths method thread-safe

Refactoring

• Factor out multiple checks of package version in message handler

Testing

• Improve message handling tests by not mocking _pull_and_enqueue_available_messages method and removing MockMessagePuller

---

Upgrade instructions

💥 Extract SRUID for child logs from subscription in message handler

This removes the service_name argument from Service.wait_for_answer. If you were using this argument, simply remove it; logs from children shown in a parent will now have the full and correct SRUID automatically.

Publish answers to question topic

Summary

This pull request removes the use of answer topics by publishing answer messages to the service revision (formerly known as question) topic and filtering subscriptions to only receive a) questions or b) response messages to a specific question. This speeds up the question asking process, reduces cloud infrastructure requirements and the permissions surface, and allows us to avoid topic number limits.

Also added is validation of all messages and their attributes against a new publicly available schema. This ensures services are communicating as they should and opens up the possibility of writing services in other languages and creating emulators.

As this, by itself, constitutes an inter-service communication breaking change, we've taken the opportunity to reduce the complexity of the codebase by removing backwards compatibility patches for service communication (i.e. we've grouped multiple breaking changes together into one).

Contents (#603)

IMPORTANT: There are 7 breaking changes.

New features

• Validate messages and their attributes against new service communication schema (see #614 for changelog - it was merged into this branch)
• Allow diagnostics (formerly known as crash diagnostics) to always be switched on for a service

Enhancements

• 💥 BREAKING CHANGE: Publish responses to questions to the service revision (question) topic instead of creating a separate answer topic
• 💥 BREAKING CHANGE: Store message number in message attributes instead of in message data
• 💥 BREAKING CHANGE: Remove question UUID from log record message body
• 💥 BREAKING CHANGE: Remove inter-service communication backwards compatibility code
• 💥 BREAKING CHANGE: Make input and output values and manifest optional
• 💥 BREAKING CHANGE: Replace boolean allow_save_diagnostics_data_on_crash argument with string/enum save_diagnostics argument in Service.ask and related methods
• Add ability to filter subscriptions
• Add question UUID attribute to all messages
• Send more possible errors to parent in Service.answer
• Add kind field to question messages
• Add sender_type attribute to all messages
• Add ability to instantiate Runner from service/app configurations

Fixes

• Stop double-JSON-encoding output manifests

Dependencies

• Update octue version in template apps' dependencies

Refactoring

• 💥 BREAKING CHANGE: Rename crash diagnostics to diagnostics
• Group message attributes in Service._send_message and MockMessage under explicit attributes argument
• Make OrderedMessageHandler._waiting_messages attribute public
• Rename various message attributes

Testing

• Store mock pub/sub messages against subscriptions instead of topics
• Add missing type field to emulated Pub/Sub questions

Operations

• Fix add-issues-to-octue-board workflow
• Stop automatically building docker images for registry in release workflow
• Add ReadTheDocs config file to fix documentation building

---

Upgrade instructions

💥 Update all Octue services in your network to use this version of octue so they're still able to communicate. Postpone upgrading until you can upgrade all services simultaneously.

💥 Replace allow_save_diagnostics_data_on_crash with save_diagnostics set to one of these values: "SAVE_DIAGNOSTICS_OFF", "SAVE_DIAGNOSTICS_ON_CRASH", or "SAVE_DIAGNOSTICS_ON"

💥 Crash diagnostics rename:

• Use the octue get-diagnostics CLI command instead of the octue get-crash-diagnostics command
• Rename crash_diagnostics_cloud_path in your service configurations to diagnostics_cloud_path

Fix signed URL and child emulator bugs

Contents (#621)

Fixes

• Make Manifest.use_signed_urls_for_datasets idempotent
• Ensure directories exist before downloading datafile from URL
• Ensure child emulators can accept input manifests

Add ability to automatically retry failed multiple questions

Contents (#608)

IMPORTANT: There is 1 breaking change.

Enhancements

• Add ability to automatically retry failed questions in Child.ask_multiple
• Add Cloud Run dockerfile with python3.11 base

Operations

• Use trusted publisher to publish to PyPi

Refactoring

• 💥 BREAKING CHANGE: Remove deprecated code

Testing

• Remove unnecessary service IDs from TestChild tests

---

Upgrade instructions

💥 Remove deprecated code

• Use datafile.local_path instead of datafile.path
• Use octue.utils.processes.run_logged_subprocess instead of octue.utils.threads.run_logged_subprocess

Require children to be registered in service registry if provided

Contents (#604)

Enhancements

• Raise error when asking a question to an unregistered service revision (when using service registries only)

Fixes

• Ensure questions can be asked to unregistered service revisions when not using a service registry

Operations

• Fix workflows name typo

Add ability to catch errors when asking multiple questions

Contents (#607)

Enhancements

• Add ability to catch errors for failed questions and keep answers for successful questions when using Child.ask_multiple

Remove dataflow deployment CLI command

Contents (#598)

IMPORTANT: There is 1 breaking change.

Operations

• Remove unnecessary checkout steps in workflows
• Run PR workflow on all PRs

Dependencies

• Remove dataflow-related dependencies

Testing

• Use live schema to test essential monitor messages

Reversions

• 💥 BREAKING CHANGE: Remove octue deploy dataflow command and related code
• Remove deployment-related attributes from ServiceConfiguration

---

Upgrade instructions

💥 Remove `octue deploy dataflow` command and related code

Use our deployment GitHub actions and workflows or contact us to deploy to Dataflow.