Switch to event-driven infrastructure and improve support for asynchronous questions
Summary
This pull request:
- Makes the SDK fully event-driven by using a single topic to emit/consume events
- Majorly refactors the event handler to facilitate asynchronous event retrieval
- Adds the ability to get and replay events from a BigQuery store
Contents (#632)
IMPORTANT: There are 4 breaking changes.
New features
- 💥 BREAKING CHANGE: Use single topic per workspace (#639)
- Add
get_events
function for retrieving events asynchronously from BigQuery - Add
EventReplayer
class to replay asynchronously-retrieved events - Add
Manifest.download
method
Enhancements
- Get subscription project name from topic by default
- Improve asking of asynchronous questions via
Child.ask
- Return download path from
Dataset.download
- Include question UUID in delivery acknowledgement log message
- Improve handling of invalid events
- Add
datetime
anduuid
attributes to all events
Fixes
- Await successful publishing of question messages
- Fix
api_access_endpoint
usage inmock_generate_signed_url
Operations
- Add test BigQuery dataset, cloud function, and IAM roles to terraform config
- Switch to reusable workflows where possible
Dependencies
- Add
google-cloud-bigquery
- Upgrade
coolname
- Add
db-dtypes
for converting bigquery rows to dataframes
Refactoring
- 💥 BREAKING CHANGE: Rename
x.received_messages
tox.received_events
- 💥 BREAKING CHANGE: Rename
record_messages
parameters torecord_events
- 💥 BREAKING CHANGE: Update
ChildEmulator
to useevent*
instead ofmessage*
- Factor out making minimal dictionary
- Factor out creating push subscription
- Factor out emitting question event in
Service.ask
- Factor out event handlers and related logic from
OrderedMessageHandler
into newAbstractEventHandler
- Move
validation
module intooctue.cloud.events
subpackage - Rename
OrderedMessageHandler
toGoogleCloudPubSubEventHandler
- Rename "message" to "event" in event handler classes
- Rename
GooglePubSubHandler
toGoogleCloudPubSubHandler
Chores
- Update licence year to 2024
Testing
- Simplify various tests
Upgrade instructions
- Update all services in your services network to this version of
octue
or later (0.53.0
+) - Replace any usages of the
received_messages
methods withreceived_events
- Replace any usages of the
record_messages
parameters withrecord_events
- Replace the word
message
withevent
in usages ofChildEmulator
methods (apart from in the case ofmonitor_message
)