Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Observability #916

Closed
grski opened this issue Nov 7, 2023 · 6 comments · Fixed by #1398
Closed

Feature: Observability #916

grski opened this issue Nov 7, 2023 · 6 comments · Fixed by #1398
Labels
Core Issues related to core FastStream functionality and affects to all brokers enhancement New feature or request

Comments

@grski
Copy link

grski commented Nov 7, 2023

We'd love to use faststream as part of our application however one thing that popped out for consideration is observability.

We currently live in the opentelemetry world and honeycomb, with some sentry on top of that. Sentry could be integrated through their basic sdk I think, however it'd be nice to have opentelemetry support of some sorts OOB.

Given it's industry standard it should be helpful to a lot of people making observability way easier.

We also thought about getting some kind of kind of a heartbeat from the workers? We'd like to know eg. if it died and wondering if it should be handled eg. a level higher on container level or sth or is it already avaialble?

Awesome work so far.

@grski grski added the enhancement New feature or request label Nov 7, 2023
@davorrunje
Copy link
Collaborator

Thanx for the suggestion, we'll start working on it very soon.

@dearkafka
Copy link

voting for this feature

@davorrunje
Copy link
Collaborator

We are looking into it right now.

@Kludex
Copy link

Kludex commented Mar 8, 2024

We are looking into it right now.

Did you find something there? 👀

@Lancetnik
Copy link
Collaborator

I found a project trying to implement this feature: https://github.com/ashambalev/opentelemetry-instrumentation-faststream

But, after 0.5.0 release, I hope, we can implement it as a broker middleware and distribute as optional dependency.

@Lancetnik
Copy link
Collaborator

Assigned to @draincoder

@davorrunje davorrunje assigned davorrunje and unassigned davorrunje Apr 22, 2024
@Lancetnik Lancetnik added the Core Issues related to core FastStream functionality and affects to all brokers label May 16, 2024
github-merge-queue bot pushed a commit that referenced this issue May 18, 2024
* Create TelemetryMiddleware

* Refactor TelemetryMiddleware

* Optimization TelemetryMiddleware

* Refactor TelemetryMiddleware

* Change msg type in __call__

* Continue process span while publish

* Change nesting and add create span

* refactor: mv useless types to TYPE_CHECKING

* refactor: add TelemetrySettingsProvider protocol & NATS impl

* chore: update distributions

* Fixed a typo in PR template

* Fixed a typo in ABCBroker: _broker_dependecies to _broker_dependencies

* Add TelemetryTestcase

* Fix invalid ruff formating

* Add telemetry tests, fix kind error

* Add metrics test, fix invalid metric error

* More specifics in the metrics test

* Changed getting a destination to methods

* Add Rabbit impl

* Ruff format

* Remove delivery_tag from publish

* Refactor extracting attributes

* Ruff format

* Add AIOKafka impl

* Add Confluent impl

* Rename test classes

* Add Redis impl

* Refactor settings providers import

* fix: correct TelemetryMiddlewares batch consuming

* Fix spans and metrics ordering in tests, switch Confluent attributes methods

* Add MESSAGING_BATCH_MESSAGE_COUNT attribute

* Add TestConsume and TestPublish to telemetry tests

* refactor: create independant TelemetryMiddleware for each broker

* tests: use broker factory instead of fixtures in Consume and Publish test cases

* tests: refactor RPC and TestClient testcases

* tests: refactor RMQ tests

* tests: refactor Kafka tests

* tests: NATS subscriber tests with patching

* tests: RMQ subscriber tests with patching

* tests: refactor Redis tests

* tests: refactor Confluent tests

* chore: run CI

* tests: fix Confluent TestClient tests

* tests: fix py3.8 compatibility

* lint: fix some mypy

* chore: fix CI

* tests: fix partition tests

* tests: fix partition real broker case

* Change supported metrics

* Separation of metrics for systems with and without batches

* Fix _subscriber_setup_extra merge conflict

* Refactor and update metrics tests, fix metrics

* Refactor metrics, need batches count in publish

* feat: count Redis consuming batch size

* lint: fix mypy

* Add batch tests, metrics ready

* chore: solve main conflicts

* fix: correct confluent batch body serialization

* Fix opentelemetry redis batch test

* lint: correct StreamRoute call annotation

* Add docs for OpenTelemetry

* docs: add otel page links style

* chore: bump version

* Refactor docs

* docs: fix styles

---------

Co-authored-by: Nikita Pastukhov <diementros@yandex.ru>
Co-authored-by: Pastukhov Nikita <nikita@pastukhov-dev.ru>
Co-authored-by: treaditup <treaditup@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core Issues related to core FastStream functionality and affects to all brokers enhancement New feature or request
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

6 participants