853 opentelemetry metrics for server mode #944
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Code Coverage (Ubuntu)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 2fa09fc |
|
Code Coverage (Windows)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 7edfabe |
dalpasso
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
2 times, most recently
from
79d31be
to
8cf83ef
Compare
sbrunato
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
from
7afb97b
to
08eee64
Compare
dalpasso
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
2 times, most recently
from
2fa09fc
to
a515210
Compare
alambare-csgroup
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
from
a515210
to
0205ce5
Compare
dalpasso
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
from
abc6c8e
to
34848a5
Compare
|
Code Coverage (Ubuntu)Diff against developResults for commit: 96e5a8a Minimum allowed coverage is ♻️ This comment has been updated with latest results |
|
Code Coverage (Windows)Diff against developResults for commit: 96e5a8a Minimum allowed coverage is ♻️ This comment has been updated with latest results |
This decorator is used by the auto-instrumentation of EODAG
The env var OTEL_EXPORTER_OTLP_ENDPOINT must be set to init the metrics
…-search operation
This reverts commit db5a991.
The attribute can be used to have a list of the providers and product types.
TODO: the callback functions continue to be called after removing the instrumentation
dalpasso
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
from
0c83fc7
to
b87329b
Compare
|
Instructions to run the metrics. Start the OTEL collector: Set the following environment variables: Run EODAG in server mode. The metrics are available on |
alambare-csgroup
requested changes
May 22, 2024
Comment on lines
148
to
159
| {{- if .Values.otel.endpoint }} | ||
| - name: OTEL_EXPORTER_OTLP_ENDPOINT | ||
| value: {{ .Values.otel.endpoint | quote }} | ||
| {{- end }} | ||
| {{- if .Values.otel.timeout }} | ||
| - name: OTEL_EXPORTER_OTLP_TIMEOUT | ||
| value: {{ .Values.otel.timeout | quote }} | ||
| {{- end }} | ||
| {{- if .Values.otel.interval }} | ||
| - name: OTEL_METRIC_EXPORT_INTERVAL | ||
| value: {{ .Values.otel.interval | quote }} | ||
| {{- end }} |
There was a problem hiding this comment.
I think we can simplify that and require OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TIMEOUT and OTEL_METRIC_EXPORT_INTERVAL if otel is enabled. No need to verify parameters individually.
eodag/rest/server.py
Outdated
Comment on lines
899
to
990
| def telemetry_init(fastapi_app: Optional[FastAPI] = None) -> None: | ||
| """Init telemetry | ||
|
|
||
| :param fastapi_app: FastAPI to automatically instrument. | ||
| :type fastapi_app: FastAPI""" | ||
|
|
||
| if not os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"): | ||
| return None | ||
| if "opentelemetry" not in sys.modules: | ||
| return None | ||
|
|
||
| # Start OTLP exporter | ||
| resource = Resource(attributes={SERVICE_NAME: "eodag-serve-rest"}) | ||
| # trace | ||
| tracer_provider = TracerProvider(resource=resource) | ||
| processor = BatchSpanProcessor(OTLPSpanExporter()) | ||
| tracer_provider.add_span_processor(processor) | ||
| trace.set_tracer_provider(tracer_provider) | ||
| # metrics | ||
| reader = PeriodicExportingMetricReader(OTLPMetricExporter()) | ||
| view_histograms: View = View( | ||
| instrument_type=Histogram, | ||
| aggregation=ExplicitBucketHistogramAggregation( | ||
| boundaries=( | ||
| 0.25, | ||
| 0.50, | ||
| 0.75, | ||
| 1.0, | ||
| 1.5, | ||
| 2.0, | ||
| 3.0, | ||
| 4.0, | ||
| 5.0, | ||
| 6.0, | ||
| 7.0, | ||
| 8.0, | ||
| 9.0, | ||
| 10.0, | ||
| ) | ||
| ), | ||
| ) | ||
| view_overhead_histograms: View = View( | ||
| instrument_type=Histogram, | ||
| instrument_name="*overhead*", | ||
| aggregation=ExplicitBucketHistogramAggregation( | ||
| boundaries=( | ||
| 0.030, | ||
| 0.040, | ||
| 0.050, | ||
| 0.060, | ||
| 0.070, | ||
| 0.080, | ||
| 0.090, | ||
| 0.100, | ||
| 0.125, | ||
| 0.150, | ||
| 0.175, | ||
| 0.200, | ||
| 0.250, | ||
| 0.500, | ||
| ) | ||
| ), | ||
| ) | ||
| meter_provider = MeterProvider( | ||
| resource=resource, | ||
| metric_readers=[reader], | ||
| views=( | ||
| view_histograms, | ||
| view_overhead_histograms, | ||
| ), | ||
| ) | ||
| metrics.set_meter_provider(meter_provider) | ||
|
|
||
| # Auto instrumentation | ||
| if fastapi_app: | ||
| logger.debug("Instrument FastAPI app") | ||
| FastAPIInstrumentor.instrument_app( | ||
| app=fastapi_app, | ||
| tracer_provider=tracer_provider, | ||
| meter_provider=meter_provider, | ||
| ) | ||
| logger.debug("Instrument EODAG app") | ||
|
|
||
| from eodag.utils.instrumentation.eodag import EODAGInstrumentor | ||
|
|
||
| EODAGInstrumentor(eodag_api).instrument( | ||
| tracer_provider=tracer_provider, | ||
| meter_provider=meter_provider, | ||
| ) | ||
| return None | ||
|
|
||
|
|
There was a problem hiding this comment.
Move this function to a separate file
Comment on lines
1
to
18
| # -*- coding: utf-8 -*- | ||
| # Copyright 2023, CS GROUP - France, https://www.csgroup.eu/ | ||
| # | ||
| # This file is part of EODAG project | ||
| # https://www.github.com/CS-SI/EODAG | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| """OpenTelemetry auto-instrumentation packages""" |
There was a problem hiding this comment.
this should be removed, we take the instrumentation from https://github.com/CS-SI/opentelemetry-instrumentation-eodag
Comment on lines
1
to
605
| :rtype: Iterable[Observation] | ||
| """ | ||
| new_available_product_types: List[str] = [ | ||
| p["ID"] for p in self._eodag_api.list_product_types() | ||
| ] | ||
| observations_dict: Dict[str, int] = { | ||
| p: 0 for p in self._last_available_product_types | ||
| } | ||
| for p in new_available_product_types: | ||
| observations_dict[p] = 1 | ||
| self._last_available_product_types = new_available_product_types | ||
| observations = [ | ||
| Observation( | ||
| v, | ||
| {"product_type_id": k}, | ||
| ) | ||
| for k, v in observations_dict.items() | ||
| ] | ||
| return observations | ||
|
|
||
| def _instrument(self, **kwargs) -> None: | ||
| """Instruments EODAG""" | ||
| tracer_provider = kwargs.get("tracer_provider") | ||
| tracer = get_tracer(__name__, tracer_provider=tracer_provider) | ||
| meter_provider = kwargs.get("meter_provider") | ||
| meter = get_meter(__name__, meter_provider=meter_provider) | ||
|
|
||
| if self._eodag_api: | ||
| meter.create_observable_gauge( | ||
| name="eodag.core.available_providers", | ||
| callbacks=[self._available_providers_callback], | ||
| description="The number available providers", | ||
| ) | ||
| meter.create_observable_gauge( | ||
| name="eodag.core.available_product_types", | ||
| callbacks=[self._available_product_types_callback], | ||
| description="The number available product types", | ||
| ) | ||
|
|
||
| request_duration_seconds = meter.create_histogram( | ||
| name="eodag.server.request_duration_seconds", | ||
| unit="s", | ||
| description="Measures the duration of the inbound HTTP request", | ||
| ) | ||
| outbound_request_duration_seconds = meter.create_histogram( | ||
| name="eodag.core.outbound_request_duration_seconds", | ||
| unit="s", | ||
| description="Measure the duration of the outbound HTTP request", | ||
| ) | ||
| request_overhead_duration_seconds = meter.create_histogram( | ||
| name="eodag.server.request_overhead_duration_seconds", | ||
| unit="s", | ||
| description="Measure the duration of the EODAG overhead on the inbound HTTP request", | ||
| ) | ||
|
|
||
| downloaded_data_counter = meter.create_counter( | ||
| name="eodag.download.downloaded_data_bytes_total", | ||
| description="Measure data downloaded from each provider and product type", | ||
| ) | ||
| _instrument_download( | ||
| tracer, | ||
| downloaded_data_counter, | ||
| request_duration_seconds, | ||
| outbound_request_duration_seconds, | ||
| request_overhead_duration_seconds, | ||
| ) | ||
|
|
||
| searched_product_types_counter = meter.create_counter( | ||
| name="eodag.core.searched_product_types_total", | ||
| description="The number of searches by provider and product type", | ||
| ) | ||
| _instrument_search( | ||
| tracer, | ||
| searched_product_types_counter, | ||
| request_duration_seconds, | ||
| outbound_request_duration_seconds, | ||
| request_overhead_duration_seconds, | ||
| ) | ||
|
|
||
| def _uninstrument(self, **kwargs) -> None: | ||
| """Uninstrument the library. | ||
|
|
||
| This only works if no other module also patches eodag""" | ||
| from eodag.rest import server | ||
|
|
||
| patches = [ | ||
| (server, "search_stac_items"), | ||
| (server, "download_stac_item"), | ||
| (QueryStringSearch, "_request"), | ||
| (Download, "progress_callback_decorator"), | ||
| ] | ||
| for p in patches: | ||
| instr_func = getattr(p[0], p[1]) | ||
| if not getattr( | ||
| instr_func, | ||
| "opentelemetry_instrumentation_eodag_applied", | ||
| False, | ||
| ): | ||
| continue | ||
| setattr(p[0], p[1], instr_func.__wrapped__) |
There was a problem hiding this comment.
remove this as well
Comment on lines
1
to
21
| # -*- coding: utf-8 -*- | ||
| # Copyright 2023, CS GROUP - France, https://www.csgroup.eu/ | ||
| # | ||
| # This file is part of EODAG project | ||
| # https://www.github.com/CS-SI/EODAG | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| _instruments = ("eodag ~= 2.0",) | ||
|
|
||
| _supports_metrics = True |
There was a problem hiding this comment.
remove this as well
dalpasso
force-pushed
the
853-opentelemetry-metrics-for-server-mode
branch
from
ae8bba6
to
96e5a8a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented OpenTelemetry metrics with Prometheus exporter.
Metrics implemented: