New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: detect monitored resources on all GCP environments #200
Merged
Merged
Changes from 22 commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
7b9d81f
added resource detection for cloud run
daniel-sanche add39c4
added new monitored resource tests
daniel-sanche 2e85755
added gcf resource detection
daniel-sanche 0e80fec
monitored resource detection for gce
daniel-sanche 97040e8
refactored
daniel-sanche eb8f309
added parens
daniel-sanche 98ca4d5
fixed env var names
daniel-sanche d7ddd5b
fixed env var names
daniel-sanche 2124310
fixed variable name
daniel-sanche 32b639f
fixed variable name
daniel-sanche 5ed2ab0
added zone to appengine results
daniel-sanche 8723cfa
upgraded environment test repo
daniel-sanche f1bb7f7
updated appengine resource creation
daniel-sanche 3a63d84
refactored, fixed unit tests
daniel-sanche 3436d06
ran blacken
daniel-sanche 91aabaa
blacken env tests
daniel-sanche b4828e8
fixed incorrect variable name
daniel-sanche 310b729
fixed lint issues
daniel-sanche 3507bc6
added unit tests for _monitored_resources
daniel-sanche 7448cf6
added unit tests
daniel-sanche bd90217
blacken tests
daniel-sanche c6dde1a
fixed failing unit tests
daniel-sanche 93be25c
fixed app engine import path
daniel-sanche 6c0db74
added comments
daniel-sanche 2cca768
CloudLoggingHandler should attempt to infer resource type
daniel-sanche aeebe3a
use variables for magic strings
daniel-sanche f4da836
fixed lint issue
daniel-sanche 2d0ff7e
pulled in go environment tests
daniel-sanche 4e0dd3c
merged submodule into main
daniel-sanche File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
159 changes: 159 additions & 0 deletions
159
google/cloud/logging_v2/handlers/_monitored_resources.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,159 @@ | ||
# Copyright 2021 Google LLC | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import os | ||
|
||
from google.cloud.logging_v2.resource import Resource | ||
from google.cloud.logging_v2._helpers import retrieve_metadata_server | ||
|
||
_GAE_SERVICE_ENV = "GAE_SERVICE" | ||
_GAE_VERSION_ENV = "GAE_VERSION" | ||
_GAE_INSTANCE_ENV = "GAE_INSTANCE" | ||
_GAE_ENV_VARS = [_GAE_SERVICE_ENV, _GAE_VERSION_ENV, _GAE_INSTANCE_ENV] | ||
"""Environment variables set in App Engine environment.""" | ||
|
||
_CLOUD_RUN_SERVICE_ID = "K_SERVICE" | ||
_CLOUD_RUN_REVISION_ID = "K_REVISION" | ||
_CLOUD_RUN_CONFIGURATION_ID = "K_CONFIGURATION" | ||
_CLOUD_RUN_ENV_VARS = [ | ||
_CLOUD_RUN_SERVICE_ID, | ||
_CLOUD_RUN_REVISION_ID, | ||
_CLOUD_RUN_CONFIGURATION_ID, | ||
] | ||
"""Environment variables set in Cloud Run environment.""" | ||
|
||
_FUNCTION_TARGET = "FUNCTION_TARGET" | ||
_FUNCTION_SIGNATURE = "FUNCTION_SIGNATURE_TYPE" | ||
_FUNCTION_NAME = "FUNCTION_NAME" | ||
_FUNCTION_REGION = "FUNCTION_REGION" | ||
_FUNCTION_ENTRY = "ENTRY_POINT" | ||
_FUNCTION_ENV_VARS = [_FUNCTION_TARGET, _FUNCTION_SIGNATURE, _CLOUD_RUN_SERVICE_ID] | ||
_LEGACY_FUNCTION_ENV_VARS = [_FUNCTION_NAME, _FUNCTION_REGION, _FUNCTION_ENTRY] | ||
"""Environment variables set in Cloud Functions environments.""" | ||
|
||
|
||
_REGION_ID = "instance/region" | ||
_ZONE_ID = "instance/zone" | ||
_GCE_INSTANCE_ID = "instance/id" | ||
"""Attribute in metadata server for compute region and instance.""" | ||
|
||
_GKE_CLUSTER_NAME = "instance/attributes/cluster-name" | ||
"""Attribute in metadata server when in GKE environment.""" | ||
|
||
|
||
def _create_functions_resource(project): | ||
region = retrieve_metadata_server(_REGION_ID) | ||
if _FUNCTION_NAME in os.environ: | ||
function_name = os.environ.get(_FUNCTION_NAME) | ||
elif _CLOUD_RUN_SERVICE_ID in os.environ: | ||
function_name = os.environ.get(_CLOUD_RUN_SERVICE_ID) | ||
else: | ||
function_name = "" | ||
resource = Resource( | ||
type="cloud_function", | ||
labels={ | ||
"project_id": project, | ||
"function_name": function_name, | ||
"region": region if region else "", | ||
}, | ||
) | ||
return resource | ||
|
||
|
||
def _create_kubernetes_resource(project): | ||
zone = retrieve_metadata_server(_ZONE_ID) | ||
cluster_name = retrieve_metadata_server(_GKE_CLUSTER_NAME) | ||
|
||
resource = Resource( | ||
type="k8s_container", | ||
labels={ | ||
"project_id": project, | ||
"location": zone if zone else "", | ||
"cluster_name": cluster_name if cluster_name else "", | ||
}, | ||
) | ||
return resource | ||
|
||
|
||
def _create_compute_resource(project): | ||
instance = retrieve_metadata_server(_GCE_INSTANCE_ID) | ||
zone = retrieve_metadata_server(_ZONE_ID) | ||
resource = Resource( | ||
type="gce_instance", | ||
labels={ | ||
"project_id": project, | ||
"instance_id": instance if instance else "", | ||
"zone": zone if zone else "", | ||
}, | ||
) | ||
return resource | ||
|
||
|
||
def _create_cloud_run_resource(project): | ||
region = retrieve_metadata_server(_REGION_ID) | ||
resource = Resource( | ||
type="cloud_run_revision", | ||
labels={ | ||
"project_id": project, | ||
"service_name": os.environ.get(_CLOUD_RUN_SERVICE_ID, ""), | ||
"revision_name": os.environ.get(_CLOUD_RUN_REVISION_ID, ""), | ||
"location": region if region else "", | ||
"configuration_name": os.environ.get(_CLOUD_RUN_CONFIGURATION_ID, ""), | ||
}, | ||
) | ||
return resource | ||
|
||
|
||
def _create_app_engine_resource(project): | ||
zone = retrieve_metadata_server(_ZONE_ID) | ||
resource = Resource( | ||
type="gae_app", | ||
labels={ | ||
"project_id": project, | ||
"module_id": os.environ.get(_GAE_SERVICE_ENV, ""), | ||
"version_id": os.environ.get(_GAE_VERSION_ENV, ""), | ||
"zone": zone if zone else "", | ||
}, | ||
) | ||
return resource | ||
|
||
|
||
def _create_global_resource(project): | ||
return Resource(type="global", labels={"project_id": project}) | ||
|
||
|
||
def detect_resource(project): | ||
gke_cluster_name = retrieve_metadata_server(_GKE_CLUSTER_NAME) | ||
gce_instance_name = retrieve_metadata_server(_GCE_INSTANCE_ID) | ||
|
||
if all([env in os.environ for env in _GAE_ENV_VARS]): | ||
# App Engine Flex or Standard | ||
return _create_app_engine_resource(project) | ||
elif gke_cluster_name is not None: | ||
# Kubernetes Engine | ||
return _create_kubernetes_resource(project) | ||
elif all([env in os.environ for env in _LEGACY_FUNCTION_ENV_VARS]) or all( | ||
[env in os.environ for env in _FUNCTION_ENV_VARS] | ||
): | ||
# Cloud Functions | ||
return _create_functions_resource(project) | ||
elif all([env in os.environ for env in _CLOUD_RUN_ENV_VARS]): | ||
# Cloud Run | ||
return _create_cloud_run_resource(project) | ||
elif gce_instance_name is not None: | ||
# Compute Engine | ||
return _create_compute_resource(project) | ||
else: | ||
# use generic global resource | ||
return _create_global_resource(project) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule environment
updated
11 files
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow this looks like quite the endeavor. Looks like it'll be very useful for users! 👏
If you haven't already, perhaps check in with some serverless folks to see if the environment variables you've chosen are what they recommend? I know some Functions variables changed between the 3.7 and 3.8 runtimes for instance:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah good point. I've been going off the docs, but I'll check with the team too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I talked with other DPEs, and this seems to be in-line with how others are doing environment detection