*Breaking* changes to the default logging resource type

[jceresini]

We have been using this client to write stackdriver logs for a 2-3 years. Those logs have been written as a global resource type.

We have Stackdriver logging sinks exporting those logs to a Pub/Sub topic, with sink filters for the resource.type (among other things). Another application is collecting those logs for analytics.

We redeployed the application with the latest client a few days ago, and the aforementioned Pub/Sub topic stopped receiving messages. I just tracked it down to a change in behavior in this client (we didn't notice it at first, and when we did tracking it down was difficult).

Logs from the current client are now logging stored with a resource.type of k8s_container.

It looks like the change is in this PR #200

Steps to reproduce

1. Run the code below using google-cloud-logging==2.2.0 in GKE
2. Upgrade google-cloud-logging to 2.3.0
3. Run the code in GKE again
4. Notice the resource.type changes from global to k8s_container

Code example

import google.cloud.logging

client = google.cloud.logging.Client(project='test-project')
logger = client.logger('test-log')
logger.log_struct(dict(foo='bar'))

The resource.type is indexed in cloud logging, and used in possibly every example log filter in the google docs. I have to believe others will be affected by this breaking change.

See, for example: https://cloud.google.com/logging/docs/view/advanced-queries

[daniel-sanche]

Thanks for reporting this, you're right, for compatibility reasons we should have kept the resource.type consistent until the next breaking 3.0.0 release. I'll put together a fix today to bring it back to the old behaviour for now.

That said, resource_type k8s_container is the expected type for code running on GKE. Is there a reason your pipeline expects a global type? I want to make sure I fully understand the issue

[jceresini]

Simple answer: its the default behavior.

I also assumed that the k8s_container type was just for stdout/stderr logs from our running containers. In this case we have an application (running in GKE) that uses the logger to send structured logs to another project (that doesn't contain a GKE cluster).

The examples referenced in the google cloud documentation don't cover using a different MonitoredResourceDescriptor. The contents of the snippets.py they reference (which is in this same repo) are:

def write_entry(logger_name):
    """Writes log entries to the given logger."""
    logging_client = logging.Client()

    # This log can be found in the Cloud Logging console under 'Custom Logs'.
    logger = logging_client.logger(logger_name)

    # Make a simple text log
    logger.log_text("Hello, world!")

    # Simple text log with severity.
    logger.log_text("Goodbye, world!", severity="ERROR")

    # Struct log. The struct can be any JSON-serializable dictionary.
    logger.log_struct(
        {
            "name": "King Arthur",
            "quest": "Find the Holy Grail",
            "favorite_color": "Blue",
        }
    )

    print("Wrote logs to {}.".format(logger.name))

[jceresini]

Another thing I just noticed, the metadata appears to be wrong. The target project we sent a log to here is cd-np-test1, the project_number of the project containing the GKE cluster is 982114195570. Those are not the same project. The resource records for the logs generated looks like this:

resource: {
    labels: {
        cluster_name: "nonprod"    
        container_name: ""    
        location: "projects/982114195570/zones/us-central1-a"    
        namespace_name: ""    
        pod_name: ""    
        project_id: "cd-np-test1"    
    }
    type: "k8s_container"   
 }

[daniel-sanche]

Ok, makes sense. I'll roll back the changes, but going forward after v3.0.0, you can expect the LogEntry's monitored_resource to default to reflect the GCP environment the code is run on. (with the option to override).

I opened a new bug for your second comment, and will take a look at that soon as well

[daniel-sanche]

This should be fixed in the new v2.3.1 release. Please re-open if you still see issues after updating