GCS service_account cannot authenticate

[Mai-Essam-Elmazaty]

Describe the bug
A clear and concise description of what the bug is.
I'm using Loki helm chart with GCS object storage, and it seems that it cannot authenticate to the bucket using the service account.

To Reproduce
Steps to reproduce the behavior:

1. Create GCS bucket and SA
2. Give the SA admin access on the bucket
3. use the following configurations:

  storage:
    bucketNames: 
      chunks: ${bucket_name}
      ruler: ${bucket_name}
      admin: ${bucket_name}   
    type: gcs
    gcs:
      bucket_name: ${bucket_name}
      service_account: ${service_account}
    
  storage_config:
    tsdb_shipper:
      active_index_directory: /var/loki/tsdb-index
      cache_location: /var/loki/tsdb-cache
    gcs:
        bucket_name: ${bucket_name}   
  schemaConfig:
    configs:
      - from: 2024-04-01
        object_store: gcs
        store: tsdb
        schema: v13
        index:
          prefix: index_
          period: 24h


  compactor:
    working_directory: /var/loki/retention
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150
    delete_request_store: gcs

4. Started Loki (chart: 6.5.1 , loki version: 3.0.0)
5. Started Promtail (chart: 6.15.5)

Expected behavior
Expect to use the service account to authenticate and work correctly.
Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

loki-0 logs
level=error caller=log.go:216 msg="error running loki" err="init compactor: failed to init delete store: googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>\nerror initialising module: compactor\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:453\nmain.main\n\t/src/loki/cmd/loki/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

[rgroothuijsen]

The service_account field is not included in the Helm chart, and since it can't find a valid service account it will instead use the cluster service account. What I could find is that by default the service account key JSON is mounted into the container as a secret. For the loki-write container for example:

write:
  extraEnv:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/etc/loki_secrets/gcp_service_account.json"
  extraVolumeMounts:
    - name: loki-secrets
      mountPath: "/etc/loki_secrets"
  extraVolumes:
    - name: loki-secrets
      secret:
        secretName: loki-secrets
        items:
          - key: gcp_service_account.json
            path: gcp_service_account.json

So if the JSON is included as a secret called loki-secrets with key gcp_service_account.json, it should use the correct service account.