Fails pulling topic in dataproc 2.2 with PERMISSION_DENIED error

[Marcus-Rosti]

Unable to read for pubsublite topic in dataproc 2.2

Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
   General, Core, and Other are also allowed as types
2. OS type and version:
3. Dataproc version: 2.2.0-RC2-debian11
4. version(s):
   Spark: 3.4
   Scala: 2.12.18
   Iceberg: 1.4
   pubsublite-spark-sql-streaming: 1.0.0
   biglake: 0.1.0

pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar
iceberg-spark-runtime-3.4_2.12-1.4.0.jar
biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar

Code example

  implicit val pubSubSource: StreamingSource[PubSubSource] = new StreamingSource[PubSubSource] {
    override def streamingSource(conf: PubSubSource)(implicit spark: SparkSession): DataFrame =
      spark
        .readStream
        .format("pubsublite")
        .option(
          "pubsublite.subscription",
          conf.subscriptionName
        )
        .load()

Stack trace

Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.PermissionDeniedException: com.google.cloud.pubsublite.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
        at repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
        at repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
        at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.getTopicPath(PslReadDataSourceOptions.java:207) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
        ... 54 more
Caused by: com.google.api.gax.rpc.PermissionDeniedException: com.google.cloud.pubsublite.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:98) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]

Thanks!