Use GCP Batch or Cloud Run Jobs for Runs?

[seandavi]

Google Cloud has a couple of batch processing environments:

• GCP Batch
• Cloud Run Jobs

Is anyone working on a Dagster integration for running jobs using either of these services, akin to the DockerRunLauncher?

[tim-carbonfarm]

I have set up a "GCP-native" Dagster deployment which:

• uses Cloud Run (services) for gRPC code location servers
• uses Cloud Run Jobs as run workers

So far it's working very well. If there is some interest I could look into open-sourcing it.

[baumann-t]

Thank you @garethbrickman for making me aware of this. Happy to work with you and @tim-carbonfarm / @timchap to see how we can move forward with the best Cloud Run Launcher!

[timchap]

Sorry, should really merge my GH accounts :D feel free to ping me on either.

The main implementation difference I see is that #21509 does not currently include support for multiple code locations. This can be added in one of several ways:

• as I have done (with a mapping from code location name -> job name)
• per my first implementation, by specifying job_name_prefix in the run launcher config. With this approach, we will look for a Cloud Run job with the name job_name_prefix + code_location_name. I moved away from this approach because I found it a bit opaque.
• or finally, by dynamically creating the Job resource for each run. I personally don't like this approach because I'm pretty deep in the declarative infrastructure game, and it seems a bit contrary to GCP's intended use of Cloud Run Jobs. But this approach would be more analogous to the EcsRunLauncher.

Perhaps there are other options, I'm open to suggestions. Otherwise, the main things to consolidate are:

• Unit tests (currently missing from Issue 15219 - Cloud Run Jobs Launcher #21509)
• Examples (currently missing from feat: add run launcher for GCP Cloud Run Jobs #21864)
• Integration tests if desired (open to your suggestions here @garethbrickman)
• Terraform code if desired (ditto)

[baumann-t]

@timchap
For the management of multiple code locations: I like your current implementation (with a mapping from code location name -> job name). As you said, it seems more declarative and less opaque than using a naming convention. Agree also on not dynamically creating a Job resource per run.

For consolidation, I would suggest:

• Let's keep your Launcher as it already includes the code location management and the unit tests
• We can add my Examples to make it easier for users (and reviewers) to try the launcher. We could either adapt my script to deploy multiple Run Job Launchers to test the feature, or add some Terraform code; but I don't think Terraform is really appropriate for Examples in Dagster (as you said, @garethbrickman can guide us on this).

Let me know what you think and we how you want to proceed with that!

[garethbrickman]

Thanks so much folks! @baumann-t would you please go ahead and close your PR and move over what's needed into #21864. I'll then get input from the Dagster engineering team about integration tests and next steps! 👍

[baumann-t]

Perfect! PR #21509 is closed. I received an invitation from @timchap to contribute to his PR. I've copied my existing Example and pushed it to his repo. I should have more time later this week to adapt my original example to the new launcher.

@timchap, I think we could add both of our approaches for the gRPC code location servers in the Cloud Run Launcher Example:

• One example using a simple VM (like mine)
• One example using Cloud Run Services (like what you mentioned)

This would showcase two different options for running Dagster on GCP. Let me know what you think!

[stickyhipp]

cc: @neilfulwiler