Support for GCP's Big Query?

[mii9000]

Currently there is a coupling to *sql.DB from the interface which is a bit difficult to get around when using the bq client library. Any ideas on how to get around it? I am more than happy to contribute a PR for this.

[dossy]

Can you provide more details regarding the issue you're encountering, or steps to reproduce what the issue is?

Could be interesting to use https://github.com/goccy/bigquery-emulator for running a local BigQuery server and https://pkg.go.dev/github.com/tozh/go-sql-bigquery as the client driver for dbmate.

[mii9000]

That is really great. I can try adding a driver based on this. @dossy any reason you recommended this fork and not the original library?

[dossy]

No reason, and I'm not recommending any particular fork as I haven't used any of them, I was simply referencing it as an example of what is currently available, based on what came up first in my quick Google search.

Both https://github.com/solcates/go-sql-bigquery and the fork at https://github.com/tozh/go-sql-bigquery seem like abandonware at this point as the last commit in both repos is nearly 4 years old, dating back to June/July 2020.

Not sure if this is a better option, but it appears to be more actively maintained: https://github.com/go-gorm/bigquery

Either way, my point is just that there are database/sql drivers for Google BigQuery, so adding support for it as a destination shouldn't be very difficult, and if the server emulator has sufficient parity with the real thing, automating tests for it in dbmate should even be reasonable to implement, too.

[mii9000]

Added a PR for this #523

[amacneil]

In theory it would be nice to refactor dbmate to remove the hard dependency on drivers requiring database/sql. The existing drivers could use that, but it would be nice to not be a hard requirement.

In practice I think that would be a pretty major refactor, and require a major version bump. Our quality bar for code and tests is high, so it might take a while to merge. If someone is willing to taking on that work it could happen, but I don't think any of the current maintainers have time to pick it up.

[dossy]

@mii9000 submitted PR #523.

[dossy]

Who is interested in helping to test the new BigQuery support? PR #523 is dev complete, and could use some real-world testing from actual users. If anyone is able to help test, either please reply here or on the PR with your experience and feedback. Thanks!

[mii9000]

@dossy I can share a service account JSON for you to test and I can have other people test it also. Which one do you think would suffice?

[dossy]

I have my own service account JSON that I'm testing with, but I have no actual projects on BigQuery that would use dbmate. I'd like someone who intends to use this for real projects to give it a try and give feedback on whether it works for them or not.

[dossy]

PR #523 has been merged, and BigQuery support should land in dbmate v2.14.0, shortly.

Update: v2.14.0 was released 2024-03-28.