Missing Rubygems Weekly Data dumps past March 11, 2019

[krithika369]

Hello,

I cannot locate the Rubygems weekly data dumps past March 11. Has this been discontinued? https://rubygems.org/pages/data
I would like to retrieve the information on the newly added versions and do some analytics. While this can be accomplished using the APIs, the data dump was a lot more convenient. Please let me know if you require any other information.

Thanks,
Krithika

[dwradcliffe]

It has been temporarily discontinued, but not exactly on purpose. We moved to a new database in March and the old process to generate the data dumps no longer works. We haven’t had a chance to setup a new system yet. It is our intention to continue providing the data dumps though. I’ll keep this open until we fix it.

[simi]

@dwradcliffe feel free to ping me if there's anything I can help with!

[dwradcliffe]

@simi Here's what I'm thinking:
We can make a simple docker image that has ruby+gems necessary for the backup process. Then we'll just run it as a cron job on the Kubernetes cluster.
Here is the previous setup: https://github.com/rubygems/rubygems-infrastructure/tree/master/cookbooks/rubygems-backups That should have all the necessary information for gems and the backup config files. We'll pass in all the secrets via ENV vars.
Let me know if you want to start work on this. Thanks!

[zackfern]

Hey there, @dwradcliffe 👋 I'd like to attempt to get this working. I had originally planned on trying to get the backup functioning in the existing rubygems.org Dockerfile, however after trying it locally it looks like Ruby version mismatches won't make that possible. 😞

Since it's looking like we will need a new Dockerfile with an older version of Ruby to get this working, I was wondering if you had a specific location you'd like it? Should it live in the rubygems.org repo or the rubygems-infrastructure repo?

[dwradcliffe]

@zackfern Thanks!! probably a new repo, I can set that up in a couple days when I’m back to my computer. 🌴

[dwradcliffe]

@zackfern Feel free to submit a PR here: https://github.com/rubygems/rubygems.org-db-backups

[zackfern]

@dwradcliffe Thanks for getting this working again! ❤️ I've noticed that data dumps have been reliably uploaded for the last week, can this issue be closed out?

[dwradcliffe]

Yep, thanks for your help!