Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI⚡️CD #52

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

CI⚡️CD #52

wants to merge 8 commits into from

Conversation

daanelson
Copy link
Collaborator

Working CI/CD + Error monitoring! Specifics:

  • CI builds and pushes a model to a staging model on Replicate, then runs integration tests.
  • The same tests will run both locally and on Replicate through CI without any user-side configuration - just run pytest tests/test_predict.py and you're good to go (assuming you have a machine that can run SDXL).
  • CD deploys model to prod when it's merged to main.
  • Error monitoring configured in Sentry.

@zeke zeke changed the title Ci cd CI ⚡️ CD Mar 6, 2024
@zeke zeke changed the title CI ⚡️ CD CI⚡️CD Mar 6, 2024
fi

# if cog.yaml changes - cog build and push. else - yolo build and push!
- name: did-it-tho
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did-it-tho

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hah the title of this step was borne out of ...debugging frustration.

but it shall stand.

Copy link
Member

@zeke zeke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So happy to see this! I took the liberty of jazzing up the PR title.

I didn't review the Python code changes in depth, but the Actions workflow looks legit. 👍🏼

cc @anotherjesse @mattt

sudo curl -o /usr/local/bin/yolo -L "https://github.com/replicate/yolo/releases/latest/download/yolo_$(uname -s)_$(uname -m)"
sudo chmod +x /usr/local/bin/yolo

# yolo as hack for pushing environment variables
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL 👀

@zeke
Copy link
Member

zeke commented May 13, 2024

@daanelson I'm excited to get this out in the world! What's blocking it at this point, if anything?

@daanelson
Copy link
Collaborator Author

@daanelson I'm excited to get this out in the world! What's blocking it at this point, if anything?

a great question! two things:

  1. doesn't seem that cog build --use-cuda-base-image=false is supported with any current version of cog within actions (see error [here | https://github.com/replicate/cog-sdxl/actions/runs/9181921032/job/25249740701], and for versions where it is supported (< 0.9.7, I believe) we get [this buildx error|https://github.com/replicate/cog-sdxl/actions/runs/9182322678/job/25250954614#step:6:1407],
  2. we need to revamp the training page for SDXL before we make it official.

Comment on lines +46 to +49
uses: replicate/setup-cog@v1.0.3
with:
token: ${{ secrets.REPLICATE_API_TOKEN }}
install-cuda: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uses: replicate/setup-cog@v1.0.3
with:
token: ${{ secrets.REPLICATE_API_TOKEN }}
install-cuda: false
uses: replicate/setup-cog@v2
with:
token: ${{ secrets.REPLICATE_API_TOKEN }}

Bumped to v2 here, where CUDA drivers are not installed by default.

Comment on lines +51 to +55
- name: cog-build
if: steps.what-changed.outputs.cog-push == 'true'
run: |
cog build

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: cog-build
if: steps.what-changed.outputs.cog-push == 'true'
run: |
cog build

Can you try taking this cog build step out entirely and letting cog push do it implicitly? I think there is some bug in Cog right now where sometimes cog build will fail on its own, but not if it's run as part of a push. Don't know what's that's about but I've seen it happen.

@zeke
Copy link
Member

zeke commented May 22, 2024

  1. doesn't seem that cog build --use-cuda-base-image=false is supported with any current version of cog

@mattt @technillogue any idea what that might be about?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants