Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup zero-shot evaluation with EAI harness #4

Open
2 tasks
slippylolo opened this issue Nov 4, 2021 · 1 comment
Open
2 tasks

Setup zero-shot evaluation with EAI harness #4

slippylolo opened this issue Nov 4, 2021 · 1 comment
Assignees

Comments

@slippylolo
Copy link

Description

For zero-shot evaluation, we would ideally want to use the EAI evaluation harness. Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or (2) connect the EAI harness to T5X.

We are currently thinking of going with (1), as it seems like an easier choice.

Action items

@slippylolo slippylolo changed the title Zero-shot evaluation with EAI harness Setup zero-shot evaluation with EAI harness Nov 4, 2021
@thomasw21
Copy link
Member

@DanielHesslow will take this up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants