Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate embeddings #11

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

Separate embeddings #11

wants to merge 45 commits into from

Conversation

lucidrains
Copy link

PR-ing to make sure tests work

sid and others added 30 commits January 13, 2021 23:18
add basic sampling code
99d11e1
Merge remote-tracking branch 'origin/main' into sample
8bb3322
add prediction input / output fns
b9fd039
@ConnorJL
get sample_autoregressive working
cf76c6c
@ConnorJL
truncate text tokens properly
c7ff6c4
@ConnorJL
log model params to tensorboard
4d51fd9
@kingoflolz
add vae decoding and write to jpeg
346871f
@kingoflolz
unshift image outputs at decode time
d13c330
@kingoflolz
dirty hack to use vae decoder params when training dalle
f8a7449
@leogao2
Move initialize_vae_weights to after lowering
ff56d12
@ConnorJL
fix vae checkpoint load in training
4c4e0e0
@ConnorJL
fix parameter count logging
2c14bde
@ConnorJL
fix image vocab size
130c26e
@ConnorJL
revert to separate embeddings for image and text
4652ef2
@leogao2
Fix masking
67247cf
@leogao2
Ignore text tokens in loss computation
a0a2828
@leogao2
Fix slicing for mtf
ca23b85
@leogao2
Fix sampling
e126b79 
Because of the mismatch between overall length and image length, it would
end up broadcasting into a weird shape. This fixes that by slicing the text
off first.
@leogao2
Implement incremental logits mask
ddeb74d
@leogao2
Fix typo
e7eb459
@ConnorJL
revert changes to sample.py
29f3006
@ConnorJL
add mask to bias op
b69ff72
@ConnorJL
update mask. (still not working :( )
7e3b6ff
@ConnorJL
mask changes
34d5326
@ConnorJL
fix label shifting
05dec26
@ConnorJL
add weight decay Adam
6c39bdf
@ConnorJL
add eval steps
69d7ef9
add slow sampling
2ec46be
add slow sampling
321b718
@lucidrains
add tests
d564853
lucidrains and others added 15 commits April 4, 2021 14:41
@lucidrains
switch to using <bos> instead of <eos>
a0d01b9
@lucidrains
separate logits for text and images
58c4d4e
@lucidrains
remove offsetted image, since it is no longer needed
9410ba9
@lucidrains
add unique pad tokens feature, hidden behind a feature flag
f13a567
@ConnorJL
Merge branch 'separate_embeddings' of
dafcb99 
https://github.com/EleutherAI/DALLE-mtf into separate_embeddings
@lucidrains
remove logits mask code
0f774e1
@lucidrains
fix syntax
2972665
@lucidrains
variable scope
2c081f2
@lucidrains
remove stop_at_tokens, since it wont be used, to avoid confusion
6a37d0f
@lucidrains
more sample cleanup
c60bf54
@lucidrains
make sure one can sample the non-cached way
02a51f9
@lucidrains
fix sampling for cached version (potentially)
beb1c60
@lucidrains
change max steps to image seq len, since it is counting from the star…
c085faf 
…t of the image
@lucidrains
fix initial position at 0
3ffc5d1
@lucidrains
make sure axial positional embedding is shifted over by one due to <bos>
ef7864c
@CLAassistant
Copy link

CLAassistant commented Apr 23, 2023 •
edited

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 5 committers have signed the CLA.

❌ sid
❌ ConnorJL
❌ kingoflolz
❌ leogao2
❌ lucidrains

sid seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@lucidrains @CLAassistant @ConnorJL @kingoflolz @leogao2