Skip to content

"A survey of Transformer" paper study πŸ‘©πŸ»β€πŸ’»πŸ§‘πŸ»β€πŸ’» KoreaUniv. DSBA Lab

Notifications You must be signed in to change notification settings

yukyunglee/Transformer_Survey_Study

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

46 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ€— Transformer_Survey_Paper_Study πŸ€—

main

"A survey of Transformer" paper study @DSBA Lab

πŸ“ƒ Paper : Lin, Tianyang, et al. "A Survey of Transformers." arXiv preprint arXiv:2106.04554 (2021) [Link]

1. μŠ€ν„°λ”” μ†Œκ°œ

1) Members (@DSBA Lab )

κ³ λ €λŒ€ν•™κ΅ μ‚°μ—…κ²½μ˜κ³΅ν•™κ³Ό Data Science & Business Analytics Lab [HomePage] [Youtube]

: Transformer와 NLPλ₯Ό μ‚¬λž‘ν•˜λŠ” 연ꡬ싀 ν•™μƒλ“€λ‘œ κ΅¬μ„±λœ μŠ€ν„°λ”” μž…λ‹ˆλ‹€ πŸ™‚

yukyung myeongsup hoonsang jina jaehyuk subin
Github Github Github Github Github Github

2. μŠ€ν„°λ”” λͺ©μ  및 λ°©ν–₯

0) μŠ€ν„°λ”” λ²”μœ„

: 이 μŠ€ν„°λ””λŠ” μ„œλ² μ΄ λ…Όλ¬Έμ—μ„œ λ‚˜μ˜€λŠ” λͺ¨λ“  λ…Όλ¬Έμ€‘μ—μ„œ λ…Έλž€μƒ‰μœΌλ‘œ ν‘œμ‹œλœ 논문을 λ‹€λ£Ήλ‹ˆλ‹€ (μ΄ˆλ‘μƒ‰: 연ꡬ싀 μ„Έλ―Έλ‚˜μ—μ„œ 닀룬 적 있음)

main

1) μŠ€ν„°λ”” λͺ©μ 

: 이 μŠ€ν„°λ””λŠ” 논문에 λ‚˜μ˜€λŠ” λͺ¨λ“  Transformer의 ꡬ쑰λ₯Ό μ΄ν•΄ν•˜κΈ° μœ„ν•œ 것이 μ•„λ‹ˆλ©°, 논문을 톡해 κΈ°μ‘΄ Transformer λŒ€λΉ„ Module level - Arch level, Pretrain levelμ—μ„œ μ–΄λ–€ λ³€ν™”κ°€ μžˆλŠ”μ§€ 흐름을 μ‚΄νŽ΄λ³΄λŠ” 것을 λͺ©μ μœΌλ‘œ ν•©λ‹ˆλ‹€.

2) μŠ€ν„°λ”” λ°©ν–₯

  • λͺ¨λ“  인원이 ν•΄λ‹Ή 논문을 1νšŒλ… ν•΄μ•Όν•©λ‹ˆλ‹€.
  • 각 μ±•ν„°λ³„λ‘œ 인원을 λ°°μ •ν•˜μ—¬ ν•΄λ‹Ή 파트λ₯Ό coreν•˜κ²Œ 곡뢀할 μ‚¬λžŒμ„ μ •ν•˜κ³ , λ°œν‘œμžλ£Œ μ œμž‘κ³Ό μŠ€ν„°λ”” λ°œν‘œλ₯Ό λ‹΄λ‹Ήν•©λ‹ˆλ‹€.
  • λ°œν‘œ 자료λ₯Ό μ œμž‘ν•˜λŠ” 이유: κ³΅λΆ€ν•œ λ‚΄μš©μ„ μ •λ¦¬ν•˜λŠ” 과정을 톡해 μ΄ν•΄ν•œ λ‚΄μš©μ„ ꡬ쑰화 ν•˜κΈ° μœ„ν•¨μž…λ‹ˆλ‹€.
  • 각 μ±•ν„°λ³„λ‘œ λ…Όλ¬Έ λ‚΄μš©μ„ μ­‰ ν›‘μ–΄μ€€ ν›„ λλ‚˜λŠ”κ²ƒμ΄ μ•„λ‹ˆλΌ, ν•΄λ‹Ή λΆ€λΆ„μ—μ„œ μ€‘μš”ν•  것 같은 '핡심'κ°œλ…μ„ μ°Ύμ•„ κ·Έ κ°œλ…μ„ μžμ„Ένžˆ μ„€λͺ…ν•΄μ£Όμ‹œλ©΄ λ©λ‹ˆλ‹€
  • μΆ”ν›„ μžμ‹ μ˜ μ±•ν„°λ‘œ λ°œν‘œμ˜μƒμ„ μ œμž‘ν•œ ν›„ 연ꡬ싀 μœ νŠœλΈŒμ— μ—…λ‘œλ“œ ν•΄μ•Όν•©λ‹ˆλ‹€

3) μŠ€ν„°λ”” κΈ°κ°„ : 2021/7/7 ~ 2021/7/30

  • 7/7 ~ 7/18 : λ…Όλ¬Έ 개인 곡뢀
  • 7/19 ~ 7/29 : μŠ€ν„°λ”” 진행
  • 7/30 : μŠ€ν„°λ”” μ˜μƒ μ—…λ‘œλ“œ (μ˜ˆμ •, DSBA Youtube channel)

3. μŠ€ν„°λ”” 운영

00) Introduction

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • λ°œν‘œμž : 이유경 박사과정
  • νŽΈμ§‘ : μœ€ν›ˆμƒ 석사과정, ν—ˆμž¬ν˜ 석사과정

01) Transformer basic (1~6 Page)

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : Vanilla Transformer
  • μŠ€ν„°λ”” 진행 : 7/20
  • λ°œν‘œμž : κΉ€μˆ˜λΉˆ 석사과정

02) Module-level : Attention 1 (6~11 Page)

03) Module-level : Attention 2 (11~15 Page)

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : Linearized / Prototype / Memory Compress Attention
  • μŠ€ν„°λ”” 진행 : 7/27
  • λ°œν‘œμž : κΉ€μ§€λ‚˜ 석박톡합과정

04) Module-level : Attention 3 (15~20 Page)

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : LowRank SeltAttention / Attention with Prior / Improved Multi-Head Mechanism
  • μŠ€ν„°λ”” 진행 : 7/28
  • λ°œν‘œμž : 이유경 석박톡합과정

05) Module-level : Others (20~26 Page)

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : Position Encoding / LayerNorm / FFN
  • μŠ€ν„°λ”” 진행 : 7/29
  • λ°œν‘œμž : ν—ˆμž¬ν˜ 석사과정

06) Arch.-level ~ end (26~33 Page)

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : Architecture-level variant
  • μŠ€ν„°λ”” 진행 : 7/30
  • λ°œν‘œμž : κΉ€λͺ…μ„­ 석사과정

07) Appendix : 당신이 λͺ¨λ₯΄λŠ” transformer의 3가지 사싀

  • μ˜μƒ / λ°œν‘œμžλ£Œ
  • 주제 : [Transformer] Complexity, Parameters, and Scaling
  • μŠ€ν„°λ”” 진행 : 7/28
  • λ°œν‘œμž : κΉ€λͺ…μ„­ 석사과정

4. μŠ€ν„°λ”” κ·œμΉ™

  1. μ–΄λ–€ μ§ˆλ¬Έμ΄λ“  ν•  수 μžˆμŠ΅λ‹ˆλ‹€
    • λͺ¨λ₯΄λŠ” 뢀뢄을 μ±„μš°κΈ° μœ„ν•΄ 기본적이고 μ‰¬μš΄ μ§ˆλ¬Έλ„ 주고받을 수 μžˆμŠ΅λ‹ˆλ‹€
    • μ–΄λ–€ μ‰¬μš΄ μ§ˆλ¬Έμ„ ν•˜λ”λΌλ„ μ§„μ§€ν•˜κ²Œ λ…Όμ˜ν•©λ‹ˆλ‹€
  2. μ§ˆλ¬Έμ„ λŒ€ν•˜λŠ” νƒœλ„
    • μ§ˆλ¬Έμ€ μ΅œλŒ€ν•œ 예의λ₯Ό κ°–μΆ”μ–΄ μ§ˆλ¬Έν•©λ‹ˆλ‹€
  3. λ°œν‘œ 자료 퀄리티
    • λ°œν‘œμžλ£ŒλŠ” μ„Έλ―Έλ‚˜μ²˜λŸΌ μ„±μ˜λ₯Ό λ‹΄μ•„ κ΅¬μ„±ν•©λ‹ˆλ‹€
    • λ‹€λ§Œ λ„ˆλ¬΄ 길게 λ§Œλ“€ ν•„μš”λŠ” μ—†μŠ΅λ‹ˆλ‹€
  4. μΆ”ν›„ 보강
    • μŠ€ν„°λ””μ—μ„œ λ…Όμ˜ν•œ λ‚΄μš©μ„ ν•΄λ‹Ή 챕터 λ‹΄λ‹Ήμžκ°€ μ •λ¦¬ν•˜μ—¬ κ³΅μœ ν•©λ‹ˆλ‹€
      • ν•¨κ»˜ λ‚˜λˆ„μ—ˆλ˜ 질문 및 λŒ€λ‹΅λ“€
    • μŠ€ν„°λ”” 쀑간에 λŒ€λ‹΅ν•˜μ§€ λͺ»ν–ˆλ˜ 정보듀은 μŠ€ν„°λ”” μ’…λ£Œ ν›„ μ •λ¦¬ν•˜μ—¬ κ³΅μœ ν•©λ‹ˆλ‹€
      • 이 λ˜ν•œ 아카이빙 ν•΄μ£Όμ„Έμš” !

About

"A survey of Transformer" paper study πŸ‘©πŸ»β€πŸ’»πŸ§‘πŸ»β€πŸ’» KoreaUniv. DSBA Lab

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published