AIGC Digital Human (Update)

This is a repository for organizing papres, codes and other resources related to AIGC Digital Human.

🔆 This project is still on-going, pull requests are welcomed!!

⭐ If you find this repo useful, please star it!!!

Contents

• AIGC Digital Human
  • Introduction
  • 2D Digital Human
    • Large Language Model
    • Text2Speech Conversion
    • Speech Clone
    • Face Driving
    • Cloth Modification
    • Style Transfer
    • Quality Assessment
  • 3D Digital Human
    • NeRF
    • Gaussian
    • 3D Quality Assessment
    • Databases

Introduction

In this documentation, we have collected papers, databases, and code resources related to AIGC Digital Human. Digital Humans refer to virtual entities generated and simulated by computers, possessing human characteristics and behaviors.

2D Digital Human

Large Language Model (LLM)

#	LLM	Paper	Code/Project
1	ChatGPT-3.5	-	ChatGPT
2	ChatGLM-6B	"GLM-130B: An Open Bilingual Pre-trained Mode"	github
3	Qwen (通义千问)	"QWEN TECHNICAL REPORT"	github

Text2Speech Conversion

#	Model	Paper	Code/Project
1	Espeaker	-	Web

Speech Clone

#	Model	Paper	Code/Project
1	MockingBird	-	github
2	Real-Time-Voice-Cloning	"Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis"	github

Face Driving

#	Model	Paper	Code/Project
1	MakeIttalk	"MakeItTalk: Speaker-Aware Talking-Head Animation"	github
2	Audio2Head	"Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion"	github
3	Sadtalker	"SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation"	github
4	Dreamtalk	"DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models"	github
5	Wav2Lip	"A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild"	github
6	Video-Retalking	"VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild"	github
7	DINet	"DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video"	github
8	IP-LAP	"Identity-Preserving Talking Face Generation with Landmark and Appearance Priors"	github

Cloth Modification

#	Model	Paper	Code/Project
1	VITON-HD	"Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis"	github

Style Transfer

#	Model	Paper	Code/Project
1	VToonify	"VToonify: Controllable High-Resolution Portrait Video Style Transfer"	github
2	DCT-Net	"DCT-Net: Domain-Calibrated Translation for Portrait Stylization"	gihub

Super Resolution

#	Model	Paper	Code/Project
1	BasicVSR++	"On the Generalization of BasicVSR++ to Video Deblurring and Denoising"	github

Quality Assessment

#	Model	Paper	Code/Project
1	VSFA	"Quality Assessment of In-the-Wild Videos"	github

3D Digital Human

NeRF

#	Model	Paper	Code/Project
1	HumanNeRF	"HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video"	github

Gaussian

#	Model	Paper	Code/Project
1	HumanGaussian	"HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting"	github

3D Quality Assessment

#	Model	Paper	Code/Project
1	-	"A No-Reference Quality Assessment Method for Digital Human Head"	-
2	-	"Geometry-Aware Video Quality Assessment for Dynamic Digital Human"	-
3	-	Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation	-
4	-	Perceptual Quality Assessment for Digital Human Heads	-

Databases

#	Database Name	Type	Title & Link	Database Link
1	NoW	Generation	"Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision"	Link
2	FaceScape	Generation	"FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction"	Link
3	Human3.6M	Generation	"Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments"	Link
4	ZJU-Mocap	Generation	"Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans"	Link
5	BEAT	Gesture Synthesis	"BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis"	Link
6	VOCA	Face Driving	"Capture, Learning, and Synthesis of 3D Speaking Styles"	Link
7	MultiFace	Face Driving	"Multiface: A Dataset for Neural Face Rendering"	Link
8	DHHQA	Quality Assessment	Perceptual Quality Assessment for Digital Human Heads	Link
9	DDH-QA	Quality Assessment	DDH-QA: A DYNAMIC DIGITAL HUMANS QUALITY ASSESSMENT DATABASE	Link
10	SJTU-H3D	Quality Assessment	Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation	Link
11	6G-DHQA	Quality Assessment	Quality-of-Experience Evaluation for Digital Twins in 6G Network Environments	Link
12	THQA	Quality Assessment	THQA: A PERCEPTUAL QUALITY ASSESSMENT DATABASE FOR TALKING HEADS	Link

Related Reference

Awesome-Talking-Face