TimeWarp - 在线视频背景替换

Replacing Real-Time the High-Resolution Meeting Background Freely.

Current video conferencing tools like Zoom can take an input feed (left) and replace the background, often introducing artifacts, as shown in the center result with close-ups of hair and glasses that still have the residual of the original background. Leveraging a frame of video without the subject (far left inset), our method produces real-time, high-resolution background matting without those common artifacts. The image on the right is our result with the corresponding close-ups, screenshot from our Zoom plugin implementation.

👉 See image results with interactive zoom-in

✨ Features

• Real-Time High-Resolution Background Matting,  ;
• Replacing our background in meeting screen;
• Replace background with any size photo you want;
• Cross mutil-platform, like zoom plugin, web-browser, miniprogram in PC, mobile and iPad, any platform with web service;
• IFLYTEK voice translation between Chinese and English;
• ...

Overview

• Updates
• Download
  • Model / Weights
  • Video / Image Examples
  • Datasets
• Build
• Demo
  • Beta
  • Scripts
  • Notebooks
• Usage / Documentation
• Training
• License
• Reference
• Community

Updates

• [Jan 09 2021] PhotoMatte85 dataset is now published.
• [Dec 21 2020] We updated our project to MIT License, which permits commercial use!

Download

Model / Weights

• Download model / weights

Video / Image Examples

• HD videos (by Sengupta et al.) (Our model is more robust on HD footage)
• 4K videos and images

Datasets

• VideoMatte240K (Coming soon)
• PhotoMatte85

🔨 Build

1. setup enviroment

# git clone https://github.com/Charmve/TimeWarp.git
# cd TimeWarp

# pip install -r requirements.txt

2. download dataset and model Download

3. run test code

• Video

# cd TimeWarp
# bash test_video.sh

• Image

# cd TimeWarp
# python3 test_image.py

Demo

🎧 Beta

demo Real-Time High-Resolution Background Matting^[1]

Have a Try !

➡️ Web Demo by Gradio: Matting your own images from your browser.

Scripts

We provide several scripts in this repo for you to experiment with our model. More detailed instructions are included in the files.

• inference_images.py: Perform matting on a directory of images.
• inference_video.py: Perform matting on a video.
• inference_webcam.py: An interactive matting demo using your webcam.

Notebooks

Additionally, you can try our notebooks in Google Colab for performing matting on images and videos.

• Image matting (Colab)
• Video matting (Colab)

Virtual Camera

We provide a demo application that pipes webcam video through our model and outputs to a virtual camera. The script only works on Linux system and can be used in Zoom meetings. For more information, checkout:

• Webcam plugin

Web Demo

Developers in the community has helped us build a web demo. See Community Projects section below.

Usage / Documentation

You can run our model using PyTorch, TorchScript, TensorFlow, and ONNX. For detail about using our model, please check out the Usage / Documentation page.

中文解读 ->

• 单块GPU实现4K分辨率每秒30帧，华盛顿大学实时视频抠图再升级，毛发细节到位
• 整个世界都是你的绿幕：这个视频抠图换背景的方法着实真假难辨

Training

Training code will be released upon acceptance of the paper.

☕ Processing

gantt
        dateFormat  YYYY-MM-DD
        title TimeWarp Development Processing
        section image matting
        architecture design          :done,    des1, 2020-11-06,2020-12-20
        video matting                :active,  des2, 2021-01-07, 3d
        chrome extension plus        :         des3, after des2, 7d
        video meeting architecture   :         des4, after des3, 5d

• More update processing details, please click HERE to see CHANGLOG.

🔔 Update Reminder

Subscribe -->

Business ✉️ email: yidazhang1@gmail.com

🍮 Community

• GitHub discussions 💬 or issues 💭

• QQ Group: 697948168 (password：TimeWarp)

• Email: yidazhang1#gmail.com

🔗 Reference

[1] Real-Time High-Resolution Background Matting. Shanchuan Lin*, Andrey Ryabtsev*, Soumyadip Sengupta. [GitHub] | [arXiv] | [data]

[2] Background Matting: The World is Your Green Screen. Soumyadip Sengupta, Vivek Jayaram, Brian Curless, et.al. CVPR 2020. [GitHub] | [arXiv]

[3] MODNet: Is a Green Screen Really Necessary for Real-Time Portrait Matting? [GitHub] | [arXiv]

[4] High-Resolution Deep Image Matting. Haichao Yu, Ning Xu, Zilong Huang. [GitHub] | [arXiv]

[5] End-to-end Animal Image Matting. Jizhizi Li, Jing Zhang, Stephen J. Maybank. [GitHub] | [arXiv].

[6] BGMv2-webcam-plugin-linux. https://github.com/andreyryabtsev/BGMv2-webcam-plugin-linux

[7] Chrome-extensions-samples. https://github.com/GoogleChrome/chrome-extensions-samples

[8] Zoom Video Ingester. https://github.com/harvard-dce/zoom-recording-ingester

[9] Zoom Recording Downloader. https://github.com/ricardorodrigues-ca/zoom-recording-downloader

[10] PaperswithCode. Image Matting. https://paperswithcode.com/task/image-matting

📜 License

This work is licensed under the Creative Commons Attribution NonCommercial ShareAlike 4.0 License.

💝 Acknowledge

Thanks for PeterL1n, et al.

Feel free to ask any questions, open a PR if you feel something can be done differently!

🌟Star this repository🌟

Created by Charmve & maiwei.ai Community | Deployed on Gradio.app