Skip to content

[ECCV 2022] Masked Discrimination for Self-Supervised Learning on Point Clouds

License

Notifications You must be signed in to change notification settings

WisconsinAIVision/MaskPoint

Repository files navigation

MaskPoint

[ECCV 2022] Masked Discrimination for Self-Supervised Learning on Point Clouds

Haotian Liu, Mu Cai, Yong Jae Lee

Please check out our paper here.

Pretrained Models

Task Dataset Config Acc. Download
Pre-training ShapeNet pretrain_shapenet.yaml -- {w/o,w/} MoCo
Classification ScanObjectNN finetune_scanobject_hardest.yaml 84.6% here
Classification ScanObjectNN finetune_scanobject_objectbg.yaml 89.3% here
Classification ScanObjectNN finetune_scanobject_objectonly.yaml 89.7% here
Classification ModelNet40 finetune_modelnet.yaml 93.8% here

3D Object Detection

Task Dataset Config AP25 AP50 Download
Pre-training ScanNet-Medium pretrain_scannet_enc3x.yaml -- -- here
Pre-training ScanNet-Medium pretrain_scannet_enc12x.yaml -- -- here
Detection ScanNetV2 finetune_scannetv2_enc3x.sh 63.4 40.6 here
Detection ScanNetV2 finetune_scannetv2_enc12x.sh 64.2 42.1 here

Usage

Requirements

  • PyTorch >= 1.7.0
  • python >= 3.7
  • CUDA >= 9.0
  • GCC >= 4.9
  • torchvision
pip install -r requirements.txt
bash install.sh

Dataset

For ModelNet40, ScanObjectNN, and ShapeNetPart datasets, we use ShapeNet for the pre-training of MaskPoint models, and then finetune on these datasets respectively.

For ScanNetV2 object detection dataset, we use ScanNet-Medium for the pre-training. Please refer to the paper Sec. 4 [Pretraining Datasets] for details.

The details of used datasets can be found in DATASET.md.

MaskPoint pre-training

To pre-train the MaskPoint models on ShapeNet, simply run:

python main.py --config cfgs/pretrain_shapenet.yaml \
    --exp_name pretrain_shapenet \
    [--val_freq 10]

val_freq controls the frequence to evaluate the Transformer on ModelNet40 with LinearSVM.

Similarly, to pre-train the MaskPoint models on ScanNet-Medium, simply run:

# Pretrain 3x encoder model
python main.py --config cfgs/pretrain_scannet_enc3x.yaml \
    --exp_name pretrain_scannet_enc3x \
    [--val_freq 10]

# Pretrain 12x encoder model
python main.py --config cfgs/pretrain_scannet_enc12x.yaml \
    --exp_name pretrain_scannet_enc12x \
    [--val_freq 10]

Fine-tuning on downstream tasks

We finetune our MaskPoint on 5 downstream tasks: Classfication on ModelNet40, Few-shot learning on ModelNet40, Transfer learning on ScanObjectNN, Part segmentation on ShapeNetPart, and Object detection on ScanNetV2.

ModelNet40

To finetune a pre-trained MaskPoint model on ModelNet40, simply run:

python main.py
    --config cfgs/finetune_modelnet.yaml \
    --finetune_model \
    --ckpts <path> \
    --exp_name <name>

To evaluate a model finetuned on ModelNet40, simply run:

bash ./scripts/test.sh <GPU_IDS>\
    --config cfgs/finetune_modelnet.yaml \
    --ckpts <path> \
    --exp_name <name>

Few-shot Learning on ModelNet40

We follow the few-shot setting in the previous work.

First, generate your own few-shot learning split or use the same split as us (see DATASET.md).

# generate few-shot learning split
cd datasets/
python generate_few_shot_data.py
# train and evaluate the MaskPoint
python main.py \
    --config cfgs/fewshot_modelnet.yaml \
    --finetune_model \
    --ckpts <path> \
    --exp_name <name> \
    --way <int> \
    --shot <int> \
    --fold <int>

ScanObjectNN

To finetune a pre-trained MaskPoint model on ScanObjectNN, simply run:

python main.py \
    --config cfgs/finetune_scanobject_hardest.yaml \
    --finetune_model \
    --ckpts <path> \
    --exp_name <name>

To evaluate a model on ScanObjectNN, simply run:

bash ./scripts/test_scan.sh <GPU_IDS>\
    --config cfgs/finetune_scanobject_hardest.yaml \
    --ckpts <path> \
    --exp_name <name>

ScanNetV2

See MaskPoint -- 3DETR Finetuning for detailed instructions.

ShapeNetPart

Coming soon..

Citation

@article{liu2022masked,
  title={Masked Discrimination for Self-Supervised Learning on Point Clouds},
  author={Liu, Haotian and Cai, Mu and Lee, Yong Jae},
  journal={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2022}
}

About

[ECCV 2022] Masked Discrimination for Self-Supervised Learning on Point Clouds

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published