Window Detection in Facades Using Heatmaps Fusion

Official implementation of our paper.

Chuan-Kang Li, Hong-Xin Zhang, Jia-Xin Liu, Yuan-Qing Zhang, Shan-Chen Zou, Yu-Tong Fang. Window Detection in Facades Using Heatmap Fusion[J].Journal of Computer Science and Technology, 2020, 35(4): 900-912.

Introduction

Window detection is a key component in many graphics and vision applications related to 3D city modeling and scene visualization. We present a novel approach for learning to recognize windows in a colored facade image. Rather than predicting bounding boxes or performing facade segmentation, our system locates keypoints of windows, and learns keypoint relationships to group them together into windows. A further module provides extra recognizable information at the window center. Locations and relationships of keypoints are encoded in different types of heatmaps, which are learned in an end-to-end network. We have also constructed a facade dataset with 3418 annotated images to facilitate research in this field. It has richly varying facade structure, occlusion, lighting conditions, and angle of view. On our dataset, our method achieves precision of 91.4% and recall of 91.0% under 50% IoU. We also make a quantitative comparison with state-of-the-art methods to verify the utility of our proposed method. Applications based on our window detector are also demonstrated, such as window blending.

Preparation

Environment

Please install PyTorch following the official webite. In addition, you have to install other necessary dependencies.

pip3 install -r requirements.txt

Dataset

The zju_facade_jcst2020 database is described in the paper, and now avaliable on BaiduYun(code: qlx5), GoogleDrive

Facade images were collected from the Internet and existing datasets including TSG-20, TSG-60, ZuBuD, CMP, ECP, and then data cleaning proceeded to ensure data quality standards. Using the open source software LabelMe, we manually annotated the positions of four corners of windows in order.

Model

You can use our trained models from BaiduYun(code: n0ev), GoogleDrive. ResNet18, MobileNetV2, ShuffleNetV2 are provided. All the configurations are written in *.yaml files and config_pytorch.py, and you can change it up to your own needs. NOTE: model filename is ended with .tar but isn't a compressed file. Just ignore the postfix and load ckpt directly.

The table concludes the performance of three models on our i7-6700K + 1080Ti platform. Note that center verification module is not used.

Architecture	#Params	FLOPs	Time	P_50	P_75	P_mean	R_50	R_75	R_mean
ShuffleNetV2 + Head	13.8M	29.5G	62ms	85.2%	62.8%	54.9%	86.2%	63.5%	55.5%
MobileNetV2 + Head	16.9M	31.2G	65ms	87.0%	64.9%	56.8%	90.0%	67.1%	58.5%
ResNet18 + Head	19.6M	32.0G	62ms	88.4%	68.4%	58.7%	91.2%	70.5%	60.5%

Usage

Train

python train.py --cfg /path/to/yaml/config \
    --data /path/to/data/root \
    --out /path/to/output/root

Test

python test.py --cfg /path/to/yaml/config --model /path/to/model \
    --data /path/to/data/root \
    --out /path/to/output/root

Inference

python infer.py --cfg /path/to/yaml/config \
                --model /path/to/model \
                --infer /path/to/image/directory

Examples

Applications

Facade Unification

We have developed a computational workflow for window texture blending based on our window detection method. Based on our technique, graphics designer can easily manipulate facade photos to create ideal building textures, while removing windows which are unsatisfactory due to their open or closed status, lighting conditions and occlusion, replacing them with the selected unified window texture.

Facade Beautification

Applying the above workflow, image beautification can be also performed to generate visually pleasant results with mixed features.

Facade Analytics

As our method can efficiently locate windows in urban facade images, it is of use for automatically analyzing semantic structure and extracting numerical information. With additional simple steps, it is easy to determine the windows in a single row or column. Furthermore, it can be adopted to predict building layers and symmetric feature lines.

Citation

If our code/dataset/models/paper helps your research, please cite with:

@article{Chuan-Kang Li:900, 
    author = {Chuan-Kang Li, Hong-Xin Zhang, Jia-Xin Liu, Yuan-Qing Zhang, Shan-Chen Zou, Yu-Tong Fang},
    title = {Window Detection in Facades Using Heatmap Fusion},
    publisher = {Journal of Computer Science and Technology},
    year = {2020},
    journal = {Journal of Computer Science and Technology},
    volume = {35},
    number = {4},
    eid = {900},
    numpages = {12},
    pages = {900},
    keywords = {facade parsing;window detection;keypoint localization},
    url = {http://jcst.ict.ac.cn/EN/abstract/article_2660.shtml},
    doi = {10.1007/s11390-020-0253-4}
}    

Acknowledgement

The major contributors of this repository include Chuankang Li, Yuanqing Zhang, Shanchen Zou, and Hongxin Zhang.