IsoBN

Code for AAAI 2021 paper IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization.

Requirements

• PyTorch (tested on 1.7.0)
• Transformers (tested on 3.4.0)
• wandb
• tqdm
• datasets

Training and Evaluation

Train the BERT-base model on GLUE with the following command:

>> python train.py --task_name TASK

The training loss and evaluation results on the dev set are synced to the wandb dashboard.