New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] There is a problem with the setting of optim_wrapper and param_scheduler, which always leads to the best results in the top 100 epochs. #1874
Comments
The log is displayed as follows: |
分支
main 分支 (mmpretrain 版本)
描述该错误
When I use Swin-large _ 16xb64 _ in1K-384px.py, I use auto_scale_lr, and the default training value is epochs=300, but the best result is always in the top 100 epochs, and the accuracy fluctuates seriously. How to set optim_wrapper and param_scheduler reasonably?
model = dict(
backbone=dict(
arch='large',
img_size=384,
stage_cfgs=dict(block_cfgs=dict(window_size=12)),
type='SwinTransformer'),
optim_wrapper = dict(
clip_grad=dict(max_norm=5.0),
optimizer=dict(
betas=(
0.9,
0.999,
),
eps=1e-08,
lr=0.001,
type='AdamW',
weight_decay=0.05),
paramwise_cfg=dict(
bias_decay_mult=0.0,
custom_keys=dict({
'.absolute_pos_embed': dict(decay_mult=0.0),
'.relative_position_bias_table': dict(decay_mult=0.0)
}),
flat_decay_mult=0.0,
norm_decay_mult=0.0))
param_scheduler = [
dict(
by_epoch=True,
convert_to_iter_based=True,
end=20,
start_factor=0.001,
type='LinearLR'),
dict(begin=20, by_epoch=True, eta_min=1e-05, type='CosineAnnealingLR'),
]
test_dataloader = dict(
batch_size=20,
train_dataloader = dict(
batch_size=16,
环境信息
Python: 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 1050081049
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda-11.6
NVCC: Cuda compilation tools, release 11.6, V11.6.124
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.0.1
PyTorch compiling details: PyTorch built with:
GCC 9.3
C++ Version: 201703
Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.7
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.5
Magma 2.6.1
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.15.2
OpenCV: 4.7.0
MMEngine: 0.10.3
其他信息
Since I use a single gpu and the batch-size setting is small, I use auto-scale-lr. But acc vibrates greatly, and I think I will miss the best result.
The text was updated successfully, but these errors were encountered: