GPU memory increases although input image size decreases in instance segmentation

[eunwoosh]

Describe the bug
GPU memory increases although input image size decreases in instance segmentation.
When I trained both instance segmentation model with 700x700 input size using wgisd-5 dataset,
GPU memory usage is higher than model with default input size.
When I trained a model using coco dataset, it didn't happen.

Steps to Reproduce

1. Train any instance segmentation models with default input size using wgisd-5 dataset.
2. Check GPU memory usage during training.
   3.Train any instance segmentation models with 700x700 input size using wgisd-5 dataset.
3. Check GPU memory usage during training.
4. Compare two GPU memory usage

Environment:

• OS: Linux Ubuntu 18.04
• Framework version: torch 1.13.1
• Python version: 3.10
• OpenVINO version: 2022.3
• CUDA/cuDNN version: 11.7
• GPU model and memory: A100