Make experiment evaluation a separate process

[prabhuteja12]

The current experimentation code in benchmarking runs evaluation in the same thread as the (subsequent) trainings. This is a problem when using DDP as the first evaluation (Line 295) creates several processes (as many GPUs) and each of them try to spawn training processes causing a problem with DDP ports clashing.

Describe the solution you'd like
The evaluation/testing should run in a separate process a la run_training_job and this issue wouldn't occur.

**Additional info:
See the discussion on Lightning forum: Lightning-AI/pytorch-lightning#2537