JENKINS-73096 Create config for check interval and use the same default

[sfc-gh-mayang]

This PR allowed us to mitigate an active incident. The previous behavior is as follows:

• terminate node is called
• updateComputerList calls updateComputer on every node
• updateComputer calls SlaveComputer.check which runs describeInstances if time is after nextCheckAfter

This isn't an issue when we are terminating less than 600 nodes. But since describeInstances take around 100ms. When UpdateComputerList is called, to update every node, the total duration will exceed the current hard coded value of 1 minutes. So every computer termination takes 1 min to finish, during this time Queue lock is taken so no new work can be accepted.

TESTING PLAN
Installed on our controllers, launched 800 nodes via minInstances, then set minInstances to 0. Instead of the controller locking up until it reaches below 700 idle instances which takes more than an hour. It cleans all nodes up within 5 min

Submitter checklist

Edit tasklist title

Beta Give feedback Tasklist Submitter checklist, more options

• Rename
• Copy markdown
• Delete tasklist

Delete tasklist

Delete tasklist block?

Are you sure? All relationships in this tasklist will be removed.

Cancel Delete

1. Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!

   Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
2. Ensure that the pull request title represents the desired changelog entry

   Ensure that the pull request title represents the desired changelog entry
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
3. Please describe what you did

   Please describe what you did
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
4. Link to relevant issues in GitHub or Jira

   Link to relevant issues in GitHub or Jira
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
5. Link to relevant pull requests, esp. upstream and downstream changes

   Link to relevant pull requests, esp. upstream and downstream changes
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
6. Ensure you have provided tests - that demonstrates feature works or fixes the issue

   Ensure you have provided tests - that demonstrates feature works or fixes the issue
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove

Add item to Submitter checklist

[mwebber]

@sfc-gh-mayang

Is it possible to document the new setting jenkins.ec2.checkIntervalMinutes somewhere, such as https://plugins.jenkins.io/ec2/ ?

You said

TESTING PLAN
Installed on our controllers, launched 800 nodes via minInstances, then set minInstances to 0. Instead of the controller locking up until it reaches below 700 idle instances which takes more than an hour. It cleans all nodes up within 5 min

What value of jenkins.ec2.checkIntervalMinutes did you use when you did the test?

[sfc-gh-mayang]

We used a value of 2 to test