ci: Refactor and optimize the CI pipelines

[purelind]

Enhancement

The current pipelines has some issues, which have brought a significant burden to the maintenance and updates of the pipelines.

• testing logic and environment preparation logic are mixed in the Groovy file
• The pipeline is shared by multiple branches, which leads to the complexity and ambiguity of Groovy in order to be compatible with different branch version.

The plan is to restructure CI Pipelines through intentional redesign, making the structure of the pipelines simpler and its execution more stable.

• Refactor the script-based Jenkins Groovy to declarative Groovy. The code of the new pipeline groovy have been reduced from 800 lines to 200.
• The environment preparation script(like prepare cache and tools) is implemented using shell script instead of Groovy script.
• Using Prow to take over the triggering of pipelines.
• Each branch has an independent pipeline configuration to avoid adding conditional logic in the pipeline that is compatible with the differences of multiple branches.

Which CI piplines will be affected?

• https://ci.pingcap.net/view/ghpr_tikv/job/tikv_ghpr_test/

New CI Pipelines

• https://do.pingcap.net/jenkins/job/tikv/job/tikv/

Which branches' pipelines will be affected?

This optimization only targets the pipeline of the master branch. The remaining release branches will be switched and updated in the coming weeks.

Which behaviors will change

After this adjustment, the trigger command of the pipeline will change, and the status of the pipeline will also change.

The CI trigger command will change
/run-test --> /test pull-unit-test

CI status will change
idc-jenkins-ci/test --> pull-unit-test

Automatic triggering of the pielines
For newly created PRs or PRs with new commits pushed, the pipeline will be triggered automatically.

How to check my pull requst ci status on prow

https://prow.tidb.net/?repo=tikv%2Ftikv&type=presubmit

Any others changes I need to pay attention to?

• This refactoring optimization only targets the master branch, currently other branches are not affected.
• The PR batch merge will be supported in the future.

[purelind]

To unify the experience across all branches, the following active release branch pipelines will also be restructured and adjusted in the next few weeks

• release-8.1
• release-7.5
• release-7.1
• release-6.5
• release-6.1

[wuhuizuo]

/assign purelind

[wuhuizuo]

/type enhancement