[SDK Stability] Add plugin-specific scaffolding test interface to decide if a plugin version should be updated or not

[itroyano]

Feature Request

Describe the problem you need a feature to resolve.

Currently a plugin can fail on basic scaffolding operator-sdk init --plugins=<plugin> returning 1 and/or creating no files, or fail partially creating an incomplete operator project which doesn't meet that specific plugin's requirements.

Describe the solution you'd like.

Add a mechanism in CI (e.g. on pull request) to test any current or future plugin for

1. Basic scaffolding and exit code 0.
2. Advanced / plugin-specific scaffolding to be implemented or delegated to be run by the plugin itself (?).

If a plugin passes both criteria - update its version in go.mod and continue to test other plugins and components.
otherwise reject the plugin version, keep the existing one, and continue to test other plugins and components.

Optionally - bot creates an issue in the repo of the plugin, so maintainers can take a look.

[varshaprasad96]

@itroyano Thanks for raising the issue. Some info on how this is done today:

When a new plugin is released (for ex. go/) and we would like to pull that into SDK, the developer:

1. Creates a new branch and tries updating the go.mod.
2. Updates the template for scaffolding plugin in cli.go -

   operator-sdk/internal/cmd/operator-sdk/cli/cli.go

   Line 84 in de40e50

   gov2Bundle, _ := plugin.NewBundleWithOptions(

3. After which, the testdata (https://github.com/operator-framework/operator-sdk/tree/de40e5091cc33bbd08592aa1da0007d9bfad6eb9/testdata) is updated using make generate. During the updation of testdata, the operator-sdk command then calls in the new plugin version and scaffolds out the files using the operator-sdk <commands> (

   operator-sdk/hack/generate/samples/generate_testdata.go

   Lines 35 to 60 in de40e50

   	flag.StringVar(&binaryPath, "bin", testutils.BinaryName, "Binary path that should be used")
   	flag.Parse()
   	// Make the binary path absolute if pathed, for reproducibility and debugging purposes.
   	if dir, _ := filepath.Split(binaryPath); dir != "" {
   	tmp, err := filepath.Abs(binaryPath)
   	if err != nil {
   	log.Fatalf("Failed to make binary path %q absolute: %v", binaryPath, err)
   	}
   	binaryPath = tmp
   	}
   	wd, err := os.Getwd()
   	if err != nil {
   	log.Fatal(err)
   	}
   	// samplesPath is the path where all samples should be generated
   	samplesPath := filepath.Join(wd, "testdata")
   	log.Infof("writing sample directories under %s", samplesPath)
   	log.Infof("creating Helm Memcached Sample")
   	helm.GenerateMemcachedSamples(binaryPath, samplesPath)
   	log.Infof("creating Go Memcached Sample with Webhooks")
   	golang.GenerateMemcachedSamples(binaryPath, samplesPath)

   ).

With this change if there is any issue during scaffolding the tests do not pass, and CI errors. There is no automation here though, but we do catch scaffolding issues before merging anything.

This works well currently with Helm and Go plugins. But with ansible (recently moved out external plugin), generation of testdata gets a bit complicated since we scaffold the templates in the external repository: https://github.com/operator-framework/ansible-operator-plugins/blob/4e8c78d21b6017ab0a280e3d274d97edf50b8fc4/hack/generate/samples/ansible/generate.go#L107. The scaffolded operator is tested in the plugin repository for any plugin update (https://github.com/operator-framework/ansible-operator-plugins/blob/main/test/e2e/ansible/suite_test.go) before updating it in SDK.

Having improvements in testing external scaffolds would be helpful, however to do so we need to discuss on how we would like to split tests between the external plugin and SDK repos.