[poc] [wip] create QE test mappings. #85

deads2k · 2023-11-01T15:07:17Z

No description provided.

openshift-ci · 2023-11-01T15:08:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

deads2k · 2023-11-01T15:08:19Z

cmd/ci-test-mapping/map.go

-			if err := tableManager.Migrate(); err != nil {
-				log.WithError(err).Fatal("could not migrate mapping table")
-			}
+			//tableManager = bigquery.NewMappingTableManager(context.Background(), bigqueryClient)


this table (renamed to be qe_component_mapping) needs to be created. The serviceaccount I have lacks permissions to do this.

deads2k · 2023-11-01T18:24:39Z

go build . && ./ci-test-mapping map --push-to-bigquery=true --mode=bigquery --google-service-account-credential-file='/home/deads/Downloads/google-sa.json' --mapping-file=qe_mapping.json

INFO[0000] fetching unique test/suite names from bigquery
INFO[0002] fetched unique test/suite names from bigquery in 2.096940488s count=8526
INFO[0002] writing results to file
INFO[0002] write complete in 15.662121ms
INFO[0002] mapping tests to ownership
INFO[0004] mapping tests to ownership complete in 2.365497963s matched=2686 unmatched=5840
INFO[0004] pushing to bigquery...
FATA[0004] could not push records to bigquery error="googleapi: Error 403: Access Denied: Table openshift-gce-devel:ci_analysis_us.qe_component_mapping: Permission bigquery.tables.updateData denied on table openshift-gce-devel:ci_analysis_us.qe_component_mapping (or it may not exist)., accessDenied"

I lack the powers

openshift-merge-robot · 2023-11-28T19:37:41Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-11-28T19:41:44Z

@deads2k: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/lint	`bd6bc4a`	link	true	`/test lint`
ci/prow/mapping	`bd6bc4a`	link	true	`/test mapping`
ci/prow/unit	`bd6bc4a`	link	true	`/test unit`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

liangxia · 2023-11-30T09:53:52Z

pkg/components/build/component.go

+			{Suite: "template related scenarios:"},
+			{Suite: "Testing the isolation during build scenarios"},
+			{Suite: "buildlogic.feature"},
+			{Suite: "admin build related features"},


Perhaps we can add BuildAPI and Workloads as one of the suites for build ?

shellyyang1989 · 2024-05-14T14:01:05Z

pkg/components/clusterversionoperator/component.go

+			{Suite: "Operators related features"},
+			{Suite: "Cluster_Operator"},
+			{Suite: "Display All Namespace Operands for Global Operators"},
+			{Suite: "OTA"},
+			{Suite: "Operators related features"},
+			{Suite: "Scenarios which will be used both for function checking and upgrade checking"},
+			{Suite: "basic verification for upgrade testing"},
+			{Suite: "cluster upgrade"},
+			{Suite: "fips enabled verification for upgrade"},
+			{Suite: "operand tests"},
+			{Suite: "Operators Installed nonlatest operator test"},
+			{Suite: "Operators related features on sts cluster mode"},


I think only OTA and cluster upgrade need to map to CVO.

Actually I think it will be easier to understand that mapping CVO and cluster upgrade to OTA. To achieve that, sounds we need update golang e2e test case keyword.

Currently we are using OTA cvo and OTA osus for the golang e2e test and cluster upgrade for the upgrade test. Maybe we can map OTA and cluster upgrade to OTA? cc @jiajliu for discussion

Maybe we can map OTA and cluster upgrade to OTA

That sounds good to me.

Mapping to OTA instead of current clusterversionoperator sounds reasonable. It looks like sub directory could be created on component dir, maybe we could use sub dirs under OTA component for a better mapping.

$ tree ota ota ├── cvo ├── oc ├── osus └── upgrade

then the ginkgo cases and upgrade could be mapping to each sub directory clearly.

anpingli · 2024-05-21T02:41:19Z

pkg/components/logging/component.go

+			{Suite: "Cases to test forward logs to external elasticsearch"},
+			{Suite: "cluster log forwarder features"},
+			{Suite: "fluentd related tests"},
+		},


I don't see feature below in Suite? shall use use these items>

verification-tests repo:

collector related tests eventrouter related test permission related test

cucushift repo:

log forwarding related tests logging permission related tests kibana web UI related cases for logging

zhaozhanqi · 2024-05-21T02:55:07Z

pkg/components/networking/multus/component.go

+			{Suite: "Multus-CNI ipv6 related scenarios"},
+			{Suite: "Multus-CNI related scenarios"},
+			{Suite: "SDN multus compoment upgrade testing"},
+			{Suite: "testing multicast scenarios"},


testing multicast scenarios is not belong to multus, please move to openshiftsdn and ovnkubernetes

Suggested change

{Suite: "testing multicast scenarios"},

Please updated to

{Suite: "CNI related features"}, {Suite: "Multus-CNI ipv6 related scenarios"}, {Suite: "Multus-CNI related scenarios"}, {Suite: "SDN multus compoment upgrade testing"}, {Suite: "Multus"},

jianlinliu · 2024-05-21T02:58:01Z

pkg/components/installer/openshiftinstaller/component.go

@@ -25,6 +25,13 @@ var OpenshiftInstallerComponent = Component{
 			{
 				IncludeAll: []string{"install should succeed"},
 			},
+			{
+				Suite:    "Install and configuration related scenarios",


installer component does not use this keyword, so I think it should be removed.

duanwei33 · 2024-05-21T03:08:29Z

pkg/components/storage/kubernetes/component.go

@@ -14,7 +14,12 @@ var KubernetesComponent = Component{
 		Name:                 "Storage / Kubernetes",
 		Operators:            []string{},
 		DefaultJiraComponent: "Storage / Kubernetes",
-		Matchers:             []config.ComponentMatcher{},
+		Matchers: []config.ComponentMatcher{


Hi, maybe I missed something, but I think we don't need a storage/kubernetes component, and all these cases can be added in storage

rioliu-rh · 2024-05-21T03:11:09Z

pkg/components/machineconfigoperator/component.go

@@ -39,6 +39,11 @@ var MachineConfigOperatorComponent = Component{
 				IncludeAll:   []string{"Pods cannot access the /config"},
 				Capabilities: []string{"Config"},
 			},
+			{Suite: "Improve MachineConfigPool list table for update status"},
+			{Suite: "MCO"},


in MCO QE dedicated jobs, only suite MCO is used in generated junit xml file e..g gcp-ipi-f9-longduration-mco-critical/openshift-extended-test-longduration/artifacts/junit/import-MCO.xml, I think other suites can be removed

zhaozhanqi · 2024-05-21T03:16:31Z

pkg/components/networking/ingressnodefirewall/component.go

@@ -14,7 +14,9 @@ var IngressNodeFirewallComponent = Component{
 		Name:                 "Networking / ingress-node-firewall",
 		Operators:            []string{},
 		DefaultJiraComponent: "Networking / ingress-node-firewall",
-		Matchers:             []config.ComponentMatcher{},
+		Matchers: []config.ComponentMatcher{
+			{Suite: "Egress-ingress related networking scenarios"},


Egress-ingress related networking scenarios belong to openshiftsdn and ovnkubernetes component

Here should be Suite: "INFW"

juzhao · 2024-05-21T03:40:45Z

pkg/components/monitoring/component.go

+			{Suite: "Symptom Detection"},
+			{Suite: "cluster monitoring related upgrade check"},
+			{Suite: "servicemonitor"},
+			{Suite: "serviceXXXitor"},


no

{Suite: "Symptom Detection"}, {Suite: "servicemonitor"}, {Suite: "serviceXXXitor"},

missed cucushift suites
Install and configuration related scenarios, see from https://github.com/openshift/cucushift/blob/master/monitoring/ci.feature#L1
alerts browser, see from https://github.com/openshift/cucushift/blob/master/web/monitoring/alerts-browser.feature#L1
query browser, see from https://github.com/openshift/cucushift/blob/master/web/monitoring/query-browser.feature#L1

also Monitoring dashboards related features for monitoring cypress UI cases
https://github.com/openshift/openshift-tests-private/blob/master/frontend/tests/monitoring/dashboards.cy.ts#L2

xiaojiey · 2024-05-21T03:41:20Z

pkg/components/apiserverauth/component.go

+			{Suite: "Group sync related scenarios"},
+			{Suite: "SCC policy related scenarios"},
+			{Suite: "Seccomp part of SCC policy should be kept and working after upgrade"},
+			{Suite: "Security_and_Compliance"},


Please drop Security_and_Compliance. I don't think apiserverauth contains this keyword.

juzhao · 2024-05-21T03:48:36Z

pkg/components/managementconsole/component.go

+			{Suite: "about cluster setting page"},
+			{Suite: "add idp from console"},
+			{Suite: "admin console api related"},
+			{Suite: "alerts browser"},


see from https://github.com/openshift-eng/ci-test-mapping/pull/85/files#r1607567823
alerts browser belongs to monitoring team

juzhao · 2024-05-21T03:49:09Z

pkg/components/managementconsole/component.go

+			{Suite: "operand form view"},
+			{Suite: "pod page"},
+			{Suite: "projects related features via web"},
+			{Suite: "query browser"},


see from https://github.com/openshift-eng/ci-test-mapping/pull/85/files#r1607567823
query browser belongs to monitoring team

zhaozhanqi · 2024-05-21T03:53:35Z

pkg/components/networking/clusternetworkoperator/component.go

+			{Suite: "IPsec upgrade scenarios"},
+			{Suite: "Network policy plugin scenarios"},
+			{Suite: "Operator related networking scenarios"},
+			{Suite: "Pod related networking scenarios"},
+			{Suite: "SCTP related scenarios"},
+			{Suite: "Service related networking scenarios"},
+			{Suite: "Service_Development_A"},
+			{Suite: "networking isolation related scenarios"},
+			{Suite: "service upgrade scenarios"},
+			{Suite: "service related scenarios"},
+			{Suite: "Egress IP related features"},


please updated to

{Suite: "Operator related networking scenarios"}, {Suite: "CNO"}

xiaojiey · 2024-05-21T03:53:56Z

The below components are missing: complianceoperator, occompliance, fileintegrityoperator and securityprofilesoperator.
Besides, there are several fips related test cases, fips-or-die, fips-check-node-scan, . Shall we create a separate component for FIPS? Thanks.

memodi

netobserv comments

memodi · 2024-05-21T15:21:36Z

pkg/components/networking/networktools/component.go

how "networktools" package name was chosen? these are are network observability test suite. I wonder how the package name will be utilized?

memodi · 2024-05-21T15:43:53Z

pkg/components/networking/networktools/component.go

+			{Suite: "Connection tracking netflow table page features"},
+			{Suite: "NETOBSERV Performances"},
+			{Suite: "NETOBSERV dashboards tests"},
+		},


How often will these be updated if we add/remove/rename suites?

jianping-shu · 2024-05-22T03:11:38Z

pkg/components/cloudcredentialoperator/component.go

@@ -21,6 +21,8 @@ var CloudCredentialOperatorComponent = Component{
 			{
 				IncludeAll: []string{"bz-Cloud Credential Operator"},
 			},
+			{Suite: "CloudCredentialOperator components upgrade tests"},
+			{Suite: "cloud credential operator"},


For below golang test case as example, it won't be mapping into the Suites?
g.It("NonHyperShiftHOST-ROSA-OSD_CCS-Author:jshu-LEVEL0-Critical-36498-CCO credentials secret change to STS-style", func()

zhaozhanqi · 2024-05-22T07:01:29Z

pkg/components/networking/ovnkubernetes/component.go

+			{Suite: "OVN related networking scenarios"},
+			{Suite: "OVNKubernetes IPsec related networking scenarios"},
+			{Suite: "OVNKubernetes Windows Container related networking scenarios"},
+			{Suite: "SDN/OVN metrics related networking scenarios"},
+			{Suite: "ipv6 dual stack cluster test scenarios"},
+			{Suite: "sdn2ovn migration testing"},


Suggested change

{Suite: "OVN related networking scenarios"},

{Suite: "OVNKubernetes IPsec related networking scenarios"},

{Suite: "OVNKubernetes Windows Container related networking scenarios"},

{Suite: "SDN/OVN metrics related networking scenarios"},

{Suite: "ipv6 dual stack cluster test scenarios"},

{Suite: "sdn2ovn migration testing"},

{Suite: "SDN"},

{Suite: "OVN related networking scenarios"},

{Suite: "OVNKubernetes IPsec related networking scenarios"},

{Suite: "OVNKubernetes Windows Container related networking scenarios"},

{Suite: "SDN/OVN metrics related networking scenarios"},

{Suite: "ipv6 dual stack cluster test scenarios"},

{Suite: "sdn2ovn migration testing"},

{Suite: "SDN compoment upgrade testing"},

{Suite: "SDN externalIP compoment upgrade testing"},

{Suite: "SDN multicast compoment upgrade testing"},

{Suite: "SDN multus compoment upgrade testing"},

zhaozhanqi · 2024-05-22T07:03:58Z

pkg/components/networking/openshiftsdn/component.go

+			{Suite: "SDN compoment upgrade testing"},
+			{Suite: "SDN externalIP compoment upgrade testing"},
+			{Suite: "SDN multicast compoment upgrade testing"},
+			{Suite: "SSDN multus compoment upgrade testingDN"},


Suggested change

{Suite: "SSDN multus compoment upgrade testingDN"},

{Suite: "SDN multus compoment upgrade testing"},

kasturinarra · 2024-05-30T17:37:03Z

pkg/components/node/kubelet/component.go

+			{Suite: "containers related features"},
+			{Suite: "kubelet restart and node restart"},
+			{Suite: "node logs related features"},
+			{Suite: "pod related features"},


This should be included in kubecontrollermanager/component.go, thanks !!

kasturinarra · 2024-05-30T18:07:55Z

Workloads:

In pkg/components/kubecontrollermanager/component.go file

remove line.no:32 as the tests are currently inactive and these does not belong to kcm.
remove line.no:33 as only two of the cases are from workloads in this file and rest others are node teams
        case OCP-11111,OCP-9778 only belongs to workloads, need to see how to run only these cases.
remove line.no:36 as when searched they give `ui related test case results"
remove line.no:38  as when searched they give `ui related test case results"
remove line.no:30 as when searched they give `ui related test case resulsts"
remove line.no:41 as when searched does not return anything in cucushift / openshift-tests-private
remove line.no:34 as these belongs to cli and not scheduler
Include case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-12975
Include {Suite: replicaSet related tests}
Include {Suite: rolling deployment related scenarios}
Include {Suite: scaling related scenarios}
Include {Suite: certificates related scenarios}
Include {Suite: pod related features} 
Include {Suite: Workloads test kcm works well}

In pkg/components/kubescheduler/component.go file

Add {Suite: "podTopologySpreadConstraints"} for scheduler/component.go file
Include {Suite: "Workloads test kubescheduler operator works well"}
Incude {Suite: Workloads test predicates and priority work well}
Include {Suite: Workloads taints and toleration works well}
Include {Suite: Workloads podTopologySpreadConstraints works well}

pkg/components/oc/component.go (A note: including oc-mirror to oc itself)

remove line.no 25, as the test belongs to storage
Include {Suite: "Event related scenarios"} in oc instead of kubeapiserver as the tests in them belong to oc
Include {Suite: "containers related features"} 
Include {Suite: "oc exports related scenarios"}
Include {Suite: "template related scenarios:"} in oc instead of component as this file has must-gather case
Include {Suite: admin build related features} in oc instead of component build.
Include {Suite: Workloads Test oc works well}
Include {Suite: Workloads test credentials work well}
Include {Suite: Workloads test syncing the groups work well}
Include {Suite: Workloads mustgather works well}
Include {Suite: Workloads oc adm command works well}
Include {Suite: Workloads ocmirror v1 works well}
Include {Suite: Workloads ocmirror v2 works well}

Have updated the g.Describe in gingko-tests to make sure every file has a unique name otherwise everything was falling under kubecontrollermanager due to the Suite: Workloads and i do not want it to show too may test failures or regression as all these tests are not actually testing kubecontrollermanager. Please let me know if i am missing something or have misunderstood things. Thanks !!

PR to address unique names for g.Describe https://github.com/openshift/openshift-tests-private/pull/16906

yapei · 2024-06-05T08:44:40Z

pkg/components/managementconsole/component.go

@@ -22,6 +22,49 @@ var ManagementConsoleComponent = Component{
 			{
 				IncludeAll: []string{"bz-Management Console"},


QE test cases all have string UserInterface
can we have a new IncludeAll matcher?

{ IncludeAll: []string{"UserInterface"}, },

then we can safely remove all {Suite} matching lines from 25 ~ 67

openshift-ci bot requested review from DennisPeriquet and neisw November 1, 2023 15:08

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 1, 2023

deads2k commented Nov 1, 2023

View reviewed changes

deads2k force-pushed the qe-tests branch 2 times, most recently from 759d883 to f048583 Compare November 15, 2023 02:22

deads2k added 3 commits November 28, 2023 13:49

switch to use qe_tables

17a8bb1

assign suites for QE tests

696df07

generated

bd6bc4a

deads2k force-pushed the qe-tests branch from f048583 to bd6bc4a Compare November 28, 2023 19:37

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2023

liangxia reviewed Nov 30, 2023

View reviewed changes

shellyyang1989 reviewed May 14, 2024

View reviewed changes

anpingli reviewed May 21, 2024

View reviewed changes

zhaozhanqi reviewed May 21, 2024

View reviewed changes

jianlinliu reviewed May 21, 2024

View reviewed changes

duanwei33 reviewed May 21, 2024

View reviewed changes

rioliu-rh reviewed May 21, 2024

View reviewed changes

zhaozhanqi reviewed May 21, 2024

View reviewed changes

juzhao reviewed May 21, 2024

View reviewed changes

xiaojiey reviewed May 21, 2024

View reviewed changes

juzhao reviewed May 21, 2024

View reviewed changes

zhaozhanqi reviewed May 21, 2024

View reviewed changes

memodi reviewed May 21, 2024

View reviewed changes

jianping-shu reviewed May 22, 2024

View reviewed changes

zhaozhanqi reviewed May 22, 2024

View reviewed changes

kasturinarra reviewed May 30, 2024

View reviewed changes

yapei reviewed Jun 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[poc] [wip] create QE test mappings. #85

[poc] [wip] create QE test mappings. #85

deads2k commented Nov 1, 2023

openshift-ci bot commented Nov 1, 2023

deads2k Nov 1, 2023

deads2k commented Nov 1, 2023

openshift-merge-robot commented Nov 28, 2023

openshift-ci bot commented Nov 28, 2023

liangxia Nov 30, 2023

shellyyang1989 May 14, 2024

jianlinliu May 21, 2024

shellyyang1989 May 21, 2024

jianlinliu May 21, 2024

jiajliu May 21, 2024

anpingli May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

jianlinliu May 21, 2024

duanwei33 May 21, 2024

rioliu-rh May 21, 2024

zhaozhanqi May 21, 2024 •

edited

juzhao May 21, 2024

xiaojiey May 21, 2024

juzhao May 21, 2024

juzhao May 21, 2024

zhaozhanqi May 21, 2024 •

edited

xiaojiey commented May 21, 2024 •

edited

memodi left a comment

memodi May 21, 2024

memodi May 21, 2024

jianping-shu May 22, 2024

zhaozhanqi May 22, 2024 •

edited

zhaozhanqi May 22, 2024

kasturinarra May 30, 2024

kasturinarra commented May 30, 2024

yapei Jun 5, 2024

yapei Jun 5, 2024

	{Suite: "SSDN multus compoment upgrade testingDN"},
	{Suite: "SDN multus compoment upgrade testing"},

		@@ -22,6 +22,49 @@ var ManagementConsoleComponent = Component{
		{
		IncludeAll: []string{"bz-Management Console"},

[poc] [wip] create QE test mappings. #85

Are you sure you want to change the base?

[poc] [wip] create QE test mappings. #85

Conversation

deads2k commented Nov 1, 2023

openshift-ci bot commented Nov 1, 2023

Choose a reason for hiding this comment

deads2k commented Nov 1, 2023

openshift-merge-robot commented Nov 28, 2023

openshift-ci bot commented Nov 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anpingli May 21, 2024 • edited

Choose a reason for hiding this comment

zhaozhanqi May 21, 2024 • edited

Choose a reason for hiding this comment

zhaozhanqi May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaozhanqi May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaozhanqi May 21, 2024 • edited

Choose a reason for hiding this comment

xiaojiey commented May 21, 2024 • edited

memodi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaozhanqi May 22, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasturinarra commented May 30, 2024

Workloads:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anpingli May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

zhaozhanqi May 21, 2024 •

edited

xiaojiey commented May 21, 2024 •

edited

zhaozhanqi May 22, 2024 •

edited