Skip to content

Commit

Permalink
Cycpress integration testing
Browse files Browse the repository at this point in the history
  • Loading branch information
@dtaylor113
dtaylor113 committed Jul 29, 2020
1 parent 85c26c2 commit 5150fde
Show file tree
Hide file tree
Showing 51 changed files with 2,432 additions and 500 deletions.
2 changes: 1 addition & 1 deletion Dockerfile
View file
@@ -1,4 +1,4 @@
FROM quay.io/coreos/tectonic-console-builder:v19 AS build
FROM quay.io/coreos/tectonic-console-builder:v20 AS build

RUN mkdir -p /go/src/github.com/openshift/console/
ADD . /go/src/github.com/openshift/console/
Expand Down
4 changes: 3 additions & 1 deletion Dockerfile.builder
View file
Expand Up @@ -27,7 +27,9 @@ RUN chmod 777 -R ${HOME}

RUN apt-get update \
&& apt-get install --no-install-recommends -y -q \
curl wget git unzip bzip2 jq
curl wget git unzip bzip2 jq \
libgtk2.0-0 libgtk-3-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
# ^^ additional Cypress dependencies: https://docs.cypress.io/guides/guides/continuous-integration.html#Dependencies

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.0/bin/linux/amd64/kubectl && \
chmod +x ./kubectl && \
Expand Down
45 changes: 42 additions & 3 deletions README.md
View file
Expand Up @@ -185,6 +185,22 @@ Run frontend tests:

### Integration Tests

#### Cypress

Cypress integration tests are run via [Cypress.io](https://www.cypress.io/).

Launch Cypress test runner:
```
cd frontend
oc login ...
yarn run test-cypress
```

This will launch the Cypress test runner where you can run one or all cypress tests.

[**_More information on Console's Cypress usage_**](frontend/packages/integration-tests-cypress/README.md)
#### Protractor

Integration tests are run in a headless browser driven by [protractor](http://www.protractortest.org/#/).
Requirements include Chrome or Firefox, a working cluster, kubectl, and bridge itself (see building above).
By default, it will look for Chrome in the system and use it, but if you want to use Firefox instead, set `BRIDGE_E2E_BROWSER_NAME` environment variable in your shell with the value `firefox`.
Expand Down Expand Up @@ -220,7 +236,6 @@ For macOS, you can use:
```
yarn run webdriver-update-macos
```

#### How the Integration Tests Run in CI

The end-to-end tests run against pull requests using [ci-operator](https://github.com/openshift/ci-operator/).
Expand All @@ -245,7 +260,7 @@ If you don't want to run the entire e2e tests, you can use a different suite fro
$ ./test-gui.sh <suite>
```

#### Hacking Integration Tests
##### Hacking Protractor Tests

To see what the tests are actually doing, it is posible to run in none `headless` mode by setting the `NO_HEADLESS` environment variable:

Expand All @@ -265,7 +280,7 @@ To avoid skipping remaining portion of tests upon encountering the first failure
$ NO_FAILFAST=true ./test-gui.sh <suite>
```

##### Debugging Integration Tests
##### Debugging Protractor Tests

1. `cd frontend; yarn run build`
2. Add `debugger;` statements to any e2e test
Expand All @@ -275,6 +290,30 @@ $ NO_FAILFAST=true ./test-gui.sh <suite>
6. Will break on any `debugger;` statements
7. Pauses browser when not using `--headless` argument!

#### How the Integration Tests Run in CI

The end-to-end tests run against pull requests using [ci-operator](https://github.com/openshift/ci-operator/).
The tests are defined in [this manifest](https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/console/openshift-console-master-presubmits.yaml)
in the [openshift/release](https://github.com/openshift/release) repo and were generated with [ci-operator-prowgen](https://github.com/openshift/ci-operator-prowgen).

CI runs the [test-prow-e2e.sh](test-prow-e2e.sh) script, which runs the cypress tests and the protractor `e2e` test suite defined in [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts).

You can simulate an e2e run against an existing 4.0 cluster with the following commands (replace `/path/to/install-dir` with your OpenShift 4.0 install directory):

```
$ oc apply -f ./frontend/integration-tests/data/htpasswd-secret.yaml
$ oc patch oauths cluster --patch "$(cat ./frontend/integration-tests/data/patch-htpasswd.yaml)" --type=merge
$ export BRIDGE_BASE_ADDRESS="$(oc get consoles.config.openshift.io cluster -o jsonpath='{.status.consoleURL}')"
$ export BRIDGE_KUBEADMIN_PASSWORD=$(cat "/path/to/install-dir/auth/kubeadmin-password")
$ ./test-gui.sh e2e
```

If you don't want to run the entire e2e tests, you can use a different suite from [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts). For instance,

```
$ ./test-gui.sh <suite>
```

### Deploying a Custom Image to an OpenShift Cluster

Once you have made changes locally, these instructions will allow you to push
Expand Down
2 changes: 1 addition & 1 deletion builder-run.sh
View file
Expand Up @@ -11,7 +11,7 @@ set -e
# Without env vars:
# ./builder-run.sh ./my-script --my-script-arg1 --my-script-arg2

BUILDER_IMAGE="quay.io/coreos/tectonic-console-builder:v19"
BUILDER_IMAGE="quay.io/coreos/tectonic-console-builder:v20"

# forward whitelisted env variables to docker
ENV_STR=()
Expand Down
182 changes: 6 additions & 176 deletions frontend/integration-tests/tests/crud.scenario.ts
View file
Expand Up @@ -4,57 +4,17 @@ import { browser, $, $$, by, ExpectedConditions as until, Key, element } from 'p
import { safeLoad, safeDump } from 'js-yaml';
import * as _ from 'lodash';
import { execSync } from 'child_process';
import { OrderedMap } from 'immutable';

import { appHost, testName, checkLogs, checkErrors } from '../protractor.conf';
import * as crudView from '../views/crud.view';
import * as yamlView from '../views/yaml.view';
import * as namespaceView from '../views/namespace.view';
import * as createRoleBindingView from '../views/create-role-binding.view';

const K8S_CREATION_TIMEOUT = 15000;

describe('Kubernetes resource CRUD operations', () => {
const testLabel = 'automatedTestName';
const leakedResources = new Set<string>();
const k8sObjs = OrderedMap<string, { kind: string; namespaced?: boolean }>()
.set('pods', { kind: 'Pod' })
.set('services', { kind: 'Service' })
.set('serviceaccounts', { kind: 'ServiceAccount' })
.set('secrets', { kind: 'Secret' })
.set('configmaps', { kind: 'ConfigMap' })
.set('persistentvolumes', { kind: 'PersistentVolume', namespaced: false })
.set('storageclasses', { kind: 'StorageClass', namespaced: false })
.set('ingresses', { kind: 'Ingress' })
.set('cronjobs', { kind: 'CronJob' })
.set('jobs', { kind: 'Job' })
.set('daemonsets', { kind: 'DaemonSet' })
.set('deployments', { kind: 'Deployment' })
.set('replicasets', { kind: 'ReplicaSet' })
.set('replicationcontrollers', { kind: 'ReplicationController' })
.set('persistentvolumeclaims', { kind: 'PersistentVolumeClaim' })
.set('statefulsets', { kind: 'StatefulSet' })
.set('resourcequotas', { kind: 'ResourceQuota' })
.set('limitranges', { kind: 'LimitRange' })
.set('horizontalpodautoscalers', { kind: 'HorizontalPodAutoscaler' })
.set('networkpolicies', { kind: 'NetworkPolicy' })
.set('roles', { kind: 'Role' });
const openshiftObjs = OrderedMap<string, { kind: string; namespaced?: boolean }>()
.set('deploymentconfigs', { kind: 'DeploymentConfig' })
.set('buildconfigs', { kind: 'BuildConfig' })
.set('imagestreams', { kind: 'ImageStream' })
.set('routes', { kind: 'Route' })
.set('user.openshift.io~v1~Group', { kind: 'user.openshift.io~v1~Group', namespaced: false });
const serviceCatalogObjs = OrderedMap<string, { kind: string; namespaced?: boolean }>().set(
'clusterservicebrokers',
{
kind: 'servicecatalog.k8s.io~v1beta1~ClusterServiceBroker',
namespaced: false,
},
);
let testObjs = browser.params.openshift === 'true' ? k8sObjs.merge(openshiftObjs) : k8sObjs;
testObjs =
browser.params.servicecatalog === 'true' ? testObjs.merge(serviceCatalogObjs) : testObjs;

afterEach(() => {
checkLogs();
Expand All @@ -63,9 +23,12 @@ describe('Kubernetes resource CRUD operations', () => {

afterAll(() => {
const leakedArray: Array<string> = [...leakedResources];
console.error(
`Leaked ${leakedArray.length} resources out of ${testObjs.size}:\n${leakedArray.join('\n')}`,
);
if (!_.isEmpty(leakedArray)) {
console.error(`Leaked ${leakedArray.length} resources\n${leakedArray.join('\n')}.`);
} else {
console.log('No resources leaked.');
}

leakedArray
.map((r) => JSON.parse(r) as { name: string; plural: string; namespace?: string })
.filter((r) => r.namespace === undefined)
Expand All @@ -78,104 +41,6 @@ describe('Kubernetes resource CRUD operations', () => {
});
});

testObjs.forEach(({ kind, namespaced = true }, resource) => {
describe(kind, () => {
const name = `${testName}-${_.kebabCase(kind)}`;
it('displays a list view for the resource', async () => {
await browser.get(
`${appHost}${
namespaced ? `/k8s/ns/${testName}` : '/k8s/cluster'
}/${resource}?name=${testName}`,
);
await crudView.isLoaded();
});

if (namespaced) {
it('has a working namespace dropdown on namespaced objects', async () => {
await browser.wait(until.presenceOf(namespaceView.namespaceSelector));
expect(namespaceView.namespaceSelector.getText()).toContain(testName);
});
} else {
it('does not have a namespace dropdown on non-namespaced objects', async () => {
expect(namespaceView.namespaceSelector.isPresent()).toBe(false);
});
}

it('displays a YAML editor for creating a new resource instance', async () => {
await crudView.clickListPageCreateYAMLButton();
const yamlLinkIsPresent = await crudView.createYAMLLink.isPresent();
if (yamlLinkIsPresent) {
await crudView.createYAMLLink.click();
}
await yamlView.isLoaded();

const content = await yamlView.getEditorContent();
const newContent = _.defaultsDeep(
{},
{ metadata: { name, labels: { [testLabel]: testName } } },
safeLoad(content),
);
await yamlView.setEditorContent(safeDump(newContent));
});

it('creates a new resource instance', async () => {
leakedResources.add(
JSON.stringify({ name, plural: resource, namespace: namespaced ? testName : undefined }),
);
await yamlView.saveButton.click();

expect(crudView.errorMessage.isPresent()).toBe(false);
});

it('displays detail view for new resource instance', async () => {
await browser.wait(until.presenceOf(crudView.resourceTitle));
expect(browser.getCurrentUrl()).toContain(`/${name}`);
expect(crudView.resourceTitle.getText()).toEqual(name);
});

it('search view displays created resource instance', async () => {
await browser.get(
`${appHost}/search/${
namespaced ? `ns/${testName}` : 'all-namespaces'
}?kind=${kind}&q=${testLabel}%3d${testName}&name=${name}`,
);
await crudView.resourceRowsPresent();
await crudView
.rowForName(name)
.element(by.linkText(name))
.click();
await browser.wait(until.urlContains(`/${name}`));
expect(crudView.resourceTitle.getText()).toEqual(name);
});

it('edit the resource instance', async () => {
if (kind !== 'ServiceAccount') {
await browser.get(
`${appHost}/search/${
namespaced ? `ns/${testName}` : 'all-namespaces'
}?kind=${kind}&q=${testLabel}%3d${testName}&name=${name}`,
);
await crudView.resourceRowsPresent();
await crudView.editRow(kind)(name);
}
});

it('deletes the resource instance', async () => {
await browser.get(
`${appHost}${namespaced ? `/k8s/ns/${testName}` : '/k8s/cluster'}/${resource}`,
);
await crudView.resourceRowsPresent();
// Filter by resource name to make sure the resource is on the first page of results.
// Otherwise the tests fail since we do virtual scrolling and the element isn't found.
await crudView.filterForName(name);
await crudView.deleteRow(kind)(name);
leakedResources.delete(
JSON.stringify({ name, plural: resource, namespace: namespaced ? testName : undefined }),
);
});
});
});

describe('Role Bindings', () => {
const bindingName = `${testName}-cluster-admin`;
const roleName = 'cluster-admin';
Expand Down Expand Up @@ -228,41 +93,6 @@ describe('Kubernetes resource CRUD operations', () => {
});
});

describe('Namespace', () => {
const name = `${testName}-ns`;

it('displays `Namespace` list view', async () => {
await browser.get(`${appHost}/k8s/cluster/namespaces`);
await crudView.isLoaded();

expect(crudView.rowForName(name).isPresent()).toBe(false);
});

it('creates the namespace', async () => {
await crudView.createYAMLButton.click();
await browser.wait(until.presenceOf($('.modal-body__field')));
await $$('.modal-body__field')
.get(0)
.$('input')
.sendKeys(name);
leakedResources.add(JSON.stringify({ name, plural: 'namespaces' }));
await $('#confirm-action').click();
await browser.wait(until.invisibilityOf($('.modal-content')), K8S_CREATION_TIMEOUT);

expect(browser.getCurrentUrl()).toContain(`/k8s/cluster/namespaces/${testName}-ns`);
});

it('deletes the namespace', async () => {
await browser.get(`${appHost}/k8s/cluster/namespaces`);
// Filter by resource name to make sure the resource is on the first page of results.
// Otherwise the tests fail since we do virtual scrolling and the element isn't found.
await crudView.filterForName(name);
await crudView.resourceRowsPresent();
await crudView.deleteRow('Namespace')(name);
leakedResources.delete(JSON.stringify({ name, plural: 'namespaces' }));
});
});

describe('CustomResourceDefinitions', () => {
const plural = `crd${testName}`;
const group = 'test.example.com';
Expand Down

0 comments on commit 5150fde

Please sign in to comment.