Test TravisCI #95

jrha · 2018-06-04T12:02:28Z

DO NOT MERGE!

Proof of concept, modifies code all over the place to get tests running.

jrha · 2018-06-04T13:57:17Z

Ok it works well enough to prove the point, I'm expecting a student to be working on this from 18th June.

* Copy HOME from standard environment * Add dependency on protobuf 3.6.0 * Install knc * Install protocols to /usr/local/lib/... * Use locally installed knc * Force service name to be aquilon * Set keytab location Change-Id: I3dabb9e17d9ad4686a19336e121b1aceb925cf0d

Let's see how far we get

jrha · 2018-06-27T10:53:20Z

It turned out to be easier to get kerberos installed and running on Travis than to fake it. We built and tested that as a separate repository https://github.com/jrha/krb5-travis and this makes use of those scripts.

There are a few places where we played around with modifying env to preserve HOME, this was to allow the git configuration to be picked up and avoid the tests dying due to:

Please tell me who you are.
Run
  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: unable to auto-detect email address (got 'travis@travis-job-42beb0f8-567c-4078-a82d-aa8d4b7d9ec1.(none)

Getting the DSDB related tests to run feels like fighting multiple Catch 22s, particularly due to this block of code, I think we've got as we can without input from experts... @urbonegi?

Change-Id: I9c010847d3b14e10d8c2f7946a3164f44011c9d0

jrha · 2018-06-28T13:23:07Z

Ok, I've added that and tried with DSDB enabled, but all the tests now fail at a lower level because they cannot import ms.dsdb.client, which cannot be easily provided as the name-space conflicts with the "bootstrapped" ms module, another Catch 22 I think.
Any suggestions before I make some bodges to try and work around this?

urbonegi · 2018-07-02T16:15:10Z

@jrha strange... The import should work cause it will import from the 'fakemodules' in case the unittest run. This is achieved with a help of this config option:
https://github.com/quattor/aquilon/blob/upstream/tests/unittest.conf.noms#L81
and this piece of code:
https://github.com/quattor/aquilon/blob/upstream/lib/twisted/plugins/aqd.py#L138

Maybe old version of the aquilon code does not have the newest unittest conf with fakemodule path. Or the fakemodele itself:
https://github.com/quattor/aquilon/blob/upstream/tests/fakemodules/ms/dsdb/client.py

I will try setup this myself, in case none of the above helps :)

urbonegi · 2018-07-25T14:30:00Z

lib/aquilon/worker/processes.py

+except ImportError:
+    pass
+else:
+    ms.version.addpkg('setuptools', '0.6c11')


why this one is added? :)

I don't know, I copied and pasted it from you?

i might have copied if from somewhere else. My bad. Actually we might release this fix ourselves next week and push to github. Will let you know then we can rebase this one :)

jrha · 2018-07-25T14:33:43Z

This is still not ready for review, I'm working on debugging the python environment still.

urbonegi · 2018-07-25T14:35:01Z

.travis.yml

@@ -6,7 +6,7 @@ before_script:
  - curl https://raw.githubusercontent.com/jrha/krb5-travis/master/install_knc | bash
  - wget https://github.com/google/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip -O /tmp/protoc.zip && cd /tmp && unzip /tmp/protoc.zip && sudo cp bin/protoc /usr/bin/ && sudo chmod ugo+x /usr/bin/protoc
  - cd /tmp && git clone https://github.com/quattor/aquilon-protocols.git && cd aquilon-protocols && git checkout upstream && ./setup.py build && sudo ./setup.py install --install-lib /usr/local/lib/aquilon/protocols/lib/python
-  - cd $TRAVIS_BUILD_DIR/tools/bootstrap_ms && python setup.py install
+  - cd $TRAVIS_BUILD_DIR/tools/bootstrap_ms && python setup.py install && ls /home/travis/virtualenv/python$TRAVIS_PYTHON_VERSION/lib/python$TRAVIS_PYTHON_VERSION/site-packages/*


Another way to check this would be to try to run the Aquilon dev server outside MS with the updated config to include [unittest] section and fake_module_location = %(srcdir)s/tests/fakemodules/
It is not easy to recreate this in MS cause we have modified python which does mess with sys path and import itself. I do not have Aquilon dev environment outside MS, if the import ms.dsdb.client keeps failing i might create one and help out with this. I have an idea that we might need to add import ms for the outside MS...

The kludge I've got now seems to be working, but yes it would be better to find a cleaner way to get it working.

Change-Id: I193a4a302c71195ef4b53c060e3ef782beacfe5f

jrha · 2018-07-25T16:06:54Z

Phew! At least the tests seem to be failing for more reasonable reasons now.

urbonegi · 2018-07-25T16:44:19Z

Awesome! You made me nervous of my python skills :) so the current failures are also about DSDB, however, these are different DSDB integration which is even more interesting. Idea is that we call dsdb_fakebin instead of real one:
https://github.com/quattor/aquilon/blob/upstream/tests/fakebin/fake_dsdb
it records the 'expected command' by adding it to a file; then 'verify command' check that the fail contains it. So question is does the echo "$*" >> "${AQTEST_DSDB_COVERAGE_DIR}/issued_dsdb_cmds" work.

Change-Id: I05db6b3f2ea1743d68753ef4d8b4a743588d77a6

jrha · 2018-07-26T09:16:44Z

Looks like the answer to that is no!

Change-Id: I7b1e4f2d9c747f4560745a1d1cad65317182a63b

jrha · 2018-07-26T09:53:59Z

At least the git based checks are working now.

Change-Id: I6f7ba5a9da3c1341bfa56788ee48510853ce0b61

XaF · 2018-07-30T23:09:46Z

Just a comment to mention I'm happy that we're moving on having tests done directly in here too!

Shaeli · 2018-08-23T21:38:21Z

Hi @jrha, I’m looking for something to contribute, can I try to continue this, as you proposed in PR 102?

jrha · 2018-08-24T08:56:25Z

Yes please, feel free to fork my branch.

jrha force-pushed the travis branch 6 times, most recently from 7f0b1c1 to bbd5b45 Compare June 4, 2018 12:24

jrha changed the title ~~Add test travis config~~ Test TravisCI Jun 4, 2018

jrha force-pushed the travis branch 4 times, most recently from a554e71 to 6968a67 Compare June 4, 2018 13:53

jrha force-pushed the travis branch 14 times, most recently from 2aaf2e7 to 8e951bc Compare June 4, 2018 16:04

jrha added 2 commits June 27, 2018 11:21

Test travis config

d3275b9

* Copy HOME from standard environment * Add dependency on protobuf 3.6.0 * Install knc * Install protocols to /usr/local/lib/... * Use locally installed knc * Force service name to be aquilon * Set keytab location Change-Id: I3dabb9e17d9ad4686a19336e121b1aceb925cf0d

Don't fail fast

185c8e7

Let's see how far we get

jrha force-pushed the travis branch from 8e951bc to 185c8e7 Compare June 27, 2018 10:21

Catch import errors as per @urbonegi's suggestion

59ece30

Change-Id: I9c010847d3b14e10d8c2f7946a3164f44011c9d0

jrha force-pushed the travis branch from 415fb87 to 59ece30 Compare June 28, 2018 13:21

jrha force-pushed the travis branch from 3be48d8 to 43957fc Compare July 25, 2018 14:29

urbonegi reviewed Jul 25, 2018

View reviewed changes

jrha force-pushed the travis branch 2 times, most recently from 3182b17 to 6585a70 Compare July 25, 2018 14:43

Inspect virtualenv

1e84317

Change-Id: I193a4a302c71195ef4b53c060e3ef782beacfe5f

jrha force-pushed the travis branch from 6585a70 to 1e84317 Compare July 25, 2018 14:47

Inspect scratch directory and fake DSDB logs

c0066e8

Change-Id: I05db6b3f2ea1743d68753ef4d8b4a743588d77a6

Fudge the environment passed to git

9b9af10

Change-Id: I7b1e4f2d9c747f4560745a1d1cad65317182a63b

jrha force-pushed the travis branch from bb1f083 to 9b9af10 Compare July 26, 2018 09:46

jrha force-pushed the travis branch 5 times, most recently from c5e9611 to 643da45 Compare July 26, 2018 10:44

Test fake DSDB

7c83bb7

Change-Id: I6f7ba5a9da3c1341bfa56788ee48510853ce0b61

jrha force-pushed the travis branch from 643da45 to 7c83bb7 Compare July 26, 2018 10:58

jrha marked this pull request as draft December 23, 2021 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test TravisCI #95

Test TravisCI #95

jrha commented Jun 4, 2018 •

edited

jrha commented Jun 4, 2018

jrha commented Jun 27, 2018 •

edited

jrha commented Jun 28, 2018

urbonegi commented Jul 2, 2018

urbonegi Jul 25, 2018

jrha Jul 25, 2018

urbonegi Jul 25, 2018

jrha Jul 25, 2018

jrha commented Jul 25, 2018

urbonegi Jul 25, 2018 •

edited

jrha Jul 25, 2018

jrha commented Jul 25, 2018

urbonegi commented Jul 25, 2018

jrha commented Jul 26, 2018

jrha commented Jul 26, 2018

XaF commented Jul 30, 2018

Shaeli commented Aug 23, 2018

jrha commented Aug 24, 2018

Test TravisCI #95

Are you sure you want to change the base?

Test TravisCI #95

Conversation

jrha commented Jun 4, 2018 • edited

DO NOT MERGE!

jrha commented Jun 4, 2018

jrha commented Jun 27, 2018 • edited

jrha commented Jun 28, 2018

urbonegi commented Jul 2, 2018

urbonegi Jul 25, 2018

Choose a reason for hiding this comment

jrha Jul 25, 2018

Choose a reason for hiding this comment

urbonegi Jul 25, 2018

Choose a reason for hiding this comment

jrha Jul 25, 2018

Choose a reason for hiding this comment

jrha commented Jul 25, 2018

urbonegi Jul 25, 2018 • edited

Choose a reason for hiding this comment

jrha Jul 25, 2018

Choose a reason for hiding this comment

jrha commented Jul 25, 2018

urbonegi commented Jul 25, 2018

jrha commented Jul 26, 2018

jrha commented Jul 26, 2018

XaF commented Jul 30, 2018

Shaeli commented Aug 23, 2018

jrha commented Aug 24, 2018

jrha commented Jun 4, 2018 •

edited

jrha commented Jun 27, 2018 •

edited

urbonegi Jul 25, 2018 •

edited