Add tutorial for running tests with Docker #2768

oleksandr-shabelnyk · 2019-06-10T10:16:49Z

@markus2330 Please review this tutorial. Unfortunately, since a I have a lot of pending changes for another issue, I could not create a separate PR just for this tutorial, so I had to do it from my different account, not from @alexsaber . I hope it is OK for you.

Basics

[x ] Short descriptions should be in the release notes (added as entry in
doc/news/_preparation_next_release.md which contains _(my name)_)
Please always add something to the the release notes.
Longer descriptions should be in documentation or in design decisions.
Describe details of how you changed the code in commit messages
(first line should have module: short statement syntax)
References to issues, e.g. close #X, should be in the commit messages.

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar none of these points below
need to be checked.

I added unit tests
I ran all tests locally and everything went fine
affected documentation is fixed
I added code comments, logging, and assertions (see Coding Guidelines)
meta data is updated (e.g. README.md of plugins and METADATA.ini)

Review

Reviewers will usually check the following:

Code is conforming to our Coding Guidelines
APIs are conforming to our Design Guidelines
Code is consistent to our Design Decisions
Documentation is concise and good to read
Examples are well chosen and understandable

Labels

Add the "work in progress" label if you do not want the PR to be reviewed yet.
Add the "ready to merge" label if the build server is happy and also you
say that everything is ready to be merged.

markus2330 · 2019-06-10T10:27:31Z

Unfortunately, since a I have a lot of pending changes for another issue, I could not create a separate PR just for this tutorial, so I had to do it from my different account, not from @alexsaber . I hope it is OK for you.

It is okay but you do not need different accounts for different PRs. You can simply create different branches and create a PR for each of the branches. See https://help.github.com/en/articles/creating-a-pull-request

alexsaber · 2019-06-10T10:35:12Z

It is okay but you do not need different accounts for different PRs.
I know this, of course, it tried it first, but it led me to nowhere after 2 hours of moving commits, changing branches, etc.

Why is the build failing this time?! I haven't changed any code in the PR..... I added the release note.

markus2330

Very nice tutorial.

markus2330 · 2019-06-10T10:29:22Z

doc/tutorials/run_all_tests_with_docker.md

@@ -0,0 +1,109 @@
+# Introduction
+
+Running all the tests like the build server does it requires to have multiple dependencies preinstalled. To overcome this problem, instead of trying to install all the necessary dependencies on your own, an appropriate Docker image should be used. This way you can easily and quickly run all the tests.


Suggested change

Running all the tests like the build server does it requires to have multiple dependencies preinstalled. To overcome this problem, instead of trying to install all the necessary dependencies on your own, an appropriate Docker image should be used. This way you can easily and quickly run all the tests.

Running all the tests like the build server requires to have multiple dependencies installed. To overcome this problem, instead of trying to install all the necessary dependencies on your own, an appropriate Docker image can be used. This way you can easily and quickly run all the tests.

markus2330 · 2019-06-10T10:29:54Z

doc/tutorials/run_all_tests_with_docker.md

+
+## Who is this guide for?
+
+For anyone who wants to run all the tests, like it is done by the build server, once changes are pushed to the repository.


Suggested change

For anyone who wants to run all the tests, like it is done by the build server, once changes are pushed to the repository.

For anyone who wants to run all the tests, like it is done by the build server.

markus2330 · 2019-06-10T10:32:43Z

doc/tutorials/run_all_tests_with_docker.md

+
+### 1. Pick a Docker image and pull it
+
+Pick one of the available Docker images of Elektra. If you do not know the difference, just pick this one --> "build-elektra-debian-stretch". Unfortunately, it will take some time to download it, since it is pretty big, but you can be sure you'll have all the needed dependencies.


Maybe also suggest some alpine image as alternative for slow connections.

But I thought it was not possible to run all tests with the alpine image. If it is possible, then I can suppest using it in the tutorial.

Yes you are right but you can also mention this limitation in the tutorial.

markus2330 · 2019-06-10T10:32:53Z

doc/tutorials/run_all_tests_with_docker.md

+docker pull <image_name>:<tag_name>
+```
+
+EXAMPLE:


Suggested change

EXAMPLE:

Example:

markus2330 · 2019-06-10T10:33:40Z

doc/tutorials/run_all_tests_with_docker.md

+<image_name>:<tag_name>
+```
+
+EXAMPLE:


Suggested change

EXAMPLE:

Example:

markus2330 · 2019-06-10T10:34:14Z

doc/tutorials/run_all_tests_with_docker.md

+### 3. Build
+
+
+After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace


Suggested change

After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace

After starting the container, you should be automatically inside it in the working directory `/home/jenkins/workspace`.

markus2330 · 2019-06-10T10:34:40Z

doc/tutorials/run_all_tests_with_docker.md

+
+After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace
+
+Create folder for building project


Sentence incomplete? Or should this be a header?

markus2330 · 2019-06-10T10:36:14Z

doc/tutorials/run_all_tests_with_docker.md

+Finally run the tests
+```sh
+make run_all
+```


Also describe how to install Elektra and run the installed tests (kdb run_all). Please also link to doc/TESTING.md.

What do you mean? Elektra is built within the container anyway. Why do we need to install it?

To also install Elektra make install and run then the installed tests. The build server also does this and it is possible that the installed tests have different results (e.g. if test data is missing). See scripts/jenkins/Jenkinsfile

Oh, I didn't know it. So basically before executing make run_all make install should be executed first?

Yes, some tests only run if Elektra is installed.

And there is a second set of tests to be executed with:

kdb run_all

These tests also need an installed Elektra.

If your tutorial should cover how to execute all tests, you should mention this.

markus2330 · 2019-06-10T10:39:35Z

Why is the build failing this time?! I haven't changed any code in the PR..... I added the release note.

Simply read what the build server tells you. (You might need to change between the pipeline steps, as not always the faulty one opens). The problem seems that you need to reformat markdown.

Will you describe the reformatting in a separate tutorial?

alexsaber · 2019-06-10T10:55:33Z

doc/tutorials/run_all_tests_with_docker.md

+
+### 1. Pick a Docker image and pull it
+
+Pick one of the available Docker images of Elektra. If you do not know the difference, just pick this one --> "build-elektra-debian-stretch". Unfortunately, it will take some time to download it, since it is pretty big, but you can be sure you'll have all the needed dependencies.


But I thought it was not possible to run all tests with the alpine image. If it is possible, then I can suppest using it in the tutorial.

alexsaber · 2019-06-10T10:57:54Z

doc/tutorials/run_all_tests_with_docker.md

+Finally run the tests
+```sh
+make run_all
+```


What do you mean? Elektra is built within the container anyway. Why do we need to install it?

alexsaber · 2019-06-10T13:01:44Z

Now these tests in mac fail

The following tests FAILED:
	 86 - testmod_dbus (Failed)
	 87 - testmod_dbusrecv (Failed)
Errors while running CTest
FAILED: tests/CMakeFiles/run_all 
cd /private/var/folders/sy/2x5qvs0n4lg18fry9jz4y21m0000gn/T/cirrus-ci-build/build && /private/var/folders/sy/2x5qvs0n4lg18fry9jz4y21m0000gn/T/cirrus-ci-build/build/scripts/run_all RelWithDebInfo
ninja: build stopped: subcommand failed.

The Cirrurs CI say it failed due to following error:
test change notification with cascaded parent key

../src/plugins/dbus/testmod_dbus.c:393: error in test_cascadedChangeNotification: string "user/tests/foo/bar" is not equal to "system/tests/testmod_dbusrecv/added"
	compared: keyName (toAdd) and context->receivedKeyName
testmod_dbus Results: 35 Tests done — 1 error.
        Start  89: testmod_directoryvalue
 68/272 Test  #87: testmod_dbusrecv ..............................***Failed    0.05 sec

What does it have to do with my tiny change that does not even involve code? If I get that many errors after trying to push a text file, I wonder if it is even possible to finish resolving my other issue with a lot of code changes...

sanssecours · 2019-06-10T13:12:58Z

What does it have to do with my tiny change that does not even involve code?

Those tests are known to fail quite regularly. I restarted the failing build job 🍎 Mac.

alexsaber · 2019-06-10T13:17:24Z

What does it have to do with my tiny change that does not even involve code?

Those tests are known to fail quite regularly. I restarted the failing build job 🍎 Mac.

Oh, thank you very much! It is good to hear.

alexsaber · 2019-06-10T20:16:41Z

@markus2330 I took into consideration all your comments and pushed some more changes. Before I did this, I saw all the build checks finally succeeded. Can you please merge this PR then?

markus2330 · 2019-06-10T20:42:42Z

We need to wait for the build server and maybe other reviewers. I did not check if the code actually works yet.

sanssecours

I followed the steps described in the tutorial. My computer reported a few failing tests:

testshell_markdown_tutorial_cascading,
testshell_markdown_tutorial_validation,
testkdb_error, and
testkdb_ensure

, and minor problems with the install procedure. Those problems might be related to my host OS (macOS 10.14.5) though.

sanssecours · 2019-06-12T07:57:14Z

doc/tutorials/run_all_tests_with_docker.md

+make install
+```
+
+The number 10 can be changed as follows: number of supported simultaneous threads by your CPU + 2. But don't worry, this can only affect the speed of the building, it cannot really break it.


I think it would make sense to move these instructions closer to the command make -j 10. Right below the command would be one good option in my opinion.

Done it. Thank you!

sanssecours · 2019-06-12T08:38:25Z

doc/tutorials/run_all_tests_with_docker.md

+Just run this command:
+
+```sh
+make install


This step fails on my machine with the error:

-- Install configuration: "RelWithDebInfo" CMake Error at cmake_install.cmake:36 (file): file cannot create directory: /usr/local/share/doc/elektra. Maybe need administrative privileges. Makefile:72: recipe for target 'install' failed make: *** [install] Error 1

. It might make sense to add the option -DINSTALL_SYSTEM_FILES=OFF to the cmake command above.

Thank you for your comment! Now I am getting this error too... Unfortunately, adding -DINSTALL_SYSTEM_FILES=OFF did not help. Does anyone have a suggestion on how to solve it? @markus2330

I think setting CMAKE_INSTALL_PREFIX to a newly created folder inside the repository (e.g. "$PWD/install") should help.

I am sorry, but I don't understand what do you mean. Can you make an example?

Can you make an example?

Just replace the current cmake command with:

cmake .. \ -DBINDINGS="ALL;-DEPRECATED;-haskell" \ -DPLUGINS="ALL;-DEPRECATED" \ -DTOOLS="ALL" \ -DENABLE_DEBUG=ON \ -DKDB_DB_HOME="$PWD" \ -DKDB_DB_SYSTEM="$PWD/.config/kdb/system" \ -DKDB_DB_SPEC="$PWD/.config/kdb/system" \ -DCMAKE_INSTALL_PREFIX="$PWD/install" \ -DINSTALL_SYSTEM_FILES=OFF

, and installing should work.

I am using Ubuntu 18.04 in a virtual machine.

I am using Ubuntu 18.04 in a virtual machine.

Interesting. Would it not make more sense to just use Docker directly? Anyway, it would be great, if someone who uses Linux as main OS, could check if the tests also fail on their machine.

I could not install Elektra on Windows, so I decided to work entirely in a Linux virtual machine.

I could not install Elektra on Windows, so I decided to work entirely in a Linux virtual machine.

What did you try? Can you write an issue about that please?

Interesting. Would it not make more sense to just use Docker directly?

Yes, I also wonder why you do not use docker directly?

Anyway, it would be great, if someone who uses Linux as main OS, could check if the tests also fail on their machine.

I can reproduce the same problem (3 failed tests:testshell_markdown_tutorial_cascading testshell_markdown_tutorial_validation testkdb_ensure). Something is tidy inside the docker image.

Please report these three issues separately, they seems to be unrelated to this tutorial.

@sanssecours any idea why our build server does not catch this problem?

@sanssecours any idea why our build server does not catch this problem?

Nope.

markus2330 · 2019-06-12T10:22:00Z

Thank you for the review!

@oleksandr-shabelnyk please fix the CMake command so that the tutorial actually works as described.

merge

markus2330

Thank you!

markus2330 · 2019-06-13T09:42:31Z

doc/tutorials/run_all_tests_with_docker.md


 ```sh
-mkdir build-docker && cd build-docker
+mkdir build-docker && cd build-docker && mkdir install


mkdir -p build-docker/install && cd build-docker is shorter and more fail-safe

But, in any way, I get:

mkdir: cannot create directory ‘build-docker’: Permission denied

because the folder is owned by uid/gid 1000, which is not the user in docker (id -u 47110)

Ok, I am going to fix it.

markus2330 · 2019-06-13T10:06:12Z

doc/tutorials/run_all_tests_with_docker.md

+Just run this command:
+
+```sh
+make install


I could not install Elektra on Windows, so I decided to work entirely in a Linux virtual machine.

What did you try? Can you write an issue about that please?

Interesting. Would it not make more sense to just use Docker directly?

Yes, I also wonder why you do not use docker directly?

Anyway, it would be great, if someone who uses Linux as main OS, could check if the tests also fail on their machine.

I can reproduce the same problem (3 failed tests:testshell_markdown_tutorial_cascading testshell_markdown_tutorial_validation testkdb_ensure). Something is tidy inside the docker image.

Please report these three issues separately, they seems to be unrelated to this tutorial.

@sanssecours any idea why our build server does not catch this problem?

markus2330 · 2019-06-13T10:20:29Z

doc/tutorials/run_all_tests_with_docker.md


 Finally run the tests. There are two sets of tests. Run the first one with this command:

 ```sh
 make run_all
 ```

-For the second set to run remember to execute `make install` from the previous step. Run the second set with this command:
+For the second set to run, remember to execute `make install` from the previous step. Run the second set with this command:


To make kdb run_all work more than setting the PATH is needed. (Also export LD_LIBRARY_PATH="$PWD/install/lib:$PATH").

But even then the testkdb_ensure fails.

What did you try? Can you write an issue about that please?

It was 4 months ago. I really do not remember what I did back then.

Yes, I also wonder why you do not use docker directly?

My whole dev environment is inside this VM. On my host OS, Windows, it is hard to map volumes. And there sometimes problems with the rights.

Please report these three issues separately, they seems to be unrelated to this tutorial.

Ok, you mean

Failing tests

Windows installation problems

What is the 3rd issue?

To make kdb run_all work more than setting the PATH is needed. (Also export LD_LIBRARY_PATH="$PWD/install/lib:$PATH").

But even then the testkdb_ensure fails.

Thank you! I will add it.

As 3rd issue, you probably meant failing tests with kdb run_all. I added it as well.

I fixed everything you requested. Is it OK now? I am really running extremely out of time with all these changes...
The building seems to get stuck. Can you restart it?

markus2330

Still not fixed?

markus2330 · 2019-06-14T20:00:26Z

doc/tutorials/run_all_tests_with_docker.md

@@ -67,7 +66,7 @@ So from your root project folder run the following:
 docker run -it --rm \
 -v "$PWD:/home/jenkins/workspace" \
 -w /home/jenkins/workspace \
-<image_name>:<tag_name>
+hub-public.libelektra.org/<image_name>:<tag_name>


Here the user discussed in the issue #2781 is still missing.

The proposed solution introduces other problems. If running with the current user, not just 1 test fails, but around 20 of them.

Seems like the problem can only be fixed if you build the docker container correctly. See scripts/docker/README.md

markus2330 · 2019-06-19T15:05:20Z

doc/tutorials/run_all_tests_with_docker.md

-
-Example:
-
-```sh


Why do you remove the example?

It showed how to run a container created from a public image. Since we do not recommend it, I thought it is not needed anymore.

I thought you will simply leave this part and say it is not recommended.

alexsaber · 2019-06-19T15:49:03Z

@markus2330 As suggested today by you, I inspected the jenkinsfile. I saw some mismatches in the cmake options and corrected/added some. This fixed the testkdb_ensure test! Now it passes!

testmod_lua is still failing though....

I literally have no idea at all how to fix it.

markus2330 · 2019-06-19T19:24:29Z

These are very good news!

I think lua needs export LUA_CPATH="${WORKSPACE}/system/lib/lua/5.2/?.so;" or maybe even more paths (see scripts/jenkins/Jenkinsfile lines 917-920)

alexsaber · 2019-06-19T19:53:57Z

These are very good news!

I think lua needs export LUA_CPATH="${WORKSPACE}/system/lib/lua/5.2/?.so;" or maybe even more paths (see scripts/jenkins/Jenkinsfile lines 917-920)

Perfect! It solved the problem! Thank you very much! All tests run with kdb run_all pass now.

alexsaber · 2019-06-19T19:57:19Z

Just tried to run another batch of the tests with make run_all. They all pass as well now! I am so happy! =)

markus2330 · 2019-06-20T08:07:22Z

Great job!

markus2330 added the cm2019s label Jun 10, 2019

markus2330 requested changes Jun 10, 2019

View reviewed changes

alexsaber approved these changes Jun 10, 2019

View reviewed changes

markus2330 requested a review from sanssecours June 10, 2019 20:42

markus2330 approved these changes Jun 10, 2019

View reviewed changes

markus2330 removed the request for review from sanssecours June 10, 2019 20:48

sanssecours added enhancement usability labels Jun 11, 2019

Prepend repository name for the image names

a4022cc

alexsaber mentioned this pull request Jun 11, 2019

Reformat with docker tutorial #2773

Merged

13 tasks

Apply changes suggested by sanssecours

72b7714

sanssecours approved these changes Jun 12, 2019

View reviewed changes

oleksandr-shabelnyk and others added 6 commits June 12, 2019 13:52

Merge pull request #1 from ElektraInitiative/master

2772384

merge

Add cmake option, minor description change

136c95a

Merge branch 'master' into run_tests_docker_tutorial

bd83eb5

Add missing ""

e356ed1

Change the way Elektra is installed

a713fea

Change a line to make the CI happy

7cae7ed

markus2330 requested changes Jun 13, 2019

View reviewed changes

change paths

13ba13b

markus2330 requested changes Jun 14, 2019

View reviewed changes

recommend to build own image like requested today

513777d

markus2330 reviewed Jun 19, 2019

View reviewed changes

fix passing testkdb_ensure test

eb1a555

Add path variable for lua lib

2c22b4c

alexsaber mentioned this pull request Jun 19, 2019

Running tests #2585

Closed

markus2330 approved these changes Jun 19, 2019

View reviewed changes

markus2330 merged commit d8747ba into ElektraInitiative:master Jun 20, 2019

		@@ -0,0 +1,109 @@
		# Introduction

		Running all the tests like the build server does it requires to have multiple dependencies preinstalled. To overcome this problem, instead of trying to install all the necessary dependencies on your own, an appropriate Docker image should be used. This way you can easily and quickly run all the tests.


		## Who is this guide for?

		For anyone who wants to run all the tests, like it is done by the build server, once changes are pushed to the repository.


		### 1. Pick a Docker image and pull it

		Pick one of the available Docker images of Elektra. If you do not know the difference, just pick this one --> "build-elektra-debian-stretch". Unfortunately, it will take some time to download it, since it is pretty big, but you can be sure you'll have all the needed dependencies.

		### 3. Build


		After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace

	After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace
	After starting the container, you should be automatically inside it in the working directory `/home/jenkins/workspace`.


		After starting the contrainer, you should be automatically inside it in the working directory we set --> /home/jenkins/workspace

		Create folder for building project

Add tutorial for running tests with Docker #2768

Add tutorial for running tests with Docker #2768

Conversation

oleksandr-shabelnyk commented Jun 10, 2019

Basics

Checklist

Review

Labels

markus2330 commented Jun 10, 2019

alexsaber commented Jun 10, 2019

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markus2330 commented Jun 10, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexsaber commented Jun 10, 2019

sanssecours commented Jun 10, 2019

alexsaber commented Jun 10, 2019

alexsaber commented Jun 10, 2019

markus2330 commented Jun 10, 2019

sanssecours left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanssecours Jun 12, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markus2330 commented Jun 12, 2019

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexsaber commented Jun 19, 2019

markus2330 commented Jun 19, 2019

alexsaber commented Jun 19, 2019

alexsaber commented Jun 19, 2019

markus2330 commented Jun 20, 2019

markus2330 commented Jun 10, 2019 •

edited

sanssecours Jun 12, 2019 •

edited