Using Docker images of Jenkins for unit tests #230

gnl42 · 2023-07-09T17:15:42Z

Following some comments in #203 regarding the state of our current Jenkins server used for the unit tests I wondered if Docker images could be used during the Mylyn build.

I've come up with a working PoC using TestContainer to launch Jenkins as docker container and using that container to run a couple of of JUnit tests successfully.

It runs fine under Eclipse but when I try build Mylyn with Maven the build blows up on the modified target file.
[ERROR] Failed to resolve target definition file:/E:/workspaces/eclipse-mylyn/mylyn-main/git/org.eclipse.mylyn/org.eclipse.mylyn-target/mylyn-e4.28.target: Artifact MavenArtifactFacade [...], IncludeDependencyScope = [compile, runtime, test], MissingManifestStrategy = GENERATE, IncludeSource = true is not a valid jar file

Also I don't know if it will run using the Mylyn Jenkins instance used for the builds?

Is this something worth exploring more? Or is it just a wild idea?

George
Originally posted by @gnl42 in #203 (comment)

The text was updated successfully, but these errors were encountered:

gnl42 · 2023-07-10T03:38:52Z

Shoot, so close:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin]) 79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null)As no valid configuration was found, execution cannot continue.

Would it be possible to have docker installed on the server?

ruspl-afed · 2023-07-10T16:25:22Z

Please have a look at https://wiki.eclipse.org/Jenkins

gnl42 · 2023-07-11T02:49:04Z

:( Doesn't sound promising

docker build has been promised 'soon' since 2018

It had such promise

ruspl-afed · 2023-07-11T10:15:48Z

@gnl42 , please describe your issue for Helpdesk, I'm sure they can provide a guidance.

BeckerFrank · 2023-07-12T09:40:54Z

@gnl42 , please describe your issue for Helpdesk, I'm sure they can provide a guidance.

I think we should ask Helpdesk the following thinks.

is it possible to have a Docker in Docker image for the build
if the answer to 1) is yes we need a network so that the two container can talk to each other

If not possible we should ask if there is an other way to host Test instances for Jenkins and Bugzilla on the Eclipse infastructure and access over an internal url.

gnl42 · 2023-07-16T19:40:56Z

Do the servers on mylyn.org talk to each other? If not, why do we need a network?
And if they do, could we not set up the test image to run them all?

How does this sound @BeckerFrank @sddavis @ruspl-afed :

===
The Mylyn project currently uses servers (gerrit, bugzilla jenkin, etc.) running on mylyn.org, a server sponsored by Tasktop.com, for the unit test validation of its code.
Tasktop have said that they no longer maintaining the server and has indicated that they will retire the server at some point in the future.

One possible solution to this is run the unit test against services running on Docker images. I've implemented a Proof Of Concept of this using TestContainers that runs on fine my local machine against a Jenkins server docker image, but when the Jenkins server on https://ci.eclipse.org/mylyn ran the code it complained with:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null) As no valid configuration was found, execution cannot continue.

When I look at https://wiki.eclipse.org/Jenkins I see:

I want to build a custom Docker image (with docker build), but it does not work. What should I do?
You cannot currently build images on the cluster (i.e. docker build does not work). We plan to address this shortcoming shortly.

This note was added to the page in 2018.

So I'm wondering if the docker engine could be installed on the Jenkins server to be used as a possible replacement for mylyn.org?

If this is not possible, is there any other way a Test instance for the servers could be set up on the Eclipse infrastructure that can be accessed by the Mylyn build?

Thanks
George

wimjongman · 2023-07-17T07:16:31Z

Great.

BeckerFrank · 2023-07-17T17:04:51Z

Do the servers on mylyn.org talk to each other? If not, why do we need a network? And if they do, could we not set up the test image to run them all?

No they do not talk to each other. But the all register to a list of available services.

How does this sound @BeckerFrank @sddavis @ruspl-afed :

=== The Mylyn project currently uses servers (gerrit, bugzilla jenkin, etc.) running on mylyn.org, a server sponsored by Tasktop.com, for the unit test validation of its code. Tasktop have said that they no longer maintaining the server and has indicated that they will retire the server at some point in the future.

One possible solution to this is run the unit test against services running on Docker images. I've implemented a Proof Of Concept of this using TestContainers that runs on fine my local machine against a Jenkins server docker image, but when the Jenkins server on https://ci.eclipse.org/mylyn ran the code it complained with:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null) As no valid configuration was found, execution cannot continue.

When I look at https://wiki.eclipse.org/Jenkins I see:

I want to build a custom Docker image (with docker build), but it does not work. What should I do?
You cannot currently build images on the cluster (i.e. docker build does not work). We plan to address this shortcoming shortly.

This note was added to the page in 2018.

So I'm wondering if the docker engine could be installed on the Jenkins server to be used as a possible replacement for mylyn.org?

We need to run the entire build in a Docker image and the container running the unit test must have access to the newly created Jenkins/Bugzilla test instance.

If this is not possible, is there any other way a Test instance for the servers could be set up on the Eclipse infrastructure that can be accessed by the Mylyn build?

I don't know if there is a place for a virtual infrastructure that can be accessed from https://ci.eclipse.org/mylyn where we can have custom Docker images with our Bugzilla/Jenkins instances.

Thanks George

gnl42 · 2023-07-18T07:14:57Z

I've created a helpdesk issue at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3432

Sounds like docker images are out of the question, but if someone else with some knowledge on how Eclipse infrastructure could jump since I'm now out of my depth as to what would fulfill our needs. Thanks

gnl42 · 2023-07-20T09:18:30Z

@BeckerFrank @ruspl-afed Could someone jump in on the GitLab issue https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3432?

BeckerFrank · 2023-08-27T08:07:58Z

@gnl42: What was the solution?
Did you try the sidecar-container way?

gnl42 · 2023-09-03T22:32:22Z

@BeckerFrank Sorry, I thought you were working on it

BeckerFrank · 2023-10-18T18:15:25Z

I have created a POC of a Mylyn Multipass setup and a subdomain but need a tester to verify that we can access this from outside of my network. If this works I can implement this when I have a new Server for hosting this. The POC use my Laptop .

gnl42 · 2023-10-18T18:18:02Z

How can I test it?

BeckerFrank · 2023-10-19T00:25:26Z

With https://mylyn.local/mylyn_idx/service I can access the services from within my network and get the json String as a result and with https://mylyn.local/mylyn_idx I get a http page with the human more readable output instead of the json String.

In a first step please test if you get a response from the following url's

When this works we know that I can share the content from a multipass vm.

Next you can change org.eclipse.mylyn.commons.sdk.util.TestConfiguration.discover(Class, String, boolean) in /org.eclipse.mylyn.commons.sdk.util/src/org/eclipse/mylyn/commons/sdk/util/TestConfiguration.java and
use "/mylyn_idx/service" instead of "/cgi-bin/services" (line 130)

If it not work please ping me so that we can make sure that my MacBook is not in sleep mode.

gnl42 · 2023-10-19T06:11:38Z

Browser issue with the certificate:

https://mylyn.frank-becker.de/mylyn_idx/service

Peer’s Certificate issuer is not recognized.

Once I accept the risk, I'm in.

Needed to change URL_SERVICES_DEFAULT as well in TestConfiguration.

9 Errors, 2 Failures. Basically two types of errors:

org.eclipse.core.runtime.CoreException: I/O Error occurred while communicating with https://mylyn.frank-becker.de/bugzilla-4_4.
UnknownHostException: mylyn.local
at org.eclipse.mylyn.internal.bugzilla.core.BugzillaClient.connectInternal(BugzillaClient.java:342)
junit.framework.ComparisonFailure: expected:<[Unable to login to https://mylyn.frank-becker.de/bugzilla-4_4.
The username or password you entered is not valid.
Please validate credentials via Task Repositories view.]> but was:<[I/O Error occurred while communicating with https://mylyn.frank-becker.de/bugzilla-4_4.
UnknownHostException: mylyn.local]>
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at junit.framework.TestCase.assertEquals(TestCase.java:260)
at org.eclipse.mylyn.bugzilla.tests.core.BugzillaClientTest.testValidateUserPlusHTTP(BugzillaClientTest.java:183)

BeckerFrank · 2023-10-20T05:36:14Z

@gnl42: Thanks for testing.

I think the problems you are experiencing are due to the multipass setup being designed for the mylyn.local domain. This domain is the one I can access from my home network and used in the the bugzilla.conf.

From within my home network I can not access the public domain because of the config in my router. I have changed the bugzilla setup so please try it again

Do you think we need to change the Certificate?

BeckerFrank · 2023-10-26T04:27:10Z

@gnl42: Did you have found time to test this again?
I have changed the bugzilla.conf and hope that this fixed the login error.

gnl42 · 2023-10-26T06:38:24Z

@BeckerFrank Sorry, battling guava.

Not being able to connect to the server at the moment.

BeckerFrank · 2023-10-26T18:57:54Z

OK, maybe my Notebook was in sleep mode. Can you try again?

gnl42 · 2023-10-26T19:01:37Z

Much better. Just one error now. Looks like missing data?:

org.eclipse.core.runtime.CoreException: Repository error from :

Bug id not found. The requested bug id does not exist.
	at org.eclipse.mylyn.internal.bugzilla.core.MultiBugReportFactory.populateReport(MultiBugReportFactory.java:63)
	at org.eclipse.mylyn.internal.bugzilla.core.BugzillaClient.getTaskData(BugzillaClient.java:2338)
	at org.eclipse.mylyn.bugzilla.tests.support.BugzillaFixture.getTask(BugzillaFixture.java:258)
	at org.eclipse.mylyn.bugzilla.tests.core.BugzillaClientTest.testLeadingZeros(BugzillaClientTest.java:277)

The test is looking for String taskNumber = "0002";

BeckerFrank · 2023-10-26T19:18:41Z

Thanks, this looks like a setup error but not a connection error.

Now we know that I can set up a server with a multipass instance for all others, but not for an access from my internal network . So I will plan when and how I replace my old internal with a new one that can host her needed Multipass Instance.

gnl42 self-assigned this Jul 10, 2023

gnl42 added the enhancement New feature or request label Jul 10, 2023

gnl42 closed this as completed Aug 26, 2023

BeckerFrank reopened this Sep 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using Docker images of Jenkins for unit tests #230

Using Docker images of Jenkins for unit tests #230

gnl42 commented Jul 9, 2023 •

edited

gnl42 commented Jul 10, 2023

ruspl-afed commented Jul 10, 2023

gnl42 commented Jul 11, 2023

ruspl-afed commented Jul 11, 2023

BeckerFrank commented Jul 12, 2023

gnl42 commented Jul 16, 2023

wimjongman commented Jul 17, 2023

BeckerFrank commented Jul 17, 2023

gnl42 commented Jul 18, 2023

gnl42 commented Jul 20, 2023

BeckerFrank commented Aug 27, 2023

gnl42 commented Sep 3, 2023

BeckerFrank commented Oct 18, 2023

gnl42 commented Oct 18, 2023

BeckerFrank commented Oct 19, 2023

gnl42 commented Oct 19, 2023

BeckerFrank commented Oct 20, 2023

BeckerFrank commented Oct 26, 2023

gnl42 commented Oct 26, 2023

BeckerFrank commented Oct 26, 2023

gnl42 commented Oct 26, 2023 •

edited

BeckerFrank commented Oct 26, 2023

Using Docker images of Jenkins for unit tests #230

Using Docker images of Jenkins for unit tests #230

Comments

gnl42 commented Jul 9, 2023 • edited

gnl42 commented Jul 10, 2023

ruspl-afed commented Jul 10, 2023

gnl42 commented Jul 11, 2023

ruspl-afed commented Jul 11, 2023

BeckerFrank commented Jul 12, 2023

gnl42 commented Jul 16, 2023

wimjongman commented Jul 17, 2023

BeckerFrank commented Jul 17, 2023

gnl42 commented Jul 18, 2023

gnl42 commented Jul 20, 2023

BeckerFrank commented Aug 27, 2023

gnl42 commented Sep 3, 2023

BeckerFrank commented Oct 18, 2023

gnl42 commented Oct 18, 2023

BeckerFrank commented Oct 19, 2023

gnl42 commented Oct 19, 2023

BeckerFrank commented Oct 20, 2023

BeckerFrank commented Oct 26, 2023

gnl42 commented Oct 26, 2023

BeckerFrank commented Oct 26, 2023

gnl42 commented Oct 26, 2023 • edited

BeckerFrank commented Oct 26, 2023

gnl42 commented Jul 9, 2023 •

edited

gnl42 commented Oct 26, 2023 •

edited