OW distribution Build/test servers #165

jmalak · 2015-04-30T14:29:39Z

jmalak
Apr 30, 2015
Maintainer

I am currently using server with 32-bit Windows for build/test that I could integrate with buildbot.
It is OK for 16/32-bit testing and for documentation building.

Generaly in build/server directory is build server which you can use for build/test OW distribution.

Take into account that full build and testing can take hours. Bellow is report from my server to be able estimate how long it takes (host is 32-bit Windows).

Open Watcom Build Report (build on win32-x86)

    Built through change   : b118abc7e65bb35d0fc067ef1c62b082e4d01caa on 2015-04-24
    Building through change: 19d0e5bdec6217fcbffc98961036da5abf51f959

Compilers and Tools

 CLEAN+BOOTSTRAP STARTED    : 2015-04-29, 21:50 UTC
 CLEAN+BOOTSTRAP COMPLETED  : 2015-04-29, 21:57 UTC

 CLEAN+BUILD STARTED        : 2015-04-29, 21:57 UTC
 CLEAN+BUILD COMPLETED      : 2015-04-30, 01:18 UTC

Build Successful

 REGRESSION TESTS STARTED   : 2015-04-30, 01:18 UTC
 REGRESSION TESTS COMPLETED : 2015-04-30, 05:19 UTC

    Fortran Compiler : Succeeded.
    C Compiler       : Succeeded.
    C++ Compiler     : Succeeded.
    WASM             : Succeeded.

Documentation Build

 CLEAN+BUILD STARTED        : 2015-04-30, 05:19 UTC
 CLEAN+BUILD COMPLETED      : 2015-04-30, 06:21 UTC

Build Successful

Installers build

 INSTALLER BUILD STARTED    : 2015-04-30, 06:21 UTC
 INSTALLER BUILD COMPLETED  : 2015-04-30, 06:27 UTC

Post tasks

 POST RUN COMPLETED         : 2015-04-30, 06:29 UTC

git Messages

Already up-to-date.

git Messages end

jmalak · 2015-04-30T15:08:07Z

jmalak
Apr 30, 2015
Maintainer Author

Jeff, what is necessary to be on the host?

0 replies

pchapin · 2015-04-30T15:20:25Z

pchapin
Apr 30, 2015
Maintainer

It is true that Jenkins is a Java program but it should, in theory, be able to build anything. Right now I don't know the system well enough to say how complicated it would be for it to build Open Watcom. I'll be able to say more in a few weeks once I learn my way around it.

0 replies

ArmstrongJ · 2015-04-30T15:23:33Z

ArmstrongJ
Apr 30, 2015
Collaborator

Basically the machine doing the building (the build slave) needs to run a server to listen for incoming instructions. There would be some configuration beyond getting the server running on the slave such as configuring a work directory and ensuring all the build tools are on the path.

The harder part is actually on the build master. That server is what would hold the build instructions. It would instruct the slave to check out the git repo and tell it what commands to execute. For Open Watcom, most of the instructions will be shell commands, which are easy enough to implement. As an example, here is the build script on the server for "global_build_rel":

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them.  Note that any particular build will
# only take place on one slave.

from buildbot.process.factory import BuildFactory
from buildbot.steps.source.git import Git
from buildbot.steps.slave import RemoveDirectory
from buildbot.steps.shell import ShellCommand
from buildbot.steps.transfer import FileDownload, FileUpload

checkout = Git(repourl='https://github.com/open-watcom/open-watcom-v2.git',
               mode='incremental',
               workdir=checkout_dir)

clean   = ShellCommand(name='Global Clean',
                       command=['builder','clean'],
                       env=ow_env,
                       workdir=ow_src_dir)

bootclean   = ShellCommand(name='Bootstrap Clean',
                           command=['builder','bootclean'],
                           env=ow_env,
                           workdir=ow_src_dir)

deployprep = FileDownload(mastersrc="~/helpers/prep.sh",
                          slavedest=Interpolate("%(prop:workdir)s/"+checkout_dir+"/prep.sh"),
                          mode=0755)

prep      = ShellCommand(name='Global Bootstrap Prep',
                         command=['./prep.sh'],
                         description='tool build',
                         env=ow_env,
                         workdir=Interpolate("%(prop:workdir)s/"+checkout_dir),
                         haltOnFailure=True)

bootstrap = ShellCommand(name='Global Bootstrap Build',
                         command=['builder','boot'],
                         description='bootstrap',
                         env=ow_env,
                         workdir=ow_src_dir,
                         haltOnFailure=True)

build     = ShellCommand(name='Global Build',
                         command=['builder','build'],
                         description='building',
                         env=ow_env,
                         workdir=ow_src_dir,
                         haltOnFailure=True)

release   = ShellCommand(name='Global Release',
                         command=['builder','rel'],
                         description='release',
                         env=ow_env,
                         workdir=ow_src_dir,
                         haltOnFailure=True)

zip       = ShellCommand(name='Zip Release',
                         description='zip release',
                         command=['zip', '-r', release_file, '.'],
                         workdir=ow_rel_dir)

upload    = FileUpload(name='Transfer Release',
                       slavesrc=release_file,
                       masterdest=os.path.join('/tmp', release_file),
                       workdir=ow_rel_dir)


factory = BuildFactory()
# check out the source
factory.addStep(checkout)
factory.addStep(deployprep)
factory.addStep(prep)
factory.addStep(clean)
factory.addStep(bootclean)
factory.addStep(bootstrap)
factory.addStep(build)
factory.addStep(release)
factory.addStep(zip)
factory.addStep(upload)

from buildbot.config import BuilderConfig

c['builders'] = []
c['builders'].append(
    BuilderConfig(name="global_build_rel",
      slavenames=["local1"],
      factory=factory))

And you might notice it ships a shell script named prep.sh. This script simply steals some of the boostrap code from our existing scripts:

#!/bin/sh

cd $OWSRCDIR/wmake
if [ ! -d $OWOBJDIR ]; then mkdir $OWOBJDIR; fi
cd $OWOBJDIR
rm -f $OWBINDIR/wmake
if [ "$OWUSENATIVETOOLS" -eq "1" ]; then
case `uname` in
        FreeBSD)
            make -f ../posmake clean
            make -f ../posmake TARGETDEF=-D__BSD__
            ;;
# Linux)
        *)
            make -f ../posmake clean
            make -f ../posmake TARGETDEF=-D__LINUX__
            ;;
    esac
else
wmake -f ../wmake clean
    wmake -f ../wmake
fi
cd $OWSRCDIR/builder
if [ ! -d $OWOBJDIR ]; then mkdir $OWOBJDIR; fi
cd $OWOBJDIR
rm -f $OWBINDIR/builder
$OWBINDIR/wmake -f ../binmake clean
$OWBINDIR/wmake -f ../binmake bootstrap=1 builder.exe

So at the moment, the build is extremely focused on building on Linux, and it never bothers creating installers. It also doesn't run tests at the moment. On the master server, things would need to be cleaned up a bit. We'd probably want a separate BuilderConfig for Windows and Linux since Windows can build more of the documentation.

I won't get to it today, but let me try creating a configuration for a Windows machine. We can start with seeing if we can get the build to work. It might also be a good idea to throw these scripts into git here (in a separate repo) so we have them under source control. That is, if we decide to try buildbot...

0 replies

ArmstrongJ · 2015-04-30T15:25:19Z

ArmstrongJ
Apr 30, 2015
Collaborator

@pchapin that's great! I know very little about Jenkins other than using it (when it was Hudson pre-forking) on a large-scale Java project probably 7 years ago. Keep us informed!

0 replies

jmalak · 2015-04-30T16:04:08Z

jmalak
Apr 30, 2015
Maintainer Author

@ArmstrongJ it doesn't hurry. I have a question, is it possible to configure buildbot slave to connect server (main site), because mostly build/test servers are after firewalls.

0 replies

ArmstrongJ · 2015-04-30T16:18:11Z

ArmstrongJ
Apr 30, 2015
Collaborator

@jmalak I'm not exactly sure if the slaves contact the master or vice versa. I suspect the slaves contact the master based on: http://buildbot.approximatrix.com/buildslaves

But I'm not sure... I'll research it.

0 replies

jmalak · 2015-04-30T16:31:07Z

jmalak
Apr 30, 2015
Maintainer Author

OK. If you need some help with testing then send me e-mail directly.
I would like to connect my current build/test server to your master.
We must also be able to run it on OS/2 to be able run tests there.
My idea is to use 3 build servers and more test servers, because build OW on each server is useless.

0 replies

staalmannen · 2015-04-30T17:33:19Z

staalmannen
Apr 30, 2015

For Linux : what about OpenSuse Build Service ( OBS )? That way one could
also have Arch linux, .deb and .rpm- based (and possibly more) repositories
"for free" in addition to acting as a build server.
Den 30 apr 2015 18:31 skrev "Jiří Malák" notifications@github.com:

OK. If you need some help with testing then send me e-mail directly.
I would like to connect my current build/test server to your master.
We must also be able to run it on OS/2 to be able run tests there.
My idea is to use 3 build servers and more test servers, because build OW
on each server is useless.

—
Reply to this email directly or view it on GitHub
#165 (comment)
.

0 replies

jmalak · 2015-04-30T18:30:12Z

jmalak
Apr 30, 2015
Maintainer Author

It is only partial solution. We need one solution for Linux (32/64-bit), Windows(16/32/64-bit), OS/2(16/32-bit) and DOS(16/32-bit) to be able maintain it with minimal resources. If we use different solution for each platform then it consumes much more resources for maintenance.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OW distribution Build/test servers #165

{{title}}

Replies: 9 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

OW distribution Build/test servers #165

jmalak Apr 30, 2015 Maintainer

Open Watcom Build Report (build on win32-x86)

Compilers and Tools

Documentation Build

Installers build

Post tasks

git Messages

Replies: 9 comments

jmalak Apr 30, 2015 Maintainer Author

pchapin Apr 30, 2015 Maintainer

ArmstrongJ Apr 30, 2015 Collaborator

ArmstrongJ Apr 30, 2015 Collaborator

jmalak Apr 30, 2015 Maintainer Author

ArmstrongJ Apr 30, 2015 Collaborator

jmalak Apr 30, 2015 Maintainer Author

staalmannen Apr 30, 2015

jmalak Apr 30, 2015 Maintainer Author

jmalak
Apr 30, 2015
Maintainer

jmalak
Apr 30, 2015
Maintainer Author

pchapin
Apr 30, 2015
Maintainer

ArmstrongJ
Apr 30, 2015
Collaborator

ArmstrongJ
Apr 30, 2015
Collaborator

jmalak
Apr 30, 2015
Maintainer Author

ArmstrongJ
Apr 30, 2015
Collaborator

jmalak
Apr 30, 2015
Maintainer Author

staalmannen
Apr 30, 2015

jmalak
Apr 30, 2015
Maintainer Author