Repeatable builds #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Thanks a lot, this version is much better than the previous one, much much better actually.
Readme should be updated, but updating that will probably be easy because you will mostly remove many things there :)
Overall this makes sense, just left some minor things.
README.md
Outdated
|
|
||
| Because of the specific logic that we have to run tests faster, we have a docker image `debbuilder` that generates a debian package for `postgresql-server-dev-{pg-major}`. The generated package is in the built docker image, specifically in `{container-name}/home/circleci/debs`. The package is pushed to [the-process](https://packagecloud.io/citus-bot/the-process). This package place is added to apt source list and pinned with with the highest priority so that it is chosen over the standard `postgresql-server-dev-{pg-major}` package. | ||
| This image contains all the artifacts required to produce a build of citus binaries for exactly 1 postgres version. This image is build for every supported Postgres version. Any scripts driving the build are contained in the citus repostiroy. |
There was a problem hiding this comment.
which one of the 3? :P
built is past tense
build is current tense
There was a problem hiding this comment.
This image is built
README.md
Outdated
| ```bash | ||
| docker push {repositoryName}/{imageName} | ||
| ``` | ||
| TODO: this image has not changed and is currently not build by the Makefile. Future work. |
| WORKDIR /build/postgresql-${PG_VERSION}/ | ||
| COPY patches/ patches/ | ||
| SHELL ["/bin/bash", "-c"] | ||
| RUN if [ -d "patches/${PG_VERSION}/" ]; \ |
There was a problem hiding this comment.
I like this idea, it is just sad that one mistake in postgres side causes many others to resolve the problem multiple times.
There was a problem hiding this comment.
Only half of the year ;)
SaitTalhaNisanci
approved these changes
Dec 18, 2020
thanodnl
added a commit
to citusdata/citus
that referenced
this pull request
Dec 18, 2020
This doesn't change anything functionally for Citus or our CI. Instead it is a quality of life change to make it easier to maintain CI and how we build. In general this patch accomplishes the following - move build and test scripts into the Citus tree instead of our CI tree. - deduplicate convoluted jobs definition by a couple generic build and test runs - standardized the output - standardized coredumps - made all coredumps and regression diffs available for download A big part of this refactor is in the `the-process` repository. citusdata/the-process#48
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR rewrites our complete build system for creating and maintaining CI images.
It uses docker layers with better reuse compared to what we used to have. There are still some ways for better reuse. In the future we can move all dockerfiles into 1 with different target names and reuse of common layers like
For now it touched all by the stylechecker.
The
Makefileincluded incirceleci\imagescontains the postgres versions to currently build for. All scripts install them from the apt archive maintained by pgdg. This should allow for stable builds over time, where we can rebuild older versions with more ease.We have removed the need to create custom debian packages that needed to be uploaded to package cloud. Instead we use multi layer docker builds to compile the specific postgres version and copy over the isolation tester binaries and the vanilla tests for us to run in CI.
By default the Makefile will build images tagged with a
-devYYYYmmddHHMMsuffix for use when changing images. Once changes are approved and landed to master we can create the final images viaRELEASE=1 make push-all.TODO