[WiP] Docker #625
Open
[WiP] Docker #625
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pose.yml with base service
… docker/Dockerfile
… single axix (for testing), all in 2 CI & dev modes, introduce NBL_UPDATE_GIT_SUBMODULE_NO_SEPARATE_SHELL which toggles updates of submodules between separate git bash shell and CMake execute process, perform tests. Update base image definition and create cache image for internal CI infrastructure production runs with cached configured build directories. The first dev mode creates bind mount point in Nabla root path and makes whole Nabla clonned to a host machine available to containers for read & write operations, therefore containers are capable of seeing real time changed source code and build host Nabla without forcing a developer to push & test - this mode doesn't cache any build directory into image, but reuses build directories that are generated on host machine. The CI mode is designed to cache build directories within instance image, containers created will start having everything configured from the point they were built. Create pipeline proxy platform build script, cross platform pipeline build script, compose files and .env files, define first base and axis services (CI mode not tested yet, focusing on dev mode first)
…on process enabled, there are a lot of tweaks due to windows container features support but it is possible
- remove dxc CT and the zip executable which will be soon replaced with Nabla Shader Compiler Tool
- use choco packet manager in docker windows image base creation
- update build python script, parse arguments, add clone, configure and build functions
- have 2 separate compose files, one for base with entrypoint set to our cross platform pipeline build script, the second one for instance services (axes) which assumes that instance image gets created from base - CI will fire the build script for configuring build directories with both static & dynamic library formats to have it cached into container and then it will commit it into instance image. The reason behind is that many nice docker features that would allow me to do it directly in one single Dockerfile just don't work with Windows containers (but do with Linux), for example you could do following in your instance Dockerfile definition
RUN --mount=type=secret,id=id_rsa,required=true,target="${THIS_PROJECT_SSH_DIRECTORY}"
it would temporary mount your keys WITHOUT caching into docker image layers (security reasons) just to do the clones and configure steps cached directly into image from Dockerfile definition, but you cannot because buildkit is required and it doesn't work on windows containers https://docs.docker.com/build/buildkit/
so then you start creating workarounds
COPY <your_key> <destination/.ssh>
but it also won't work since even if you grant OS permissions & setup agent in image then you find you cannot use ssh while image building unless you have buildkit and use some tricks with it XD
so maybe you would do
# Allow rewrite protocol request https://git-scm.com/docs/git-config#Documentation/git-config.txt-protocolallow
RUN `
git config --system protocol.*.allow always`
`
# Request rewrite of all urls to https
`
&& git config --system url."https://github.com/".insteadOf "git@github.com:"
to force https protocol rewrite, but you would end of commits it cannot checkout,
so the best for CI will be to create instance image for services from a temporary container which will prepare the cache
…, perform tests on (still dummy) nabla.static.release.x86_64 service
…opment and runs, mark TODOs
…d more build arguments and adjust compose service definitions
…ot include nbl's builtinResources.h header directly, this exposition makes all builtin resource targets depend on Nabla's BR target, correct it and make them compile independently
… build.py - update submodules only once and let 2 configures process without the update requests
…pdate CMakeLists.txt
…ckages by default
…debugpy, adjust build.py script to work in server & client mode, add service ipv4 discovery, define nablaNetwork for containers, update Dockerfile
… compose files, have one debug server port per service, add Debug Listen Port argument to build.py script
…vice corresponds to a groovy stage now. Split python build.py into modules, update proxy script. Create docker compose dependency chains between services. Add Kazoo Apache Keeper Server for atomic operations between containers, implement kazoo.py and integrate into cluster. Create healthy checks granting proper service policy executions. Update Dockerfile, begin performing tests
…configuration file. Attempt to making this Docker Pipeline Codebase Cross-Platform & capable of scaling, update all dev modules and their implementations, perform tests, fix bugs, perform local pipeline runtime tests. Have top dev compose file including modular compose configurations
…rvice copy the key to persistent ssh volume
…an be invoked from cmd, remove .repository & .artifactory persistent volumes, bind whole Nabla directory from host and write directly to it from containers therefore make easy development - containers see local changes, make sure submodules are updated by host to prevent issues with .git shared between containers and host, correct dependency service chain
…an issue on Kitware SCM regarding buggy penv/env inheritance in build presets, it forced me to create separate configs), add registry long paths enabled feature to base image definition, add CMake services extended to cover configurations, output build directory per library format type and configuration therefore make sure no containers from different setup writes to another build directory (well, some targets compiled with configuration X reuse configuration Y), update modules
…work with host's codebase, submodules update operation which requires authentication is performed by dev.py script on host
…v.${libraryType}", "dev.${libraryType}.${config}", grant flexibility for local CI pipelines execution
…g to write to SCM by locking & releasing examples_tests and zlib directory. Perform tests executing all axes at once with docker. TODO: we could make atomic lock guard per file, though directory is fine as a solution - implement CMake NBL_CONFIGURE_FILE_ATOMIC(*) and NBL_FILE_ATOMIC(*) in future
…axis, update modules, update .gitignore
…e kazoo serves after package module is done, let compose down services
AnastaZIuk
force-pushed
the
master
branch
3 times, most recently
from
a07e79f
to
ffbd843
Compare
… shutdown implementation which works
…late.cmake, script component install rules has been added creating interface install variables in separate files which can be used with find_package. TODO: global configuration Nabla file including the component files if found in package, makes flexible search requests for package parts
…make & licence/template.cmake, update apache zookeeper version
…IBRARY_TYPE>_<CONFIG> install package variable, perform tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.