Skip to content

Commit

Permalink
Merge branch 'master' into 4.x
Browse files Browse the repository at this point in the history
# Conflicts:
#	CHANGELOG.md
  • Loading branch information
jafarre-bi committed Nov 24, 2022
2 parents 63642fb + 41445e8 commit 005ed8b
Show file tree
Hide file tree
Showing 52 changed files with 2,159 additions and 251 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/continuous-integration-workflow.yml
Expand Up @@ -122,7 +122,7 @@ jobs:
name: Download golangci-lint
run: |
curl -sSfL --output /tmp/golangci-lint.sh https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh
cat /tmp/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.46.2
cat /tmp/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.49.0
-
name: Run linter
working-directory: jenkins/webhook-proxy
Expand Down Expand Up @@ -201,7 +201,7 @@ jobs:
name: Verify all Go tests pass linting
uses: golangci/golangci-lint-action@v3
with:
version: v1.46.2
version: v1.49.0
working-directory: tests
args: --timeout=10m
-
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Expand Up @@ -2,6 +2,13 @@

## Unreleased

## [4.1.1] - 2022-11-24

- Fix CI/CD problems in Jenkins pipelines ([#1177](https://github.com/opendevstack/ods-core/pull/1177))
- Fixes Python agent does not seems to have java in the path ([#685](https://github.com/opendevstack/ods-quickstarters/issues/685))
- Removes existing differences between jenkins agent base image in Centos 7 and UBI 8 ([#1181](https://github.com/opendevstack/ods-core/pull/1181))
- Upgrade to Java 11 Jenkins master and agents and increase logging to know why they sometimes die ([#1185](https://github.com/opendevstack/ods-core/pull/1185))

## [4.1] - 2022-11-17

- Create Dockerfile.rhel7 and Dockerfile.centos7 with respectives plugins.rhel7.txt and plugins.centos7.txt definitions ([1000](https://github.com/opendevstack/ods-core/issues/1000))
Expand Down
9 changes: 9 additions & 0 deletions README.md
Expand Up @@ -48,3 +48,12 @@ b) inside the [tests](tests) directory. </p> The tests can be started with `make

1. [ODS Development Environment / ODS in a box](ods-devenv)<br>
ODS also ships as Amazon AMI - ready to go. The scripts to create the AMI can be found in ods-devenv. These scripts can be used also be used to install a `developer` version of ODS on a plain linux vm. Simply execute [bootstrap.sh](ods-devenv/scripts/bootstrap.sh)

## Current AMI build logs
the log files contain color coding, they are best viewed using a tool supporting color coding, like tail. E.g.:


```
# after untaring view the log file 'current' like so:
tail -fn +1 current
```
2 changes: 1 addition & 1 deletion create-projects/create-cd-jenkins.sh
Expand Up @@ -7,7 +7,7 @@ set -eu
# As this script is executed within the context of Jenkins, which has some
# env vars exposed (via the DeploymentConfig, but also from inside the image).
# It might be surprising to have them alter what the script does without seeing
# them passed/set in the Jenkinsfile. That's why we reset all env vars here
# them passed/set in the Jenkinsfile_createBuildBot. That's why we reset all env vars here
# and require them to be passed as parameters to the script.

TAILOR="tailor"
Expand Down
2 changes: 1 addition & 1 deletion create-projects/create-projects.sh
Expand Up @@ -4,7 +4,7 @@ set -e
# As this script is executed within the context of Jenkins, which has some
# env vars exposed (via the DeploymentConfig, but also from inside the image).
# It might be surprising to have them alter what the script does without seeing
# them passed/set in the Jenkinsfile. That's why we reset all env vars here
# them passed/set in the Jenkinsfile_createBuildBot. That's why we reset all env vars here
# and require them to be passed as parameters to the script.

PROJECT_ID=""
Expand Down
57 changes: 38 additions & 19 deletions jenkins/agent-base/Dockerfile.centos7
Expand Up @@ -10,29 +10,43 @@ ENV SONAR_SCANNER_VERSION=3.1.0.1141 \
HELM_PLUGIN_SECRETS_VERSION=3.3.5 \
GIT_LFS_VERSION=2.6.1 \
SKOPEO_VERSION=0.1.37-3 \
OSTREE_VERSION=2018.5-1
OSTREE_VERSION=2018.5-1 \
JNLP_JAVA_OPTIONS="-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true"

ARG APP_DNS
ARG SNYK_DISTRIBUTION_URL
ARG AQUASEC_SCANNERCLI_URL

RUN yum -y install \
openssl \
&& yum clean all \
&& rm -rf /var/cache/yum/*

ENV JAVA_HOME=/usr/lib/jvm/jre
# Workaroud we use when running docker build behind proxy
# Basically we put the proxy certificates in certs folder
# COPY certs/* /etc/pki/ca-trust/source/anchors/
# RUN update-ca-trust force-enable && update-ca-trust extract

RUN yum -y install java-1.8.0-openjdk-devel.x86_64 \
COPY ensure_java_jre_is_adequate.sh /usr/local/bin/
RUN rm -fv /etc/yum.repos.d/CentOS-Media.repo /etc/yum.repos.d/origin-local-release.repo \
&& ensure_java_jre_is_adequate.sh \
&& yum -y install openssl \
&& yum -y update \
&& yum clean all \
&& rm -rf /var/cache/yum/*

# Copy use java scripts.
COPY use-j*.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/use-j*.sh && \
chmod ugo+s /usr/local/bin/use-j*.sh && \
sh -c 'chmod ugo+s $(which alternatives)' && \
ls -la /usr/local/bin/use-j*.sh && \
echo "--- STARTS JDK 11 TESTS ---" && \
use-j11.sh && \
echo "--- ENDS JDK 11 TESTS ---"

COPY ./import_certs.sh /usr/local/bin/import_certs.sh
RUN import_certs.sh
COPY ./fix_java_certs_permissions.sh /usr/local/bin/fix_java_certs_permissions.sh
RUN import_certs.sh && fix_java_certs_permissions.sh

# Install Sonar Scanner.
RUN cd /tmp \
&& curl -LOv https://repo1.maven.org/maven2/org/sonarsource/scanner/cli/sonar-scanner-cli/${SONAR_SCANNER_VERSION}/sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& curl -sSLOv https://repo1.maven.org/maven2/org/sonarsource/scanner/cli/sonar-scanner-cli/${SONAR_SCANNER_VERSION}/sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& unzip sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& mv sonar-scanner-${SONAR_SCANNER_VERSION} /usr/local/sonar-scanner-cli \
&& rm -rf sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
Expand All @@ -41,27 +55,26 @@ ENV PATH=/usr/local/sonar-scanner-cli/bin:$PATH

# Add sq cnes report jar.
RUN cd /tmp \
&& curl -Lv https://github.com/cnescatlab/sonar-cnes-report/releases/download/${CNES_REPORT_VERSION}/sonar-cnes-report-${CNES_REPORT_VERSION}.jar -o cnesreport.jar \
&& curl -sSLv https://github.com/cnescatlab/sonar-cnes-report/releases/download/${CNES_REPORT_VERSION}/sonar-cnes-report-${CNES_REPORT_VERSION}.jar -o cnesreport.jar \
&& mkdir /usr/local/cnes \
&& mv cnesreport.jar /usr/local/cnes/cnesreport.jar \
&& chmod 777 /usr/local/cnes/cnesreport.jar

# Install Tailor.
RUN cd /tmp \
&& curl -LOv https://github.com/opendevstack/tailor/releases/download/v${TAILOR_VERSION}/tailor-linux-amd64 \
&& curl -sSLOv https://github.com/opendevstack/tailor/releases/download/v${TAILOR_VERSION}/tailor-linux-amd64 \
&& mv tailor-linux-amd64 /usr/local/bin/tailor \
&& chmod a+x /usr/local/bin/tailor \
&& tailor version

# Install Helm.
RUN cd /tmp \
&& mkdir -p /tmp/helm \
&& curl -LO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz \
&& curl -sSLO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz \
&& tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz -C /tmp/helm \
&& mv /tmp/helm/linux-amd64/helm /usr/local/bin/helm \
&& chmod a+x /usr/local/bin/helm \
&& helm version \
&& helm env \
&& helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION} \
&& helm plugin install https://github.com/jkroepke/helm-secrets --version v${HELM_PLUGIN_SECRETS_VERSION} \
&& sops --version \
Expand All @@ -70,15 +83,15 @@ RUN cd /tmp \
# Install GIT-LFS extension https://git-lfs.github.com/.
RUN cd /tmp \
&& mkdir -p /tmp/git-lfs \
&& curl -LOv https://github.com/git-lfs/git-lfs/releases/download/v${GIT_LFS_VERSION}/git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz \
&& curl -sSLOv https://github.com/git-lfs/git-lfs/releases/download/v${GIT_LFS_VERSION}/git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz \
&& tar -zxvf git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz -C /tmp/git-lfs \
&& bash /tmp/git-lfs/install.sh \
&& git lfs version \
&& rm -rf /tmp/git-lfs*

# Optionally install snyk.
RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ; else echo 'Installing snyk... getting binary from' $SNYK_DISTRIBUTION_URL \
&& curl -Lv $SNYK_DISTRIBUTION_URL --output snyk \
&& curl -sSLv $SNYK_DISTRIBUTION_URL --output snyk \
&& mv snyk /usr/local/bin \
&& chmod +rwx /usr/local/bin/snyk \
&& mkdir -p $HOME/.config/configstore/ \
Expand All @@ -90,7 +103,7 @@ RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ;

# Optionally install Aquasec.
RUN if [ -z $AQUASEC_SCANNERCLI_URL ] ; then echo 'Skipping AquaSec installation!' ; else echo 'Installing AquaSec... getting binary from' $AQUASEC_SCANNERCLI_URL \
&& wget $AQUASEC_SCANNERCLI_URL -O aquasec \
&& curl -sSL $AQUASEC_SCANNERCLI_URL --output aquasec \
&& mv aquasec /usr/local/bin \
&& chmod +rwx /usr/local/bin/aquasec \
&& echo 'AquaSec CLI version:' \
Expand All @@ -102,7 +115,14 @@ RUN if [ -z $AQUASEC_SCANNERCLI_URL ] ; then echo 'Skipping AquaSec installation
COPY set_java_proxy.sh /tmp/set_java_proxy.sh
RUN . /tmp/set_java_proxy.sh && echo $JAVA_OPTS

RUN mv /usr/local/bin/run-jnlp-client /usr/local/bin/openshift-run-jnlp-client
# The following line fix incorrect behaviours in the base image.
# It is setting the variable JAVA_TOOL_OPTIONS while it should not.
# Besides, we need to know if this variable has not been set.
# It is a problem very difficult to detect...
COPY fix_openshift_run_jnlp_client.sh /usr/local/bin/fix_openshift_run_jnlp_client.sh
RUN mv /usr/local/bin/run-jnlp-client /usr/local/bin/openshift-run-jnlp-client \
&& fix_openshift_run_jnlp_client.sh /usr/local/bin/openshift-run-jnlp-client

COPY ods-run-jnlp-client.sh /usr/local/bin/run-jnlp-client

# Add skopeo.
Expand All @@ -119,4 +139,3 @@ RUN mkdir -p /home/jenkins/.config && chmod -R g+w /home/jenkins/.config \
&& mkdir -p /home/jenkins/.cache && chmod -R g+w /home/jenkins/.cache \
&& mkdir -p /home/jenkins/.sonar && chmod -R g+w /home/jenkins/.sonar

RUN chmod g+w $JAVA_HOME/lib/security/cacerts
42 changes: 31 additions & 11 deletions jenkins/agent-base/Dockerfile.ubi8
Expand Up @@ -9,7 +9,7 @@ ENV SONAR_SCANNER_VERSION=3.1.0.1141 \
HELM_PLUGIN_DIFF_VERSION=3.3.2 \
HELM_PLUGIN_SECRETS_VERSION=3.3.5 \
GIT_LFS_VERSION=2.6.1 \
JAVA_HOME=/usr/lib/jvm/jre
JNLP_JAVA_OPTIONS="-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true"

ARG APP_DNS
ARG SNYK_DISTRIBUTION_URL
Expand All @@ -18,17 +18,35 @@ ARG AQUASEC_SCANNERCLI_URL
# Add UBI repositories.
COPY yum.repos.d/ubi.repo /etc/yum.repos.d/ubi.repo

COPY ensure_java_jre_is_adequate.sh /usr/local/bin/
RUN cd /etc/yum.repos.d && rm -f localdev-* ci-rpm-mirrors.repo \
&& ensure_java_jre_is_adequate.sh \
&& yum -y install make glibc-langpack-en openssl \
&& yum -y update \
&& yum clean all \
&& rm -rf /var/cache/yum/*

#
# WARNING: We do not install java 8 nor java 11 in this image because they are already intalled in it.
#

# Copy use java scripts.
COPY use-j*.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/use-j*.sh && \
chmod ugo+s /usr/local/bin/use-j*.sh && \
sh -c 'chmod ugo+s $(which alternatives)' && \
ls -la /usr/local/bin/use-j*.sh && \
echo "--- STARTS JDK 11 TESTS ---" && \
use-j11.sh && \
echo "--- ENDS JDK 11 TESTS ---"

COPY ./import_certs.sh /usr/local/bin/import_certs.sh
RUN import_certs.sh
COPY ./fix_java_certs_permissions.sh /usr/local/bin/fix_java_certs_permissions.sh
RUN import_certs.sh && fix_java_certs_permissions.sh

# Install Sonar Scanner.
RUN cd /tmp \
&& curl -LO https://repo1.maven.org/maven2/org/sonarsource/scanner/cli/sonar-scanner-cli/${SONAR_SCANNER_VERSION}/sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& curl -sSLO https://repo1.maven.org/maven2/org/sonarsource/scanner/cli/sonar-scanner-cli/${SONAR_SCANNER_VERSION}/sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& unzip sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
&& mv sonar-scanner-${SONAR_SCANNER_VERSION} /usr/local/sonar-scanner-cli \
&& rm -rf sonar-scanner-cli-${SONAR_SCANNER_VERSION}.zip \
Expand All @@ -37,22 +55,22 @@ ENV PATH=/usr/local/sonar-scanner-cli/bin:$PATH

# Add sq cnes report jar.
RUN cd /tmp \
&& curl -L https://github.com/cnescatlab/sonar-cnes-report/releases/download/${CNES_REPORT_VERSION}/sonar-cnes-report-${CNES_REPORT_VERSION}.jar -o cnesreport.jar \
&& curl -sSL https://github.com/cnescatlab/sonar-cnes-report/releases/download/${CNES_REPORT_VERSION}/sonar-cnes-report-${CNES_REPORT_VERSION}.jar -o cnesreport.jar \
&& mkdir /usr/local/cnes \
&& mv cnesreport.jar /usr/local/cnes/cnesreport.jar \
&& chmod 777 /usr/local/cnes/cnesreport.jar

# Install Tailor.
RUN cd /tmp \
&& curl -LO https://github.com/opendevstack/tailor/releases/download/v${TAILOR_VERSION}/tailor-linux-amd64 \
&& curl -sSLO https://github.com/opendevstack/tailor/releases/download/v${TAILOR_VERSION}/tailor-linux-amd64 \
&& mv tailor-linux-amd64 /usr/local/bin/tailor \
&& chmod a+x /usr/local/bin/tailor \
&& tailor version

# Install Helm.
RUN cd /tmp \
&& mkdir -p /tmp/helm \
&& curl -LO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz \
&& curl -sSLO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz \
&& tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz -C /tmp/helm \
&& mv /tmp/helm/linux-amd64/helm /usr/local/bin/helm \
&& chmod a+x /usr/local/bin/helm \
Expand All @@ -66,15 +84,15 @@ RUN cd /tmp \
# Install GIT-LFS extension https://git-lfs.github.com/.
RUN cd /tmp \
&& mkdir -p /tmp/git-lfs \
&& curl -LO https://github.com/git-lfs/git-lfs/releases/download/v${GIT_LFS_VERSION}/git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz \
&& curl -sSLO https://github.com/git-lfs/git-lfs/releases/download/v${GIT_LFS_VERSION}/git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz \
&& tar -zxvf git-lfs-linux-amd64-v${GIT_LFS_VERSION}.tar.gz -C /tmp/git-lfs \
&& bash /tmp/git-lfs/install.sh \
&& git lfs version \
&& rm -rf /tmp/git-lfs*

# Optionally install snyk.
RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ; else echo 'Installing snyk... getting binary from' $SNYK_DISTRIBUTION_URL \
&& curl -L $SNYK_DISTRIBUTION_URL --output snyk \
&& curl -sSL $SNYK_DISTRIBUTION_URL --output snyk \
&& mv snyk /usr/local/bin \
&& chmod +rwx /usr/local/bin/snyk \
&& mkdir -p $HOME/.config/configstore/ \
Expand All @@ -86,7 +104,7 @@ RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ;

# Optionally install Aquasec.
RUN if [ -z $AQUASEC_SCANNERCLI_URL ] ; then echo 'Skipping AquaSec installation!' ; else echo 'Installing AquaSec... getting binary from' $AQUASEC_SCANNERCLI_URL \
&& wget $AQUASEC_SCANNERCLI_URL -O aquasec \
&& curl -sSL $AQUASEC_SCANNERCLI_URL --output aquasec \
&& mv aquasec /usr/local/bin \
&& chmod +rwx /usr/local/bin/aquasec \
&& echo 'AquaSec CLI version:' \
Expand All @@ -99,7 +117,10 @@ COPY set_java_proxy.sh /tmp/set_java_proxy.sh
RUN . /tmp/set_java_proxy.sh && echo $JAVA_OPTS

# Customize entrypoint.
RUN mv /usr/local/bin/run-jnlp-client /usr/local/bin/openshift-run-jnlp-client
COPY fix_openshift_run_jnlp_client.sh /usr/local/bin/fix_openshift_run_jnlp_client.sh
RUN mv /usr/local/bin/run-jnlp-client /usr/local/bin/openshift-run-jnlp-client \
&& fix_openshift_run_jnlp_client.sh /usr/local/bin/openshift-run-jnlp-client

COPY ods-run-jnlp-client.sh /usr/local/bin/run-jnlp-client

# Add skopeo.
Expand All @@ -113,4 +134,3 @@ RUN mkdir -p /home/jenkins/.config && chmod -R g+w /home/jenkins/.config \
&& mkdir -p /home/jenkins/.cache && chmod -R g+w /home/jenkins/.cache \
&& mkdir -p /home/jenkins/.sonar && chmod -R g+w /home/jenkins/.sonar

RUN chmod g+w $JAVA_HOME/lib/security/cacerts
67 changes: 67 additions & 0 deletions jenkins/agent-base/ensure_java_jre_is_adequate.sh
@@ -0,0 +1,67 @@
#!/bin/bash
set -eu -o pipefail

ME="$(basename $0)"
JAVA_INSTALLED_PKGS_LOGS="/tmp/java_installed_pkgs.log"
JAVA_11_INSTALLED_PKGS_LOGS="/tmp/java_11_installed_pkgs.log"
rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}

NEEDS_DEVEL=${1-""}
PKG_NAME_TAIL="headless"
if [ ! -z "${NEEDS_DEVEL}" ] && [ "" != "${NEEDS_DEVEL}" ]; then
NEEDS_DEVEL="true"
PKG_NAME_TAIL="devel"
else
NEEDS_DEVEL="false"
PKG_NAME_TAIL="headless"
fi

echo "${ME}: Needs development packages? ${NEEDS_DEVEL}"
echo " "
echo "${ME}: Listing versions of java installed: "
yum list installed | grep -i "\(java\|jre\)" | tee -a ${JAVA_INSTALLED_PKGS_LOGS}
touch ${JAVA_11_INSTALLED_PKGS_LOGS}
grep -i "java-11" ${JAVA_INSTALLED_PKGS_LOGS} > ${JAVA_11_INSTALLED_PKGS_LOGS} || echo "No java 11 packages found."

NEEDS_INSTALLATION="true"
if [ -f ${JAVA_11_INSTALLED_PKGS_LOGS} ]; then
if grep -qi "${PKG_NAME_TAIL}" ${JAVA_11_INSTALLED_PKGS_LOGS} ; then
NEEDS_INSTALLATION="false"
fi
fi

# We need devel package in masters to have jar binary.
if [ "true" == "${NEEDS_INSTALLATION}" ]; then
echo "${ME}:Java-11 is *not* installed. Installing..."
if [ "true" == "${NEEDS_DEVEL}" ]; then
yum -y install java-11-openjdk-devel
else
yum -y install java-11-openjdk-headless
fi
else
echo "${ME}: Java-11 is already installed."
fi

if grep -qi "java-1.8" ${JAVA_INSTALLED_PKGS_LOGS} ; then
echo "${ME}: Java-8 is installed. Removing..."
yum -y remove java-1.8*
else
echo "${ME}: Java-8 is not installed. Correct."
fi

rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}

echo " "
echo "${ME}: Checking java tool versions: "
if [ "true" == "${NEEDS_DEVEL}" ]; then
jar --version
fi

NO_JAVA_LINK="false"
java -version || NO_JAVA_LINK="true"
if [ "true" == "${NO_JAVA_LINK}" ]; then
JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-11-openjdk-11.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_HOME="/usr/lib/jvm/${JAVA_HOME_FOLDER}"
alternatives --set java ${JAVA_HOME}/bin/java
fi
java -version

0 comments on commit 005ed8b

Please sign in to comment.