From 1133d0a2790c0247a8f3540212b02d27358e2e0a Mon Sep 17 00:00:00 2001 From: Yuxiang Zhu Date: Aug 09 2018 10:40:37 +0000 Subject: Add dev pipeline to run CI/CD on OpenShift This PR includes OpenShift Templates to produce a new WaiverDB dev CI/CD pipeline in OpenShift. This dev build/test/deploy Pipeline migrates the the existing Jenkinsfile with some procedure changes to match the C3I Phase 3. When starting a new pipeline build, the Jenkins agent and whole testing environment will be dynamically created as OpenShift pods. Concurrent builds are also allowed (but currently disabled) to increase the overall throughput. Dev pipeline is a part of the WaiverDB Pipeline, covers the following steps: - Run Flake8 and Pylint checks - Run unit tests - Build Docs - Publish Docs - Build SRPM - Build RPM - Invoke Rpmlint - Build container - Run functional tests - Push container Instructions are located at `openshift/README.md`. --- diff --git a/openshift/README.md b/openshift/README.md new file mode 100644 index 0000000..8301f72 --- /dev/null +++ b/openshift/README.md @@ -0,0 +1,127 @@ +# OpenShift Manifests for CI/CD Pipelines and Deployment + +This directory includes OpenShift manifests and Dockerfiles that can be used +to deploy a CI/CD pipeline and a test environment for WaiverDB. + +## WaiverDB Pipeline + +WaiverDB Pipeline gives you control over building, testing, deploying, and promoting WaiverDB on OpenShift. + +It uses [OpenShift Pipeline][], a technology powered by OpenShift, using a combination of the [Jenkins Pipeline Build Strategy][], [Jenkinsfiles][], and the OpenShift Domain Specific Language (DSL) powered by the [OpenShift Jenkins Pipeline (DSL) Plugin][] (previously called the OpenShift Jenkins Client Plugin). + +When starting a new pipeline build, the Jenkins slave and the whole testing environment will be dynamically created as OpenShift pods. And after the build is complete, all of them will be destroyed. Concurrent builds are also allowed (but currently disabled) to increase the overall throughput. + +Before using the Pipeline, please ensure your Jenkins master has the following plugins installed and configured: +- [Openshift Sync plugin][] [1] +- [Openshift Client plugin][OpenShift Jenkins Pipeline (DSL) Plugin] [1] +- [Kubernetes plugin][] [1] +- [SSH Agent plugin][]: + +Notes: +[1]: Those plugins are preinstalled if you are using the Jenkins master shipped with OpenShift. + +### Dev Pipeline +Dev Pipeline is a part of the WaiverDB Pipeline, covers the following steps: + +- Run Flake8 and Pylint checks +- Run unit tests +- Build Docs +- Publish Docs +- Build SRPM +- Build RPM +- Invoke Rpmlint +- Build container +- Run functional tests +- Push container + +Whenever a new Git commit comes in, the dev pipeline is run against the commit to ensure that this commit can be merged into mainline, and a container is built and pushed to the specified registry. + +#### Installation +To install the pipeline to a project on OpenShift, just login to your +OpenShift cloud, switch to a project (I would suggest using a separated +project instead of sharing the real dev/stage/prod environment): + +```bash + oc login + oc project waiverdb-test # your project name +``` + +Then instantiate the template with default parameters +(this will set up a pipeline to build the upstream repo and publish the result to upstream registries): + +```bash + oc process --local -f ./waiverdb-dev-pipeline.yml | oc apply -f - +``` + +Or you want to set up a pipeline for your own fork: + +```bash + NAME=waiverdb-dev-pipeline + oc process --local -f ./waiverdb-dev-pipeline-template.yml \ + -p NAME="$NAME" \ + -p WAIVERDB_GIT_REPO="https://pagure.io/forks//waiverdb.git" \ + -p WAIVERDB_GIT_REF="" \ + -p WAIVERDB_DEV_IMAGE_DESTINATIONS="docker.io//waiverdb:latest" \ + -p PAGURE_DOC_REPO_NAME="forks//waiverdb" \ + | oc apply -f - +``` + +#### Configure Secrets for Push Containers +This section is optional if publishing containers is not needed. + +To publish the container built by the pipeline, you need to set up a secret for your target container registries. + +- Please go to your registry dashboard and create a robot account. +- Backup your docker-config-json file (`$HOME/.docker/config.json`) if present. +- Run `docker login` with the robot account you just created to produce a new docker-config-json file. +- Create a new [OpenShift secret for registries][] named `factory2-pipeline-registry-credentials` from your docker-config-json file: +```bash + oc create secret generic dockerhub \ + --from-file=.dockerconfigjson="$HOME/.docker/config.json" \ + --type=kubernetes.io/dockerconfigjson +``` + +#### Configure Secrets for Publishing Pagure Docs +This section is optional if publishing Pagure Docs is not needed. + +- Please go to your Pagure repository settings and activate the project documentation. +- Create an SSH key pair for publishing Pagure Docs and show the public key: +```bash +ssh-keygen -f "$HOME/.ssh/id_rsa_pagure" +cat "$HOME/.ssh/id_rsa_pagure.pub" +``` +- Go to your Pagure repository settings, and add the public key as the deploy key. +- Add the private key to your OpenShift project by creating a new [OpenShift secret for SSH key authentication][]: +```bash + oc create secret generic pagure-doc-secret \ + --from-file=ssh-privatekey="$HOME/.ssh/id_rsa_pagure" \ + --type=kubernetes.io/ssh-auth + # This label is used by Jenkins OpenShift Sync Plugin for synchonizing OpenShift secrets with Jenkins Credentials. + oc label secret pagure-doc-secret credential.sync.jenkins.openshift.io=true +``` + +#### Build Jenkins slave container image +Before running the pipeline, you need to build a container image for Jenkins slave pods. +This step should be repeated every time you change +the Dockerfile for the Jenkins slave pods. +```bash + oc start-build waiverdb-dev-pipeline-jenkins-slave +``` + +#### Trigger A Pipeline Build +To trigger a pipeline build, start the `waiverdb-dev-pipeline` BuildConfig with the Git commit ID or branch that you want to +test against: +```bash + oc start-build waiverdb-dev-pipeline -e WAIVERDB_GIT_REF= +``` +You can go to the OpenShift Web console to check the details of the pipeline build. + +[OpenShift Pipeline]: https://docs.okd.io/3.9/dev_guide/openshift_pipeline.html +[Jenkins Pipeline Build Strategy]: https://docs.openshift.com/container-platform/3.9/dev_guide/dev_tutorials/openshift_pipeline.html +[Jenkinsfiles]: https://jenkins.io/doc/book/pipeline/jenkinsfile/ +[OpenShift Jenkins Pipeline (DSL) Plugin]: https://github.com/openshift/jenkins-client-plugin +[Openshift Sync plugin]: https://github.com/openshift/jenkins-sync-plugin +[Kubernetes plugin]: https://github.com/jenkinsci/kubernetes-plugin +[SSH Agent plugin]: https://github.com/jenkinsci/ssh-agent-plugin +[OpenShift secret for registries]:https://docs.openshift.com/container-platform/3.9/dev_guide/builds/build_inputs.html#using-docker-credentials-for-private-registries +[OpenShift secret for SSH key authentication]: https://docs.openshift.com/container-platform/3.9/dev_guide/builds/build_inputs.html#source-secrets-ssh-key-authentication diff --git a/openshift/containers/jenkins-slave/Dockerfile b/openshift/containers/jenkins-slave/Dockerfile new file mode 100644 index 0000000..e90e29b --- /dev/null +++ b/openshift/containers/jenkins-slave/Dockerfile @@ -0,0 +1,46 @@ +#FIXME: The Pipeline is currently using this Dockerfile to produce images. The one located at the project root is not changed in case of breaking anything. +FROM docker-registry.engineering.redhat.com/devops-automation/rad-slave-fedora-27:latest +LABEL name="waiverdb-jenkins-slave" \ + description="Jenkins slave for WaiverDB dev tests" \ + vendor="WaiverDB Developers" \ + license="GPLv2+" + +USER root + +# Add dependency Tini, a tiny but valid init for containers +ARG TINI_VERSION=0.18.0 +# OpenShift Jenkins Pipeline (DSL) Plugin requires oc binary to be present in the PATH environment variable of all Jenkins nodes +ARG OPENSHIFT_CLIENTS_VERSION=3.10.0 +ARG OPENSHIFT_CLIENTS_COMMIT=dd10d17 + +RUN dnf -y install 'dnf-command(builddep)' dnf-utils git mock-core-configs tar gzip skopeo \ + wget postgresql make rpmdevtools rpmlint \ + python3-flake8 python3-pylint python3-pytest \ + python3-sphinx python3-sphinxcontrib-httpdomain \ + && mkdir -p /usr/local/src/waiverdb \ + && TEMPDIR=$(mktemp -d) \ + # install oc + && wget -O "$TEMPDIR"/openshift-origin-client-tools.tar \ + "https://github.com/openshift/origin/releases/download/v${OPENSHIFT_CLIENTS_VERSION}/openshift-origin-client-tools-v${OPENSHIFT_CLIENTS_VERSION}-${OPENSHIFT_CLIENTS_COMMIT}-linux-64bit.tar.gz" \ + && tar -xvf "$TEMPDIR"/openshift-origin-client-tools.tar -C "$TEMPDIR" --strip-components=1 \ + && cp "$TEMPDIR"/oc /usr/local/bin/oc && chmod +rx /usr/local/bin/oc \ + # install tini + && wget -O /usr/local/bin/tini "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini" \ + && chmod +rx /usr/local/bin/tini \ + # install wait-for-it.sh, to allow containers to wait for other services to come up + && wget -O /usr/local/bin/wait-for-it "https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh" \ + && chmod +rx /usr/local/bin/tini /usr/local/bin/wait-for-it \ + # clean up + && rm -rf "$TEMPDIR" \ + && dnf clean all + +# install build dependencies for WaiverDB +COPY waiverdb.spec /usr/local/src/waiverdb/waiverdb.spec +RUN cd /usr/local/src/waiverdb \ + && dnf -y builddep waiverdb.spec \ + && dnf clean all \ + && chgrp -R root /usr/local/src/waiverdb \ + && chmod -R g+rwX /usr/local/src/waiverdb +WORKDIR /var/lib/jenkins/ +ENTRYPOINT ["/usr/local/bin/tini", "--", "jenkins-slave"] +USER 1000 diff --git a/openshift/containers/waiverdb/Dockerfile b/openshift/containers/waiverdb/Dockerfile new file mode 100644 index 0000000..2980337 --- /dev/null +++ b/openshift/containers/waiverdb/Dockerfile @@ -0,0 +1,56 @@ +FROM fedora:28 +LABEL \ + name="waiverdb" \ + description="WaiverDB application" \ + vendor="WaiverDB developers" \ + license="GPLv2+" \ + build-date="" + +# Installing WaiverDB dependencies +RUN dnf -y install python3-gunicorn \ + python3-flask \ + python3-sqlalchemy \ + python3-flask-restful \ + python3-flask-sqlalchemy \ + python3-psycopg2 \ + python3-gssapi \ + python3-mock \ + python3-flask-oidc \ + python3-click \ + python3-flask-migrate \ + python3-stomppy \ + python3-fedmsg \ + && dnf -y clean all + +ARG WAIVERDB_GIT_REPO=https://pagure.io/waiverdb.git +ARG WAIVERDB_GIT_REF=master +ARG WAIVERDB_CACERT_URL= +ARG WAIVERDB_VERSION= +ENV WAIVERDB_VERSION=$WAIVERDB_VERSION + +# Installing CA certificate +RUN if [ -n "$WAIVERDB_CACERT_URL" ]; then \ + cd /etc/pki/ca-trust/source/anchors \ + && curl -O --insecure --location "$WAIVERDB_CACERT_URL" \ + && update-ca-trust extract; \ + fi + +# Installing WaiverDB +RUN dnf -y install git python3-pip \ + && mkdir -p /usr/local/src \ + && git clone "$WAIVERDB_GIT_REPO" /usr/local/src/waiverdb \ + && cd /usr/local/src/waiverdb \ + && git fetch origin "$WAIVERDB_GIT_REF" \ + && git checkout -f "$WAIVERDB_GIT_REF" \ + && pip3 install --no-deps . \ + && mkdir -p /etc/waiverdb \ + && cp conf/settings.py.example /etc/waiverdb/settings.py \ + && cp conf/client.conf.example /etc/waiverdb/client.conf \ + && dnf -y history undo last \ + && dnf -y clean all \ + && cd / && rm -rf /usr/local/src/waiverdb + +USER 1001 +EXPOSE 8080 + +CMD ["/usr/bin/gunicorn-3", "--bind", "0.0.0.0:8080", "--access-logfile", "-", "--enable-stdio-inheritance", "waiverdb.wsgi:app"] diff --git a/openshift/waiverdb-container-template.yml b/openshift/waiverdb-container-template.yml new file mode 100644 index 0000000..06f434f --- /dev/null +++ b/openshift/waiverdb-container-template.yml @@ -0,0 +1,83 @@ +# Template to produce a new BuildConfig and ImageStream for WaiverDB container builds. + +--- +apiVersion: v1 +kind: Template +metadata: + name: waiverdb-container-template +labels: + template: "waiverdb-container-template" +parameters: +- name: NAME + displayName: Short unique identifier for the templated instances. + required: true + value: "waiverdb-container" +- name: WAIVERDB_GIT_REPO + displayName: WaiverDB Git repo URL + description: Default WaiverDB Git repo URL in which to run dev tests against + required: true + value: https://pagure.io/waiverdb.git +- name: WAIVERDB_GIT_REF + displayName: WaiverDB Git repo ref + description: Default WaiverDB Git repo ref in which to run dev tests against + required: true + value: master +- name: WAIVERDB_IMAGESTREAM_NAME + displayName: ImageStream name of the resulting image + required: true + value: waiverdb +- name: WAIVERDB_IMAGE_TAG + displayName: Tag of resulting image + required: true + value: latest +- name: WAIVERDB_IMAGESTREAM_NAMESPACE + displayName: Namespace of ImageStream for WaiverDB container images + required: false +- name: WAIVERDB_VERSION + displayName: WaiverDB app version + required: false + value: '' +objects: +- apiVersion: v1 + kind: ImageStream + metadata: + name: "${WAIVERDB_IMAGESTREAM_NAME}" + labels: + app: "${NAME}" +- kind: "BuildConfig" + apiVersion: "v1" + metadata: + name: "${NAME}" + metadata: + labels: + app: "${NAME}" + spec: + runPolicy: "Parallel" + completionDeadlineSeconds: 1800 + strategy: + dockerStrategy: + forcePull: true + buildArgs: + - name: "WAIVERDB_GIT_REPO" + value: "${WAIVERDB_GIT_REPO}" + - name: "WAIVERDB_GIT_REF" + value: "${WAIVERDB_GIT_REF}" + - name: "WAIVERDB_VERSION" + value: "${WAIVERDB_VERSION}" + dockerfilePath: openshift/containers/waiverdb/Dockerfile + resources: + requests: + memory: "768Mi" + cpu: "300m" + limits: + memory: "1Gi" + cpu: "500m" + source: + git: + uri: "${WAIVERDB_GIT_REPO}" + ref: "${WAIVERDB_GIT_REF}" + output: + to: + kind: "ImageStreamTag" + name: "${WAIVERDB_IMAGESTREAM_NAME}:${WAIVERDB_IMAGE_TAG}" + namespace: "${WAIVERDB_IMAGESTREAM_NAMESPACE}" diff --git a/openshift/waiverdb-dev-pipeline-template.yml b/openshift/waiverdb-dev-pipeline-template.yml new file mode 100644 index 0000000..49241e2 --- /dev/null +++ b/openshift/waiverdb-dev-pipeline-template.yml @@ -0,0 +1,629 @@ +# Template to produce a new WaiverDB dev CI/CD pipeline in OpenShift. +# +# Dev pipeline is a part of the WaiverDB Pipeline, covers the following steps: +# +# - Run Flake8 and Pylint checks +# - Run unit tests +# - Build Docs +# - Publish Docs +# - Build SRPM +# - Build RPM +# - Invoke Rpmlint +# - Build container +# - Run functional tests +# - Push container +# +# Required Jenkins Plugins: +# - Openshift Sync plugin +# - Openshift Client plugin +# - Kubernetes plugin +# - SSH Agent plugin +# - Timestamper plugin +# +--- +apiVersion: v1 +kind: Template +metadata: + name: waiverdb-dev-pipeline +parameters: +- name: NAME + displayName: Short unique identifier for the templated instances + description: This field is used to deploy multiple pipelines to one OpenShift project from this template. + required: true + value: waiverdb-dev-pipeline +- name: WAIVERDB_GIT_REPO + displayName: WaiverDB Git repo URL + description: Default WaiverDB Git repo URL in which to run dev tests against + required: true + value: "https://pagure.io/waiverdb.git" +- name: WAIVERDB_GIT_REF + displayName: WaiverDB Git repo ref + description: Default WaiverDB Git repo ref in which to run dev tests against + required: true + value: master +- name: WAIVERDB_MAIN_BRANCH + displayName: Name of the main branch. + description: If WAIVERDB_MAIN_BRANCH equals WAIVERDB_GIT_REF, publishing steps will be automatically performed. + value: master + required: true +- name: JENKINS_AGENT_CLOUD_NAME + displayName: Name of OpenShift cloud in Jenkins master configuration + required: true + value: openshift +- name: JENKINS_AGENT_IMAGE + displayName: Container image for Jenkins slave pods + required: true + value: docker-registry.engineering.redhat.com/factory2/waiverdb-jenkins-slave:latest +- name: PAGURE_DOC_REPO_NAME + displayName: namespace/project of Pagure doc repo for publishing docs + description: If not emptry and WAIVERDB_GIT_REF is master, docs will be published to the specified Pagure doc repo + required: false + value: waiverdb +- name: PAGURE_DOC_SECRET + displayName: Name of the OpenShift SSH secret for publishing docs to Pagure. + required: false + value: pagure-waiverdb-deploy-key +- name: WAIVERDB_DEV_IMAGE_DESTINATIONS + displayName: Comma seperated list of container repository/tag to which the built WaiverDB dev image will be pushed + description: OpenShift registries must be prefixed with 'atomic:' + required: false + value: "atomic:docker-registry.engineering.redhat.com/factory2/waiverdb:latest,quay.io/factory2/waiverdb:latest" +- name: CONTAINER_REGISTRY_CREDENTIALS + displayName: Secret name of container registries used for pulling and pushing images + value: factory2-pipeline-registry-credentials + required: false +- name: WAIVERDB_DEV_IMAGE_TAG + displayName: Tag name of the resulting container image for development environment + value: "latest" + required: true +- name: WAIVERDB_IMAGESTREAM_NAME + displayName: Name of ImageStream for WaiverDB container images + required: true + value: waiverdb +- name: WAIVERDB_IMAGESTREAM_NAMESPACE + displayName: Namespace of ImageStream for WaiverDB container images + required: false +- name: FORCE_PUBLISH_IMAGE + displayName: Whether to push the resulting image regardless of the Git branch + value: "false" + required: true +- name: FORCE_PUBLISH_DOCS + displayName: Whether to publish docs regardless of the Git branch + value: "false" + required: true +- name: TAG_INTO_IMAGESTREAM + displayName: Whether to tag the pushed image as dev + value: "true" + required: true +labels: + template: waiverdb-dev-pipeline +objects: +- kind: "BuildConfig" + apiVersion: "v1" + metadata: + name: "${NAME}-jenkins-slave" + labels: + app: "${NAME}" + spec: + runPolicy: "Serial" + completionDeadlineSeconds: 1800 + strategy: + dockerStrategy: + forcePull: true + dockerfilePath: openshift/containers/jenkins-slave/Dockerfile + resources: + requests: + memory: "512Mi" + cpu: "300m" + limits: + memory: "768Mi" + cpu: "500m" + source: + git: + uri: "${WAIVERDB_GIT_REPO}" + ref: "${WAIVERDB_GIT_REF}" + output: + to: + kind: "DockerImage" + name: "${JENKINS_AGENT_IMAGE}" + pushSecret: + name: "${CONTAINER_REGISTRY_CREDENTIALS}" + +- kind: ServiceAccount + apiVersion: v1 + metadata: + name: "${NAME}-jenkins-slave" + labels: + app: "${NAME}" + +- kind: RoleBinding + apiVersion: v1 + metadata: + name: "${NAME}-jenkins-slave_edit" + labels: + app: "${NAME}" + subjects: + - kind: ServiceAccount + name: "${NAME}-jenkins-slave" + roleRef: + name: edit + +- kind: "BuildConfig" + apiVersion: "v1" + metadata: + name: "${NAME}" + labels: + app: "${NAME}" + spec: + runPolicy: "Serial" # FIXME: Parallel is supported, but we have limited quota in UpShift. + completionDeadlineSeconds: 1800 + strategy: + type: JenkinsPipeline + source: + type: None + jenkinsPipelineStrategy: + env: + - name: "WAIVERDB_GIT_REPO" + value: "${WAIVERDB_GIT_REPO}" + - name: "WAIVERDB_GIT_REF" + value: "${WAIVERDB_GIT_REF}" + - name: "JENKINS_AGENT_CLOUD_NAME" + value: "${JENKINS_AGENT_CLOUD_NAME}" + - name: "JENKINS_AGENT_SERVICE_ACCOUNT" + value: "${NAME}-jenkins-slave" + - name: "WAIVERDB_DEV_IMAGE_DESTINATIONS" + value: "${WAIVERDB_DEV_IMAGE_DESTINATIONS}" + - name: "FORCE_PUBLISH_IMAGE" + value: "${FORCE_PUBLISH_IMAGE}" + - name: "FORCE_PUBLISH_DOCS" + value: "${FORCE_PUBLISH_DOCS}" + - name: "TAG_INTO_IMAGESTREAM" + value: "${TAG_INTO_IMAGESTREAM}" + jenkinsfile: |- + pipeline { + agent { + kubernetes { + cloud params.JENKINS_AGENT_CLOUD_NAME + label "jenkins-slave-${UUID.randomUUID().toString()}" + serviceAccount params.JENKINS_AGENT_SERVICE_ACCOUNT + defaultContainer 'jnlp' + yaml """ + apiVersion: v1 + kind: Pod + metadata: + labels: + app: "${NAME}" + factory2-pipeline-kind: "waiverdb-dev-pipeline" + factory2-pipeline-build-number: "${env.BUILD_NUMBER}" + spec: + containers: + - name: jnlp + image: "${JENKINS_AGENT_IMAGE}" + imagePullPolicy: Always + tty: true + env: + - name: REGISTRY_CREDENTIALS + valueFrom: + secretKeyRef: + name: "${CONTAINER_REGISTRY_CREDENTIALS}" + key: '.dockerconfigjson' + # Set up NSS Wrapper to generate a fake user name for the random UID assigned by OpenShift + - name: LD_PRELOAD + value: '/usr/lib64/libnss_wrapper.so' + - name: NSS_WRAPPER_PASSWD + value: '/tmp/passwd' + - name: NSS_WRAPPER_GROUP + value: '/etc/group' + imagePullSecrets: + - "${CONTAINER_REGISTRY_CREDENTIALS}" + volumeMounts: + - name: postgresql-socket + mountPath: /var/run/postgresql + resources: + requests: + memory: 768Mi + cpu: 300m + limits: + memory: 1Gi + cpu: 500m + - name: db + image: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest + imagePullPolicy: Always + imagePullSecrets: + - "${CONTAINER_REGISTRY_CREDENTIALS}" + env: + - name: POSTGRESQL_USER + value: waiverdb + - name: POSTGRESQL_PASSWORD + value: waiverdb + - name: POSTGRESQL_DATABASE + value: waiverdb + volumeMounts: + - name: postgresql-socket + mountPath: /var/run/postgresql + resources: + requests: + memory: 256Mi + cpu: 100m + limits: + memory: 384Mi + cpu: 200m + volumes: + - name: postgresql-socket + emptyDir: {} + """ + } + } + options { + timestamps() + timeout(time: 30, unit: 'MINUTES') + } + stages { + stage('Prepare') { + steps { + checkout([$class: 'GitSCM', + branches: [[name: params.WAIVERDB_GIT_REF]], + userRemoteConfigs: [[url: params.WAIVERDB_GIT_REPO]], + ]) + script { + env.WAIVERDB_GIT_COMMIT_ID = sh(returnStdout: true, script: 'git rev-parse HEAD').trim() + // Generate a version-release number for the target Git commit + def versions = sh(returnStdout: true, script: 'source ./version.sh && echo -en "$WAIVERDB_VERSION\n$WAIVERDB_CONTAINER_VERSION"').split('\n') + env.WAIVERDB_VERSION = versions[0] + env.WAIVERDB_CONTAINER_VERSION = versions[1] + env.TEMP_TAG = env.WAIVERDB_CONTAINER_VERSION + '-jenkins-' + currentBuild.id + env.PIPELINE_NAMESPACE = readFile('/run/secrets/kubernetes.io/serviceaccount/namespace').trim() + env.PIPELINE_USERNAME = sh(returnStdout: true, script: 'id -un').trim() + } + sh 'cp conf/settings.py.example conf/settings.py' + } + } + stage('Run checks') { + failFast false + parallel { + stage('Invoke Flake8') { + steps { + sh 'flake8' + } + } + stage('Invoke Pylint') { + steps { + sh 'pylint-3 --reports=n waiverdb' + } + } + } + } + stage('Run unit tests') { + steps { + // wait for the test datebase to come up + sh 'wait-for-it -s -t 300 127.0.0.1:5432' + // create a database role + sh 'psql -h 127.0.0.1 -U "postgres" -q -d "waiverdb" -c "CREATE ROLE \"$PIPELINE_USERNAME\" WITH LOGIN SUPERUSER;"' + // run unit tests + sh 'py.test-3 -v --junitxml=junit-tests.xml tests' + } + post { + always { + junit 'junit-tests.xml' + } + } + } + stage('Build Artifacts') { + failFast false + parallel { + stage('Branch Docs') { + stages { + stage('Build Docs') { + steps { + sh 'make -C docs html' + } + post { + always { + archiveArtifacts artifacts: 'docs/_build/html/**' + } + } + } + stage('Publish Docs') { + when { + expression { + return "${PAGURE_DOC_REPO_NAME}" && (params.WAIVERDB_GIT_REF == "${WAIVERDB_MAIN_BRANCH}" || env.FORCE_PUBLISH_DOCS == "true") + } + } + steps { + sshagent (credentials: [env.PIPELINE_NAMESPACE + '-${PAGURE_DOC_SECRET}']) { + sh ''' + mkdir -p ~/.ssh/ + touch ~/.ssh/known_hosts + ssh-keygen -R pagure.io + echo 'pagure.io ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC198DWs0SQ3DX0ptu+8Wq6wnZMrXUCufN+wdSCtlyhHUeQ3q5B4Hgto1n2FMj752vToCfNTn9mWO7l2rNTrKeBsELpubl2jECHu4LqxkRVihu5UEzejfjiWNDN2jdXbYFY27GW9zymD7Gq3u+T/Mkp4lIcQKRoJaLobBmcVxrLPEEJMKI4AJY31jgxMTnxi7KcR+U5udQrZ3dzCn2BqUdiN5dMgckr4yNPjhl3emJeVJ/uhAJrEsgjzqxAb60smMO5/1By+yF85Wih4TnFtF4LwYYuxgqiNv72Xy4D/MGxCqkO/nH5eRNfcJ+AJFE7727F7Tnbo4xmAjilvRria/+l' >>~/.ssh/known_hosts + rm -rf docs-on-pagure + git clone ssh://git@pagure.io/docs/${PAGURE_DOC_REPO_NAME}.git docs-on-pagure + rm -rf docs-on-pagure/* + cp -r docs/_build/html/* docs-on-pagure/ + cd docs-on-pagure + git config user.name 'Pipeline Bot' + git config user.email "pipeline-bot@localhost.localdomain" + git add -A . + if [[ "$(git diff --cached --numstat | wc -l)" -eq 0 ]] ; then + exit 0 # No changes, nothing to commit + fi + git commit -m 'Automatic commit of docs built by Jenkins job ${env.JOB_NAME} #${env.BUILD_NUMBER}' + git push origin master + ''' + } + } + } + } + } + stage('Build SRPM') { + steps { + sh './rpmbuild.sh -bs' + } + post { + success { + archiveArtifacts artifacts: 'rpmbuild-output/*.src.rpm' + } + } + } + stage('Branch RPM') { + stages { + stage('Build RPM') { + steps { + sh './rpmbuild.sh -bb' + } + post { + success { + archiveArtifacts artifacts: 'rpmbuild-output/*/*.rpm' + } + } + } + stage('Invoke Rpmlint') { + steps { + sh 'rpmlint -f rpmlint-config.py rpmbuild-output/*/*.rpm' + } + } + } + } + } + } + stage('Build container') { + //options { + // timeout(time: 15, unit: 'MINUTES') + //} + environment { + BUILDCONFIG_INSTANCE_ID = "waiverdb-container-build-${currentBuild.id}" + } + steps { + script { + openshift.withCluster() { + // OpenShift BuildConfig doesn't support specifying a tag name at build time. + // We have to create a new BuildConfig for each container build. + // Create a BuildConfig from a seperated Template. + echo 'Creating a BuildConfig...' + def template = readYaml file: 'openshift/waiverdb-container-template.yml' + def processed = openshift.process(template, + "-p", "NAME=" + env.BUILDCONFIG_INSTANCE_ID, + '-p', 'WAIVERDB_GIT_REPO=' + params.WAIVERDB_GIT_REPO, + '-p', 'WAIVERDB_GIT_REF=' + env.WAIVERDB_GIT_COMMIT_ID, + '-p', 'WAIVERDB_IMAGE_TAG=' + env.TEMP_TAG, + '-p', 'WAIVERDB_VERSION=' + env.WAIVERDB_VERSION, + '-p', 'WAIVERDB_IMAGESTREAM_NAME=${WAIVERDB_IMAGESTREAM_NAME}', + '-p', 'WAIVERDB_IMAGESTREAM_NAMESPACE=${WAIVERDB_IMAGESTREAM_NAMESPACE}', + ) + def created = openshift.apply(processed) + def bc = created.narrow('bc') + echo 'Starting a container build from the created BuildConfig...' + buildSelector = bc.startBuild() + // `buildSelector.logs()` can be dumb when the OpenShift Build is not started. + // Let's wait for it to be started or completed (failed). + echo 'Waiting for the container build to be started...' + timeout(5) { // 5 min + buildSelector.watch { + return !(it.object().status.phase in ["New", "Pending"]) + } + } + echo 'Following container build logs...' + // This function sometimes hangs infinitely. + // Not sure it is a problem of OpenShift Jenkins Client plugin + // or OpenShift. + // FIXME: logs() step may fail with unknown reasons. + timeout(time: 15, activity: false) { + buildSelector.logs('-f') + } + // Ensure the build is stopped + echo 'Waiting for the container build to be fully stopped...' + timeout(5) { // 5 min + buildSelector.watch { + return it.object().status.phase != "Running" + } + } + // Assert build result + def ocpBuild = buildSelector.object() + if (ocpBuild.status.phase != "Complete") { + error("Failed to build container image for ${env.TEMP_TAG}, .status.phase=${ocpBuild.status.phase}.") + } + echo 'Container build is complete.' + env.RESULTING_IMAGE_REF = ocpBuild.status.outputDockerImageReference + env.RESULTING_IMAGE_DIGEST = ocpBuild.status.output.to.imageDigest + def imagestream= created.narrow('is').object() + env.RESULTING_IMAGE_REPO = imagestream.status.dockerImageRepository + env.RESULTING_TAG = env.TEMP_TAG + } + } + } + post { + cleanup { + script { + openshift.withCluster() { + echo 'Tearing down...' + openshift.selector('bc', [ + 'app': env.BUILDCONFIG_INSTANCE_ID, + 'template': 'waiverdb-container-template', + ]).delete() + } + } + } + } + } + stage('Run functional tests') { + environment { + // Jenkins BUILD_TAG could be too long (> 63 characters) for OpenShift to consume + TEST_ID = "jenkins-${currentBuild.id}" + ENVIRONMENT_LABEL = "test-${env.TEST_ID}" + } + steps { + script { + openshift.withCluster() { + def template = readYaml file: 'openshift/waiverdb-test-template.yaml' + def webPodReplicas = 1 // The current quota in UpShift is agressively limited + def models = openshift.process(template, + '-p', "TEST_ID=${env.TEST_ID}", + '-p', "WAIVERDB_APP_IMAGE_REPO=${env.RESULTING_IMAGE_REPO}", + '-p', "WAIVERDB_APP_VERSION=${env.RESULTING_TAG}", + '-p', "WAIVERDB_REPLICAS=${webPodReplicas}", + ) + def objects = openshift.apply(models) + echo "Waiting for test pods with label environment=${env.ENVIRONMENT_LABEL} to become Ready" + //def rm = dcSelector.rollout() + def dcs = openshift.selector('dc', ['environment': env.ENVIRONMENT_LABEL]) + def rm = dcs.rollout() + def pods = openshift.selector('pods', ['environment': env.ENVIRONMENT_LABEL]) + timeout(15) { + pods.untilEach(webPodReplicas + 1) { + def pod = it.object() + if (pod.status.phase in ["Pending", "Unknown"]) { + return false + } + if (pod.status.phase == "Running") { + for (cond in pod.status.conditions) { + if (cond.type == 'Ready' && cond.status == 'True') { + return true + } + } + return false + } + error("Test pod ${pod.metadata.name} is not running. Current phase is ${pod.status.phase}.") + } + } + // Run functional tests + def route_hostname = objects.narrow('route').object().spec.host + echo "Running tests against https://${route_hostname}/" + withEnv(["WAIVERDB_TEST_URL=https://${route_hostname}/"]) { + sh 'py.test-3 -v --junitxml=junit-functional-tests.xml functional-tests/' + } + } + } + } + post { + always { + script { + junit 'junit-functional-tests.xml' + openshift.withCluster() { + /* Extract logs for debugging purposes */ + openshift.selector('deploy,pods', ['environment': env.ENVIRONMENT_LABEL]).logs() + } + } + } + cleanup { + script { + openshift.withCluster() { + /* Tear down everything we just created */ + echo "Tearing down test resources..." + openshift.selector('dc,deploy,configmap,secret,svc,route', + ['environment': env.ENVIRONMENT_LABEL]).delete() + } + } + } + } + } + stage('Push container') { + when { + expression { + return params.FORCE_PUBLISH_IMAGE == 'true' || + params.WAIVERDB_GIT_REF == "${WAIVERDB_MAIN_BRANCH}" + } + } + steps { + script { + def destinations = env.WAIVERDB_DEV_IMAGE_DESTINATIONS ? + env.WAIVERDB_DEV_IMAGE_DESTINATIONS.split(',') : [] + openshift.withCluster() { + def sourceImage = env.RESULTING_IMAGE_REPO + ":" + env.RESULTING_TAG + if (env.REGISTRY_CREDENTIALS) { + dir ("${env.HOME}/.docker") { + writeFile file:'config.json', text: env.REGISTRY_CREDENTIALS + } + } + // pull the built image from imagestream + echo "Pulling container from ${sourceImage}..." + def registryToken = readFile(file: '/var/run/secrets/kubernetes.io/serviceaccount/token') + withEnv(["SOURCE_IMAGE_REF=${sourceImage}", "TOKEN=${registryToken}"]) { + sh '''set -e +x # hide the token from Jenkins console + mkdir -p _build + skopeo copy \ + --src-cert-dir=/var/run/secrets/kubernetes.io/serviceaccount/ \ + --src-creds=serviceaccount:"$TOKEN" \ + docker://"$SOURCE_IMAGE_REF" dir:_build/waiverdb_container + ''' + } + // push to registries + def pushTasks = destinations.collectEntries { + ["Pushing ${it}" : { + def dest = it + // Only docker and atomic registries are allowed + if (!it.startsWith('atomic:') && !it.startsWith('docker://')) { + dest = 'docker://' + it + } + echo "Pushing container to ${dest}..." + withEnv(["DEST_IMAGE_REF=${dest}"]) { + /* Pushes to the internal registry can sometimes randomly fail + * with "unknown blob" due to a known issue with the registry + * storage configuration. So we retry up to 5 times. */ + retry(5) { + sh 'skopeo copy dir:_build/waiverdb_container "$DEST_IMAGE_REF"' + } + } + }] + } + parallel pushTasks + } + } + } + } + stage('Tag into image stream') { + when { + expression { + return "${WAIVERDB_DEV_IMAGE_TAG}" && params.TAG_INTO_IMAGESTREAM == "true" && + (params.FORCE_PUBLISH_IMAGE == 'true' || params.WAIVERDB_GIT_REF == "${WAIVERDB_MAIN_BRANCH}") + } + } + steps { + script { + openshift.withCluster() { + openshift.withProject("${WAIVERDB_IMAGESTREAM_NAMESPACE}") { + def sourceRef = "${WAIVERDB_IMAGESTREAM_NAME}:${env.RESULTING_TAG}" + def destRef = "${WAIVERDB_IMAGESTREAM_NAME}:${WAIVERDB_DEV_IMAGE_TAG}" + echo "Tagging ${sourceRef} as ${destRef}..." + openshift.tag("${sourceRef}", "${destRef}") + } + } + } + } + } + } + post { + cleanup { + script { + if (env.RESULTING_TAG) { + echo "Removing tag ${env.RESULTING_TAG} from the ImageStream..." + openshift.withCluster() { + openshift.withProject("${WAIVERDB_IMAGESTREAM_NAMESPACE}") { + openshift.tag("${WAIVERDB_IMAGESTREAM_NAME}:${env.RESULTING_TAG}", + "-d") + } + } + } + } + } + } + }