Right now, everything in this PR is working as far as deploying the test environment from template manually using /usr/bin/oc.

However the Jenkins pieces are currently failing with this error which makes no sense to me:

ERROR: process returned an error;
{reference={}, err=error: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt for  due to open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory, verb=process, cmd=oc process openshift/waiverdb-test-template.yaml -p TEST_ID=jenkins-waiverdb-dcallagh8-592 -p WAIVERDB_APP_VERSION=0.1.2.dev27-git.baec9a5 -f -o=json --server=https://open.paas.redhat.com/ --namespace=waiverdb-test --token=XXXXX , out=, status=1}

What I am not sure of right now is, why is it expecting /var/run/secrets/kubernetes.io/serviceaccount/ca.crt to exist and where should it be coming from? It seems to be a bug in the OpenShift Client Jenkins plugin but it needs more investigation.

That error may indicate that the OpenShift cluster isn't configured properly.

https://github.com/openshift/jenkins-client-plugin#configuring-an-openshift-cluster

Oh yeah I think you're right. When I filled in the cluster config into our Jenkins, I left the SSL certificate field blank, because I thought it would just use the system-wide trust store as normal. Seems like it doesn't do that though. A bit annoying... that's why we have a system-wide trust store, so that I don't have to configure the CA every single time. Oh well.

I have added the CA cert to the config and it seems to have got past that error now. Next problem is that openshift.process('-f', '<filename>'... does not work. I think we need to load the YAML template first and pass it in...

Okay, I have updated the last commit to 5a3dfec37eea6d27dd19b5640824e97da4246911 which gets a little further.

Have to wait for pods to be Ready, not just Running (they can start Running before the db is up, in which they will go into a crash loop until the db is reachable).

I got my selectors and stuff messed up, fixed those.

Now I am just running afoul of some Groovy scoping rules that I don't understand...

groovy.lang.MissingPropertyException: No such property: podReady for class: groovy.lang.Binding

😕

Could you prefix this with WAIVERDB, so that it's called WAIVERDB_SECRET_KEY?

As far as I understand in this example https://github.com/openshift/jenkins-client-plugin/blob/master/examples/coverage.groovy,
this should be written like this
pods.untilEach(3) {
def ready_cond = it.object().status.conditions.find { it.type == 'Ready' }
return ready_cond?.status == true
}

This may fix this error

groovy.lang.MissingPropertyException: No such property: podReady for class: groovy.lang.Binding

Nope I had that originally but you can't call .find in a closure because of JENKINS-26481... hence that @NonCPS business.

This pipeline stuff sounds promising in theory but is full of lots of nastiness once you start trying to use it for real...

Is this output from the pipeline or an openshift log?

Can you provide some more details if they are available?

Environment variables in OpenShift are scoped to each container so I don't think there is any potential for confusion if we just call this SECRET_KEY. Same as we don't have WAIVERDB_ prefixed on the database password, or krb5 keytab, etc.

Yes, but perhaps we shouldn't assume this will always be run in a container. I'm fairly certain you guys are deploying or have deployed a dev instance on a VM.

@alivigni the MissingPropertyException was from Jenkins. It was just a mistake in my Jenkinsfile. My Groovy fu is weak.

I was trying to pass my podReady function as if it were a first-class object like in Python. But in Groovy it's actually a method and you have to convert it to a closure using .& operator.

Anyway now my podReady method is still failing with com.cloudbees.groovy.cps.impl.CpsCallableInvocation although I had to hack it full of echo statements to even figure that out. I think this time I'm hitting JENKINS-31314... I've gotta say, this Pipeline stuff sounds nice in theory but is full of a lot of ridiculous gotchas...

Yeah on a traditional deployment the SECRET_KEY would just go into the config file, not the environment, so it's not an issue there.

Okay, I think I got it. Amended commit ec7f848f avoids all the CPS gotchas. It passes!

... and one more rebase to fix conflicts. Rebased series ends in commit 0e08e677.

nice, :thumbsup: