releng

#12029 [STG Koji] osbuild builders can't authenticate with Koji using a keytab file
Opened by thozza. Modified

Open
Close Issue
  • Describe the issue
    osbuild builders use keytab file to authenticate with Koji instance. The principal that we use for STG koji is "osbuild-automation-bot@STG.FEDORAPROJECT.ORG".

I'm observing an issue in STG Koji environment with our builders not being able to authenticate with the keytab and getting error "Post \"https://koji.stg.fedoraproject.org/kojihub/ssllogin\": No credentials were supplied, or the credentials were unavailable or inaccessible - Cannot find KDC for realm \"STG.FEDORAPROJECT.ORG\"" e.g.:

https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=120070798
https://kojipkgs.stg.fedoraproject.org//work/tasks/798/120070798/compose-status.json

The same setup (with a different keytab and principal) works just fine with PROD Koji. And in addition, the STG setup worked for us before. Not sure if something changed in the meantime.

  • When do you need this? (YYYY/MM/DD)
    The sooner, the better.

  • When is this no longer needed or useful? (YYYY/MM/DD)
    N/A

  • If we cannot complete your request, what is the impact?
    We will have to test changes to our infra in the production environment, which is not great.

Metadata Update from @phsmoura:
- Issue tagged with: low-gain, low-trouble, ops

Hey @thozza,

Can you check if the issue persists? This might have happened when we had an IPA apocalypse week back ¯_(ツ)_/¯

Hey.

The problem seems to be fixed for x86_64 and aarch64 builders, which are maintained by us in our AWS account.

However, I still see the problem on our ppc64le builder, which is part of Fedora infra in IAD2. Specifically buildvm-ppc64le-osbuild02.iad2.fedoraproject.org.

I submitted https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=120148017 and for ppc64le I can see:

 {
    "details": "org.osbuild.koji",
    "id": 2,
    "reason": "failed to authenticate with Koji server \"koji.stg.fedoraproject.org\": Post \"https://koji.stg.fedoraproject.org/kojihub/ssllogin\": No credentials were supplied, or the credentials were unavailable or inaccessible - Cannot find KDC for realm \"STG.FEDORAPROJECT.ORG\""
 }

All of our builders use the same keytab file and principal to authenticate with koji.stg.fedoraproject.org. But it does not work only for the ppc64le worker.

Previously, I saw the same issue on all builders, as noted in the issue description.

Hey. This is still an issue. Any chance that someone could look at it? Maybe @kevin has some ideas :innocent:

Yeah, sorry... this has been lower on my list since it's staging...

So, first thing to try: we need to move your builders from fedora-38 to fedora-40. ;)
So, lets try just reprovisioning it and see if that fixes things. If it does not, we can debug more from there. I don't see why it would just suddenly stop working off hand tho.

When would be a good time to reinstall? I (or others) can take down/remove the existing instances and you should just be able to run the playbook to deploy.
Or I can do so if you like and give me a time window.
Metadata
None

low-gain low-trouble ops

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.