koji

#2271 Resolve a single Hub's IP so GSSAPI authentication and all requests for a session end up on the same host
Closed by alexi. Opened by alexi.
alexi/koji fix-2250  into  master

Download 2271.patch

This change allows you to have GSSAPI authentication and multiple hubs behind the same DNS name by making sessions "sticky".

The Problem

python-gssapi always resolves the hostname of the machine you're trying to talk to in order to obtain its kerberos ticket. With python-krbV this was controllable from the Koji client by using the krb_rdns = false and krb_canon_host = false options, but those options don't work for python-gssapi (see issue #2250). The only way I know of right now to control this behavior is by setting rdns = false in /etc/krb5.conf, but this would affect all kerberos requests on that machine and we can't do this on user's machines that have the Koji CLI installed.

If we can't set krb_rdns = false in the CLI, the Hubs need to have a keytab with their real hostnames (as opposed to the DNS alias) in order for the authentication to work. When the Koji CLI is used, a session is started with kojihub.example.com. This DNS name is resolved to one of the two Hubs (let's say kojihub01.example.com), and a kerberos ticket is obtained for it. When the CLI's actual request is sent, kojihub.example.com is resolved again and this time you might get a different Hub (kojihub02.example.com). When kojihub02.example.com gets the request, it will receive an authorization header for kojihub01.example.com, which it can't decrypt and therefore the request will fail. Sometimes you'll get lucky and all your requests will go to the same hub, but this is non-deterministic.

This means you can either have GSSAPI authentication or multiple Koji Hubs, but not both. I want both.

My Solution

The Koji code needs to be modified to only resolve the Hub's name once and use that resolution for the kerberos authentication and all subsequent requests withing the same session, guaranteeing that they all go to the same Hub. In lines 22-30 of koji/__init__.py , the hub's alias is extracted and resolved to all the IP addresses that are behind it. Those IPs are tested to see if they have a listening service, and the first one that does is chosen as the hub to talk to. In line 51, the Host header is set to the original hub alias so that SSL certificate validation will still work as expected. Note that for this to work, a new dependency (python-requests-toolbelt) has to be introduced. The original hub alias is preserved to be displayed for cosmetics purposes (line 6 of cli/koji_cli/commands.py) and also so that subsessions (created by Koji Builders) have the chance to get started on a different hub (line 42 of koji/__init__.py).

Is "kojihub.example.com" a single DNS A record with two IPs (that have PTR records to kojihub01 and kojihub02)?

What happens when you use a koji hub keytab with entries for all three service principal names? You can write them into one keytab and store the keytab on both hubs.

HTTP/kojihub.example.com
HTTP/kojihub01.example.com
HTTP/kojihub02.example.com

Yes, that's correct, kojihub.example.com resolves to two DNS A records.

We would like to avoid having a single keytab with all entries because of the management issues it creates if we modify any of the machines that are part of the alias. We'd need to manually regenerate the keytabs, merge them and redeploy them.

I think it could be done this way, nevertheless I would add a configuration option for that for cases where load-balancing is done in the way ktdreyer describes. In such case it could interfere with that (especially if hubs are spawn semi-randomly e.g. in openshift)

...and I would avoid using requests-toolbelt. It looks that repo is for a year untouched and we're using only simple function from there.

2 new commits added

  • Add new session_balance option
  • Drop dependency of python-requests-toolbelt

Ok, I've removed the dependency on requests-toolbelt and added an option to control this load-balancing behavior.

1 new commit added

  • Add new option everywhere

This feels kind of fragile and not something I think we should carry in Koji. The root problem seems to be a limitation in python-gssapi. It seems from reading through #2250 that there are other workarounds already.
Edited by mikem

Also it seems like overkill to change core behavior for the entire session to address an issue that only happens in one call for gssapi_login()

pretty please pagure-ci rebuild

I think we can close this, right?

Personally, I'm still using this patch as I still have to deal with clients with rdns = true and I haven't found a better solution.

Closing, the solution proposed in #2250 is better.

Pull-Request has been closed by alexi
Metadata
None
No Tags
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.