koji

#4064 [1.33.0] Unable to use it in multi realm environments.
Closed: Fixed by frank-mdc. Opened by frank-mdc.

Closed: Fixed
Reopen Issue

When I try to use it in krb5 environment whit more then one realm, koji fails because it will mix both realms.
Environment:
- DNS domain:
- foo.bar
- KRB5 realms:
- FOO.BAR
- KOJI.FOO.BAR
Hub: hub.foo.bar
KDC for KOJ.FOO.BAR: kdc-koji.foo.bar
Sample:
~~~~
klist -a
klist: Credentials cache 'KCM:1000' not found
~~~~
~~~~
kinit user@KOJI.FOO.BAR
Password for user@KOJI.FOO.BAR:
~~~~
~~~~
klist -a
Ticket cache: KCM:1000
Default principal: user@KOJI.FOO.BAR

Valid starting Expires Service principal
03/27/24 07:31:59 03/28/24 07:31:59 krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
renew until 03/27/24 07:31:59
Addresses: (none)
~~~~
~~~~
KRB5_TRACE=/dev/stdout koji --principal=user@KOJI.FOO.BAR hello
[18588] 1711521285.890168: Matching user@KOJI.FOO.BAR in collection with result: 0/Success
[18588] 1711521285.890169: Matching user@KOJI.FOO.BAR in collection with result: 0/Success
[18588] 1711521285.890170: Getting credentials user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@ using ccache KCM:1000
[18588] 1711521285.890171: Retrieving user@KOJI.FOO.BAR -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890172: Retrieving user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@ from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890173: Retrying user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@KOJI.FOO.BAR with result: -1765328243/Matching credential not found
[18588] 1711521285.890174: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890175: Retrieving user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR from KCM:1000 with result: 0/Success
[18588] 1711521285.890176: Starting with TGT for client realm: user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890177: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@MFOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890178: Requesting TGT krbtgt/FOO.BAR@KOJI.FOO.BAR using TGT krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890179: Generated subkey for TGS request: aes256-cts/DBE9
[18588] 1711521285.890180: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[18588] 1711521285.890182: Encoding request body and padata into FAST request
[18588] 1711521285.890183: Sending request (1125 bytes) to KOJI.FOO.BAR
[18588] 1711521285.890184: Resolving hostname kdc-koji.foo.bar
[18588] 1711521285.890185: Initiating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890186: Sending TCP request to stream x.x.x.x:88
[18588] 1711521285.890187: Received answer (454 bytes) from stream x.x.x.x:88
[18588] 1711521285.890188: Terminating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890189: Sending DNS URI query for _kerberos.KOJI.FOO.BAR.
[18588] 1711521285.890190: No URI records found
[18588] 1711521285.890191: Sending DNS SRV query for _kerberos-master._udp.KOJI.FOO.BAR.
[18588] 1711521285.890192: Sending DNS SRV query for _kerberos-master._tcp.KOJI.FOO.BAR.
[18588] 1711521285.890193: No SRV records found
[18588] 1711521285.890194: Response was not from primary KDC
[18588] 1711521285.890195: Decoding FAST response
[18588] 1711521285.890196: TGS request result: -1765328377/Server krbtgt/FOO.BAR@KOJI.FOO.BAR not found in Kerberos database
[18588] 1711521285.890197: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890198: Retrieving user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR from KCM:1000 with result: 0/Success
[18588] 1711521285.890199: Starting with TGT for client realm: user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890200: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890201: Requesting TGT krbtgt/FOO.BAR@KOJI.FOO.BAR using TGT krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890202: Generated subkey for TGS request: aes256-cts/33CC
[18588] 1711521285.890203: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[18588] 1711521285.890205: Encoding request body and padata into FAST request
[18588] 1711521285.890206: Sending request (1125 bytes) to KOJI.FOO.BAR
[18588] 1711521285.890207: Resolving hostname kdc-koji.foo.bar
[18588] 1711521285.890208: Initiating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890209: Sending TCP request to stream x.x.x.x:88
[18588] 1711521285.890210: Received answer (454 bytes) from stream x.x.x.x:88
[18588] 1711521285.890211: Terminating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890212: Sending DNS URI query for _kerberos.KOJI.FOO.BAR.
[18588] 1711521285.890213: No URI records found
[18588] 1711521285.890214: Sending DNS SRV query for _kerberos-master._udp.KOJI.FOO.BAR.
[18588] 1711521285.890215: Sending DNS SRV query for _kerberos-master._tcp.KOJI.FOO.BAR.
[18588] 1711521285.890216: No SRV records found
[18588] 1711521285.890217: Response was not from primary KDC
[18588] 1711521285.890218: Decoding FAST response
[18588] 1711521285.890219: TGS request result: -1765328377/Server krbtgt/FOO.BAR@KOJI.FOO.BAR not found in Kerberos database
[18588] 1711521285.890220: Matching user@KOJI.FOO.BAR in collection with result: 0/Success
[18588] 1711521285.890221: Matching user@KOJI.FOO.BAR in collection with result: 0/Success
[18588] 1711521285.890222: Getting credentials user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@ using ccache KCM:1000
[18588] 1711521285.890223: Retrieving user@KOJI.FOO.BAR -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890224: Retrieving user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@ from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890225: Retrying user@KOJI.FOO.BAR -> HTTP/hub.foo.bar@KOJI.FOO.BAR with result: -1765328243/Matching credential not found
[18588] 1711521285.890226: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890227: Retrieving user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR from KCM:1000 with result: 0/Success
[18588] 1711521285.890228: Starting with TGT for client realm: user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890229: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890230: Requesting TGT krbtgt/FOO.BAR@KOJI.FOO.BAR using TGT krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890231: Generated subkey for TGS request: aes256-cts/EDBA
[18588] 1711521285.890232: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[18588] 1711521285.890234: Encoding request body and padata into FAST request
[18588] 1711521285.890235: Sending request (1125 bytes) to FOO.BAR
[18588] 1711521285.890236: Resolving hostname kdc-koji.foo.bar
[18588] 1711521285.890237: Initiating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890238: Sending TCP request to stream x.x.x.x:88
[18588] 1711521285.890239: Received answer (454 bytes) from stream x.x.x.x:88
[18588] 1711521285.890240: Terminating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890241: Sending DNS URI query for _kerberos.KOJI.FOO.BAR.
[18588] 1711521285.890242: No URI records found
[18588] 1711521285.890243: Sending DNS SRV query for _kerberos-master._udp.KOJI.FOO.BAR.
[18588] 1711521285.890244: Sending DNS SRV query for _kerberos-master._tcp.KOJI.FOO.BAR.
[18588] 1711521285.890245: No SRV records found
[18588] 1711521285.890246: Response was not from primary KDC
[18588] 1711521285.890247: Decoding FAST response
[18588] 1711521285.890248: TGS request result: -1765328377/Server krbtgt/FOO.BAR@KOJI.FOO.BAR not found in Kerberos database
[18588] 1711521285.890249: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890250: Retrieving user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR from KCM:1000 with result: 0/Success
[18588] 1711521285.890251: Starting with TGT for client realm: user@KOJI.FOO.BAR -> krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890252: Retrieving user@KOJI.FOO.BAR -> krbtgt/FOO.BAR@FOO.BAR from KCM:1000 with result: -1765328243/Matching credential not found
[18588] 1711521285.890253: Requesting TGT krbtgt/FOO.BAR@KOJI.FOO.BAR using TGT krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
[18588] 1711521285.890254: Generated subkey for TGS request: aes256-cts/B9C1
[18588] 1711521285.890255: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[18588] 1711521285.890257: Encoding request body and padata into FAST request
[18588] 1711521285.890258: Sending request (1125 bytes) to KOJI.FOO.BAR
[18588] 1711521285.890259: Resolving hostname kdc-koji.foo.bar
[18588] 1711521285.890260: Initiating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890261: Sending TCP request to stream x.x.x.x:88
[18588] 1711521285.890262: Received answer (454 bytes) from stream x.x.x.x:88
[18588] 1711521285.890263: Terminating TCP connection to stream x.x.x.x:88
[18588] 1711521285.890264: Sending DNS URI query for _kerberos.KOJI.FOO.BAR.
[18588] 1711521285.890265: No URI records found
[18588] 1711521285.890266: Sending DNS SRV query for _kerberos-master._udp.KOJI.FOO.BAR.
[18588] 1711521285.890267: Sending DNS SRV query for _kerberos-master._tcp.KOJI.FOO.BAR.
[18588] 1711521285.890268: No SRV records found
[18588] 1711521285.890269: Response was not from primary KDC
[18588] 1711521285.890270: Decoding FAST response
[18588] 1711521285.890271: TGS request result: -1765328377/Server krbtgt/FOO.BAR@KOJI.FOO.BAR not found in Kerberos database
2024-03-27 07:34:45,991 [ERROR] koji: (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://hub.foo.bar/kojihub/ssllogin)
Use following documentation to debug kerberos/gssapi auth issues. https://docs.pagure.org/koji/kerberos_gssapi_debug/
2024-03-27 07:34:45,999 [ERROR] koji: GSSAPIAuthError: unable to obtain a session (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://hub.foo.bar/kojihub/ssllogin)
Use following documentation to debug kerberos/gssapi auth issues. https://docs.pagure.org/koji/kerberos_gssapi_debug/
~~~~
For me it looks like the call for the TGS are wrong.
I think it must be krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR instead of of krbtgt/FOO.BAR@KOJI.FOO.BAR
This would match the error on the KDC for KOJI.FOO.BAR:
~2024-03-27T07:34:45+0100 krb5kdc[3445088]: TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 141.80.121.45: UNKNOWN_SERVER: authtime 0, user@KOJI.FOO.BAR for krbtgt/FOO.BAR@KOJI.FOO.BAR, Server not found in Kerberos database 2024-03-27T07:34:45+0100 krb5kdc[3445088]: closing down fd 13~~~~`

From the command line I can request the TGS:
~~~~
kvno krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR
krbtgt/KOJI.FOO.BAR@KOJI.FOO.BAR: KVNO = 1
~~~~

I constantly have krb credentials for multiple unrelated domains and I do not see this problem

Metadata Update from @mikem:
- Custom field Size adjusted to None

Koji itself doesn't deal with deep krb details. We rely on httpd to validate credentials on the hub side, and on python-requests-gssapi to handle auth on the client side.

As I see in the log of the KDC, the source of it will be the koji client itself.
So for me it looks like the hub don't try to talk to the kdc. Also an tcpdump of the krb5 traffic will show this. I will open an bug report at the python-requests-gssapi developers.

After long searching the problem was the cache file used by the parts.

Metadata Update from @frank-mdc:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4064

Please continue any further discussion there.
Metadata
None
None
None
None
Low
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.