All the justification here seems to be based on development requirements. None of it seems to actually say "doing this will make Fedora 42 better in any appreciable way". Ultimately the request is to ship a new version of a key component in a stable Fedora release in order to make an upstream CI system happy, which I don't think is a good rationale.

Yeah, I have to agree with @adamwill based on what is written so far.

I also suspect the sentence "This allows us to be sure Fedora stable release is always supported by FreeIPA." is missing a lot of important context. I have a suspicion that you're talking about FreeIPA servers running on F42 and F43 needing this change to be able to interoperate as replicas, but since you didn't say that, I'm only making a guess.

I'm -1 for this request as it stands right now (which means it goes on the upcoming meeting agenda).

BTW, some context from discussions of this elsewhere: the change between dogtag-pki 11.6 and 11.7 is pretty substantial, this isn't a minor bump. If you want to see for yourself the repo is https://github.com/dogtagpki/pki and the tags are v11.6.0 and v11.7.0.

"This allows us to be sure Fedora stable release is always supported by FreeIPA" - that's a laudable goal, but it would be a mistake to use it as a justification for making changes to Fedora stable releases which aren't otherwise appropriate. That's getting things backwards.

There's no inherent reason new releases of FreeIPA should always work on the current stable Fedora, after all. This is a perfect example of a case where it's appropriate for it not to work on the current stable Fedora. The changes to pki-core are major development work; they're not inherently appropriate as updates to a stable operating system. Porting FreeIPA to use them upstream is a perfectly sensible thing to do, but it naturally means the new FreeIPA upstream code no longer works on an older stable operating system. But that's OK! That's how things are meant to be. In this situation, "make sure tests of the new upstream code pass on older stable distributions" is no longer a sensible goal, IMHO.

BTW, some context from discussions of this elsewhere: the change between dogtag-pki 11.6 and 11.7 is pretty substantial, this isn't a minor bump. If you want to see for yourself the repo is https://github.com/dogtagpki/pki and the tags are v11.6.0 and v11.7.0.

"This allows us to be sure Fedora stable release is always supported by FreeIPA" - that's a laudable goal, but it would be a mistake to use it as a justification for making changes to Fedora stable releases which aren't otherwise appropriate. That's getting things backwards.

There's no inherent reason new releases of FreeIPA should always work on the current stable Fedora, after all. This is a perfect example of a case where it's appropriate for it not to work on the current stable Fedora. The changes to pki-core are major development work; they're not inherently appropriate as updates to a stable operating system. Porting FreeIPA to use them upstream is a perfectly sensible thing to do, but it naturally means the new FreeIPA upstream code no longer works on an older stable operating system. But that's OK! That's how things are meant to be. In this situation, "make sure tests of the new upstream code pass on older stable distributions" is no longer a sensible goal, IMHO.

Again, I agree. I just want confirmation that that's what this means. "FreeIPA 4.13 can not be installed on Fedora 42" is different from "FreeIPA 4.12 and 4.13 cannot co-exist in the same domain" or "FreeIPA 4.12 on F42 cannot upgrade to FreeIPA 4.13 on F43 without the intermediate step of first switching to Dogtag v11.7.0". (This last one would be Really Bad™️)

Part of those changes in v11.7 is to provide a different API for operations that otherwise trigger a path in 389-ds that is performing bad or losing data with lmdb backend. Fedora 42 already defaults to lmdb backend, so we know it might be an issue (we saw it for some customers already). This is one concrete problem that we aim to solve (dropping VLV queries from Dogtag side that LDAP server with lmdb backend is not able to handle well).

Again, this all should have landed in Fedora 42 at least 6 months ago because it did land in CentOS 10 Stream and RHEL. But it wasn't, so now we stuck with both upstream releases and Fedora updates.

Regarding 'no inherent reason', this will be the first Fedora release in past 30 or so that we would break this rule on FreeIPA side. We proud in making a release possible on the stable Fedora from the master branch upstream. CA integration makes it impossible now unless we get v11.7 in.

And we know that the effect to existing users will not be devastating. This is because we ran the https://github.com/freeipa/freeipa/pull/7781 PR on a F42+v11.7 COPR and we also ran whole set of upstream and downstream tests in RHEL for both v11.6 and v11.7 environments already.

Regarding cross-version co-existence. Technically, each IPA server talks to its own CA API locally but sometimes it forwards requests through a different IPA server to reach a CA (CA is optional per IPA replica). It was addressed in v11.7 to make it sure such communication is possible and our testing in RHEL shows we have no problems with that.

@rcritten knows more but he is on vacation until next Tuesday.

@rcritten @abbra

You're trying to convince us that the upgrade is safe, but what we really need to know is what the effect is if we do not accept this request. How will Fedora be negatively affected if F42 remains on Dogtag 11.6 for its full lifetime?

Historically, we've only accepted requests like this if the package in question had a serious flaw that cannot be fixed with a backported patch. If that's the case, please explain the flaw and why it can't be fixed in 11.6. Is there some other, similar negative outcome that affects Fedora if we don't permit this upgrade?

389-ds dropped support for Berkley DB with 3.2.0.

PKI relies on VLV to determine the next value in a number of operations: serial number, request id, etc.

The 389-ds team has warned us that VLV performance will not be very good with larger databases. That limit AFAIK has not been determined.

PKI 11.7.0 provides a new RPC API that doesn't rely on VLV. This API requires changes on the IPA side to be used.

So we're trying to avoid performance issues with LMDB and updating PKI is the only solution for existing sequential serial number installations. Those IPA installations using random serial numbers are not impacted but few users, particularly those with older installations, are using it. Switching to random serial numbers for users is possible but is involved.

It would be a two step process or a side-tag. Update pki to provide the new API and update IPA to utilize it. Note that pki can be updated separately as it still supports its older RPC APIs for backwards compatibility.

The PKI team tells us that the changes are involved enough that cherry-picking patches is not possible.

Since I had to look it up, "VLV" stands for "Virtual List View", an optional LDAP extension an older mode where all the serial numbers were... serial and sequential and a newer mechanism where they are randomized. Changing existing systems from sequential to randomized is prohibitively difficult and should not be attempted via RPM updates. Ok, that explains what not to do here.

OK, so 389 DS switched from Berkley DB to LMDB in 3.2.0 and this introduces a performance regression in the VLV extension when used with very large LDAP databases that are using the older, sequential serial numbers. Converting them to use the newer format would avoid the performance issue, but as noted above, this is too dangerous to attempt in an RPM transaction.

PKI 11.7 introduces a new API that gets the information that would have come through via VLV with some new mechanism that is not vulnerable to the same performance problem. I'm unclear on whether FreeIPA can speak both the VLV and new RPC API, or if that's part of the problem here.

In either case, if I'm reading this correctly, the downside to not updating PKI in the stable release is that there is a potentially serious performance degradation possible in very large databases, but we don't know how large or how serious that degradation will be. Do we have any statistics on how likely it will be for deployments of FreeIPA on Fedora to have large enough databases to trigger this concern?

Thank you, @sgallagh, for expanding the dense details (we should have done that ourselves!).

VLV in 389-ds was quite fragile in past as well, see https://frasertweedale.github.io/blog-redhat/posts/2020-09-17-dogtag-vlv-corruption.html for some of the Dogtag PKI-related details. The aim of use of PKI 11.7 is to let both Dogtag and FreeIPA to switch to a new REST API provided by PKI 11.7 that will avoid calling into VLV functions.

Performance wise, one of bigger 'offenders' is a certificate management through ACME. I know a lot of people do use ACME in FreeIPA to provide automated certificate issuance for their deployments in homelabs and companies. What we also figured out is that a lot of non-Fedora users do run FreeIPA in their environments through FreeIPA containers which are built on Fedora as well. That's their use of FreeIPA with clients which otherwise aren't based on Fedora.

10K-50K certificates is something that is not a very large database. You can easily get to that range within an organization with ~100 resources that need regularly requested certificates. ACME certificate issuance is at least 4 times a year, so you get to 10K range quite soon if there are production and test environments to cover with something like Kubernetes.

I'm a weak +1 here. The confluence of the issues makes me think it's enough to allow.

I remain -1.

this all should have landed in Fedora 42 at least 6 months ago because it did land in CentOS 10 Stream and RHEL. But it wasn't, so now we stuck with both upstream releases and Fedora updates.

Yeah, this is something to reconsider for the future. Pushing fixes downstream before Fedora just makes everything complicated and also means that the downstreams don't get the expected testing and benefit from Fedora.

OTOH, if this code has been in Stream and RHEL for a while, it should be good for F42 too.

This isn't great that we have to have this update, but it sounds like this will allow upstream work to be unblocked and move things forward in general. Even though this is an artificial creation by upstream policies, I think we should try to accomodate this.

Weak +1

Do we have actiual bugzillas on this for F42. If not, this is probably not a feature that is used by anyone, considering that F42 has been out for nearly 5 months.
The lack of clarity on the exact impact and reasons to do (or problems if it is not done) the upgrade leave me concerned about this change.

-1

Weak +1

+1

+1

Exception request for updating Dogtag PKI to 11.7 approved (+5, 1, -2)