#351 Beta Software Released On Main EPEL Channel
Opened by scuttlefish42. Modified

I am concerned that openvpn beta (openvpn-2.7_beta2-1.el10_0.x86_64) is being released on the standard EPEL channel.

"The OpenVPN community project team is proud to release OpenVPN 2.7_beta2. This is the second Beta release for the feature release 2.7.0. As the Beta name implies this is an early release build, it is not intended for production use."

If I do 'dnf update' on a production server I don't expect to get beta software.

"The goal of EPEL is to make high quality Fedora packages available for RHEL and compatible derivatives."

Beta software is surely not "high quality".

I feel this is a breach of the trust we place on EPEL.

This is mission critical privacy and security software.


That was done by the upstream software developers who maintain the package in Fedora and EPEL.

That said, I am weirded out by @dsommers and @flichtenheld both releasing newer versions into EPEL 10 but not into Fedora. Something is going wrong here.

Some clarifications are needed here. The concerns are valid, but it is not as severe as it might seem at first glance. I am an upstream OpenVPN contributor and has been involved in OpenVPN 2.x development earlier plus have done Fedora openvpn packaging for a long while. Due to this, I see this from a quite broad perspective.

First, @ngompa is correct! It's a glaring misstep to not have the 2.7 versions in Fedora 43 and Rawhide. We will correct this for the Rawhide very soon and look into if it is too late for the F43 cycle or not.

So some background on EPEL-10 and the OpenVPN 2.7 releases. The OpenVPN community had planned the 2.7 release to happen earlier this summer. And I understood the RHEL-10 release was due this summer as well (it was released in May 20205). So for the RHEL 10 Beta, which was released in December 2024, we prepared the EPEL-10 repo to use the OpenVPN 2.7 baseline.

The reason was simply to provide newer feature which was already prepared and more which was coming, in time for the planned 2.7 release this summer. These includes support for the kernel support for the new ovpn kernel module, which was accepted into the mainline kernel earlier this year (6.16 kernel). We do have the ovpn-backport project providing this driver for older kernels, such as the RHEL-10 kernel-6.12.

Another important feature is the possibility to listen to multiple sockets (ports and protocols), a feature which has been requested for many years already.

There are also crypto related enhancements making the VPN tunnels more efficient and future proof.

On top of that, there are changes to the configuration options which only happens between major releases (2.6 to 2.7 is a major release in the OpenVPN project). That restricts us from upgrading from 2.6 to 2.7 version during the lifetime of EPEL-10. And then it is the maintenance support which will hit the EPEL package maintainers once the 2.6 release reaches an EOL phase after the 2.7 release has happened.

So all in all, it was considered reasonable to ship the OpenVPN 2.7 base line in EPEL-10 to provide important features and functionality for the life time of RHEL-10. And since the initial release plan was very close between OpenVPN 2.7 and RHEL-10, this seemed reasonable.

Unfortunately, the OpenVPN 2.7 release has been delayed. The last beta release is planned in coming soon and the final release is just around the corner after that.

In regards to OpenVPN development, it is done in a very conservative way with several tests being performed for before each change/pull-request being accepted and merged. When the OpenVPN project releases beta and RC releases, they are so close to release that they are generally just as stable as the final release. From the first alpha release, there are only bug fixes accepted and as the beta phase enters, there are often even more restrictions to changing internal code - even though that does not impact the user interface in any way. This is a pretty hard line to ensure the code is stabilised as quickly as possible while being tested quite broadly by the overall community.

With that in mind, I do not agree that «[OpenVPN] Beta [versions] is not "high quality"». In contrary, the history since I got involved in the OpenVPN project over 15 years ago has shown that there are very seldom found critical bugs in beta and RC releases.

And rest assure - if a highly critical issue would ever be found in the EPEL-10 openvpn package, it would be patched outside of an OpenVPN 2.7 beta/rc/final release if the official update from the project would be delayed. @flichtenheld is very active in the OpenVPN development these days, and I am not far behind either if needed. We are not packaging openvpn from the outside of the upstream OpenVPN project.

So from that context, knowing the history of OpenVPN development, the development process of the project and the stability of the code base, aligned with both RHEL-10 and OpenVPN 2.7 release schedules, I stand quite solid in the decision we did on providing the 2.7 codebase in EPEL-10.

The other alternative I did consider was to not provide any openvpn package at all until the 2.7 release would be ready - and have only a Fedora Copr repository available in the mean time. But already early during the RHEL-10 Beta release, there were lots of people looking for the openvpn package.

And, for reasons mentioned earlier, shipping OpenVPN 2.6 for EPEL-10 - knowing EPEL-10 has a 10 year support time line and OpenVPN 2.6 will be unsupported by the OpenVPN project 18 months after OpenVPN 2.7 is released, I felt it was very irresponsible to not pave the way for OpenVPN 2.7 in EPEL-10. The recent OpenVPN major releases has usually had minimum 5 years of support by the project, the 2.6 release will probably end some time before the summer of 2027. RHEL-10 will be supported until 2035.

More details about the OpenVPN 2.7 version progress: https://community.openvpn.net/Development/StatusOfOpenvpn27

Latest OpenVPN community meeting: https://community.openvpn.net/Meetings/2025/2025-10-01

More details about the supported versions: https://community.openvpn.net/Pages/Supported%20versions

As a side remark, a discussion has started recently to clarify the expected support level for the alpha/beta/rc releases - which ends up as final release anyhow.

First, @ngompa is correct! It's a glaring misstep to not have the 2.7 versions in Fedora 43 and Rawhide. We will correct this for the Rawhide very soon and look into if it is too late for the F43 cycle or not.

It should be possible to get it into F43 if you get it done before freeze on the 7th.

The reason was simply to provide newer feature which was already prepared and more which was coming, in time for the planned 2.7 release this summer. These includes support for the kernel support for the new ovpn kernel module, which was accepted into the mainline kernel earlier this year (6.16 kernel). We do have the ovpn-backport project providing this driver for older kernels, such as the RHEL-10 kernel-6.12.

Have you engaged with Red Hat to try to get it backported and shipped in the RHEL kernel in some way?

@ngompa Not yet. I wanted to see it in Fedora first and have a proven case with the ovpn-backport project first.

First, @ngompa is correct! It's a glaring misstep to not have the 2.7 versions in Fedora 43 and Rawhide. We will correct this for the Rawhide very soon and look into if it is too late for the F43 cycle or not.

It should be possible to get it into F43 if you get it done before freeze on the 7th.

Ahh, brilliant! I'll get that rolling!

First, @ngompa is correct! It's a glaring misstep to not have the 2.7 versions in Fedora 43 and Rawhide. We will correct this for the Rawhide very soon and look into if it is too late for the F43 cycle or not.

It should be possible to get it into F43 if you get it done before freeze on the 7th.

Ahh, brilliant! I'll get that rolling!

Fedora Rawhide is ready: https://koji.fedoraproject.org/koji/taskinfo?taskID=137806799

I'm not so sure this is a good change for Fedora 43, since it's the final freeze before GA (line 132) coming up in 2 days. I'm concerned about users who is running F43 Beta with OpenVPN 2.6 working and then breaking on the GA release.

In addition, due to how NetworkManager-openvpn handles the OpenVPN configurations, this might break there as well. Even though, that might be somewhat covered via EPEL-10 users - but it's not a full guarantee.

It feels a bit too risky and rushed and might upset Fedora 43 users much more. I feel Fedora users deserves better. But I'm truly sorry we missed this opportunity for Fedora 43. We didn't pay enough attention to the Fedora release schedule, as some internal talks months ago highlighted Fedora 43 as an ideal Fedora target release due to the ovpn kernel module being available there.

The @OpenVPN/openvpn-beta Copr repo provides the OpenVPN 2.7 builds for the time being.

You must have already figured out how to handle these transitions between EL9 and EL10, so wouldn't F42 to F43 benefit from that?

You must have already figured out how to handle these transitions between EL9 and EL10, so wouldn't F42 to F43 benefit from that?

Based on prior experience, the OpenVPN configurations used by Fedora users varies a lot more than EPEL users. There are much more options in use.

Had we've been awake and pushed out the earlier 2.7 codebase before Fedora 43 Beta was released, we could have tackled issues related to that upgrade in a better way.

From the OpenVPN project's perspective, breaking configuration changes are acceptable if they can't be avoided only in new major releases. So as an extension to that, when you upgrade to a Fedora Beta release we find it acceptable that a new OpenVPN major release might bring in breaking configuration changes. And that is also one of the reasons we decided to bring in the 2.7 codebase into EPEL-10 as well, so RHEL-10 users would end up on a stable 2.7 baseline once the needed configuration adoptions was done.

Most users may expects some rough edges in the beginning of a major distribution upgrade. But pushing a package update possibly breaking something very late to the distribution release cycle can come as a much bigger surprise. Possibly causing a regression going from a beta release to a GA release feels very wrong to me, especially when the most likely resolution would be to downgrade the package.

Metadata Update from @carlwgeorge:
- Issue tagged with: meeting

It's a glaring misstep to not have the 2.7 versions in Fedora 43 and Rawhide.

I'm glad this part was a misstep and not intentional. Generally I would advise releasing new versions in Rawhide first, then a stable Fedora release, then EPEL releases. This staged approach can reduce the impact of a regression if one were to occur.

The reason was simply to provide newer feature which was already prepared and more which was coming, in time for the planned 2.7 release this summer.

This is understandable, but IMO the right answer would have been to wait to add openvpn 2.7 to EPEL until it was released. I could understand maybe adding it once it reached RC status, but the update history shows this was first added as a pre-alpha git snapshot, then updated to alpha, then updated to beta. That's not inline with the spirit of the EPEL updates policy. The EPEL Steering Committee is working on making this explicit in the policy (likely an RFC 2119 "SHOULD NOT" to allow for exceptions).

These includes support for the kernel support for the new ovpn kernel module, which was accepted into the mainline kernel earlier this year (6.16 kernel). We do have the ovpn-backport project providing this driver for older kernels, such as the RHEL-10 kernel-6.12.

This should not be a factor for the EPEL package, because EPEL packages are intended to work with stock CentOS/RHEL and are not allowed to depend on things outside of the base distro and EPEL. I hope openvpn 2.7 doesn't outright require this kernel module.

That restricts us from upgrading from 2.6 to 2.7 version during the lifetime of EPEL-10.

Indeed, a regular update requiring configuration changes would not be appropriate for EPEL. However be aware there is also the incompatible upgrades process for when it is unavoidable.

With that in mind, I do not agree that «[OpenVPN] Beta [versions] is not "high quality"».

Whether or not openvpn betas are high quality or not is not the point here. EPEL is intended to be used on enterprise distros in production environments, and enterprise users generally do not wish to deploy pre-release software in those environments. The upcoming policy changes will reflect that.

The other alternative I did consider was to not provide any openvpn package at all until the 2.7 release would be ready - and have only a Fedora Copr repository available in the mean time.

This would have been a much better approach IMO. I took the same route with some of my packages, providing them in copr until conditions lined up for them to be appropriate for EPEL.

And, for reasons mentioned earlier, shipping OpenVPN 2.6 for EPEL-10 - knowing EPEL-10 has a 10 year support time line and OpenVPN 2.6 will be unsupported by the OpenVPN project 18 months after OpenVPN 2.7 is released, I felt it was very irresponsible to not pave the way for OpenVPN 2.7 in EPEL-10.

This affects virtually all software in EPEL. I'm not aware of any software packaged in EPEL that have as long a lifecycle as EPEL itself. Just like CentOS/RHEL packages, it's totally normal for EPEL packages to stay on a version that is no longer maintained upstream. This is how stability is achieved, with a bias towards backporting required fixes instead of rebasing to a new version.

The recent OpenVPN major releases has usually had minimum 5 years of support by the project, the 2.6 release will probably end some time before the summer of 2027. RHEL-10 will be supported until 2035.

So that means 2.7 will likely run into the same problem in 2030?

It feels a bit too risky and rushed and might upset Fedora 43 users much more. I feel Fedora users deserves better.

This statement makes me really worried about this being in EPEL. Fedora users are far more adaptable to change than enterprise users.

And I understood the RHEL-10 release was due this summer as well (it was released in May 20205).

One other thing I need to clarify here. While RHEL 10 wasn't released until May 2025, EPEL 10 was released in December 2024 alongside CentOS Stream 10. CentOS users should be given the same consideration as RHEL users, and should not have been provided pre-release software via EPEL.

Metadata Update from @rcallicotte:
- Issue assigned to rcallicotte

Metadata Update from @tdawson:
- Issue untagged with: meeting

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/epel/steering/issues/351

Please continue any further discussion there.

Metadata