#215 remove singularity obsoletes from apptainer
Closed: Fixed by tdawson. Opened by carlwgeorge.

While discussing #214 at the EPEL Steering Committee today, it was suggested that the apptainer package should also remove the obsoletes for the singularity package. Originally this was treated as a simple package rename from singularity to apptainer, but this transition involves some incompatibilities. Specifically, it changed the config directory, without carrying forward previous config settings. It also switched to unprivileged user namespaces, which is not enabled by default on the RHEL7 kernel, meaning manual changes are required there to keep using the software. Neither of these are acceptable for a regular EPEL update. Instead of following the incompatible upgrades process, the maintainer made the change with no announcement, and even admitted later that he did it in spite of being fully aware of the incompatibilities. Several weeks after the update was promoted to stable, a late announcement was sent about the incompatibilities.

The suggestion at the meeting today was that even though this has already negatively impacted users, there are likely many more users that haven't updated yet. We can reduce the impact of this incompatible change by removing the obsoletes from apptainer. That means that users with singularity installed will continue to have it working, but will have to take manual action to switch to a maintained SIF runtime, such as apptainer (with the manual changes to account for its incompatibilities) or singularity-ce (which is reported to be more compatible with the original singularity).

In #214 the EPEL Steering Committee members are voting about removing the singularity provides from apptainer EPEL branches. In this issue we will vote on removing the singularity obsoletes from apptainer EPEL branches. These are independent votes.


Removing both Provides and Obsoletes is a non-starter for security reasons. It would leave existing singularity users without security updates.

Here are more user reports that appear to be about the incompatible changes.

https://github.com/apptainer/apptainer/issues/769
https://github.com/apptainer/apptainer/issues/789
https://github.com/apptainer/apptainer/issues/778
https://github.com/apptainer/apptainer/issues/868
https://github.com/apptainer/apptainer/issues/871
https://github.com/apptainer/apptainer/issues/876

See my comment here
https://pagure.io/epel/issue/214#comment-832564

From Fedora renaming guidelines, and the points I talk about in the comment

apptainer should NOT have a Provides but it SHOULD have an Obsoletes

After giving this some thought, I think we have more options than just removing or not removing the obsoletes. Here are all the possible the choices we have.

  1. Leave the singularity obsoletes in apptainer. This exposes more users to the disruptive update, which we've already seen is causing problems. Some users are already blaming this on EPEL overall rather than on the specific package.

  2. Remove the singularity obsoletes from apptainer. This mean that users that haven't updated yet will be spared from the disruptive update, but are then stuck with an unmaintained singularity package until they take manual action (yum swap singularity apptainer, yum swap singularity apptainer-suid, or yum swap singularity singularity-ce).

  3. Move the singularity obsoletes from apptainer to apptainer-suid. This is still a disruptive update (manual config migration, manual setting environment variable for some behavior, manual config to continue pulling from Sylabs remote endpoint, etc.) but the the most disruptive change of switching unprivileged user namespaces would be avoided for users that haven't updated yet.

  4. Move the singularity obsoletes from apptainer to singularity-ce. This would provide the most compatible upgrade path based on the information I've seen so far.

Number 3 is an intriguing idea. Number 2 is very bad because of a lack of security updates. Number 4 is bad because it would lead to a confused mix of people who have updated to different projects, and people from now on will be switched to a different project without their knowledge.

I REALLY wish you had kept everything on one issue. We now have two separate issues discussing the exact same thing. Very confusing.

For those that haven't skipped over to the other issue to see what I wrote, I will re-write it here, because it is relevant to my replies.

First point. Is Apptainer a rename of Singularity?
From everything I can see, legally, yes it is.

Second point. Does it provide a similar functionality?
From everything I can see, yes, in the overall sense.

Third point. Is it backwards compatible?
Without modifications on the user machine. No.
With modification on the user machine. Yes

So the real question here is, how incompatible does something have to be before we do not allow it to have a "Provides" and/or "Obsoletes"?

For that we should go to the Fedora Renaming guidelines.
https://docs.fedoraproject.org/en-US/packaging-guidelines/#renaming-or-replacing-existing-packages

Reading those guidelines we see

If a package is being renamed without any functional changes, or is a compatible enough replacement to an existing package (where "enough" means that it includes only changes of magnitude that are commonly found in version upgrade changes), provide clean upgrade paths and compatibility with:

Provides: oldpackagename = $provEVR
Obsoletes: oldpackagename < $obsEVR

If a package supersedes/replaces an existing package without being a sufficiently compatible replacement as defined above, use only the Obsoletes: line from the above example.

To me, it looks like this rename is not a sufficiently compatible replacement. Thus it should NOT have a Provides, but it SHOULD have an Obsoletes.

After giving this some thought, I think we have more options than just removing or not removing the obsoletes. Here are all the possible the choices we have.

  1. Leave the singularity obsoletes in apptainer. This exposes more users to the disruptive update, which we've already seen is causing problems. Some users are already blaming this on EPEL overall rather than on the specific package.

If they do, by policy, it is their right to do so.

  1. Remove the singularity obsoletes from apptainer. This mean that users that haven't updated yet will be spared from the disruptive update, but are then stuck with an unmaintained singularity package until they take manual action (yum swap singularity apptainer, yum swap singularity apptainer-suid, or yum swap singularity singularity-ce).

Again, by policy, it is the app maintainers right to keep it in, or not. That is not something the EPEL Steering Committee should have a say in.

  1. Move the singularity obsoletes from apptainer to apptainer-suid. This is still a disruptive update (manual config migration, manual setting environment variable for some behavior, manual config to continue pulling from Sylabs remote endpoint, etc.) but the the most disruptive change of switching unprivileged user namespaces would be avoided for users that haven't updated yet.

I personally like this idea. But as I said in the first two, by policy, it is the app maintainers right to keep it in, or not. That is not something the EPEL Steering Committee should have a say in.

  1. Move the singularity obsoletes from apptainer to singularity-ce. This would provide the most compatible upgrade path based on the information I've seen so far.

This shouldn't even be in this discussion. singularity-ce is a fork. apptainer is a rename.
If singularity-ce wants to put an obsoletes in their package, then they should follow the obsoletes policy.

I REALLY wish you had kept everything on one issue. We now have two separate issues discussing the exact same thing. Very confusing.

I asked at the last committee meeting if this should be all in #214 or separated into another issue to have an separate vote. One person said they preferred it be separate, and no one voiced a preference to keep it all in the same issue. If you want to combine them at this point it's fine by me.

To me, it looks like this rename is not a sufficiently compatible replacement. Thus it should NOT have a Provides, but it SHOULD have an Obsoletes.

I am in favor of this outcome, possibly slightly modified to the apptainer-suid idea.

That is not something the EPEL Steering Committee should have a say in.

The maintainer asked for the committee to weigh in.

My apologies @tdawson

The items being in two different tickets was my fault. Having an argument over whether a package should remove a provides which was in line with other FESCO policies seemed separate from GOING beyond what was normal by removing an obsoletes. Especially since this was asking for the package to have it removed in EPEL but not asking for it in Fedora.

I would also like to sincerely apologize to @dwd . I should have made it clear that a discussion on this required the packager time to answer in a ticket and to make sure all sides had time to think about things.

singularity-ce is a fork. apptainer is a rename.

They are both forks.

In terms of EPEL it was handled as a rename yes, but apptainer is a fork of the original singularity, and singularity-ce is a fork of the original singularity.

Now that I am feeling better, I think I can reply politely and sensibly.

Now that I've had time to think of it, I like having two issues, one for provides and one for obsoletes. Since there are different criteria for each, it allows us to talk about each independently.

As for obsoletes. I think the policy is pretty clear for this. apptainer SHOULD use an obsoletes.

From the discussion in here, the underlying dispute is whether Apptainer is considered sufficiently compatible to retain Obsoletes: singularity.

The discussion in #214 and the references in this ticket seem to indicate that the answer is "no".

If an Obsoletes is used, it needs to express a reasonably compatible upgrade path, which I have yet to be convinced exists.

No, that is not correct. From the Fedora Policy (and quoted above)

If a package supersedes/replaces an existing package without being a sufficiently compatible replacement as defined above, use only the Obsoletes: line from the above example.

Thus, if a package isn't compatible enough, it ONLY has Obsoletes.

No, that is not correct. From the Fedora Policy (and quoted above)

If a package supersedes/replaces an existing package without being a sufficiently compatible replacement as defined above, use only the Obsoletes: line from the above example.

Thus, if a package isn't compatible enough, it ONLY has Obsoletes.

Hmm, my (RPM-level) understanding of Obsoletes clashes with this policy statement. I think this is something I have to bring up to FPC to figure out, but you're right.

So Obsoletes is probably fine by the strict reading of our policy.

The EPEL Steering Commitee agrees that and Obsoletes should be in the package, preferably in the apptainer-suid package.

Metadata Update from @tdawson:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/epel/steering/issues/215

Please continue any further discussion there.

Metadata