fedora-infrastructure

#12733 Pull requests from a fork cannot be merged
Closed: Fixed by psloboda. Opened by psloboda.

Closed: Fixed
Reopen Issue

NOTE

If your issue is for security or deals with sensitive info please
mark it as private using the checkbox below.

Please check https://www.fedorastatus.org for known outages
before filing a ticket about an outage.

Describe what you would like us to do:

I have created a fork of mysql8.0 [0] and every time I create a pull request from said fork against the main repo the pull request cannot be merged as it says that it cannot be rebased, I have tried deleting and recreating the fork but the issue persists. We also tried with Michal Schorm to cherry pick the commit into his fork and create the pull request from there but the issue persists. This has occurred on every pull request I have created in the last ~month. I am sure I have branches I create the pull requests from up to date to the target branch but it does not seem to help. This makes my work on mysql8.0 very difficult as every pull request has to be cherry-picked into the main repo instead of being merged. This is very time consuming.

[0] https://src.fedoraproject.org/fork/psloboda/rpms/mysql8.0

When do you need this to be done by? (YYYY/MM/DD)

Eventually

Metadata Update from @kevin:
- Issue assigned to mschorm

Metadata Update from @james:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble

Metadata Update from @nphilipp:
- Issue tagged with: src.fp.o

Something similar also happened to @lecris with the pybind11 package, and I looked into it yesterday.

Here’s what I found:

  • I forked pybind11 myself and ran into the same issues (I couldn’t even push to my fork)
  • I took a peek at the database and found his and quite some other forked repos to have read_only set
  • Some affected entries in the projects table are quite old, some of the older ones even duplicated, i.e. same namespace/name for the same user
  • In the logs, I found entries where Pagure had trouble forking/cloning the repos:
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: 2025-08-19 15:24:11,751 [ERROR] celery.app.trace: Task pagure.lib.tasks.fork[5faea334-88e8-4b13-aa31-103d184e7d65] raised unexpected: PagureException('Did not manage to rebase this pull-request',)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: Traceback (most recent call last):
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/repo.py", line 35, in run_command
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     command, stderr=subprocess.STDOUT, cwd=cwd
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     **kwargs).stdout
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib64/python3.6/subprocess.py", line 438, in run
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     output=stdout, stderr=stderr)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: subprocess.CalledProcessError: Command '['git', 'clone', '/srv/git/repositories/rpms/pybind11.git', '/tmp/pagure-fork-j3ylb_8b']' returned non-zero exit status 128.
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: During handling of the above exception, another exception occurred:
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: Traceback (most recent call last):
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     R = retval = fun(*args, **kwargs)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     return self.run(*args, **kwargs)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/tasks_utils.py", line 39, in decorated_function
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     return function(self, session, *args, **kwargs)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/tasks.py", line 498, in fork
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     repo_from, "main", "fork"
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/git.py", line 1028, in __enter__
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     PagureRepo.clone(self._origpath, self.repopath)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/repo.py", line 78, in clone
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     run_command(cmd)
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:   File "/usr/lib/python3.6/site-packages/pagure/lib/repo.py", line 47, in run_command
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]:     "Did not manage to rebase this pull-request"
Aug 19 15:24:11 pkgs01.rdu3.fedoraproject.org celery[1887]: pagure.exceptions.PagureException: Did not manage to rebase this pull-request
  • This morning, I cleaned up behind myself (removed my fork of pybind11 in the DB, removed the corresponding forked package, tickets and requests repos, and tried again and it worked for me (I could push and submit a PR).
  • Coordinating with lecris, I unset the read_only flag on his fork after checking that the set of repos was present and looked okay. Then he could use it normally, too.
  • There are a number of affected forks (>40), these go back to 2019. About 10 of them are recent (July 2025 or later). Some of the forks have the corresponding repositories in place, for others, they are missing.

@psloboda Yours is one of the cases where the repositories on disk are missing. With your permission, I would delete the entry of your fork of mysql8.0 from the database, then you can again attempt to fork the repo, so we can see if the issue still persists for you.
Edited by nphilipp

For the record, the repo paths on pkgs01 are /srv/git/repositories/{,*/}forks/$USER/rpms/$PKG.git, i.e. the package repo itself and the auxiliary ones for requests, tickets and (optionally) docs.

Thank you @nphilipp, you can go ahead and delete it, there is currently nothing useful on the fork. Thanks.

Metadata Update from @nphilipp:
- Issue assigned to nphilipp (was: mschorm)

Hi! I’ve removed the entry from the database, please try to fork the repo again and report back.

Metadata Update from @nphilipp:
- Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

For the record, and if other forked packages need fixing, here’s what I did:

1) In the DB, check whether or not the fork is one of the affected projects:

pagure=> select u.user, p.id, p.namespace, p.name, p.date_created from projects p, users u where p.read_only=true and p.user_id=u.id order by p.date_created;

If not, bail out :wink:.

2) On pkgs01, check if the various repo directories (the forked pkg itself, requests, tickets and optionally docs) exist:

ls -l /srv/git/repositories/{,*/}forks/$USER/rpms/$NAME.git

3.1) If not, …

3.1.1) Delete the entry in the projects table:

pagure=> begin;
BEGIN
pagure=*> delete from projects where id=<the ID>;

Then run the query from 1) again to verify nothing bad happened, and commit if everything is okay:

pagure=*> select …;
pagure=*> commit;

3.1.2) Ask the reporter to recreate the fork and verify it works.

3.2) If the repo directories exist, …

3.2.1) Unset read_only for the project:

pagure=> begin;
BEGIN
pagure=*> update projects set read_only=false where id=<the ID>;

Then run the query from 1) again to verify everything worked fine, and commit:

pagure=*> select …;
pagure=*> commit;

3.2.2) Ask the reporter to verify that the fork is functional now (i.e. changes/branches can be pushed into it and PRs can be submitted from it).

NB: For accessing the DB, I ran psql from batcave because the version on pkgs01 is too old, so that commands to inspect the DB structure didn’t work. I grepped the credentials from /etc/pagure/pagure.cfg on pkgs01.

Thank you, after forking again it said that the fork already exists, I was instead redirected to it and it seems to be empty. I have tried to make simple testing commit to test out whether I can create a functional pr but it does not show up in the commits. The files folder shows No content found in this repository and there is a big banner on the top of the page:

 The permissions on this repository are being updated. This may take a while. During this time, you or some of the project's contributors may not be able to push to this repository.

So I guess I'll have to wait a bit before the changes take place. Still that is strange as I didn't have this issue with any other fork.
  • I forked pybind11 myself and ran into the same issues (I couldn’t even push to my fork)
  • I took a peek at the database and found his and quite some other forked repos to have read_only set

That usually means something failed in the task for creating the fork (the error
you've found seems to be about a rebase, so it's a little weird for me, but I
didn't check the pagure code in a while...).
Note that there is a pagure-admin command to change this flag (no need to poke
the DB directly).

Maybe related to https://pagure.io/releng/issue/12832

Yeah that's likely:

# ls -ld /srv/git/repositories/rpms/mysql8.0.git/objects/tmp_objdir-incoming-bWtwjG/
drwx--S---. 3 mschorm xpackager 18 Apr  5 21:15 /srv/git/repositories/rpms/mysql8.0.git/objects/tmp_objdir-incoming-bWtwjG/

Fixing the perms.

There are others, fixing as well:

# ls -ld /srv/git/repositories/rpms/*.git/objects/tmp_objdir-incoming-*
drwx--S---.  6 acme       xpackager 48 Jan 23  2025 /srv/git/repositories/rpms/dwarves.git/objects/tmp_objdir-incoming-PCmqku
drwx--S---.  3 atim       xpackager 18 Jun 23 19:55 /srv/git/repositories/rpms/icewm.git/objects/tmp_objdir-incoming-zQ2zAx
drwx--S---.  5 daxelrod   xpackager 38 Jul 21  2024 /srv/git/repositories/rpms/jowl.git/objects/tmp_objdir-incoming-fX5EvO
drwx--S---.  3 alciregi   xpackager 18 Jun 23 20:43 /srv/git/repositories/rpms/lite-xl.git/objects/tmp_objdir-incoming-evobum
drwxrwsr-x.  3 mschorm    xpackager 18 Apr  5 21:15 /srv/git/repositories/rpms/mysql8.0.git/objects/tmp_objdir-incoming-bWtwjG
drwx--S---.  8 bengal     xpackager 68 Jan 21  2025 /srv/git/repositories/rpms/NetworkManager.git/objects/tmp_objdir-incoming-F9Yyn3
drwx--S---.  3 steve      xpackager 18 Jan  2  2025 /srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Anb6ij
drwx--S---.  3 steve      xpackager 18 Jan  2  2025 /srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Ddhm7m
drwx--S---.  3 steve      xpackager 18 Jan  2  2025 /srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Eje8VQ
drwxrwsr-x.  7 tchaikov   xpackager 58 Jul 15  2024 /srv/git/repositories/rpms/pybind11.git/objects/tmp_objdir-incoming-kq3cjb
drwx--S---.  6 cheimes    xpackager 48 Mar 23  2024 /srv/git/repositories/rpms/pyOpenSSL.git/objects/tmp_objdir-incoming-kBujbF
drwx--S---.  3 sagitter   xpackager 18 Jun 23 19:56 /srv/git/repositories/rpms/python-reportlab.git/objects/tmp_objdir-incoming-qjUbRk
drwx--S---.  3 fmaurer    xpackager 18 Jun 23 20:49 /srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-3g9Car
drwx--S---.  3 fmaurer    xpackager 18 Jun 23 20:34 /srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-HJ9a8p
drwx--S---.  3 fmaurer    xpackager 18 Jun 23 20:40 /srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-svldMi
drwx--S---.  3 decathorpe xpackager 18 Jan 23  2025 /srv/git/repositories/rpms/rust-cargo_metadata0.18.git/objects/tmp_objdir-incoming-dct742
drwx--S---.  3 decathorpe xpackager 18 Jan 23  2025 /srv/git/repositories/rpms/rust-cargo_metadata0.18.git/objects/tmp_objdir-incoming-U5XNoR
drwx--S---. 10 pemensik   xpackager 88 Aug  3  2024 /srv/git/repositories/rpms/rust-criterion.git/objects/tmp_objdir-incoming-qnmsnn
drwx--S---.  3 1494357580 xpackager 18 Jun 28 15:21 /srv/git/repositories/rpms/rust-gitui.git/objects/tmp_objdir-incoming-tJsr8Z
drwx--S---.  3 music      xpackager 18 Apr 24 00:32 /srv/git/repositories/rpms/rust-prodash.git/objects/tmp_objdir-incoming-73VZF3
drwx--S---.  3 music      xpackager 18 Apr 21 00:25 /srv/git/repositories/rpms/sleef.git/objects/tmp_objdir-incoming-mBTcXi
drwx--S---.  6 1494357580 xpackager 48 Jun 23 19:44 /srv/git/repositories/rpms/tomcat-native.git/objects/tmp_objdir-incoming-RMPZRn
drwx--S---.  3 mtasaka    xpackager 18 May 23  2023 /srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-M0VQmJ
drwx--S---.  3 mtasaka    xpackager 18 May 23  2023 /srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-mHoq3Z
drwx--S---.  3 mtasaka    xpackager 18 May 23  2023 /srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-Xd42Az
drwx--S---.  9 neil       xpackager 78 Sep  1  2023 /srv/git/repositories/rpms/zeek.git/objects/tmp_objdir-incoming-jooBBz

Running this to give them the same perms as other files in the folder:

# chmod =2775 /srv/git/repositories/rpms/*.git/objects/tmp_objdir-incoming-*
Edited by abompard

Could you please try again?

If my assumptions are correct, all directories under /srv/git/repositories should have mode 2775 (u=rwx,g=rws,o=rx) and, running find /srv/git/repositories -type d -a \! -perm 2775, I find quite a number that don’t. Should I fix them, too?
Edited by nphilipp

There are others, fixing as well:

# ls -ld /srv/git/repositories/rpms/*.git/objects/tmp_objdir-incoming-*

It seems that these shouldn’t exist at all unless part of an ongoing git operation: https://stackoverflow.com/questions/60735574/cleaning-git-dir-objects-incoming-temporary-directories#comment107459332_60735574
Edited by nphilipp

If my assumptions are correct, all directories under /srv/git/repositories should have mode 2775 (u=rwx,g=rws,o=rx) and, running find /srv/git/repositories -type d -a \! -perm 2775, I find quite a number that don’t. Should I fix them, too?

Yeah, I suppose.

There are others, fixing as well:

# ls -ld /srv/git/repositories/rpms/*.git/objects/tmp_objdir-incoming-*

It seems that these shouldn’t exist at all unless part of an ongoing git operation: https://stackoverflow.com/questions/60735574/cleaning-git-dir-objects-incoming-temporary-directories#comment107459332_60735574

Right, I guess we can actually delete those. Maybe using find and a limit on the mtime, because we don't want to run the command right when someones does a git operation.
Edited by abompard

Right, I guess we can actually delete those. Maybe using find and a limit on the mtime, because we don't want to run the command right when someones does a git operation.

I removed all of them older than a day (-mmin +1440), which removed all of them:

/srv/git/repositories/forks/fweimer/rpms/glibc.git/objects/tmp_objdir-incoming-h2hytd
/srv/git/repositories/forks/gotmax23/rpms/matrix-synapse.git/objects/tmp_objdir-incoming-DmFkAG
/srv/git/repositories/rpms/lite-xl.git/objects/tmp_objdir-incoming-evobum
/srv/git/repositories/rpms/mysql8.0.git/objects/tmp_objdir-incoming-bWtwjG
/srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-3g9Car
/srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-HJ9a8p
/srv/git/repositories/rpms/rust-bzip2-rs.git/objects/tmp_objdir-incoming-svldMi
/srv/git/repositories/rpms/rust-cargo_metadata0.18.git/objects/tmp_objdir-incoming-U5XNoR
/srv/git/repositories/rpms/rust-cargo_metadata0.18.git/objects/tmp_objdir-incoming-dct742
/srv/git/repositories/rpms/rust-criterion.git/objects/tmp_objdir-incoming-qnmsnn
/srv/git/repositories/rpms/rust-gitui.git/objects/tmp_objdir-incoming-tJsr8Z
/srv/git/repositories/rpms/sleef.git/objects/tmp_objdir-incoming-mBTcXi
/srv/git/repositories/rpms/supernovas.git-20250221/objects/tmp_objdir-incoming-hv849q
/srv/git/repositories/rpms/NetworkManager.git/objects/tmp_objdir-incoming-F9Yyn3
/srv/git/repositories/rpms/dwarves.git/objects/tmp_objdir-incoming-PCmqku
/srv/git/repositories/rpms/icewm.git/objects/tmp_objdir-incoming-zQ2zAx
/srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Anb6ij
/srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Ddhm7m
/srv/git/repositories/rpms/perl-Cache-Mmap.git/objects/tmp_objdir-incoming-Eje8VQ
/srv/git/repositories/rpms/pyOpenSSL.git/objects/tmp_objdir-incoming-kBujbF
/srv/git/repositories/rpms/pybind11.git/objects/tmp_objdir-incoming-kq3cjb
/srv/git/repositories/rpms/tomcat-native.git/objects/tmp_objdir-incoming-RMPZRn
/srv/git/repositories/rpms/jowl.git/objects/tmp_objdir-incoming-fX5EvO
/srv/git/repositories/rpms/python-reportlab.git/objects/tmp_objdir-incoming-qjUbRk
/srv/git/repositories/rpms/rust-prodash.git/objects/tmp_objdir-incoming-73VZF3
/srv/git/repositories/rpms/ttyd.git-20250219/objects/tmp_objdir-incoming-C3fzAl
/srv/git/repositories/rpms/ttyd.git-20250219/objects/tmp_objdir-incoming-C4LdMw
/srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-M0VQmJ
/srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-Xd42Az
/srv/git/repositories/rpms/ugene.git/objects/tmp_objdir-incoming-mHoq3Z
/srv/git/repositories/rpms/zeek.git/objects/tmp_objdir-incoming-jooBBz

@psloboda I’ve removed the broken forked project and with @abompard’s cleaning up the source repo, forking mysql8.0 should work again. Would you give it a try?

Metadata Update from @nphilipp:
- Assignee reset

Metadata Update from @nphilipp:
- Issue assigned to nphilipp

Everything seems to be working now, I can even create a pr[0] and the merge button is green (so I hope it works as I don't want to merge some nonsense into the repo). Thank you.

[0] https://src.fedoraproject.org/rpms/mysql8.0/pull-request/12

Metadata Update from @psloboda:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

medium-gain medium-trouble src.fp.o

None
None
High
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.