fedora-infrastructure

#11147 PDC unhappy? pre-receive hooks crash when git-pushing to dist-git on src.fp.o
Closed: Fixed by decathorpe. Opened by decathorpe.

Closed: Fixed
Reopen Issue

It appears that PDC is in some kind of unhappy state. About 50% of my "git push" operations to dist-git repositories on src.fp.o randomly fail today with errors like this:

remote: Unspecified ref refs/heads/f37 is blocked
remote: Denied push for ref 'refs/heads/f37' for user 'decathorpe'
remote: All changes have been rejected
To ssh://pkgs.fedoraproject.org/rpms/rust-zstd-safe
 ! [remote rejected] f37 -> f37 (pre-receive hook declined)
error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/rust-zstd-safe'

(which is wrong, f37 is a valid branch), or this:

 ! [remote rejected] f36 -> f36 (hook declined)
error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/rust-trust-dns-resolver'

Retrying the operation usually succeeds.

I also got a PDC crash when attempting to request a branch with "fedpkg request-branch", which returned just a HTTP 500 "internal server error" page, but a retried request also succeeded.

From the logs:

[Fri Feb 24 00:47:05.340294 2023] [:error] [pid 10445] 2023-02-24 00:47:05,339 10445 [base.py -- base.handle_uncaught_exception:256] [ERROR]- Internal Server Error: /compose/17940/rpms/container/x86_64/
[Fri Feb 24 00:47:05.340333 2023] [:error] [pid 10445] Traceback (most recent call last):
[Fri Feb 24 00:47:05.340335 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 125, in get_response
[Fri Feb 24 00:47:05.340338 2023] [:error] [pid 10445]     response = middleware_method(request, callback, callback_args, callback_kwargs)
[Fri Feb 24 00:47:05.340340 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/pdc/apps/changeset/middleware.py", line 73, in process_view
[Fri Feb 24 00:47:05.340342 2023] [:error] [pid 10445]     response = view_func(request, *view_args, **view_kwargs)
[Fri Feb 24 00:47:05.340344 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
[Fri Feb 24 00:47:05.340346 2023] [:error] [pid 10445]     return self.dispatch(request, *args, **kwargs)
[Fri Feb 24 00:47:05.340348 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
[Fri Feb 24 00:47:05.340350 2023] [:error] [pid 10445]     return handler(request, *args, **kwargs)
[Fri Feb 24 00:47:05.340352 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 52, in get
[Fri Feb 24 00:47:05.340354 2023] [:error] [pid 10445]     return self.post(request,  *args,  **kwargs)
[Fri Feb 24 00:47:05.340356 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 215, in post
[Fri Feb 24 00:47:05.340358 2023] [:error] [pid 10445]     return self.form_valid(form)
[Fri Feb 24 00:47:05.340360 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 70, in form_valid
[Fri Feb 24 00:47:05.340361 2023] [:error] [pid 10445]     self.queryset = self.get_queryset().filter(query)
[Fri Feb 24 00:47:05.340363 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/pdc/apps/compose/views.py", line 90, in get_queryset
[Fri Feb 24 00:47:05.340365 2023] [:error] [pid 10445]     variant_arch = variant_arch.get(arch=arch)
[Fri Feb 24 00:47:05.340367 2023] [:error] [pid 10445]   File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
[Fri Feb 24 00:47:05.340369 2023] [:error] [pid 10445]     self.model._meta.object_name
[Fri Feb 24 00:47:05.340371 2023] [:error] [pid 10445] DoesNotExist: VariantArch matching query does not exist.
[Fri Feb 24 00:54:48.813319 2023] [:error] [pid 13267] 2023-02-24 00:54:48,813 13267 [base.py -- base.handle_uncaught_exception:256] [ERROR]- Internal Server Error: /compose/17941/images/cloud/aarch64/
[Fri Feb 24 00:54:48.813345 2023] [:error] [pid 13267] Traceback (most recent call last):
[Fri Feb 24 00:54:48.813347 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 125, in get_response
[Fri Feb 24 00:54:48.813350 2023] [:error] [pid 13267]     response = middleware_method(request, callback, callback_args, callback_kwargs)
[Fri Feb 24 00:54:48.813352 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/pdc/apps/changeset/middleware.py", line 73, in process_view
[Fri Feb 24 00:54:48.813354 2023] [:error] [pid 13267]     response = view_func(request, *view_args, **view_kwargs)
[Fri Feb 24 00:54:48.813356 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
[Fri Feb 24 00:54:48.813367 2023] [:error] [pid 13267]     return self.dispatch(request, *args, **kwargs)
[Fri Feb 24 00:54:48.813369 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
[Fri Feb 24 00:54:48.813371 2023] [:error] [pid 13267]     return handler(request, *args, **kwargs)
[Fri Feb 24 00:54:48.813373 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 52, in get
[Fri Feb 24 00:54:48.813375 2023] [:error] [pid 13267]     return self.post(request,  *args,  **kwargs)
[Fri Feb 24 00:54:48.813376 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 215, in post
[Fri Feb 24 00:54:48.813378 2023] [:error] [pid 13267]     return self.form_valid(form)
[Fri Feb 24 00:54:48.813380 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 70, in form_valid
[Fri Feb 24 00:54:48.813382 2023] [:error] [pid 13267]     self.queryset = self.get_queryset().filter(query)
[Fri Feb 24 00:54:48.813384 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/pdc/apps/compose/views.py", line 127, in get_queryset
[Fri Feb 24 00:54:48.813385 2023] [:error] [pid 13267]     variant_arch = variant_arch.get(arch=arch)
[Fri Feb 24 00:54:48.813387 2023] [:error] [pid 13267]   File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
[Fri Feb 24 00:54:48.813389 2023] [:error] [pid 13267]     self.model._meta.object_name
[Fri Feb 24 00:54:48.813391 2023] [:error] [pid 13267] DoesNotExist: VariantArch matching query does not exist.
[Fri Feb 24 01:45:26.859391 2023] [:error] [pid 10445] 2023-02-24 01:45:26,859 10445 [renderers.py -- renderers.get_field_type:343] [ERROR]- Undocumented field BranchNameField()
[Fri Feb 24 02:51:14.216522 2023] [:error] [pid 10470] 2023-02-24 02:51:14,216 10470 [renderers.py -- renderers.get_field_type:343] [ERROR]- Undocumented field BranchNameField()
[Fri Feb 24 03:22:48.452790 2023] [:error] [pid 21807] 2023-02-24 03:22:48,451 21807 [base.py -- base.handle_uncaught_exception:256] [ERROR]- Internal Server Error: /compose/17920/rpms/atomichost/x86_64/
[Fri Feb 24 03:22:48.452806 2023] [:error] [pid 21807] Traceback (most recent call last):
[Fri Feb 24 03:22:48.452808 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 125, in get_response
[Fri Feb 24 03:22:48.452809 2023] [:error] [pid 21807]     response = middleware_method(request, callback, callback_args, callback_kwargs)
[Fri Feb 24 03:22:48.452811 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/pdc/apps/changeset/middleware.py", line 73, in process_view
[Fri Feb 24 03:22:48.452813 2023] [:error] [pid 21807]     response = view_func(request, *view_args, **view_kwargs)
[Fri Feb 24 03:22:48.452814 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
[Fri Feb 24 03:22:48.452816 2023] [:error] [pid 21807]     return self.dispatch(request, *args, **kwargs)
[Fri Feb 24 03:22:48.452817 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
[Fri Feb 24 03:22:48.452819 2023] [:error] [pid 21807]     return handler(request, *args, **kwargs)
[Fri Feb 24 03:22:48.452820 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 52, in get
[Fri Feb 24 03:22:48.452822 2023] [:error] [pid 21807]     return self.post(request,  *args,  **kwargs)
[Fri Feb 24 03:22:48.452823 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 215, in post
[Fri Feb 24 03:22:48.452825 2023] [:error] [pid 21807]     return self.form_valid(form)
[Fri Feb 24 03:22:48.452826 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/kobo/django/views/generic.py", line 70, in form_valid
[Fri Feb 24 03:22:48.452827 2023] [:error] [pid 21807]     self.queryset = self.get_queryset().filter(query)
[Fri Feb 24 03:22:48.452834 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/pdc/apps/compose/views.py", line 90, in get_queryset
[Fri Feb 24 03:22:48.452836 2023] [:error] [pid 21807]     variant_arch = variant_arch.get(arch=arch)
[Fri Feb 24 03:22:48.452837 2023] [:error] [pid 21807]   File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
[Fri Feb 24 03:22:48.452839 2023] [:error] [pid 21807]     self.model._meta.object_name
[Fri Feb 24 03:22:48.452840 2023] [:error] [pid 21807] DoesNotExist: VariantArch matching query does not exist.
[Fri Feb 24 03:35:09.407847 2023] [:error] [pid 29997] [client 192.168.1.190:37456] mod_wsgi (pid=29997): Exception occurred processing WSGI script '/usr/lib/python2.7/site-packages/pdc/wsgi.py'.
[Fri Feb 24 03:35:09.408525 2023] [:error] [pid 29997] [client 192.168.1.190:37456] IOError: failed to write data
[Fri Feb 24 03:41:02.361051 2023] [core:notice] [pid 525] AH00052: child pid 10454 exit signal Segmentation fault (11

I think @humaton fixed this? can anyone chime in if you still see it?

I haven't seen any more pre-receive hook failures in the past few hours, but I haven't pushed many commits, either. Let's close this ticket for now, if it happens again, I'll come back :)

Metadata Update from @decathorpe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.