fedora-infrastructure

#10391 is copr backend down?
Closed: Fixed by vanorman. Opened by vanorman.

Closed: Fixed
Reopen Issue

I saw there was a planned outage recently, but everything is listed as being up. However, I cannot access https://copr-be.cloud.fedoreproject.org.

~$ curl -v https://copr-be.cloud.fedoraproject.org/results/sbonazzo/EL8_collection/epel-8-x86_64/repodata/repomd.xml
*   Trying 52.44.175.77:443...
* TCP_NODELAY set
* connect to 52.44.175.77 port 443 failed: Connection refused
*   Trying 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83: Network is unreachable
*   Trying 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83: Network is unreachable
* Failed to connect to copr-be.cloud.fedoraproject.org port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to copr-be.cloud.fedoraproject.org port 443: Connection refused

I was in the middle of recovering from an ovirt engine (hosted) failure and this is blocking any installs or updates of oVirt.

I encountered the same problem here. Returns 502 on my end.

> curl -fSL https://download.copr.fedorainfracloud.org/results/kwizart/kernel-longterm-5.10/fedora-35-x86_64/02990248-kernel-longterm/fedora-35-x86_64/repodata/repomd.xml
curl: (22) The requested URL returned error: 502

Opening the link in a web browser shows a CloudFront error page:
Screenshot_from_2021-12-02_09-35-28.png

It should be back up.

For some reason lighttpd was failed on copr-be. ;( 

Dec 02 00:05:06 copr-be.aws.fedoraproject.org lighttpd[851]: 2021-12-02 00:05:06: (configfile.c.273) Warning: please add "mod_openssl" to server.modules list in lighttpd.conf.  A future release of lighttpd 1.4.x *will not* automatically load mod_openssl and lighttpd *will not* use SSL/TLS where your lighttpd.conf contains ssl.* directives
Dec 02 00:05:06 copr-be.aws.fedoraproject.org lighttpd[851]: 2021-12-02 00:05:06: (mod_openssl.c.752) SSL: BIO_read_filename('/etc/lighttpd/copr-be.cloud.fedoraproject.org.intermediate.cert') failed
Dec 02 00:05:06 copr-be.aws.fedoraproject.org lighttpd[851]: 2021-12-02 00:05:06: (mod_openssl.c.2814) SSL: error:0909006C:PEM routines:get_name:no start line /etc/lighttpd/copr-be.cloud.fedoraproject.org.intermediate.cert
Dec 02 00:05:06 copr-be.aws.fedoraproject.org lighttpd[851]: 2021-12-02 00:05:06: (server.c.1282) Initialization of plugins failed. Going down.
Dec 02 00:05:06 copr-be.aws.fedoraproject.org systemd[1]: lighttpd.service: Main process exited, code=exited, status=255/EXCEPTION
Dec 02 00:05:06 copr-be.aws.fedoraproject.org systemd[1]: lighttpd.service: Failed with result 'exit-code'.
Dec 02 00:05:06 copr-be.aws.fedoraproject.org systemd[1]: lighttpd.service: Consumed 4h 37min 2.702s CPU time.

cc @praiskup

I can confirm I can access it now, thank you!

Metadata Update from @vanorman:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Uuuuh :-( thank you for the report.

This never happened before, as far as I remember. So something new with F35.

Offhand, I think we could try service config: Restart=on-failure?

We also had a typo in monitoring, so fixing this took pretty long.. I fixed this now, hopefully.

@kevin, have you just systemctl start lighttpd?

Logrotate probably triggered this:

Dec 02 00:00:00 copr-be.aws.fedoraproject.org systemd[1]: Starting Rotate log files...
Dec 02 00:04:41 copr-be.aws.fedoraproject.org systemd[1]: logrotate.service: Deactivated successfully.
Dec 02 00:04:41 copr-be.aws.fedoraproject.org systemd[1]: Finished Rotate log files.
Dec 02 00:04:41 copr-be.aws.fedoraproject.org systemd[1]: logrotate.service: Consumed 4min 36.259s CPU time.

Yeah, I only did a 'systemctl restart lighttpd'

Commit d05a9558 relates to this ticket

Commit 19b8f82f fixes this issue

Should we reopen this issue? It looks like the solution that was implemented didn't actually work, because the same isssue occurred again yesterday.

From #fedora-admin on irc:

21:18:30 <gotmax> .oncall
21:18:30 <zodbot> siddharthvipul is oncall. My normal hours are 0700 UTC to 1600 UTC Monday through Friday. If I do not answer or it is outside those hours, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
21:18:46 <gotmax> The Copr 502 issue is happening again
21:20:07 <gotmax> https://pagure.io/fedora-infrastructure/issue/10391
21:34:23 <gotmax> Last time, nirik fixed it by running "systemctl restart lighthttpd"
21:35:10 <nirik> and I did just now again. 
21:35:23 * nirik goes back to his movie
21:36:07 <gotmax> Thank you so much! Sorry to bother you.
22:17:05 — 22:44:37 jcline has joined, mymindstorm has quit
22:45:48 <VipulSiddharth[m> gotmax: glad it was fixed

(Times are in CST/UTC-6 on Sat Dec 4)

This is being tracked here:
https://pagure.io/fedora-infrastructure/issue/10391

I disabled logrotate till we have the fix, though logrotate seems started again, :-(
Dunno what is happening, it's my mistake that I haven't disabled this in ansible.
I was trying now ... though I can't fetch from pagure.io for some reason.

Sorry, tracked here:
https://pagure.io/copr/copr/issue/2001

disabled in ansible
https://pagure.io/fedora-infra/ansible/c/f4e09eac78ac761c5ac81147f065869165e26d4a?branch=main
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.