fedora-infrastructure

#8096 aarch64 epel-7, fedora-29 and fedora-30 repository problems on copr builders
Closed: Fixed by kevin. Opened by praiskup.

Closed: Fixed
Reopen Issue

From dnf.log:

2019-08-15T10:55:00Z INFO --- logging initialized ---
2019-08-15T10:55:00Z DDEBUG timer: config: 12 ms
2019-08-15T10:55:00Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
2019-08-15T10:55:00Z DEBUG DNF version: 4.2.7
2019-08-15T10:55:00Z DDEBUG Command: yum --installroot /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/ --releasever 7 install @buildsys-build 
2019-08-15T10:55:00Z DDEBUG Installroot: /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/
2019-08-15T10:55:00Z DDEBUG Releasever: 7
2019-08-15T10:55:00Z DEBUG cachedir: /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/var/cache/dnf
2019-08-15T10:55:00Z DDEBUG Base command: install
2019-08-15T10:55:00Z DDEBUG Extra commands: ['--installroot', '/var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/', '--releasever', '7', 'install', '@buildsys-build']
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration value: failovermethod=priority in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf; Configuration: OptionBinding with id "failovermethod" does not exist
2019-08-15T10:55:00Z DEBUG Unknown configuration option: best = 1 in /var/lib/mock/843342-epel-7-aarch64-1565866489.665174/root/etc/dnf/dnf.conf
2019-08-15T10:55:00Z DEBUG repo: downloading from remote: copr_base
2019-08-15T10:55:00Z DEBUG copr_base: using metadata from Wed 14 Aug 2019 04:34:40 PM UTC.
2019-08-15T10:55:00Z DEBUG repo: downloading from remote: base
2019-08-15T10:55:09Z DEBUG base: using metadata from Tue 27 Nov 2018 09:05:10 AM UTC.
2019-08-15T10:55:09Z DEBUG repo: downloading from remote: updates
2019-08-15T10:55:18Z DEBUG updates: using metadata from Wed 31 Jul 2019 10:18:15 AM UTC.
2019-08-15T10:55:18Z DEBUG repo: downloading from remote: epel
2019-08-15T10:55:18Z DEBUG error: Status code: 404 for http://mirror.cogentco.com/pub/linux/epel/7/aarch64/repodata/8ea14f2301169860df48191b2cca0c9ef346caa3970a70fb7fd966712a755a86-primary.xml.gz (http://mirror.cogentco.com/pub/linux/epel/7/aarch64/repodata/8ea14f2301169860df48191b2cca0c9ef346caa3970a70fb7fd966712a755a86-primary.xml.gz).
2019-08-15T10:55:18Z DEBUG error: Status code: 404 for http://mirror.cogentco.com/pub/linux/epel/7/aarch64/repodata/142a8658c2da5bc9a57aa80ba7b60ed74d6cd8c894d35094a883afaaf504078c-filelists.xml.gz (http://mirror.cogentco.com/pub/linux/epel/7/aarch64/repodata/142a8658c2da5bc9a57aa80ba7b60ed74d6cd8c894d35094a883afaaf504078c-filelists.xml.gz).
2019-08-15T10:55:18Z DEBUG error: Status code: 404 for http://mirror.nodesdirect.com/epel/7/aarch64/repodata/8ea14f2301169860df48191b2cca0c9ef346caa3970a70fb7fd966712a755a86-primary.xml.gz (http://mirror.nodesdirect.com/epel/7/aarch64/repodata/8ea14f2301169860df48191b2cca0c9ef346caa3970a70fb7fd966712a755a86-primary.xml.gz).
[snip ~200 errors with 404]
2019-08-15T10:55:39Z DEBUG Cannot download 'http://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=aarch64': Yum repo downloading error: Downloading error(s): repodata/8ea14f2301169860df48191b2cca0c9ef346caa3970a70fb7fd966712a755a86-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/142a8658c2da5bc9a57aa80ba7b60ed74d6cd8c894d35094a883afaaf504078c-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/59fda8159234b95d1891c1e0003db041ffe9c959c933b38a00722aade3a4a62c-prestodelta.xml.gz - Cannot download, all mirrors were already tried without success; repodata/d370878dcea587ad1a720619d162d84fc268b44e65721ff72fd872465d4f49de-updateinfo.xml.bz2 - Cannot download, all mirrors were already tried without success.
2019-08-15T10:55:39Z ERROR Failed to download metadata for repo 'epel'
2019-08-15T10:55:39Z DDEBUG Cleaning up.
2019-08-15T10:55:39Z SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/repo.py", line 552, in load
    ret = self._repo.load()
  File "/usr/lib64/python3.7/site-packages/libdnf/repo.py", line 394, in load
    return _repo.Repo_load(self)
RuntimeError: Failed to download metadata for repo 'epel'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 65, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 98, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 114, in cli_run
    cli.run()
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 1118, in run
    self._process_demands()
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 816, in _process_demands
    load_available_repos=self.demands.available_repos)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 406, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 136, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python3.7/site-packages/dnf/repo.py", line 558, in load
    raise dnf.exceptions.RepoError(str(e))
dnf.exceptions.RepoError: Failed to download metadata for repo 'epel'
2019-08-15T10:55:39Z CRITICAL Error: Failed to download metadata for repo 'epel'

I tried with both this:

# rpm -q dnf librepo libdnf rpm 
dnf-4.2.7-2.fc30.noarch
librepo-1.10.5-1.fc30.aarch64
libdnf-0.35.1-3.fc30.aarch64
rpm-4.14.2.1-4.fc30.1.aarch64

and this:

# rpm -q dnf librepo libdnf rpm 
dnf-4.2.2-2.fc30.noarch
librepo-1.9.6-2.fc30.aarch64
libdnf-0.28.1-1.fc30.aarch64
rpm-4.14.2.1-4.fc30.1.aarch64

I'm not sure whether this is related to #8084, can you confirm?

  • inet 38.145.48.106/23
  • default /etc/hosts

No, it shouldn't have anything to do with 8084... looks more like some kind of network problem.

This is odd: 2019-08-15T10:55:18Z DEBUG updates: using metadata from Wed 31 Jul 2019 10:18:15 AM UTC.
Thats 2 weeks ago? Is the time set right there?

When did this start? Is it still happpening?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: mirrorlists

Thats 2 weeks ago?

Weird.... there are no mock caches baked into the image.

Is the time set right there?

Yes.

When did this start? Is it still happpening?

Few days back, and it is still happening. Sometimes it happens for F29 and F30,
but reproducible for EPEL7.

The reason is that aarch64 builders pick broken mirror [1] for repomd.xml download (2019-08-11 20:17 5.4K), and consider that file relevant for downloading the rest of metadata.

Can [1] be dropped from mirror manager?

[1] http://mirror.cogentco.com/pub/linux/epel/7/aarch64/

Out of curiosity, is this bug in mirrormanager (crawler?) that the mirror isn't put aside? The metadata is 5 days old. i'd expect that mirror should be synced within 24h after change, and if it is not - mirrormanager should mark it outdated (I admit I haven't looked at the code yet).

Or should this be solved in librepo as well (download the repomd.xml from multiple sources, and compare)?

This bug slightly touches this topic: https://bugzilla.redhat.com/show_bug.cgi?id=1737709

I have disabled that mirror and mailed the admin for it. It should drop out in 30-60min.

There's various reasons it wouldn't have dropped, perhaps @adrian can see why?

I talked to them and they now seem to be up to date.

Can you confirm the problem has stopped?

Looks good, thank you! What can we do to prevent this situation from happening in future? It is pretty hard to notice, debug and resolve on copr side. Was this caused by mirror manager bug then?

I'm not sure what caused it. In the past where this has happened it's because a mirror has some issue and gets out of sync, but is sending to report_mirror that they are ok. so the crawler takes them out, report_mirror adds them back and you get this cycle. I think we may need to look at not using report_mirror for any public mirrors? Not sure.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

I'm not sure what caused it. In the past where this has happened it's because a mirror has some issue and gets out of sync, but is sending to report_mirror that they are ok. so the crawler takes them out, report_mirror adds them back and you get this cycle. I think we may need to look at not using report_mirror for any public mirrors? Not sure.

About a year ago I wanted to disable report_mirror for non-private mirrors, but did not finish it. I still think this would be a good idea. Need to finally implement it.

Is there some bug/issue for this?

Perhaps https://github.com/fedora-infra/mirrormanager2/issues/85 ?
Metadata
None

mirrorlists

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.