fedora-infrastructure

#8869 Additional checks on mirrors
Closed: Upstream by kevin. Opened by specing.

Closed: Upstream
Reopen Issue

Describe what you would like us to do:

Some mirrors make it difficult to create further downstream local caching proxies
by including "Cache-Control: no-cache", "Cache-Control: max-age=0" headers
and setting Expires to right now.

wget -S -O /dev/stdout 'https://mirrors.uni-ruse.bg/fedora/linux/'
--2020-04-28 12:56:08--  https://mirrors.uni-ruse.bg/fedora/linux/
Resolving mirrors.uni-ruse.bg (mirrors.uni-ruse.bg)... 91.210.88.42, 2001:67c:220c::2a
Connecting to mirrors.uni-ruse.bg (mirrors.uni-ruse.bg)|91.210.88.42|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx
  Date: Tue, 28 Apr 2020 10:56:02 GMT
  Content-Type: text/html
  Transfer-Encoding: chunked
  Connection: keep-alive
  Expires: Tue, 28 Apr 2020 10:56:02 GMT
  Cache-Control: max-age=0
  Cache-Control: no-cache
Length: unspecified [text/html]

I have also seen one https mirror downgrade connections to http via a 301
Location: redirect, but I cannot find it right now.

Additionally, it seems that some mirrors in the public mirror list are not
actual mirrors but rather caching proxies (note the X-Cache-Status: MISS):

wget -S -O /dev/stdout 'https://fedora.mirror.garr.it/fedora/linux/releases/'
--2020-04-28 12:49:52--  https://fedora.mirror.garr.it/fedora/linux/releases/                                                                                                     
Resolving fedora.mirror.garr.it (fedora.mirror.garr.it)... 90.147.160.72, 90.147.160.73, 90.147.160.70, ... 
Connecting to fedora.mirror.garr.it (fedora.mirror.garr.it)|90.147.160.72|:443... connected.              
HTTP request sent, awaiting response...                                                                                                                                           
  HTTP/1.1 200 OK                                                                                                                                                                 
  Server: nginx                                                                                                                                                                     Date: Tue, 28 Apr 2020 10:49:51 GMT                                                                                                                                             
  Content-Type: text/html                                                                                                                                                           Transfer-Encoding: chunked                                                                                                                                                      
  Connection: keep-alive
  Vary: Accept-Encoding
  Vary: Accept-Encoding
  X-Cache-Status: MISS
Length: unspecified [text/html]

The above may or may not be a problem, but it could at least be noted in the
mirror list. I use a local caching proxy due to metered internet and many
installations. This requirement makes the Ruse university mirror unusable for me.

When do you need this to be done by? (YYYY/MM/DD)

There is no deadline.

PS: I was told that Github is a better place for these issues, but I believe
that using proprietary software to develop libre software is morally wrong
and should be avoided as libre replacements exist.

Curious, how are you running your caching proxy? If you are using mirrorlist/metalink for input the URL changes every time you hit another mirror and when I used something like squid the caching only worked if the URL stays the same. From that point of view it would make more sense to select a specific mirror and use that as upstream for your caching proxy. Then you can select a mirror that works for you.

For us the main mirror consumer is dnf/yum and as far as I know that works for most mirrors.

If you have ideas and code how to efficiently check mirrors and how to include this result in mirrorlist/metalink, patches are welcome.

Filed https://github.com/fedora-infra/mirrormanager2/issues/285 as an upstream copy of this one and I am going to close it.

However, note, you can feel free to exchange further comments here instead of the github issue if you like...

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

The solution for changing upstream server is using proxy_cache_key "$request_uri". That makes nginx save the proxy url instead of upstream url,
i.e. KEY: /fedora-31-amd64-upd/Packages/l/libidn2-2.3.0-1.fc31.x86_64.rpm. But I still use a fixed upstream.

proxy_cache_path /data/nginx/fedora-31-amd64 levels=2 keys_zone=fedora-cache-31-amd64:4m max_size=10000m inactive=1y loader_files=1000 loader_threshold=10000;
proxy_cache_path /data/nginx/fedora-31-amd64-upd levels=2 keys_zone=fedora-cache-31-amd64-upd:4m max_size=10000m inactive=1y loader_files=1000 loader_threshold=10000;

    proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
    proxy_cache_revalidate on; # use conditiona If-Modified-Since requests to check for new content
    proxy_cache_lock on;
##  proxy_cache_use_state ...
##  proxy_cache_background_update on;
    # Returns info on where a file was fetched from/what is going on:
    add_header X-Cache-Status $upstream_cache_status;
    # Ignore Cache-Control headers? Must include proxy_cache_valid directive, nginx does not cache
    # stuff that has no expiration
    # proxy_ignore_headers Cache-Control;
    # Default key is $scheme$proxied_host$request_uri
    proxy_cache_key "$request_uri";
    # Defaults: Date, Server, X-Pad, X-Accel-*
    #proxy_hide_header Location;
    # for items with response codes 200 (Success:OK) and 302 (Redirection:Found)
    proxy_cache_valid 200 301 302 3y;
    # for 404 (ClientError:Not Found)
    proxy_cache_valid 404 1m;
    location /fedora-31-amd64 {
        proxy_pass https://mirrors.netix.net/fedora/linux/releases/31/Everything/x86_64;
#       proxy_pass https://ftp.halifax.rwth-aachen.de/fedora/linux/releases/31/Everything/x86_64;
#       proxy_set_header Host ftp.halifax.rwth-aachen.de;
#       proxy_redirect default;
#       proxy_redirect https://mirrors.uni-ruse.bg /;
        proxy_cache fedora-cache-31-amd64;
    }
    location /fedora-31-amd64-upd {
        proxy_pass https://mirrors.netix.net/fedora/linux/updates/31/Everything/x86_64;
        proxy_cache fedora-cache-31-amd64-upd;
    }
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.