#1936 issues/1880: send Range: header to accelerate duplicate/partial downloads
Closed by tkopecek. Opened by fche.
fche/koji issue1880  into  master

Download 1936.patch

See also issue #1880.

Metadata Update from @tkopecek:
- Pull-request tagged with: testing-ready

Metadata Update from @tkopecek:
- Pull-request untagged with: testing-ready

This is not working with older python.

Old enough that you need

('bytes=%d-' % pos)

instead of the f'' expression?

1 new commit added

  • Range: header: use old school fmt % operator

Old enough that you need
('bytes=%d-' % pos)

instead of the f'' expression?

yes :-)

:thumbsup:

Metadata Update from @tkopecek:
- Pull-request tagged with: testing-ready

pretty please pagure-ci rebuild

pretty please pagure-ci rebuild

Metadata Update from @tkopecek:
- Pull-request untagged with: testing-ready

Hmm, it needs more work. It fails on already downloaded file (requests range beyond the EOF). Further, it should check downloaded content check_rpm_file. At least archives and rpms has some info, which can be further checked (filesize, payload checksum). For filetypes which don't have any such info (logs, etc.) appending should be forbidden.

I've ended with few corrupted files while testing this.

Hmm, it needs more work. It fails on already downloaded file (requests range beyond the EOF).

Fails how? It does send a request beyond the EOF, and that's okay, the server returns the EOF indication, and then the file is done. That is normal for range-request type resume-download queries.

Further, it should check downloaded content check_rpm_file. At least archives and rpms has some info, which can be further checked (filesize, payload checksum). For filetypes which don't have any such info (logs, etc.) appending should be forbidden.

Good idea. The filetype to be deduced from url/filename extension directly?

I've ended with few corrupted files while testing this.

Can you elaborate please?

Hmm, it needs more work. It fails on already downloaded file (requests range beyond the EOF).

Fails how? It does send a request beyond the EOF, and that's okay, the server returns the EOF indication, and then the file is done. That is normal for range-request type resume-download queries.

Hmm, I've had misconfigured web server, which caused 'invalid range' instead. I've tried against fedora's koji and it works well.

Further, it should check downloaded content check_rpm_file. At least archives and rpms has some info, which can be further checked (filesize, payload checksum). For filetypes which don't have any such info (logs, etc.) appending should be forbidden.

Good idea. The filetype to be deduced from url/filename extension directly?

I would create some wrappers based on filetype. download_file is used only in few places (archive, rpm, logs) now and for each type we know some additional info (at least file size). So, download_archive and download_rpm could benefit from them.

I've ended with few corrupted files while testing this.

Can you elaborate please?

Problem is, that I'm often downloading different types (e.g differently signed rpms). As download function can't compare, that file being appended is really the same as what I'm downloading, it can mix the content. It could be fixed by checking the resulting file via up-mentioned methods.

I've fixed problems I had and extended it a bit in the #2080

Pull-Request has been closed by tkopecek

Metadata