Repo mirrors are painfully slow

[toreanderson]

The SourceForge mirrors seem to be extremely slow or even totally unreachable. This makes UnitedRPMs slow down essentially all dnf operations.

It's been like this for a very long time. Years. I've held back on submitting an issue since it is not really an UnitedRPMs bug per se, but it makes having the UnitedRPMs repo enabled by default very painful. Perhaps it would be possible to find another mirror system.

For the record, I have a fast Internet connection (500Mbps FTTH), and all other DNF repos (both the upstream Fedora ones as well as multiple other third-party ones). It is only the UnitedRPMs repos that are slow.

A few examples from today:

curl simply getting a connect timeout

[:~] 35 $ curl --verbose --trace-time https://vorboss.dl.sourceforge.net/project/unitedrpms/31/x86_64/repodata/repomd.xml.asc
12:43:55.869747 *   Trying 5.10.152.194:443...
12:43:55.870048 * TCP_NODELAY set
12:46:07.959269 * connect to 5.10.152.194 port 443 failed: Connection timed out
12:46:07.959462 * Failed to connect to vorboss.dl.sourceforge.net port 443: Connection timed out
12:46:07.959635 * Closing connection 0
curl: (7) Failed to connect to vorboss.dl.sourceforge.net port 443: Connection timed out

(I got this URL from a failing dnf command, for what it is worth.)

same command as above retried, but this time it eventually connected, but hit some other timeout

[:~] $ curl --verbose --trace-time https://vorboss.dl.sourceforge.net/project/unitedrpms/31/x86_64/repodata/repomd.xml.asc
12:40:32.915722 *   Trying 5.10.152.194:443...
12:40:32.916334 * TCP_NODELAY set
12:41:04.900047 * Connected to vorboss.dl.sourceforge.net (5.10.152.194) port 443 (#0)
12:41:04.904025 * ALPN, offering h2
12:41:04.904140 * ALPN, offering http/1.1
12:41:04.931467 * successfully set certificate verify locations:
12:41:04.931653 *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
12:41:04.932427 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
12:42:18.626711 * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vorboss.dl.sourceforge.net:443 
12:42:18.626927 * Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vorboss.dl.sourceforge.net:443 

dnf giving up on downloads because of transfer speed dropping below 1Kb/s

[:~] $ sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=unitedrpms update
[...]
Downloading Packages:
[MIRROR] chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm: Curl error (28): Timeout was reached for https://netcologne.dl.sourceforge.net/project/unitedrpms/31/x86_64/chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
[MIRROR] chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm: Curl error (28): Timeout was reached for https://netcologne.dl.sourceforge.net/project/unitedrpms/31/x86_64/chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
(1/4): chromium-freeworld-81.0.4044.138-604.1.x86_64.rpm      609 kB/s | 121 MB     03:23    
(2/4): chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm  39 kB/s | 8.4 MB     03:43    
(3/4): chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm      35 kB/s | 8.0 MB     03:52    
(4/4): unitedrpms-31-16.fc31.noarch.rpm                       1.6 kB/s |  11 kB     00:06    
----------------------------------------------------------------------------------------------
Total                                                         587 kB/s | 137 MB     03:59     
[...]

dnf update with only unitedrpms repo enabled: timer: sack setup: 17664 ms

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=unitedrpms update
timer: config: 3 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=unitedrpms update 
Installroot: /
Releasever: 31
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=unitedrpms', 'update']
User-Agent: constructed: 'libdnf (Fedora 31; generic; Linux.x86_64)'
unitedrpms 31 - x86_64                                         51  B/s | 870  B     00:17    
reviving: 'unitedrpms' can be revived - repomd matches.
unitedrpms: using metadata from fr. 15. mai 2020 kl. 11.57 +0200.
timer: sack setup: 17664 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 39 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m18,107s
user    0m1,039s
sys     0m0,199s

compare the above with only the (much larger!) fedora repo enabled: timer: sack setup: 1900 ms

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=fedora update
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=fedora update 
Installroot: /
Releasever: 31
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=fedora', 'update']
User-Agent: constructed: 'libdnf (Fedora 31; generic; Linux.x86_64)'
Fedora 31 - x86_64                                             46 kB/s |  25 kB     00:00    
reviving: 'fedora' can be revived - metalink checksums match.
fedora: using metadata from to. 24. okt. 2019 kl. 00.52 +0200.
timer: sack setup: 1900 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 523 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m3,119s
user    0m2,277s
sys     0m0,246s

[sergiomb2]

yeah I also noticed this issue , why other mirror is not the main mirror ?
Thank you

[kuboosoft]

The mirror is already reported. I detected various mirrors down recently (I reported it; and some hours they solved); I am not sure the cause; maybe overload in this pandemic in some mirrors.

[kuboosoft]

@sergiomb2 Issues in mirrorlist + unnecessary space with debug files... Our second mirror reduced quota... (Ho yes, we have actually big size) 😺

[toreanderson]

The UnitedRPM mirrors have been very slow since way before the pandemic struck, for what it is worth. Even with the vorboss one disabled, the UnitedRPMs repo is still is way slower than the much larger fedora repo. To me it seems like the SourceForge mirror network is just very slow overall.

unitedrpms repo enabled only

[:~] $ time sudo dnf -v --refresh --disablerepo='*' --enablerepo=unitedrpms update
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
cachedir: /var/cache/dnf
User-Agent: constructed: 'libdnf (Fedora 32; generic; Linux.x86_64)'
unitedrpms 32 - x86_64                                                                                                                                                                                         104  B/s | 870  B     00:08    
reviving: 'unitedrpms' can be revived - repomd matches.
unitedrpms: using metadata from fr. 15. mai 2020 kl. 10.11 +0200.
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

real    0m9,108s
user    0m0,790s
sys     0m0,134s

fedora repo enabled only

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=fedora update
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=fedora update 
Installroot: /
Releasever: 32
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=fedora', 'update']
User-Agent: constructed: 'libdnf (Fedora 32; generic; Linux.x86_64)'
countme: no event for fedora: window already counted
Fedora 32 - x86_64                                                                                                                                                                                              20 kB/s |  24 kB     00:01    
reviving: 'fedora' can be revived - metalink checksums match.
fedora: using metadata from to. 23. april 2020 kl. 00.22 +0200.
timer: sack setup: 2265 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 509 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m3,460s
user    0m2,010s
sys     0m0,245s

[sergiomb2]

(Ho yes, we have actually big size)

What we are talking about ? 100 Gigas ? , so just mirror non-debug rpms on https://osdn.net/projects/unitedrpms/ ... I guess , It is possible , is it possible .

Anyway mirrorlist issues was fixed
Anyway 2, metalink with mirror manager are now in use in Fedora and RPMFusion which is the preferred choice, because works well

Thanks

[kuboosoft]

@sergiomb2 aprox... we don't need debug files (checking the traffic of each debug). Next step, debug of the debug?... 😄 Yes is possible avoid debug (some hours needs).

About metalink; in Fedora and other third-party; is irrelevant; the people choice the useful for him/her... and UnitedRPMs has great and big software.

@toreanderson If anybody want speed mirrors, collaborate with it is a great idea (We need mirrors)... the mirrors in SF uses automatic geographic election; and here we only can report it.
What do you expect from a self-sustaining and collaborative infrastructure? 😄

[sergiomb2]

@sergiomb2 aprox... we don't need debug files (checking the traffic of each debug). Next step, debug of the debug?... smile Yes is possible avoid debug (some hours needs).

Debug files are separated because we save a lot of bandwidth , disk and even performance etc and for common use is rare that someone install it and we can leave without it. if we don't want debug app crashes .

we just need it to debug a crash / segfault , to find the symbols and lines of code that make the crash of the computer .

I sometime install it , usually on Beta version under Virtualbox, but after debug it , I remove it again to save disk space and bandwith , yeah I bearably update debug packages after test it I remove it , it always gigas and gigas of downloads .

[sergiomb2]

https://github.com/UnitedRPMs/unitedrpms/blob/master/unitedrpms.repo#L4 where we can use

baseurl=https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

hum and why osdn.net is not in first on https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt ? I use mirrorlist in my dnf configuration btw

and we may remove debug files from osdn.net if we remove it also from here https://github.com/UnitedRPMs/unitedrpms/blob/master/mirrorlist_debug_30.txt . and maybe do the same to the sources rpms ?

What do you think ?

[toreanderson]

What do you expect from a self-sustaining and collaborative infrastructure? smile

To be clear, I do not have any demands or hard expectations per se. I am fully aware that I get what I pay for when using UnitedRPMs. So the reason for submitting this issue is just to make you aware of it, in case you weren't already. If the conclusion is that «yeah, SourceForge is slow, but it is nevertheless the best free alternative, so we'll keep using it», then that is fair enough as far as I am concerned.

[sergiomb2]

@toreanderson thank you for these words

[kuboosoft]

@toreanderson I am testing some changes; we need secure the changes. Thanks.

@sergiomb2 I need secure the limits and if you orbserve, osdn is incomplete (repodata) and others. But is possible enable again the mirrorlist; I remember is programmed a sync in repositories, cross fingers with some lines changed in the sync task.

[kuboosoft]

@toreanderson Done. Can you test now the speed?

[toreanderson]

It seems much faster now! The sudo dnf -v --refresh --disablerepo='*' --enablerepo=unitedrpms update commands seems to complete in around 4 seconds now, which is a major speedup compared to how it used to be.

However, it is still (slightly) slower than the much larger fedora repo. While watching the below screen recording (play locally with asciinema play https://asciinema.org/a/irlpk0mlsiA2kvulym5vY81M4), you can see how the progress bar starts out stalled with --- B/s transfer speed). Indeed, it seems like the vast majority of the dnf execution time is spent in that state - once it starts moving, it finishes almost instantaneously.

By comparison, when updating the fedora repo, things start moving right away.

Unfortunately I have not found out how to make dnf spit out sufficient amounts of debugging info to figure out what exactly it is waiting for when it is stalled. Any tips here would be appreciated.

[kuboosoft]

@toreanderson Hi, thanks. Is only conjectures about the delay; the repository/repodata is signed (for security). Read about it and about how to works repo_gpgcheck here:

https://blog.centos.org/2018/07/improving-centos-package-delivery-security-with-signed-repository-metadata/

https://linux.die.net/man/5/yum.conf#repo_gpgcheck

• "mirrorlist" + "repo_gpgcheck" caused in 2017-2018 our repository down... Reason was urgent necesary go back to "baseurl"; and now I checking the issue was solved and works again. Hopefully not happen other regression.