[Dev] tiered mirrors schema

bill-auger bill-auger at peers.community
Thu Feb 25 03:57:05 GMT 2021

i have had this message ready to send to mirror operators for
since before the last two mirrors joined the network - another
joined this week; and this concern really cant be postponed any
longer - the reason why this is important is because new mirrors
puts a significant extra load onto the repo server - in 2018,
lukeshu told me that we were already pushing the bandwidth
limits on 1984's generosity, and the number of mirrors has
nearly doubles since then - tiering the mirrors in this way,
is a matter of respect, appreciation, and gratitude to 1984

no one on this list needs to do anything; but i thought people
may be interested or would like to comment - i also changed the
wiki page quite a bit, to be more aligned with the arch article,
and added some suggestions, such as seeding the p2p network
instead or creating more complete mirrors - the one thing i
anticipate that people may have an opinion about, is how i made
the requirements more precise and stringent (isacdaavid already
did) - anticipating that some may wonder why: the best answer
is that the mirror network is very healthy now, healthy than
ever - if these are going to be tiered now, then theres no
reason not to raise the standard for tier-1 and tier-2, while
retaining the original loose requirements for tier-3


this is the message i sent to all mirror operators:

== new tiered mirrors schema ==

due to an increasing number of mirrors joining the parabola
mirror network, the sync schedules are now overlapping in some
places - we would like to reorganize them into a more uniform
structured schedule for the future - this would reduce the load
on our repo server, and also improve the service for parabola
users - the increasing number of mirrors is, of course, a good
thing - presumably, it indicates that parabola is gaining in
popularity; but parabola does not have any funding, so it is
putting a strain on our infrastructure

currently, all mirrors are pulling directly from parabola; which
is pushing the limit of what our extremely generous hosting
benefactor can offer for gratis - the plan is such that the
relatively modest hosts will be asked to be on "tier-2" (pulling
regularly from some relatively faster host, other than
parabola.nu); and the higher-capacity hosts will be asked if it
is feasible for them to become dedicated "tier-1" mirrors (ones
which some number of the tier-2 mirrors will pull from regularly)

initially, we would like to have three to six tier-1 mirrors,
synchronizing with the parabola tier-0 server, every thirty
minutes (5-10 minutes separation); and for each tier-2 mirror to
sync once or twice per hour, distributed however the tier-1
mirrors can accommodate them

the intended effect is such that delivery times for new packages
will be improved for users significantly; with 404 errors being
much less likely, due to all mirrors being within a tighter
temporal proximity to each other - although a few of the newer
mirrors are already on a rapid schedule, most are currently
syncing much less frequently - the end-to-end delay is up to two
days, for clients of those mirrors; and 404s are common when
using the default re-director; because of the loosely-knit

the accelerated sync rates should have no adverse impact on
mirrors - before each sync attempt, the recommended mirroring
script downloads a timestamp file, and cancels the complete sync
if there are no changes - that is because some megabytes are
transferred even when there are no changes - with that guard in
place, the total incoming bandwidth for any mirror is essentially
the same, regardless of the rate at which the sync script is run
- its use was recently added to the wiki as a hard criteria for
endorsed mirrors

it is, of course, at each mirror operator's discretion to
throttle any connections from unknown clients, or those using
excessive bandwidth - managing the precise sync schedules of
tier-2 mirrors, will be at the discretion of each tier-1 mirror
operator - we ask only that tier-1 mirror operators give
priority on the rsync channel, to the tier-2 mirrors to which
they are feeding, providing that the admins are responsive and
agreeable to reasonable terms of use suggestions

== mailing list ==

some of the current mirror operators have suggested a new
low-volume mailing list, dedicated for only important
communication among the mirror operators, rather than expecting
everyone to subscribe to the development mailing list, and/or
publishing contact info publicly - it would be extremely rare
for parabola to need to send a mass email such as this one;
but it could be useful, for occasional coordination of sync
schedules, such as now - you should receive an invitation soon
- it may be somewhat busy during this initial migration into
tiers; but the volume will be close to nil afterward

== actions requested ==

at this time, we have incomplete information on many of the
mirrors; so there is no request for action on your part yet -
this is the initial attempt to gather all of the information
needed to coordinate the network, and subscribe mirror operators
to the new mailing list

More information about the Dev mailing list