[Dev] tiered mirrors schema

[bill-auger]

i have had this message ready to send to mirror operators for
since before the last two mirrors joined the network - another
joined this week; and this concern really cant be postponed any
longer - the reason why this is important is because new mirrors
puts a significant extra load onto the repo server - in 2018,
lukeshu told me that we were already pushing the bandwidth
limits on 1984's generosity, and the number of mirrors has
nearly doubles since then - tiering the mirrors in this way,
is a matter of respect, appreciation, and gratitude to 1984

no one on this list needs to do anything; but i thought people
may be interested or would like to comment - i also changed the
wiki page quite a bit, to be more aligned with the arch article,
and added some suggestions, such as seeding the p2p network
instead or creating more complete mirrors - the one thing i
anticipate that people may have an opinion about, is how i made
the requirements more precise and stringent (isacdaavid already
did) - anticipating that some may wonder why: the best answer
is that the mirror network is very healthy now, healthy than
ever - if these are going to be tiered now, then theres no
reason not to raise the standard for tier-1 and tier-2, while
retaining the original loose requirements for tier-3

https://wiki.parabola.nu/Creating_a_mirror

this is the message i sent to all mirror operators:

== new tiered mirrors schema ==

due to an increasing number of mirrors joining the parabola
mirror network, the sync schedules are now overlapping in some
places - we would like to reorganize them into a more uniform
structured schedule for the future - this would reduce the load
on our repo server, and also improve the service for parabola
users - the increasing number of mirrors is, of course, a good
thing - presumably, it indicates that parabola is gaining in
popularity; but parabola does not have any funding, so it is
putting a strain on our infrastructure

currently, all mirrors are pulling directly from parabola; which
is pushing the limit of what our extremely generous hosting
benefactor can offer for gratis - the plan is such that the
relatively modest hosts will be asked to be on "tier-2" (pulling
regularly from some relatively faster host, other than
parabola.nu); and the higher-capacity hosts will be asked if it
is feasible for them to become dedicated "tier-1" mirrors (ones
which some number of the tier-2 mirrors will pull from regularly)

initially, we would like to have three to six tier-1 mirrors,
synchronizing with the parabola tier-0 server, every thirty
minutes (5-10 minutes separation); and for each tier-2 mirror to
sync once or twice per hour, distributed however the tier-1
mirrors can accommodate them

the intended effect is such that delivery times for new packages
will be improved for users significantly; with 404 errors being
much less likely, due to all mirrors being within a tighter
temporal proximity to each other - although a few of the newer
mirrors are already on a rapid schedule, most are currently
syncing much less frequently - the end-to-end delay is up to two
days, for clients of those mirrors; and 404s are common when
using the default re-director; because of the loosely-knit
federation

the accelerated sync rates should have no adverse impact on
mirrors - before each sync attempt, the recommended mirroring
script downloads a timestamp file, and cancels the complete sync
if there are no changes - that is because some megabytes are
transferred even when there are no changes - with that guard in
place, the total incoming bandwidth for any mirror is essentially
the same, regardless of the rate at which the sync script is run
- its use was recently added to the wiki as a hard criteria for
endorsed mirrors

it is, of course, at each mirror operator's discretion to
throttle any connections from unknown clients, or those using
excessive bandwidth - managing the precise sync schedules of
tier-2 mirrors, will be at the discretion of each tier-1 mirror
operator - we ask only that tier-1 mirror operators give
priority on the rsync channel, to the tier-2 mirrors to which
they are feeding, providing that the admins are responsive and
agreeable to reasonable terms of use suggestions


== mailing list ==

some of the current mirror operators have suggested a new
low-volume mailing list, dedicated for only important
communication among the mirror operators, rather than expecting
everyone to subscribe to the development mailing list, and/or
publishing contact info publicly - it would be extremely rare
for parabola to need to send a mass email such as this one;
but it could be useful, for occasional coordination of sync
schedules, such as now - you should receive an invitation soon
- it may be somewhat busy during this initial migration into
tiers; but the volume will be close to nil afterward


== actions requested ==

at this time, we have incomplete information on many of the
mirrors; so there is no request for action on your part yet -
this is the initial attempt to gather all of the information
needed to coordinate the network, and subscribe mirror operators
to the new mailing list