[Dev] Let's talk about Infrastructure and Hosting

[Vikings GmbH]

On Sun, 17 Nov 2019 01:26:19 +0100
Andreas Grapentin <andreas at grapentin.org> wrote:

> 
> Hi,
> 
> it's been almost two weeks since the first announcement was made, and
> I believe we have now a good understanding of our infrastructure and
> our hosting needs, and are ready to move forward.
> 
> We have discussed internally, and have come to the conclusion that a
> good course of action could be to keep winston running, but start
> migrating some of the essential services off it one by one,
> streamlining their deplopment in the process.
> 
> To get started, we are currently looking at getting a single VPS, with
> specs comparable to, or slightly improved upon, the specs of winston
> itself.
> 
> Winston currently has:
> 
>  - (nominally) about 4GiB of RAM.
> 
>    we believe that 4GiB is currently the lower bound of what our
>    services need to function. Winston is currently struggling with
> RAM, but this seems to be mainly because he's not having the nominal
>    amount actually available for use.
> 
>    To get some room to breathe, we're hoping for an upgrade here
> towards the region of 6 or 8GiB of ram, mainly to make sure the
> machine doesn't need to swap as much as it does now.
> 
>  - 4 QEMU virtual CPUS @ 2.2GhZ
> 
>    winston is never really computationally busy. iirc, vikings is
>    allowing only for 1 CPU VPS's (please correct me if I'm wrong) so

Wrong :) Vikings can do anything between 1 and 32 CPUs. When we say
"CPU" we refer to the physical core and not some over-subscribed,
mass-hosting shenanigans.

>    we're not looking for an upgrade here. Realistically, anything
> that's not a potato should be sufficient to deal with our workload.
> 
>  - 500GiB of storage
> 
>    storage is getting a bit tight. We are planning to extend our repos
>    to more architectures, chiefly ppc64le and riscv64, and we would
> not be able to host all the built packages due to limited storage
>    capacity.
> 
>    Ideally, we would like to see an extended capacity of at least 1TiB
>    here, in order to work towards our medium-term goals. Even more
>    storage would allow us to move forward with other projects, such as
>    providing a source packgaes repository, or maintaining a larger
>    archive of older packages. But we can discuss these separately once
>    the respective projects move forward.
> 
>  - virtually unlimited network traffic
> 
>    as far as I know, winston is not limited in terms of network
> traffic. And we do incur a LOT of network traffic. The order of
> magnitude should be somewhere in the single-digit TiB's per month,
> although it's hard to be sure since we don't have a monitoring in
> place.
> 
>    Most of this traffic is because of a growing number of busily
> syncing parabola mirrors, and we could cut that down by better
> structuring our network of mirrors, and giving that more of a
> hierarchy instead of having everyone pull from the root.
> 
>    We should talk to the vikings people here, what sort of traffic is
>    allowed on their network, to get a feeling how much we need to cut
>    down here. Also, we should start putting better traffic monitoring
> in place.
> 
> So much for the rationale. In summary, we believe that we currently
> need a single VPS with:
> 
>    - a single core
>    - at least 8GiB of RAM
>    - at least 1TiB of storage
>    - a reasonable agreement on traffic limits
> 
> If I have missed or forgot anything, let me know. Otherwise, we should
> probably get the vikings people in the loop and discuss next steps!
> 
> Best,
> Andreas (oaken-source)
> 
> 
> 
> On Tue, Nov 05, 2019 at 11:28:36PM +0100, Andreas Grapentin wrote:
> > 
> > Hi,
> > 
> > As you know, a good chunk of the infrastructure that makes parabola
> > is currently hosted on a virtual machine called winston, which is
> > generously provided by the hosting provider 1984. We used to have
> > proton as well, but when that died, the things that it hosted were
> > migrated over to winston.
> > 
> > It's come to our attention that this machine is currently
> > struggling to cope with the load it has to bear, as is evident by
> > the fact that it is as of this writing at ~70% swap utilization,
> > and requires more and more frequent reboots just to keep our basic
> > infrastructure operational. As you can probably imagine, this is
> > not a healthy mode of operation for a distribution, and is
> > something that is eating a bunch of hacker time to keep on top of.
> > 
> > These developments have prompted freemor, bill-auger and me to keep
> > an eye open for other hosting options that would maybe give our
> > services and bit more breathing room, and maybe some space to grow
> > and expand.
> > 
> > bill-auger had floated the idea to get in touch with the people
> > over at vikings (vikings.net) -- in CC -- who offer (among other
> > things) hosting services on exclusively libre-friendly server
> > hardware, which would be perfect for parabola. We did already have
> > a brief chat with their team to figure out what their available
> > hosting options are, and now we need to discuss amongs ourselves
> > what sort of infrastructure we want going forwards.
> > 
> > I think it would make sense to start by listing all the services
> > that the new infrastructure should host, and estimating from there
> > what sort of physical or virtual machine or machines we need, and
> > how we distribute the services.
> > 
> > Off the top of my head, these are the services that I know we run:
> > 
> >   - webserver for parabola.nu, including
> >      - wiki
> >      - main website
> >      - git repositories
> >      - issue tracker (redmine?)
> >      - forum
> >   - Tier 0 package repositories
> >      - repo importer
> >   - mailman
> > 
> > But I'm probably missing a bunch of things. This is where I'd like
> > someone more experienced with our infrastructure to step in and
> > explain what we have and need. I would suggest we start maintaining
> > this information in the wiki as well, assuming no such article
> > exists already. This might come in handy down the line.
> > 
> > My hunch as a Software Engineer and Architect would be that it could
> > make sense to partition these services in two, or maybe three VMs,
> > just to have a more fine-grained control over fault tolerance and
> > recovery in case something does go belly-up, and to be able to
> > separate the critical from the non-critical infrastructure. But
> > this is something we need to discuss down the line, if we have a
> > clear idea about our service landscape.
> > 
> > While we do this, I would also suggest we take another look at which
> > part of our infrastructure could and / or should be converted to
> > packages, instead of unaccounted for files. We could conceivably
> > add a separate undocumented repo for server-only packages. And
> > while we're at that, I would also like to take a gander at
> > automatic provisioners, such as ansible, that would allow us to
> > describe our server roles and then automatically deploy them as
> > needed, to have a self-documenting deployment of our services.
> > 
> > But first, let's complete the list of services that parabola needs
> > to function, and figure out how we want to partition these and what
> > sort of resources they need. Everything beyond that is still far
> > away. :)
> > 
> > 
> > I have to say; I am personally very excited to see these changes
> > unfold. I believe that parabola is currently at a place where we
> > need to stabilize our infrastructure and establish ourselves as a
> > serious distribution. And a collaboration with vikings could be a
> > great chance to take a leap toward that direction.
> > 
> > However, I also appreciate that parabola is an adhocracy, and that
> > each of you has their own unique views on these issues and options.
> > If you have any concerns about these developments, please bring
> > them to our attention as well.
> > 
> > I'm looking forwards to everyones input.
> > 
> > Best,
> > Andreas
> > 
> > oaken-source
-- 
Best regards/Mit freundlichen Grüßen/Cordialement/Met vriendelijke groet
The Vikings Team

Web: https://www.vikings.net/
Store: https://store.vikings.net/
Phone: +49 69 247 54 91 0
Fax: +49 69 247 50 339

Follow us on Mastodon: https://social.vikings.net/@vikings
Follow us on Twitter: https://twitter.com/vikingslibre

Vikings GmbH
Sauerackerweg 14, 60529 Frankfurt/Main, Germany
Register Court: AG Frankfurt am Main
Register No.: HR B 84497, CEO: Thomas Umbach
Tax Office: Frankfurt am Main, VATIN: DE254094338

GPG key fingerprint: FD5E 2543 EE62 7395 00A6 9FBB F228 7A48 7CA2 3FA5
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.parabola.nu/pipermail/dev/attachments/20191117/eafba32d/attachment.sig>