[Dev] Let's talk about Infrastructure and Hosting

Andreas Grapentin andreas at grapentin.org
Sun Nov 17 00:26:19 GMT 2019


Hi,

it's been almost two weeks since the first announcement was made, and I
believe we have now a good understanding of our infrastructure and our
hosting needs, and are ready to move forward.

We have discussed internally, and have come to the conclusion that a
good course of action could be to keep winston running, but start
migrating some of the essential services off it one by one, streamlining
their deplopment in the process.

To get started, we are currently looking at getting a single VPS, with
specs comparable to, or slightly improved upon, the specs of winston
itself.

Winston currently has:

 - (nominally) about 4GiB of RAM.

   we believe that 4GiB is currently the lower bound of what our
   services need to function. Winston is currently struggling with RAM,
   but this seems to be mainly because he's not having the nominal
   amount actually available for use.

   To get some room to breathe, we're hoping for an upgrade here towards
   the region of 6 or 8GiB of ram, mainly to make sure the machine
   doesn't need to swap as much as it does now.

 - 4 QEMU virtual CPUS @ 2.2GhZ

   winston is never really computationally busy. iirc, vikings is
   allowing only for 1 CPU VPS's (please correct me if I'm wrong) so
   we're not looking for an upgrade here. Realistically, anything that's
   not a potato should be sufficient to deal with our workload.

 - 500GiB of storage

   storage is getting a bit tight. We are planning to extend our repos
   to more architectures, chiefly ppc64le and riscv64, and we would not
   be able to host all the built packages due to limited storage
   capacity.

   Ideally, we would like to see an extended capacity of at least 1TiB
   here, in order to work towards our medium-term goals. Even more
   storage would allow us to move forward with other projects, such as
   providing a source packgaes repository, or maintaining a larger
   archive of older packages. But we can discuss these separately once
   the respective projects move forward.

 - virtually unlimited network traffic

   as far as I know, winston is not limited in terms of network traffic.
   And we do incur a LOT of network traffic. The order of magnitude
   should be somewhere in the single-digit TiB's per month, although
   it's hard to be sure since we don't have a monitoring in place.

   Most of this traffic is because of a growing number of busily syncing
   parabola mirrors, and we could cut that down by better structuring
   our network of mirrors, and giving that more of a hierarchy instead
   of having everyone pull from the root.

   We should talk to the vikings people here, what sort of traffic is
   allowed on their network, to get a feeling how much we need to cut
   down here. Also, we should start putting better traffic monitoring in
   place.

So much for the rationale. In summary, we believe that we currently need
a single VPS with:

   - a single core
   - at least 8GiB of RAM
   - at least 1TiB of storage
   - a reasonable agreement on traffic limits

If I have missed or forgot anything, let me know. Otherwise, we should
probably get the vikings people in the loop and discuss next steps!

Best,
Andreas (oaken-source)



On Tue, Nov 05, 2019 at 11:28:36PM +0100, Andreas Grapentin wrote:
> 
> Hi,
> 
> As you know, a good chunk of the infrastructure that makes parabola is
> currently hosted on a virtual machine called winston, which is
> generously provided by the hosting provider 1984. We used to have proton
> as well, but when that died, the things that it hosted were migrated
> over to winston.
> 
> It's come to our attention that this machine is currently struggling to
> cope with the load it has to bear, as is evident by the fact that it is
> as of this writing at ~70% swap utilization, and requires more and more
> frequent reboots just to keep our basic infrastructure operational. As
> you can probably imagine, this is not a healthy mode of operation for a
> distribution, and is something that is eating a bunch of hacker time to
> keep on top of.
> 
> These developments have prompted freemor, bill-auger and me to keep an
> eye open for other hosting options that would maybe give our services
> and bit more breathing room, and maybe some space to grow and expand.
> 
> bill-auger had floated the idea to get in touch with the people over at
> vikings (vikings.net) -- in CC -- who offer (among other things) hosting
> services on exclusively libre-friendly server hardware, which would be
> perfect for parabola. We did already have a brief chat with their team
> to figure out what their available hosting options are, and now we need
> to discuss amongs ourselves what sort of infrastructure we want going
> forwards.
> 
> I think it would make sense to start by listing all the services that
> the new infrastructure should host, and estimating from there what sort
> of physical or virtual machine or machines we need, and how we
> distribute the services.
> 
> Off the top of my head, these are the services that I know we run:
> 
>   - webserver for parabola.nu, including
>      - wiki
>      - main website
>      - git repositories
>      - issue tracker (redmine?)
>      - forum
>   - Tier 0 package repositories
>      - repo importer
>   - mailman
> 
> But I'm probably missing a bunch of things. This is where I'd like
> someone more experienced with our infrastructure to step in and explain
> what we have and need. I would suggest we start maintaining this
> information in the wiki as well, assuming no such article exists
> already. This might come in handy down the line.
> 
> My hunch as a Software Engineer and Architect would be that it could
> make sense to partition these services in two, or maybe three VMs, just
> to have a more fine-grained control over fault tolerance and recovery in
> case something does go belly-up, and to be able to separate the critical
> from the non-critical infrastructure. But this is something we need to
> discuss down the line, if we have a clear idea about our service
> landscape.
> 
> While we do this, I would also suggest we take another look at which
> part of our infrastructure could and / or should be converted to
> packages, instead of unaccounted for files. We could conceivably add a
> separate undocumented repo for server-only packages. And while we're at
> that, I would also like to take a gander at automatic provisioners, such
> as ansible, that would allow us to describe our server roles and then
> automatically deploy them as needed, to have a self-documenting
> deployment of our services.
> 
> But first, let's complete the list of services that parabola needs to
> function, and figure out how we want to partition these and what sort of
> resources they need. Everything beyond that is still far away. :)
> 
> 
> I have to say; I am personally very excited to see these changes unfold.
> I believe that parabola is currently at a place where we need to
> stabilize our infrastructure and establish ourselves as a serious
> distribution. And a collaboration with vikings could be a great chance
> to take a leap toward that direction.
> 
> However, I also appreciate that parabola is an adhocracy, and that each
> of you has their own unique views on these issues and options. If you
> have any concerns about these developments, please bring them to our
> attention as well.
> 
> I'm looking forwards to everyones input.
> 
> Best,
> Andreas
> 
> oaken-source
> 
> 
> 
> -- 
> 
> ------------------------------------------------------------------------------
> my GPG Public Key:                 https://files.grapentin.org/.gpg/public.key
> ------------------------------------------------------------------------------



> _______________________________________________
> Dev mailing list
> Dev at lists.parabola.nu
> https://lists.parabola.nu/mailman/listinfo/dev


-- 

------------------------------------------------------------------------------
my GPG Public Key:                 https://files.grapentin.org/.gpg/public.key
------------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.parabola.nu/pipermail/dev/attachments/20191117/919f3e9f/attachment-0001.sig>


More information about the Dev mailing list