[Dev] Let's talk about Infrastructure and Hosting
Megver83
megver83 at hyperbola.info
Sun Nov 17 01:49:16 GMT 2019
El dom, 17-11-2019 a las 01:26 +0100, Andreas Grapentin escribió:
> Hi,
>
> it's been almost two weeks since the first announcement was made, and
> I
> believe we have now a good understanding of our infrastructure and
> our
> hosting needs, and are ready to move forward.
>
> We have discussed internally, and have come to the conclusion that a
> good course of action could be to keep winston running, but start
> migrating some of the essential services off it one by one,
> streamlining
> their deplopment in the process.
>
> To get started, we are currently looking at getting a single VPS,
> with
> specs comparable to, or slightly improved upon, the specs of winston
> itself.
>
> Winston currently has:
>
> - (nominally) about 4GiB of RAM.
>
> we believe that 4GiB is currently the lower bound of what our
> services need to function. Winston is currently struggling with
> RAM,
> but this seems to be mainly because he's not having the nominal
> amount actually available for use.
>
> To get some room to breathe, we're hoping for an upgrade here
> towards
> the region of 6 or 8GiB of ram, mainly to make sure the machine
> doesn't need to swap as much as it does now.
>
+1
> - 4 QEMU virtual CPUS @ 2.2GhZ
>
> winston is never really computationally busy. iirc, vikings is
> allowing only for 1 CPU VPS's (please correct me if I'm wrong) so
> we're not looking for an upgrade here. Realistically, anything
> that's
> not a potato should be sufficient to deal with our workload.
>
+1
> - 500GiB of storage
>
> storage is getting a bit tight. We are planning to extend our
> repos
> to more architectures, chiefly ppc64le and riscv64, and we would
> not
> be able to host all the built packages due to limited storage
> capacity.
Regarding the ppc64le and riscv64 ports, we have to discuss that
because we need more people for this *unless* we set up a functional
autobuilder for all our packages in a very capable machine (mainly
because there are some *very heavy* software). We can create a git repo
for modified PKGBUILDs like Arch32 and ALARM do, but for testing these
in real hardware (because IMO it's useless to waste time in porting to
architectures that are only going to be used in VMs or emulators) we
need testers (could be some Parabola devs with this hardware or very
collaborative users that own it). Plus, some packages won't cross-
compile using qemu (like libretools does with armv7h), which is sth.
that has happened to me with some armv7 pkgs, so I build them in my
Banana Pi and works.
Idk how will we do for ppc64le, because the TALOS II is too expensive,
but there are other PPC computers (I'm digging a bit about them, I
already know the discontinued PowerMacs from Apple but I want to find
something more)
>
> Ideally, we would like to see an extended capacity of at least
> 1TiB
> here, in order to work towards our medium-term goals. Even more
> storage would allow us to move forward with other projects, such
> as
> providing a source packgaes repository, or maintaining a larger
> archive of older packages. But we can discuss these separately
> once
> the respective projects move forward.
> - virtually unlimited network traffic
>
> as far as I know, winston is not limited in terms of network
> traffic.
> And we do incur a LOT of network traffic. The order of magnitude
> should be somewhere in the single-digit TiB's per month, although
> it's hard to be sure since we don't have a monitoring in place.
>
> Most of this traffic is because of a growing number of busily
> syncing
> parabola mirrors, and we could cut that down by better structuring
> our network of mirrors, and giving that more of a hierarchy
> instead
> of having everyone pull from the root.
>
> We should talk to the vikings people here, what sort of traffic is
> allowed on their network, to get a feeling how much we need to cut
> down here. Also, we should start putting better traffic monitoring
> in
> place.
>
> So much for the rationale. In summary, we believe that we currently
> need
> a single VPS with:
>
> - a single core
> - at least 8GiB of RAM
> - at least 1TiB of storage
> - a reasonable agreement on traffic limits
>
> If I have missed or forgot anything, let me know. Otherwise, we
> should
> probably get the vikings people in the loop and discuss next steps!
>
> Best,
> Andreas (oaken-source)
>
>
>
> On Tue, Nov 05, 2019 at 11:28:36PM +0100, Andreas Grapentin wrote:
> > Hi,
> >
> > As you know, a good chunk of the infrastructure that makes parabola
> > is
> > currently hosted on a virtual machine called winston, which is
> > generously provided by the hosting provider 1984. We used to have
> > proton
> > as well, but when that died, the things that it hosted were
> > migrated
> > over to winston.
> >
> > It's come to our attention that this machine is currently
> > struggling to
> > cope with the load it has to bear, as is evident by the fact that
> > it is
> > as of this writing at ~70% swap utilization, and requires more and
> > more
> > frequent reboots just to keep our basic infrastructure operational.
> > As
> > you can probably imagine, this is not a healthy mode of operation
> > for a
> > distribution, and is something that is eating a bunch of hacker
> > time to
> > keep on top of.
> >
> > These developments have prompted freemor, bill-auger and me to keep
> > an
> > eye open for other hosting options that would maybe give our
> > services
> > and bit more breathing room, and maybe some space to grow and
> > expand.
> >
> > bill-auger had floated the idea to get in touch with the people
> > over at
> > vikings (vikings.net) -- in CC -- who offer (among other things)
> > hosting
> > services on exclusively libre-friendly server hardware, which would
> > be
> > perfect for parabola. We did already have a brief chat with their
> > team
> > to figure out what their available hosting options are, and now we
> > need
> > to discuss amongs ourselves what sort of infrastructure we want
> > going
> > forwards.
> >
> > I think it would make sense to start by listing all the services
> > that
> > the new infrastructure should host, and estimating from there what
> > sort
> > of physical or virtual machine or machines we need, and how we
> > distribute the services.
> >
> > Off the top of my head, these are the services that I know we run:
> >
> > - webserver for parabola.nu, including
> > - wiki
> > - main website
> > - git repositories
> > - issue tracker (redmine?)
> > - forum
> > - Tier 0 package repositories
> > - repo importer
> > - mailman
> >
> > But I'm probably missing a bunch of things. This is where I'd like
> > someone more experienced with our infrastructure to step in and
> > explain
> > what we have and need. I would suggest we start maintaining this
> > information in the wiki as well, assuming no such article exists
> > already. This might come in handy down the line.
> >
> > My hunch as a Software Engineer and Architect would be that it
> > could
> > make sense to partition these services in two, or maybe three VMs,
> > just
> > to have a more fine-grained control over fault tolerance and
> > recovery in
> > case something does go belly-up, and to be able to separate the
> > critical
> > from the non-critical infrastructure. But this is something we need
> > to
> > discuss down the line, if we have a clear idea about our service
> > landscape.
> >
> > While we do this, I would also suggest we take another look at
> > which
> > part of our infrastructure could and / or should be converted to
> > packages, instead of unaccounted for files. We could conceivably
> > add a
> > separate undocumented repo for server-only packages. And while
> > we're at
> > that, I would also like to take a gander at automatic provisioners,
> > such
> > as ansible, that would allow us to describe our server roles and
> > then
> > automatically deploy them as needed, to have a self-documenting
> > deployment of our services.
> >
> > But first, let's complete the list of services that parabola needs
> > to
> > function, and figure out how we want to partition these and what
> > sort of
> > resources they need. Everything beyond that is still far away. :)
> >
> >
> > I have to say; I am personally very excited to see these changes
> > unfold.
> > I believe that parabola is currently at a place where we need to
> > stabilize our infrastructure and establish ourselves as a serious
> > distribution. And a collaboration with vikings could be a great
> > chance
> > to take a leap toward that direction.
> >
> > However, I also appreciate that parabola is an adhocracy, and that
> > each
> > of you has their own unique views on these issues and options. If
> > you
> > have any concerns about these developments, please bring them to
> > our
> > attention as well.
> >
> > I'm looking forwards to everyones input.
> >
> > Best,
> > Andreas
> >
> > oaken-source
> >
> >
> >
> > --
> >
> > -----------------------------------------------------------------
> > -------------
> > my GPG Public Key:
> > https://files.grapentin.org/.gpg/public.key
> > -----------------------------------------------------------------
> > -------------
>
>
> > _______________________________________________
> > Dev mailing list
> > Dev at lists.parabola.nu
> > https://lists.parabola.nu/mailman/listinfo/dev
>
> _______________________________________________
> Dev mailing list
> Dev at lists.parabola.nu
> https://lists.parabola.nu/mailman/listinfo/dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.parabola.nu/pipermail/dev/attachments/20191116/3bd3458b/attachment.sig>
More information about the Dev
mailing list