[Dev] [RFC/FYI] dbscripts plan, please read

[Luke Shumaker]

TL;DR: Over the next 2 or more weeks, there will some changes in the
pacman repos/dbscripts.  Please keep an eye out for anything amiss,
and reply in this email thread.  This going smoothly depends on
packager feedback if something is wrong.

Hi all,

There are some changes to dbscripts that I've talked about forever.
They haven't happened yet because:
 1. dbscripts is gross and scary
 2. dbscripts is under-tested
 3. Mistakes can cause critical breakage for... everyone
 4. Each change seems to be blocked by a different change.

Well,
 1. I'm more familiar with dbscripts now than I ever have been
 2. The test-suite is more complete now than it ever has been (though
    db-import-pkg is not tested at all)
 3. I'll be really careful, including communicating changes.
 4. I have a plan (this plan).

This plan is small in scope, to enable the future evolution of
dbscripts.

Objectives:

 - Have db-import-pkg use db-update & db-remove, rather than
   manipulating the repos directly.
 - Have proper user-separation, instead of having everything run as repo@

Plan:

 1. Change: Adjust `db-import-pkg` to have separate `UPSTREAM`s of
    `packages` and `community`, rather than lumping them together in
    the `UPSTREAM=archlinux`.

    Rationale: `db-update` will need to be run with different
    `PKGPOOL`/`SRCPOOL` settings for these cases.

    Timeframe: tonight/tomorrow

 2. Change: Add
    `config.local.{parabola,packages,community,archlinux32,archlinuxarm}`
    files, symlink `config.local` → `config.local.parabola`.  Move
    `PKGREPOS`, `PKGPOOL`, `SRCPOOL` to these files.
    
    Rationale: Separate `DBSCRIPTS_CONFIG` files will need to exist to
    set the per-UPSTREAM `PKGPOOL`/`SRCPOOL` values.  This allows us
    to set them in a uniform way.  The `config.local` symlink allows
    us to preserve current behavior, for clients that don't set `DBSCRIPTS_CNFIG`
    
    Timeframe: tonight/tomorrow
    
 3. Change:
     a. Create a `repo` group on repo.parabola.nu
     b. Change the primary group of `repo`: `users` → `repo`
     c. `chgrp -R repo /srv/repo/main/`
     d. Ensure that the SGID bit is set on all directories in `/srv/repo/main/`
 
    Rationale:

       All hackers' user accounts will automatically be members of the
       group (per `/etc/parabola-hackers.yml`).  Given the `002` umask
       that `db-functions` sets, write access to the repos will now be
       governed by membership in the `repo` NSS group, rather than
       restricted to the `repo` user (access to which is governed by
       the `repo` hackers.git group).

       This is a good idea, security-wise anyway; since `db-functions`
       runs `chmod g+w` the `.db` files, currently all local users have
       write access to the `.db` files.
       
       Packagers *should* then be able to adjust their
       `libretools.conf:REPODEST` to be
       `ssh://$LIBREUSER@repo.parabola.nu:1863/~/staging/` instead of
       `ssh://repo@repo.parabola.nu:1863/~/staging/$LIBREUSER/staging/`.
       This will be opt-in for a period, so that we can identify any
       issues before rolling the change out to all packagers.
       
       Doing user-separation for human packagers first, before doing
       it for db-import, allows us to limit large-scale breakage in
       case anything goes wrong, and ensures that a human is looking
       at what is going on during the initial days of it.
       
    Timeframe: tomorrow

 4. Change: Adjust librerelease to set
    `DBSCRIPTS_CONFIG=/etc/dbscripts/config.local.parabola`
    
    Rationale: This needs to happen after the `config.local.parabola`
    file is created (step #1).  This will allow us to eventually
    remove the `config.local` symlink, once all packagers have
    upgraded to a version of libretools that has this.

    Timeframe: tomorrow

 5. Change: Include systemd files with dbscripts to run cron-jobs as
    systemd timers.
 
    Rationale: This will allow us to easily set User=, ProtectSystem=,
    and such, to enforce security policy, and ensure scripts are
    well-behaved (once we're ready for that).  It will also supply
    complete logs in the journal, for better debugging.
    
    Timeframe: in a few days

 6. Change:
 
      Rework `db-import-pkg` to set up a `db-update`-style staging
      directory, rather than manipulating the repos directly.  At this
      point, `db-import-pkg` is still running as `repo`.  However, set
      `ProtectSystem=strict`, limit it to a staging & scratch
      directory, and have it
    
          ssh localhost DBSCRIPTS_CONFIG=...${UPSTREAM} db-update
        
      to add/remove packages.  This will involve adding an ssh key for
      repo.
    
    Rationale: Obviously, this is a primary objective.  Using
    ssh+ProtectSystem allows us to ensure that the objective is met.
    
    Concern: Currently, it rsyncs against the local repo, to avoid
    downloading up-to-date packages.  If not working in-place, it
    can't do that.  One solution is to copy (or hardlink?) the
    relevant directories from "$FTP_BASE" to "$WORKDIR".

    Timeframe: A few days after that
    
 7. Change: Adjust the default `libretools.conf:REPODEST` to be
    `ssh://$LIBREUSER@repo.parabola.nu:1863/~/staging/` instead of
    `ssh://repo@repo.parabola.nu:1863/~/staging/$LIBREUSER/staging/`.
    
    Rationale: This will change the user-separated `db-update` from
    opt-in to opt-out.
    
    DISRUPTION: This will make the default configuration unsuitable
    for packagers whose local username doesn't match their username in
    hackers.git.  They will need to manually adjust their
    `libretools.conf:REPODEST` to have the correct username.
    
    Timeframe: at least a week from now [NB: no sooner than 2018-05-01]

 8. Change: Add
    `db-import-{packages,community,archlinux32,archlinuxarm}` users,
    members of the `repo` group; have `db-import-pkg` run as them.
    
    Rationale: Mission accomplished
    
    Timeframe: at least a week after that [NB: no sooner than 2018-05-08]

Bonus: If someone is willing to help, we can have a copy of the new
version of `db-import-pkg` run against a shadow copy of the repos, and
generate daily reports on if the new version and old version disagree.

Comments?  Concerns?

-- 
Happy hacking,
~ Luke Shumaker