[Dev] dbscripts plan v2

Luke Shumaker lukeshu at lukeshu.com
Wed Aug 22 22:41:42 GMT 2018

TL;DR: Over the next 2 or more weeks, there will some changes in the
pacman repos/dbscripts.  Please keep an eye out for anything amiss,
and reply in this email thread.  This going smoothly depends on
packager feedback if something is wrong.

Hi all,

Back in May there was a ~2-week "sprint" to improve dbscripts.
dbscripts did get a lot better, and while we did hit the true
objective of "improve db-import-pkg and add testing such that we
aren't afraid to touch it", we didn't hit either of the stated
concrete objectives.  Let's try again.


 1. Have db-import-pkg use db-update/db-remove/db-move, rather than
    manipulating the repos directly.  We couldn't hit this before
    because db-remove was subtly broken, and db-move was not-so-subtly
    broken.  These were fixed in dbscripts v20180716.

 2. Have proper user-separation, instead of having everything run as
    repo at .  We didn't hit this before because there were so many files
    with funny permissions.  That list is now much smaller.


 1. Adjust users/groups/ownership on winston.parabola.nu

     a. Create the `repo` group

        Change: `groupadd --system repo`
	Rationale: This is a supporting change to the later bits of step 1.
        Timeframe: already done, 2018-08-22

     b. Correct any funny permissions in /srv/repo


            rm -f -- /srv/repo/main/other/kodi-libre/.kodi-libre-17.6-Krypton.tar.gz
            chmod 0644 -- /srv/repo/main/other/kodi-libre/addons/krypton/makesums.sh


           I have a script in my home directory on winston that looks
           for any funny permissions in /srv/repo.  The output should
           be self-explanatory in what it's looking for:

               $ ./find-funny 
               ==> Files with funny types: (expecting d/f/l)
               ==> Files with funny ownership: (expecting repo:users or lukeshu:users)
               ==> Directories with funny perms: (expecting 0755/2755/2775)
               ==> DB files with funny perms: (expecting 0664)
               ==> Non-DB Files with funny perms: (expecting 0644)
                    1	-rw------- 1 repo users 14417920 Mar 19 20:14 /srv/repo/main/other/kodi-libre/.kodi-libre-17.6-Krypton.tar.gz.9zxQwS
                    2	-rwxr-xr-x 1 repo users      175 Oct  1  2017 /srv/repo/main/other/kodi-libre/addons/krypton/makesums.sh
               ==> Links with funny perms: (expecting 0777)

           The file `.kodi-libre-17.6-Krypton.tar.gz.9zxQwS` is a
           truncated version of `kodi-libre-17.6-Krypton.tar.gz`.

	   The commands given should make `./find-funny` come back
	   clean.  We should verify that the permissions are what we
	   expect before we go recursively changing them.

	   The files that are owned by 'lukeshu' instead of 'repo' are
	   from when files were deleted when someone left the project,
	   and I restored them from a mirror.

        Timeframe: Tonight or tomorrow; after 1 final snapshot of the
        way thing currently are is taken by beefcake.

     c. Switch over to using the `repo` group


            usermod --gid=repo repo
            chgrp -R repo /srv/repo/main
            find /srv/repo/main -type d -exec chmod 2775 -- {} +


           All hackers' user accounts are automatically be members of
           the new 'repo' group (per `/etc/parabola-hackers.yml`),
           following its creation in step 1.a.  Given the `002` umask
           that `db-functions` sets, write access to the repos will
           now be governed by membership in the `repo` NSS group,
           rather than restricted to the `repo` user (access to which
           is governed by the `repo` hackers.git group).

           This is a good idea (security-wise) anyway; since
           `db-functions` runs `chmod g+w` the `.db` files, currently
           all local users have write access to the `.db` files.
           Packagers *should* then be able to adjust their
           `libretools.conf:REPODEST` to be
           `ssh://$LIBREUSER@repo.parabola.nu:1863/~/staging/` instead of
           This will be opt-in for a period, so that we can identify any
           issues before rolling the change out to all packagers.
           Doing user-separation for human packagers first, before
           doing it for db-import, allows us to limit large-scale
           breakage in case anything goes wrong, and ensures that a
           human is looking at what is going on during the initial
           days of it.
        Timeframe: Tonight or tomorrow; after 1 final snapshot of the
        way thing currently are is taken by beefcake.

 2. Use db-{move,update,remove}

       Rework `db-import-pkg` to set up a `db-update`-style staging
       directory, rather than manipulating the repos directly.  At
       this point, `db-import-pkg` is still running as `repo`.
       However, set `ProtectSystem=strict`, limit it to a staging &
       scratch directory, and have it
           ssh localhost DBSCRIPTS_CONFIG=...${UPSTREAM} db-update
       to add/remove packages.  This will involve adding an ssh key
       for repo.
    Rationale: Obviously, this is a primary objective.  Using
    ssh+ProtectSystem allows us to ensure that the objective is met.

    Concerns: Testing this will probably involve running an SSH server
    in the test suite.

    Timeframe: As soon as I can get it done after step 1; a couple of
 3. Migrate humans off of repo@

    Change: Adjust the default `libretools.conf:REPODEST` to be
    `ssh://$LIBREUSER@repo.parabola.nu:1863/~/staging/` instead of
    Rationale: This will change the user-separated `db-update` from
    opt-in (following step 1) to opt-out.
    DISRUPTION: This will make the default configuration unsuitable
    for packagers whose local username doesn't match their username in
    hackers.git.  They will need to manually adjust their
    `libretools.conf:REPODEST` to have the correct username.
    Timeframe: at least a week from now [NB: no sooner than 2018-08-29]

 4. Migrate robots off of repo@

    Change: Add
    `db-import-{packages,community,archlinux32,archlinuxarm}` users,
    members of the `repo` group; have `db-import-pkg` run as them.
    Rationale: Mission accomplished
    Timeframe: at least a week after that [NB: no sooner than 2018-09-05]

In summary:

  [-] 1. Set up the `repo` group        [eta: 2018-08-23]
  [ ] 2. Use db-{move,update,remove}    [eta: ???]
  [ ] 3. Migrate humans off of repo@    [eta: no sooner than 2018-08-29]
  [ ] 4. Migrate robots off of repo@    [eta: no sooner than 2018-09-05]

Comments?  Concerns?

Happy hacking,
~ Luke Shumaker

More information about the Dev mailing list