[Dev] [RFC] Redesign and reimplement librerelease and db-update

Joshua Ismael Haase Hernandez hahj87 at gmail.com
Tue Apr 10 18:53:14 GMT 2012

On Mon, 09 Apr 2012 23:24:06 +0200, mtjm at mtjm.eu (=?utf-8?Q?Micha=C5=82_Mas=C5=82owski?=) wrote:
> fauno and I discussed on #parabola changing how db-update works to
> improve its reliability and usability.

I like the idea, it seems feasible and easy to implement, however I
think we should document the whole process of repo maintaining, because
most people (me included) don't know exactly how it works.

> My proposal is to make librerelease run a script on repo and pipe to
> it the files to upload.

+1 server site script. Just some questions, Can multiple packagers upload
their packages at the same time? How will this be addresed?

This is now addresed by uploading the packages to a «staging» area, but
I do think we do need a way to stop outdated or same package version to
be uploaded.

> The script verifies them and fails if one is incorrectly signed.
> Then, when all files are ready, it locks all databases used by them,
> checks if the same version is available for any of them, fails if yes
> (explicitly recommending adding .1 to pkgrel if the new version is
> needed), then updates the packages and unlocks the databases.
> librerelease outputs errors from the repo-side script.
> If multiple uploads including the same package are done at once, only
> at most one succeeds, others fail, the library uploaded must be the one
> From the successful run.
> The server-side script might log its output to the maintenance list.
> Locks:
> - one for all repos, or
> - one per repo and arch, needs correct ordering of their acquisitionn
> Benefits:
> - no duplicated uploads of a package (these contributed to the great
>   pcre breakage affecting glib and grep on mips64el and made many
>   corrupted package errors)
> - immediate error messages on failed upload, for the user who did the
>   upload
> - no (theoretical yet, although possible) problems like updating
>   xorg-server and drivers, while having only drivers updated due to
>   missing xorg-server signature or (not sure if rsync prevents this)
>   if the connection is interrupted while uploading xorg-server after
>   drivers

I think this require use ABS or another thing as control for deps.

> Some requirements:
> - a single (repo, arch, pkgname, epoch, pkgver, pkgrel) tuple refers to
>   exactly zero or one released package
> - multiple uploads at once work, usually at least one succeeds
> - all errors make just the affected upload do nothing (and notify
>   appropriate users), they don't affect other, correct, uploads
> - no deadlocks; interrupted uploads don't cause other problems
> - runs are quick, except for waiting for the files to be uploaded (no
>   waiting for files of other users)
> Possible issues:
> - continuing interrupted uploads
> fauno recommended checking lockfile-utils for locking support (writing
> our would make more opportunities for bugs).  We need lock-or-fail and
> unlock operations for a single lock or a set of locks.
> Any comments?  Is it too complex, or too unreliable, are the
> requirements unsatisfiable, or ...?

More information about the Dev mailing list