[Dev] [RFC] Redesign and reimplement librerelease and db-update

Nicolás Reynolds fauno at kiwwwi.com.ar
Tue Apr 10 22:14:02 GMT 2012


On Tue, 10 Apr 2012 13:53:14 -0500, Joshua Ismael Haase Hernandez <hahj87 at gmail.com> wrote:
> 
> On Mon, 09 Apr 2012 23:24:06 +0200, mtjm at mtjm.eu (=?utf-8?Q?Micha=C5=82_Mas=C5=82owski?=) wrote:
> > fauno and I discussed on #parabola changing how db-update works to
> > improve its reliability and usability.
> 
> I like the idea, it seems feasible and easy to implement, however I
> think we should document the whole process of repo maintaining, because
> most people (me included) don't know exactly how it works.
> 
> > My proposal is to make librerelease run a script on repo and pipe to
> > it the files to upload.
> 
> +1 server site script. Just some questions, Can multiple packagers upload
> their packages at the same time? How will this be addresed?

I just commited a librerelease change that immediately runs db-update on the
server. it's `ssh parabola dbscripts/db-update`.

> 
> This is now addresed by uploading the packages to a «staging» area, but
> I do think we do need a way to stop outdated or same package version to
> be uploaded.
> 

I think the idea is not to change this but to make it more reliable

> > The script verifies them and fails if one is incorrectly signed.
> >
> > Then, when all files are ready, it locks all databases used by them,
> > checks if the same version is available for any of them, fails if yes
> > (explicitly recommending adding .1 to pkgrel if the new version is
> > needed), then updates the packages and unlocks the databases.
> >
> > librerelease outputs errors from the repo-side script.
> >
> > If multiple uploads including the same package are done at once, only
> > at most one succeeds, others fail, the library uploaded must be the one
> > From the successful run.
> >
> > The server-side script might log its output to the maintenance list.
> >
> > Locks:
> >
> > - one for all repos, or
> >
> > - one per repo and arch, needs correct ordering of their acquisitionn
> >
> > Benefits:
> >
> > - no duplicated uploads of a package (these contributed to the great
> >   pcre breakage affecting glib and grep on mips64el and made many
> >   corrupted package errors)
> >
> > - immediate error messages on failed upload, for the user who did the
> >   upload
> >
> > - no (theoretical yet, although possible) problems like updating
> >   xorg-server and drivers, while having only drivers updated due to
> >   missing xorg-server signature or (not sure if rsync prevents this)
> >   if the connection is interrupted while uploading xorg-server after
> >   drivers
> 
> I think this require use ABS or another thing as control for deps.
> 

yes, but the problem is keeping abs up to date. original arch dbscripts
rely on svn. i changed it to simply use abs, but abs is updated once a
day from svn so our dbscripts ended up forbidding package releases and
annoying me constantly; i disabled this functionality.

> > Some requirements:
> >
> > - a single (repo, arch, pkgname, epoch, pkgver, pkgrel) tuple refers to
> >   exactly zero or one released package
> >
> > - multiple uploads at once work, usually at least one succeeds
> >
> > - all errors make just the affected upload do nothing (and notify
> >   appropriate users), they don't affect other, correct, uploads
> >
> > - no deadlocks; interrupted uploads don't cause other problems
> >
> > - runs are quick, except for waiting for the files to be uploaded (no
> >   waiting for files of other users)
> >
> > Possible issues:
> >
> > - continuing interrupted uploads
> >
> > fauno recommended checking lockfile-utils for locking support (writing
> > our would make more opportunities for bugs).  We need lock-or-fail and
> > unlock operations for a single lock or a set of locks.
> >
> > Any comments?  Is it too complex, or too unreliable, are the
> > requirements unsatisfiable, or ...?

i agree if the implementation is kiss :P

-- 
:{
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: <https://lists.parabola.nu/pipermail/dev/attachments/20120410/ee65ef04/attachment.sig>


More information about the Dev mailing list