[Dev] Fauno's workflow

[Luke T. Shumaker]

At Fri, 30 Nov 2012 14:28:44 -0300,
Nicolás Reynolds wrote:
> 
> [1  <text/plain (quoted-printable)>]
> "Luke T. Shumaker" <lukeshu at sbcglobal.net> writes:
> 
> > Fauno put this file in /doc/ in libretools, but I felt the best way to
> > discuss it was an email
> >
> > Fauno wrote:
> >> ## fauno's way
> >> 
> >> During packaging, I don't usually restart a build from scratch if I have to
> >> make changes to the PKGBUILD. I use a lot of commenting out commands already
> >> ran, `makepkg -R`, etc. When I used `libremakepkg` I ended up using a lot more
> >> `librechroot` and working from inside the unconfigured chroot, because
> >> `makechrootpkg` (the underlying technology for `libremakepkg`) tries to be too
> >> smart.
> >
> > I'm not sure that's a "safe" way to build packages. I might do that
> > while getting a package to build, but I always build from scratch at
> > the end. "ccache" makes this bearable.
> 
> i'm a daredevil B)
> 
> > Also, libremakepkg no longer uses makechrootpkg, and is *very*
> > transparent and doesn't try to be too smart.
> >
> >  * Copies the PKGBUILD and sources in (ok, some clever things happen
> >    to copy the sources)
> 
> which ones?

The PKGBUILD in the current directory, and the sources named in it;
the ones `makepkg` would care about.

> >  * Runs checks on the PKGBUILD (currently pkgbuild-check-nonfree,
> >    eventually librenamcap)
> >  * Runs "makepkg -o"
> >  * Runs checks on the source directory
> 
> what kind of checks?

It runs the functions:

 * libre_check_pkgbuild
    currently:
     * "pkgbuild-check-nonfree"
    eventually:
     * "pkgbuild-check-nonfree"
     * "namcap"
 * libre_check_src
    currently:
     * nothing
    eventually:
     * "jh checksource"
 * libre_check_pkg
    currently:
     * nothing
    eventually:
     * "namcap"

And possibly more. The checking will eventually be encapsulated by
"librenamcap".

> >  * Runs "makepkg -e" with networking dissabled in the chroot
> 
> sounds fine
> 
> >  * Copies the resulting packages to SRCDEST.
> 
> ?

It copies the packages in ${chrootdir}/srcdest to your configured
SRCDEST outside of the chroot; where normal `makepkg` would put them.

I forgot to mention it, but it also runs `libre_check_pkg` on the
packages before it copies them.

> >> When I started writing `treepkg` I found that mounting what I need
> >> directly on the chroot and working from inside it was much more
> >> comfortable and simple than having a makepkg wrapper doing funny
> >> stuff (for instance, mangling makepkg.conf and breaking everything.)
> >
> > There is still some mangling of pacman.conf that I don't understand
> > (maybe it is important; I'm affraid to remove code I don't understand)
> > https://labs.parabola.nu/issues/256
> >
> > However, all that happens to makepkg.conf is it sets PKGDEST and SRCDEST.
> 
> i remember it also rewrites the makepkg entirely everytime, so if you
> use distcc (it requires setting DISTCC_HOSTS) it will disable it.

Maybe it used to, but it no longer does. (I did rewrite it after all)

> >> This is how the chroot is configured:
> >> 
> >> * Create the same user (with same uid) on the chroot that the one I
> >> use regularly.
> >> 
> >> * Give it password-less sudo on the chroot.
> >> 
> >> * Bind mount /home to /chroot/home, where I have the abslibre-mips64el clone.
> >
> > I'm not sure that's a good idea; maybe it works for you, but
> >  * A lot of tools look at configuration in $HOME; a chroot means it is
> >    a fresh directory... unless you bind mount /home.
> 
> it may also bind mount a subdir on home where you store abslibre and
> libretools (i use the git version) and that's clean enough. it's never
> been a problem though.

Eh, I'm a fan of getting an up-to-date libretools on the repos.

> >  * A chroot also protects you from a misbehaving build script messing
> >    with your $HOME.
> 
> i usually check pkgbuilds from aur and trust the ones that come from
> abslibre...

Not just the PKGBUILDS, but the Makefiles, ant scripts, etc that are
part of the source.

> >> * Bind mount /var/cache/pacman/pkg to /chroot/var/cache/pacman/pkg
> >> * Put these on system's fstab so I don't have to do it everytime
> >
> > archroot (librechroot's backend) bind mounts that for you so you don't have to
> > have it in fstab or do it manually.
> 
> but it's done several times, on fstab it's done once at boot.
> 
> >> * Configure makepkg.conf to PKGDEST=CacheDir and SRCDEST to something on my home.
> >> 
> >> Workflow:
> >> 
> >> * Enter the chroot with `systemd-nspawn -D/chroot` and `su - fauno`.
> >> 
> >> * From another shell (I use tmux) edit the abslibre or search for updates with
> >>   `git log --no-merges --numstat`.
> >> 
> >> * Pick a package and run `treepkg` from its dir on the chroot, or retake
> >>   a build with `treepkg /tmp/package-treepkg-xxxx`. (Refer to doc/treepkg
> >>   here).
> >> 
> >> What this allows:
> >> 
> >> * Not having to worry about the state of the chroot. `chcleanup` removes and
> >>   adds packages in a smart way so shared dependencies stay and others move
> >>   along (think of installing and removing qt for a complete kde
> >>   rebuild).
> >
> > `librechroot -c` now uses chcleanup to clean the chroot.
> 
> chcleanup is run automatically here, not optionally. -c is needed when
> the chroot is considered dirty if not pristine so it may remove the
> packages you already installed if this is the second run, and this isn't
> the case: chroot is considered dirty when more than the needed packages
> are installed.

How is it done automatically?

> >> * Building many packages in a row without recreating a chroot for every one of
> >>   them.
> >
> > libremakepkg did not recreate the chroot automatically before, and
> > can't now.
> 
> recreating here means "mounting and umounting, removing files, spawning
> shells...", why has this to be done many times when i can do it once or
> less? (yeeloongs aren't the quickest machines).

I wish I hade a yeeloong to verify...

> >> * Knowing that any change you made to the chroot stays as you want (no one
> >>   touches your makepkg.conf)
> >
> > For someone who doesn't work from inside the chroot, only setting
> > PKGDEST and SRCDEST seems to be a good balance between "don't mess
> > with my stuff" and "just work".
> >
> >> * Hability to run regular commands, not through a chroot wrapper. I can `cd` to
> >>   a dir and use `makepkg -whatever` on it and nothing breaks.
> 
> also rm packages from the build order (see treepkg), add or remove
> packages from the repo, etc.
> 
> >> 
> >> * No extra code spent on wrappers.
> >
> > Again, I'm working to make libremakepkg provide more checks than
> > simply working in a clean chroot does.
> 
> checks are ok, i just feel chroot wrappers add complexity that's not
> needed if you take the chroot as a working environment. less bugs to
> trace down!

It's pretty minimal now.  The most complicated parts are:
 * handling signals to umount and exit gracefully
 * the legacy code that only runs if systemd-nspawn isn't available

And some of the "checks" can only be done with a chroot, such as
turning networking off during the build and package stages.

Happy hacking,
~ Luke Shumaker