[Dev] dbscripts changes/plans

[Luke Shumaker]

Hi guys,

There was talk on IRC tonight about improving dbscripts.  So I figured
I'd share where we're at, and where I think we should go.

== Right now ==

I've just now post some changes on dbscripts 'master' and updated repo
to that.  Pretty conservative stuff.  Probably the biggest part is
that there's a decent README now.

I've also created the 'pending' branch.  I'm pretty sure it's good to
go, but I'm not quite confident enough to deploy it right before I go
to bed.

Bigger changes are still on the 'lukeshu/xbs' branch.  The differences
between 'master'/'pending' and 'lukeshu/xbs' are really small now![1]
There are 2 big changes sitting there:

 - Stricter integration with xbs (abs/abslibre).  I'm holding that
   back until xbs is working a little better.
 - db-import: A better way of importing packages from Arch Linux
   (replacing the handful of db-import-archlinux-* scripts). Relies on
   xbs a bit more.  It needs to be tested.

== Next Steps ==

From there, either modify db-import to also support Arch Linux ARM, or
rename it to db-import-archlinux and create db-import-archlinuxarm
(either way, replacing db-import-archlinuxarm-pkg).

I think that a reading of the README suggests a number of places to
improve.  I think these are probably also good ways to get more/new
people contributing in dbscripts!  Search for the word "should"!  It
wouldn't be direct-feature improvement, but improving the code base
would make maintenance, troubleshooting, and development much easier.

Something the default config on 'lukeshu/xbs' hints at is that I would
like to have separate user accounts for different tasks.  I think this
would help with making it hard to accidentally screw things up.
Perhaps nshd[2] plays a role in this.

== Road forward ==

In my opinion, real improvement is blocked by getting 'lukeshu/xbs'
merged.  Right now, every db-import-* implements its own raw access to
the repos, which is aweful and error prone.  It also means that if we
want to make a change (keep reading), it has to be changed in a huge
list of places, which aren't documented.  'lukeshu/xbs' db-import uses
db-{update,move,remove} to access the repo--as everything should.
Once we have those 3 programs as the single way to write to the repo,
we can start improving.

As long as we've supported things other than i686 and x86_64 (remember
mips?), there has always been pain with arch=(any) packages.  If
everything writing to the repos used db-{update,move,remove} as it
should, this would have never been a problem.

Several of the cron jobs (namely make_repo_torrents) can just become a
hook that runs on package upload.  Everything becomes more fluid.
Fewer cron jobs and batches.

I think that a good other place to work is that moving from
abslibre.git to a PBS-based[3] system would be good.  PBS was designed
to address the ways that git is shitty for how we use it to track
PKGBUILDs.  Somewhere on ArchWiki there was a page about all of the
things they'd have to do to use git instead of svn, from a tooling
perspective, and I was like "Oh all of the things I'm doing with PBS.
Glad to know we're on the same page."  Anyway, that would mean
actually finishing/testing pbs-tools.  Notes on that: I was trying to
do the conversions with `git rewrite-branch` (basically a wrapper
around `git filter-branch`, lives in the pbs-tools git repo).  This
was a mistake.  It's just too slow.  The real version will need to use
`git fast-export`/`fast-import`.  We may need to implement a utility
to complement them, in the same way that `git rewrite-branch`
complements `git filter-branch`.

I say this because XBS was originally invented to abstract away the
differences between pbs-tools, git/abslibre.git, and svn (as Arch
Linux uses it).

Using `pbs-tools` would also make adding AUR packages to our repos a
lot nicer.

-- 
Happy hacking,
~ Luke Shumaker

[1]: Starting a bit over a year ago, changes have slowly been
     propegating between 'lukeshu/xbs' and 'master' -- representing
     the first time that changes have made their way from Arch Linux
     dbscripts into Parabola dbscripts since 2012!  It's been
     challenging to identify where differences originated.  Did they
     add code, or did we delete it?  How do we unify the two
     increasingly different codebases?  When 'lukeshu/xbs' finally
     gets merged into 'master' that will be the first time that commit
     from Arch Linux since 2012-01-08 will be a git ancestor of a
     commit in Parabola's dbscripts!

[2]: https://projects.parabola.nu/~lukeshu/nshd.git/
[3]: https://projects.parabola.nu/packages/pbs-tools.git/