[Dev] [RFC] Adding DLAGENTS for git and other VCSs to makepkg.conf

Luke T.Shumaker lukeshu at sbcglobal.net
Thu Apr 12 05:43:55 GMT 2012


Abstract

   Given Parabola's policy that sources be downloaded prior to running
   a PKGBUILD file's build() function, we are creating multiple
   boiler-plate SRCBUILD files that all check out a specific commit
   from a version control system (VCS), and generate a tarball from it.

   Given that most VCSs are able to identify a commit just as
   permanently as a URL, or more so, it makes sense to to add a
   URL-handler ("DLAGENT") for URLs with a VCS SCM scheme, such as
   "git://".

Permanence of availability concerns

   One of the primary objections to getting files directly from an VCS
   has been that there must be a permanence of availability, a user
   must have a guarantee that the source he is downloading is the
   source originally used to build the package.

   If we give the DLAGENT the ability to specify a commit ID as part
   of the URL (not traditionally done by VCS tools, more below), the
   commit is just as unlikely to change as any other URL is.

   Further, distributed VCSs (DCVS) typically have cryprographically
   secure hash IDs, for example

     https://github.com/nothingmuch/git-svn-abandon.git#42014bf555fdac67c8c82ed69c4733ab9dba0c8a

   is cryptographically assured to always refer to the same code.

URL schemes for VCS tools

   VCS tools frequently use "pseudo-URLs". This necessitates using a
   wrapper program to (trivially) map between a URL and a VCS "remote
   repository", as well as getting the appropriate commit from the
   repository.

   Following are definitions for mappings for common VCS
   tools.another prot

   Git:
       git://${authority}/${path}[#${refspec}]
       git+${proto}://${authority}/${path}[#${refspec}]
     where
       * ${proto} is the scheme of the protocol being used to access
         the repository, if not git, sush as "ssh" or "http".
       * ${authority} is the "[userinfo@]host[:port]" string we all know
         so well, defined in RFC 3986.
       * ${path} is the path-part after the authority
       * ${refspec} is the git refspec of the commit. For our
         purposes, this is a commit hash, but it could be a branch or
         tag.
   SVN:
       svn+${proto}://${authority}/${path}[#${revnum}]
     where
       * ${revnum} is the revision number of the commit we are after

Implications

   This would require adding a few lines to the DLAGENTS array in
   /etc/makepkg.conf, as well as requiring that a few wrapper scripts
   be installed.

Other ideas

   I have some more ideas about decent caching/tar'ing of commit trees,
   but I'll have to tinker first.

Questions

   What do you think of this idea?
   If you like it, should the wrapper scripts be packaged with pacman,
   or be a separate package?

Sample implementation

   The following implementation is from an unrelated project I was
   working on that required pacman-like functionality. It is
   unsuitable for use by `makepkg`, however, displays the concept
   well.

case "$scheme" in
	http|https|ftp)
		wget -c "$url"
		bsdtar -xf "${url##*/}"
		;;
	svn+*)
		host="`echo "$url"|sed -r 's at .*://([^:/]*)[:/].*@\1@'`"
		data="`grep -F "$host" -- "$FRCUTILDIR/etc/svn-auth"`"
		if [ -n "$data" ]; then
			data="`echo "$data"|sed -r 's/\s+/\t/g'|cut -f 2-`"
			set -- $data
			auth="--username $1 --password $2"
		fi
		mkdir -p "$FRCUTILDIR/tmp/svn-config"
		svn co $auth --config-dir "$FRCUTILDIR/tmp/svn-config" "${url#svn+}" "$package"
		;;
	git+*|git)
		nurl=${url#git+}
		base=${nurl%%#*}
		frag=${nurl#*#}
		frc msg2 "Pulling from GIT repository..."
		if [ ! -d "$package" ]; then
			git clone "$base" "$package"
		else
			cd "$package" && {
				git pull "$base"
				cd ..
			}
		fi
		if [ -n "$frag" ]; then
			frc msg2 "Checking out proper commit/tag/branch..."
			cd "$package" && {
				git checkout "$frag"
				cd ..
			}
		fi
		;;
esac



More information about the Dev mailing list