From mtjm at mtjm.eu Sat Aug 4 08:34:10 2012 From: mtjm at mtjm.eu (=?utf-8?Q?Micha=C5=82_Mas=C5=82owski?=) Date: Sat, 04 Aug 2012 10:34:10 +0200 Subject: [Dev] [Fsf-Debian] Comparison with Parabola In-Reply-To: (Ramana Kumar's message of "Sat, 4 Aug 2012 07:49:06 +0100") References: Message-ID: <87ehnn12f1.fsf@mtjm.eu> > I'd like to know more about why Parabola GNU/Linux is endorsed by the FSF as > a free distribution, because I think it might provide new ideas for routes > by which Debian could be also. The reason is, Parabola has something > somewhat analogous to [nonfree] in the form of all the Arch repositories > ([core], [extra], [community]) and the AUR which are all not restricted to > free software. It doesn't. Our [core], [extra], [community] are stripped of packages which we change or remove and we add new packages/replacements for removed packages to [libre]. (This doesn't apply to mips64el which isn't interesting here, there all architecture-specific packages are built by us so there are changes in all "stable" repos.) > There are two ways Parabola excludes nonfree software: > 1. Putting the [libre] repo before the Arch repos, so a lot of > packages with free replacements are automatically superseded and > become invisible. The same would be done by putting them into the [core], [extra] and [community] repos. > 2. Maintaining the your-freedom package, which must be explicitly > installed, which conflicts with any remaining known nonfree packages. "must be explicitly installed" suggests that (0) it's not installed by default, (1) nonfree packages are available in Parabola, or (2) the user is forced to install it. All there cases are false, except for a package being removed (there is a notification of a conflict/replacement on update, unlike when we stop replacing a package, leading to various problems). > Parabola currently does nothing about the AUR, but there is a feature > request for dealing with it in their bug tracker. AUR isn't a part of Parabola, the only recommendation of it I noticed is a script that fetches a package from it, checking if it's known to be nonfree (or depending on nonfree packages) and asks the user to check if it's free. I believe such a script might be ok. > Why does the FSF endorse Parabola given the situation above where > nonfree packages are available? These packages aren't included in the distro and they aren't recommended (unless you can show a specific example of this, there might be bugs). You can install nonfree software using any distro, the availability of external repositories with nonfree packages doesn't affect the endorsement (as also the Fedora comment suggests and other distros on the list). > Is it possible for Debian to gain endorsement by using something like > the your-freedom package (or maybe a switch in the package manager) > which conflicts with the installation of known nonfree packages? Having to explicitly enable nonfree or contrib seems very like a "switch in the package manager". Just making a default package conflicting with all non-FSDG packages without changing some of them will probably just make any user wanting to install a Mozilla-based program remove the conflicting package (these show an addon list site recommending nonfree addons, there are more examples of such packages, although I don't know a list of them not listing many packages needing branding changes for a derived FSDG-compatible distro). (Ccing the Parabola development list, since this suggests the documentation being misleading and probably other contributors understand these issues better than me.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From lukeshu at sbcglobal.net Sun Aug 5 04:15:51 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Sun, 05 Aug 2012 00:15:51 -0400 Subject: [Dev] [Fsf-Debian] Comparison with Parabola In-Reply-To: <87ehnn12f1.fsf@mtjm.eu> References: <87ehnn12f1.fsf@mtjm.eu> Message-ID: <878vdux9c8.wl%lukeshu@sbcglobal.net> > > Parabola currently does nothing about the AUR, but there is a feature > > request for dealing with it in their bug tracker. > > AUR isn't a part of Parabola, the only recommendation of it I noticed is > a script that fetches a package from it, checking if it's known to be > nonfree (or depending on nonfree packages) and asks the user to check if > it's free. I believe such a script might be ok. I believe you are referring to the `aur` script that is part of libretools. The script is not meant to be used by 'normal users', but by Parabola packagers (like all of the libretools package). On that note, should we patch wget and gcc to warn to make sure that what you're downloading and compiling is free? :) > > > Why does the FSF endorse Parabola given the situation above where > > nonfree packages are available? > > These packages aren't included in the distro and they aren't recommended > (unless you can show a specific example of this, there might be bugs). > You can install nonfree software using any distro, the availability of > external repositories with nonfree packages doesn't affect the > endorsement (as also the Fedora comment suggests and other distros on > the list). We now have two separate bug trackers for bug-bugs and freedom-related bugs. However, on the old bug tracker, anything freedom-related was automatically 'critical'; more important than actual bugs. If you know of any nonfree non-free code that is available from our repositories, please let us know. > (Ccing the Parabola development list, since this suggests the > documentation being misleading and probably other contributors > understand these issues better than me.) Happy hacking! ~ Luke Shumaker From ramana at member.fsf.org Sun Aug 5 08:23:36 2012 From: ramana at member.fsf.org (Ramana Kumar) Date: Sun, 5 Aug 2012 09:23:36 +0100 Subject: [Dev] [Fsf-Debian] Comparison with Parabola In-Reply-To: <878vdux9c8.wl%lukeshu@sbcglobal.net> References: <87ehnn12f1.fsf@mtjm.eu> <878vdux9c8.wl%lukeshu@sbcglobal.net> Message-ID: OK It appears I misunderstood Parabola somehow. Thanks for the replies! :) One remaining question I have there is: if everything in Parabola repositories is free, then what is the point of the your-freedom package? But I don't want to discuss that too much on fsf-collab-discuss - we can resume on a Parabola list (I subscribe to dev). May I redirect this topic to my question pertaining to Debian: Can Debian achieve FSF's view of being a free distribution (relatively easily) by doing something akin to a your-freedom package? On Sun, Aug 5, 2012 at 5:15 AM, Luke T.Shumaker wrote: >> > Parabola currently does nothing about the AUR, but there is a feature >> > request for dealing with it in their bug tracker. >> >> AUR isn't a part of Parabola, the only recommendation of it I noticed is >> a script that fetches a package from it, checking if it's known to be >> nonfree (or depending on nonfree packages) and asks the user to check if >> it's free. I believe such a script might be ok. > > I believe you are referring to the `aur` script that is part of > libretools. The script is not meant to be used by 'normal users', but > by Parabola packagers (like all of the libretools package). > > On that note, should we patch wget and gcc to warn to make sure that > what you're downloading and compiling is free? :) > >> >> > Why does the FSF endorse Parabola given the situation above where >> > nonfree packages are available? >> >> These packages aren't included in the distro and they aren't recommended >> (unless you can show a specific example of this, there might be bugs). >> You can install nonfree software using any distro, the availability of >> external repositories with nonfree packages doesn't affect the >> endorsement (as also the Fedora comment suggests and other distros on >> the list). > > We now have two separate bug trackers for bug-bugs and freedom-related > bugs. However, on the old bug tracker, anything freedom-related was > automatically 'critical'; more important than actual bugs. If you know > of any nonfree non-free code that is available from our repositories, > please let us know. > >> (Ccing the Parabola development list, since this suggests the >> documentation being misleading and probably other contributors >> understand these issues better than me.) > > Happy hacking! > ~ Luke Shumaker From cer at parlementum.net Thu Aug 16 20:30:32 2012 From: cer at parlementum.net (Charles Roth) Date: Thu, 16 Aug 2012 13:30:32 -0700 Subject: [Dev] new GPG key for encyclomundi Message-ID: <20120816203031.GA19912@gramarye.hsd1.wa.comcast.net> Hi Guys, My new Key id is: DD312BBE Attached is a copy CER -- Charles Roth Cultural Detective and Curious Antiquary General Editor of The Encyclomundi Wiki Email: cer at parlementum.net Microblog: http://parlementum.net/parlementum About Me: http://encyclomundi.org/wiki/User:Encyclom "Like so many Victorian gentlemen of leisure, he published pamphlets" -------------- next part -------------- A non-text attachment was scrubbed... Name: cerpubkey.gpg Type: application/octet-stream Size: 10572 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From quiliro at congresolibre.org Thu Aug 16 20:52:05 2012 From: quiliro at congresolibre.org (=?ISO-8859-1?Q?Quiliro_Ord=F3=F1ez?=) Date: Thu, 16 Aug 2012 15:52:05 -0500 Subject: [Dev] [Fsf-Debian] Comparison with Parabola In-Reply-To: References: <87ehnn12f1.fsf@mtjm.eu> <878vdux9c8.wl%lukeshu@sbcglobal.net> Message-ID: <502D5D75.9050306@congresolibre.org> El 05/08/12 03:23, Ramana Kumar escribi?: > OK It appears I misunderstood Parabola somehow. Thanks for the replies! :) > One remaining question I have there is: if everything in Parabola > repositories is free, then what is the point of the your-freedom > package? It is an extra safe measure to tell the user s/he has installed a non-free package from somewhere else. It is useful to remind the user his/her freedom is at peril. > > But I don't want to discuss that too much on fsf-collab-discuss - we > can resume on a Parabola list (I subscribe to dev). > > May I redirect this topic to my question pertaining to Debian: > Can Debian achieve FSF's view of being a free distribution (relatively > easily) by doing something akin to a your-freedom package? That would not make Debian FSF free. It would warn the users that they have installed a non-free package that they have. In case the Debian version of your-freedom package is up to date and correct it would be great to include that package. -- Saludos libres, Quiliro Ord??ez Presidente Asociaci?n de Software Libre del Ecuador - ASLE Av de la Prensa N58-219 y Crist?bal Vaca de Castro Quito, Ecuador (593)2-253 5534 (593)2-340 1517 (593)8-454 8078 From fauno at kiwwwi.com.ar Thu Aug 23 15:21:22 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Thu, 23 Aug 2012 12:21:22 -0300 Subject: [Dev] [FYI] [arch-dev-public] linux-3.5.x status Message-ID: <87pq6hy6q5.fsf@kiwwwi.com.ar> An embedded message was scrubbed... From: Tobias Powalowski Subject: no subject Date: Thu, 23 Aug 2012 16:10:26 +0200 Size: 1039 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From fauno at kiwwwi.com.ar Fri Aug 24 12:58:01 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Fri, 24 Aug 2012 09:58:01 -0300 Subject: [Dev] [FYI] [arch-dev-public] [RFC] systemd service files policy Message-ID: <876288wip2.fsf@kiwwwi.com.ar> An embedded message was scrubbed... From: Tom Gundersen Subject: [arch-dev-public] [RFC] systemd service files policy Date: Thu, 23 Aug 2012 17:32:44 +0200 Size: 3679 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From jorgean at lavabit.com Fri Aug 24 22:21:05 2012 From: jorgean at lavabit.com (Jorge Araya Navarro) Date: Fri, 24 Aug 2012 16:21:05 -0600 Subject: [Dev] [FYI] [arch-dev-public] [RFC] systemd service files policy In-Reply-To: <876288wip2.fsf@kiwwwi.com.ar> References: <876288wip2.fsf@kiwwwi.com.ar> Message-ID: <1345846865.1590.0.camel@abril.local> El vie, 24-08-2012 a las 09:58 -0300, Nicol?s Reynolds escribi?: > mensaje de correo electr?nico adjunto ([arch-dev-public] [RFC] systemd > service files policy) > > > --------- Mensaje reenviado -------- > > De: Tom Gundersen > > Reply-to: Public mailing list for Arch Linux development > > > > Para: Public mailing list for Arch Linux development > > > > Asunto: [arch-dev-public] [RFC] systemd service files policy > > Fecha: Thu, 23 Aug 2012 17:32:44 +0200 > > > > > > Hi guys, > > > > We are hard at work adding systemd service files, and I thought it > > might be worth discussing a few guidelines for how to do it (should > > eventually end up in the wiki I suppose, but thought it would be best > > to start the discussion here). > > > > > > > > I propose (based on input gathered on IRC) something like: > > > > > > 1) Use the upstream service files whenever they exist. > > > > 2) Don't do anything Arch-specific, if possible. This will maximize > > our chances of not having to change behavior in the future once the > > service file is provided by upstream. In particular avoid > > EnvironmentFile= especially if it points to the Arch-specific > > /etc/conf.d/*. > > > > Not using an EnvironmentFile= is ok if: > > > > 2a) either the daemon has its own configuration file where the same > > settings can be specified. > > 2b) the default service file "just works" in the most common case. > > Users who want to change the behavior should then override the default > > service file. > > > > If it is not possible to provide a sane default service file, I > > suppose we should discuss it on a case-by-case basis. > > > > > > > > > > > > A few comments about service files (assuming we just want to preserve > > roughly the current behavior and not do anything fancy (if you want to > > get fancy, I'll assume you know what you are doing)): > > > > 3) If your service requires the network to be configured before it > > starts use After=network.target. Do not use > > Wants/Requires=network.target. > > > > 4) Use Type=forking, unless you know it is not necessary. Most daemons > > use double-forking to signal that they are ready, so to minimize the > > chance of problems it is probably safest to use this mode. To make > > sure that systemd is able to figure out which process is the main > > process, it is best to tell the daemon to write a pidfile and point > > systemd to it using PIDFile=. (If the daemon in question is dbus or > > socket activated, that's a different matter, but I guess that is only > > the case for the minority of daemons). > > > > 5) Our rc scripts do not support dependencies, but with systemd we > > should add them where necessary. The most typical case is that A > > requires the service B to be running before A is started. In that case > > add Requires=B and After=B to A. If the dependency is optional then > > add Wants=B and After=B instead. Dependencies are typically placed on > > services and not on targets (see 3). > > > > > > > > > > Example of a simple conversion: > > > > rc script > > ===== > > > > #!/bin/bash > > > > . /etc/rc.conf > > . /etc/rc.d/functions > > > > case "$1" in > > start) > > stat_busy "Starting NIS Server" > > /usr/sbin/ypserv > > if [ $? -gt 0 ]; then > > stat_fail > > else > > add_daemon ypserv > > stat_done > > fi > > ;; > > stop) > > stat_busy "Stopping NIS Server" > > killall -q /usr/sbin/ypserv > > if [ $? -gt 0 ]; then > > stat_fail > > else > > rm_daemon ypserv > > stat_done > > fi > > ;; > > restart) > > $0 stop > > sleep 1 > > $0 start > > ;; > > *) > > echo "usage: $0 {start|stop|restart}" > > esac > > > > systemd service file > > ============ > > > > [Unit] > > Description=NIS/YP (Network Information Service) Server > > Requires=rpcbind.service > > After=network.target rpcbind.service > > > > [Service] > > Type=forking > > PIDFile=/run/ypserv.pid > > ExecStart=/usr/sbin/ypserv > > > > [Install] > > WantedBy=multi-user.target > > > > > > > > ____________________________________________________________________________________ > > The Best Resource for Articles and Information on Any Topic! > > http://click.lavabit.com/nmcrgte97tk69f1xpm1ms9eggq8osgk7kbx3ekwohn9wgaom6cwy/ > > ____________________________________________________________________________________ > > _______________________________________________ Dev mailing list Dev at lists.parabolagnulinux.org https://lists.parabolagnulinux.org/mailman/listinfo/dev That's why I'm using systemd xd -- Jorge Araya Navarro Universitario, idealista y pseudo-activista del Software Libre. Siquirres, Lim?n, Costa Rica. http://swt.encyclomundi.net Diaspora*: http://diasp.org/u/shackra identi.ca: http://parlementum.net/sweet Jabber: shackra at jabberes.org Skype: ?De ninguna manera, tras de privativo, te esp?an!. el software privativo en GNU/Linux, al igual que en Windows o en MacOs, te hace un ser no-libre. Deja de enga?arte, ??despierta ahora!!: http://www.gnu.org/distros/free-distros.html http://replicant.us/about/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From LukeShu at sbcglobal.net Sat Aug 25 04:02:07 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 00:02:07 -0400 Subject: [Dev] [PATCH 0/3] makepkg: Alternate implementation of VCS URLs in sources array. Message-ID: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> A while ago I started working on a derivative of makepkg to support having 'git://...' type urls in the sources=() array. When preparing to file this patch, I did a `git rebase`, and noticed that Allan McRae began working on a similar feature. Our implementations are in many ways similar. Hopefully mine will be useful. My implementation makes minimal changes to makepkg itself (only adding blob expansion to DLAGENTS, allowing for things like "git+*::""). Instead I added a `vcsget` tool which generates a tarball from the VCS repo, in a very similar manner to the way Allan's implementation does so within makepkg. It looks as if Allan's download_*() functions are more verbose than mine about what failed when there is an error. His svn and hg handlers are likely more robust--though my git is pretty solid. I also have a half-written handler for for bzr. An advantage of my design is that it does allow for integrity checks of VCS packages, rather than inserting 'SKIP' into the md5sums array. This is very important to the derivative distribution Parabola. (However, the 'SKIP' option is still valuable for URLs that track a branch) Happy hacking, ~ Luke Shumaker Luke Shumaker (3): Add a `vcsget` tool to download source from VCS repositories. makepkg: do blob expansion in DLAGENTS maps makepkg.conf: add vcsget DLAGENTS etc/makepkg.conf.in | 8 +- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/makepkg.sh.in | 13 ++- scripts/vcsget.sh.in | 295 ++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 317 insertions(+), 4 deletions(-) create mode 100644 scripts/vcsget.sh.in -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 04:02:08 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 00:02:08 -0400 Subject: [Dev] [PATCH 1/3] Add a `vcsget` tool to download source from VCS repositories. In-Reply-To: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> References: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345867330-27557-2-git-send-email-LukeShu@sbcglobal.net> It would support * cvs * git * svn * bzr * hg But cvs support is yet to be written, and bzr support is half-way written. Git is the only mode that I'm thoroughly confident in, the others need testing. --- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/vcsget.sh.in | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 299 insertions(+), 1 deletion(-) create mode 100644 scripts/vcsget.sh.in diff --git a/scripts/.gitignore b/scripts/.gitignore index 9e403bf..e59551e 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -6,3 +6,4 @@ pkgdelta repo-add repo-elephant repo-remove +vcsget diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 29c81aa..8e6d037 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -14,7 +14,8 @@ OURSCRIPTS = \ pacman-key \ pacman-optimize \ pkgdelta \ - repo-add + repo-add \ + vcsget EXTRA_DIST = \ makepkg.sh.in \ @@ -23,6 +24,7 @@ EXTRA_DIST = \ pacman-optimize.sh.in \ pkgdelta.sh.in \ repo-add.sh.in \ + vcsget.sh.in \ $(LIBRARY) LIBRARY = \ diff --git a/scripts/vcsget.sh.in b/scripts/vcsget.sh.in new file mode 100644 index 0000000..d33c9ec --- /dev/null +++ b/scripts/vcsget.sh.in @@ -0,0 +1,295 @@ +#!/bin/bash +# +# vcsget - downloader agent that generates tarballs from VCS repos +# @configure_input@ +# +# Copyright (c) 2012 Luke Shumaker +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# gettext initialization +export TEXTDOMAIN='pacman-scripts' +export TEXTDOMAINDIR='@localedir@' + +unset CDPATH + +m4_include(library/output_format.sh) + +## +# usage : in_array( $needle, $haystack ) +# return : 0 - found +# 1 - not found +## +in_array() { + local needle=$1; shift + local item + for item in "$@"; do + [[ $item = "$needle" ]] && return 0 # Found + done + return 1 # Not Found +} + +usage() { + echo "Usage: $0 " + echo "" + echo "Talks with multiple version control systems (VCSs) to create a" + echo "tarball of a specific commit." + echo "" + echo " is a plain, uncompressed tarball. Given the same" + echo "commit, the tarball will always have the same checksum." + echo "" + echo " is the repository, and possibly commit to download and tar." + echo "The following schemes are recognized:" + echo " * cvs://" + echo " * git://" + echo " * git+PROTO://" + echo " * svn+PROTO://" + echo " * bzr://" + echo " * hg+PROTO://" + echo "" + echo "URLs to be consumed by $0 are not always in the format of the" + echo "relevent VCS program, but normalized to proper URLs. Simply, this" + echo "means:" + echo "" + echo " scheme://[authinfo@]host[:port]/path[#fragment]" + echo "" + echo "Where is in the format \"username[:password]\" and" + echo " is the commit ID, branch, or tag to be returned." +} + +## +# Get the column "$2" from the space-delimited string "$1". +# This has the advantage over `set -- $1` and `arr=($1)` that it separates on +# a single space, instead of '\s+', which allows it to have empty columns. +## +get_field() { + local str=$1 + local i=$2 + echo "$str"|cut -d' ' -f $i +} + +## +# Parse a URL into parts, according to RFC 3986, "URI Generic Syntax". +# Sets the variables `scheme`, `authority`, `path`, `query` and `fragment`. +## +parse_url() { + local url=$1 + + # This regex is copied from RFC 3986 + local regex='^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?' + local parts=$(echo "$url"|sed -r "s@$regex@\2 \4 \5 \7 \9@") + + scheme=$( get_field "$parts" 1) + authority=$(get_field "$parts" 2) + path=$( get_field "$parts" 3) + query=$( get_field "$parts" 4) + fragment=$( get_field "$parts" 5) +} + +## +# Parse an authority from a URL. +# Sets the variables `user`, `pass`, `sock`, `host` and `port`. +# `sock` is socket, which is defined as `hostport` in the RFC. +## +parse_authority() { + local authority=$1 + + local regex='^(([^:]*)(:([^@]*))?@)?(([^:]+)(:([0-9]*))?)' + # 12 3 4 56 7 8 + # 1 = userinfo@ + # 2 = user + # 3 = :pass + # 4 = pass + # 5 = hostport + # 6 = host + # 7 = :port + # 8 = port + local parts=$(echo "$authority"|sed -r "s%$regex%\2 \4 \5 \6 \8%") + user=$(get_field "$parts" 1) + pass=$(get_field "$parts" 2) + sock=$(get_field "$parts" 3) + host=$(get_field "$parts" 4) + port=$(get_field "$parts" 5) +} + +download_cvs() { + # TODO + error "$(gettext 'CVS not implemented')" + exit 1 +} + +download_git() { + msg "$(gettext 'Getting source from git')" + : ${fragment:=master} # default to 'master' if not set + send_url="${scheme#git+}://${authority}${path}" + + # Where to checkout the code to + dir="$basedir/git/$send_url" + if [[ "${dir##*.}" != git ]]; then + dir="${dir}/.git" + fi + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial clone')" + mkdir -p "$dir" + git clone --bare "$send_url" "$dir" + fi + + cd "$dir" + refs=($(git show-ref|sed 's at .*/@@'|sort -u)) + if in_array "$fragment" "${refs[@]}"; then + # The ref is symbolic, so it may have changed upstream + msg2 "$(gettext 'fetching updates')" + git fetch --all + fi + + # Generate tarball + archive=(git archive --format=tar --prefix="$base/" + "$fragment" -o "$tarball") + msg2 "$(gettext 'trying to generate tarball')" + if "${archive[@]}" 2>/dev/null; then + # success + msg2 "$(gettext 'success')" + exit 0 + else + # the relevent commit is probably newer than we have + msg2 "$(gettext 'failure, forcing an update')" + git fetch --all + msg2 "$(gettext 'generating tarball')" + "${archive[@]}" + local s=$? + exit $? + fi +} + +download_svn() { + msg "$(gettext 'Getting source from Subversion')" + parse_authority "$authority" + send_url="${scheme#svn+}://${sock}${path}" + args=('--config-dir' "$basedir/svn/.config") + if [ -n "$user" ]; then + args+=('--username' "$user") + fi + if [ -n "$pass" ]; then + args+=('--password' "$pass") + fi + if [ -n "$fragment" ]; then + args+=('-r' "$fragment") + fi + + # Where to checkout the code to + dir="$basedir/svn/$send_url#$fragment" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + svn co "${args[@]}" "$send_url" "$dir" + else + cd "$dir" + svn up "${args[@]}" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + find "$dir.tmp$$/$base" -name .svn -exec rm -rf '{}' + + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +download_bzr() { + # TODO finish bzr support + error "$(gettext 'Bazaar not implemented')" + exit 1 + + send_url="${url%%#*}" + if [ "$scheme" != 'bzr+ssh' ]; then + send_url="${send_url#bzr+}" + fi + + dir="$basedir/bzr/$send_url" + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial checkout')" + mkdir -p "$dir" + bzr checkout "$send_url" "$dir" + fi + + cd "$dir" + bzr update + + # Generate tarball +} + +download_hg() { + msg "$(gettext 'Getting source from Mercurial')" + send_url="${url#hg+}" + hg clone -U "$send_url" + dir="$basedir/hg/${send_url%%#*}" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + hg clone "$send_url" "$dir" + else + cd "$dir" + hg update -r "$fragment" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + rm -rf "$dir.tmp$$/$base/.hg" + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +main() { + url=$1 + tarball=$(readlink -m "$2") + + base="${tarball##*/}" + base="${base%%.*}" + basedir="${tarball%/*}/vcs-cache" + + if ( in_array '-h' "$@" || in_array '--help' "$@" ); then + usage + exit 0 + fi + if [ "$#" -ne 2 ]; then + usage + exit 1 + fi + + msg "$(printf "$(gettext "Saving '%s' to '%s'")" "$url" "$tarball")" + + parse_url "$url" + case "$scheme" in + cvs) download_cvs;; + git+*|git) download_git;; + svn+*) download_svn;; + bzr) download_bzr;; + hg+*) download_hg;; + *) + error "$(gettext 'Unable to match a VCS with scheme'): $scheme" + exit 1;; + esac +} + +main "$@" -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 04:02:09 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 00:02:09 -0400 Subject: [Dev] [PATCH 2/3] makepkg: do blob expansion in DLAGENTS maps In-Reply-To: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> References: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345867330-27557-3-git-send-email-LukeShu@sbcglobal.net> This allows vcs-* schemes to be used for vcsget. --- scripts/makepkg.sh.in | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b30e9d0..971a8b2 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -394,16 +394,25 @@ source_has_signatures() { return 1 } +string_matches_blob() { + local string=$1 + local blob=$2 + case "$string" in + $blob) return 0;; + esac + return 1 +} + get_downloadclient() { # $1 = URL with valid protocol prefix local url=$1 local proto="${url%%://*}" - # loop through DOWNLOAD_AGENTS variable looking for protocol + # loop through DLAGENTS (download agents) variable looking for protocol local i for i in "${DLAGENTS[@]}"; do local handler="${i%%::*}" - if [[ $proto = "$handler" ]]; then + if string_matches_blob "$string" "$handler"; then local agent="${i##*::}" break fi -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 04:02:10 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 00:02:10 -0400 Subject: [Dev] [PATCH 3/3] makepkg.conf: add vcsget DLAGENTS In-Reply-To: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> References: <1345867330-27557-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345867330-27557-4-git-send-email-LukeShu@sbcglobal.net> --- etc/makepkg.conf.in | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dcec6f5..7d19dfb 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -12,7 +12,13 @@ DLAGENTS=('ftp::/usr/bin/curl -qfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o 'http::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'https::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') + 'scp::/usr/bin/scp -C %u %o' + 'cvs::/usr/bin/vcsget %u %o' + 'git::/usr/bin/vcsget %u %o' + 'git+*::/usr/bin/vcsget %u %o' + 'svn+*::/usr/bin/vcsget %u %o' + 'bzr::/usr/bin/vcsget %u %o' + 'hg+*::/usr/bin/vcsget %u %o') # Other common tools: # /usr/bin/snarf -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:33:12 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:33:12 -0400 Subject: [Dev] [PATCH 0/3] makepkg: Alternate implementation of VCS URLs in sources array. Message-ID: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> A while ago I started working on a derivative of makepkg to support having 'git://...' type urls in the sources=() array. When preparing to file this patch, I did a `git rebase`, and noticed that Allan McRae began working on a similar feature. Our implementations are in many ways similar. Hopefully mine will be useful. My implementation makes minimal changes to makepkg itself (only adding blob expansion to DLAGENTS, allowing for things like "git+*::""). Instead I added a `vcsget` tool which generates a tarball from the VCS repo, in a very similar manner to the way Allan's implementation does so within makepkg. It looks as if Allan's download_*() functions are more verbose than mine about what failed when there is an error. His svn and hg handlers are likely more robust--though my git is pretty solid. I also have a half-written handler for for bzr. An advantage of my design is that it does allow for integrity checks of VCS packages, rather than inserting 'SKIP' into the md5sums array. This is very important to the derivative distribution Parabola. (However, the 'SKIP' option is still valuable for URLs that track a branch) Happy hacking, ~ Luke Shumaker Luke Shumaker (3): Add a `vcsget` tool to download source from VCS repositories. makepkg: do glob expansion in DLAGENTS maps makepkg.conf: add vcsget DLAGENTS etc/makepkg.conf.in | 8 +- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/makepkg.sh.in | 13 ++- scripts/vcsget.sh.in | 294 ++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 316 insertions(+), 4 deletions(-) create mode 100644 scripts/vcsget.sh.in -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:33:13 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:33:13 -0400 Subject: [Dev] [PATCH 1/3] Add a `vcsget` tool to download source from VCS repositories. In-Reply-To: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> References: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345915995-31160-2-git-send-email-LukeShu@sbcglobal.net> It would support * cvs * git * svn * bzr * hg But cvs support is yet to be written, and bzr support is half-way written. Git is the only mode that I'm thoroughly confident in, the others need testing. Signed-off-by: Luke Shumaker --- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/vcsget.sh.in | 294 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 scripts/vcsget.sh.in diff --git a/scripts/.gitignore b/scripts/.gitignore index 9e403bf..e59551e 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -6,3 +6,4 @@ pkgdelta repo-add repo-elephant repo-remove +vcsget diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 29c81aa..8e6d037 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -14,7 +14,8 @@ OURSCRIPTS = \ pacman-key \ pacman-optimize \ pkgdelta \ - repo-add + repo-add \ + vcsget EXTRA_DIST = \ makepkg.sh.in \ @@ -23,6 +24,7 @@ EXTRA_DIST = \ pacman-optimize.sh.in \ pkgdelta.sh.in \ repo-add.sh.in \ + vcsget.sh.in \ $(LIBRARY) LIBRARY = \ diff --git a/scripts/vcsget.sh.in b/scripts/vcsget.sh.in new file mode 100644 index 0000000..f8f6e4b --- /dev/null +++ b/scripts/vcsget.sh.in @@ -0,0 +1,294 @@ +#!/bin/bash +# +# vcsget - downloader agent that generates tarballs from VCS repos +# @configure_input@ +# +# Copyright (c) 2012 Luke Shumaker +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# gettext initialization +export TEXTDOMAIN='pacman-scripts' +export TEXTDOMAINDIR='@localedir@' + +unset CDPATH + +m4_include(library/output_format.sh) + +## +# usage : in_array( $needle, $haystack ) +# return : 0 - found +# 1 - not found +## +in_array() { + local needle=$1; shift + local item + for item in "$@"; do + [[ $item = "$needle" ]] && return 0 # Found + done + return 1 # Not Found +} + +usage() { + echo "Usage: $0 " + echo "" + echo "Talks with multiple version control systems (VCSs) to create a" + echo "tarball of a specific commit." + echo "" + echo " is a plain, uncompressed tarball. Given the same" + echo "commit, the tarball will always have the same checksum." + echo "" + echo " is the repository, and possibly commit to download and tar." + echo "The following schemes are recognized:" + echo " * cvs://" + echo " * git://" + echo " * git+PROTO://" + echo " * svn+PROTO://" + echo " * bzr://" + echo " * hg+PROTO://" + echo "" + echo "URLs to be consumed by $0 are not always in the format of the" + echo "relevent VCS program, but normalized to proper URLs. Simply, this" + echo "means:" + echo "" + echo " scheme://[authinfo@]host[:port]/path[#fragment]" + echo "" + echo "Where is in the format \"username[:password]\" and" + echo " is the commit ID, branch, or tag to be returned." +} + +## +# Get the column "$2" from the space-delimited string "$1". +# This has the advantage over `set -- $1` and `arr=($1)` that it separates on +# a single space, instead of '\s+', which allows it to have empty columns. +## +get_field() { + local str=$1 + local i=$2 + echo "$str"|cut -d' ' -f $i +} + +## +# Parse a URL into parts, according to RFC 3986, "URI Generic Syntax". +# Sets the variables `scheme`, `authority`, `path`, `query` and `fragment`. +## +parse_url() { + local url=$1 + + # This regex is copied from RFC 3986 + local regex='^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?' + local parts=$(echo "$url"|sed -r "s@$regex@\2 \4 \5 \7 \9@") + + scheme=$( get_field "$parts" 1) + authority=$(get_field "$parts" 2) + path=$( get_field "$parts" 3) + query=$( get_field "$parts" 4) + fragment=$( get_field "$parts" 5) +} + +## +# Parse an authority from a URL. +# Sets the variables `user`, `pass`, `sock`, `host` and `port`. +# `sock` is socket, which is defined as `hostport` in the RFC. +## +parse_authority() { + local authority=$1 + + local regex='^(([^:]*)(:([^@]*))?@)?(([^:]+)(:([0-9]*))?)' + # 12 3 4 56 7 8 + # 1 = userinfo@ + # 2 = user + # 3 = :pass + # 4 = pass + # 5 = hostport + # 6 = host + # 7 = :port + # 8 = port + local parts=$(echo "$authority"|sed -r "s%$regex%\2 \4 \5 \6 \8%") + user=$(get_field "$parts" 1) + pass=$(get_field "$parts" 2) + sock=$(get_field "$parts" 3) + host=$(get_field "$parts" 4) + port=$(get_field "$parts" 5) +} + +download_cvs() { + # TODO + error "$(gettext 'CVS not implemented')" + exit 1 +} + +download_git() { + msg "$(gettext 'Getting source from git')" + : ${fragment:=master} # default to 'master' if not set + send_url="${scheme#git+}://${authority}${path}" + + # Where to checkout the code to + dir="$basedir/git/$send_url" + if [[ "${dir##*.}" != git ]]; then + dir="${dir}/.git" + fi + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial clone')" + mkdir -p "$dir" + git clone --bare "$send_url" "$dir" + fi + + cd "$dir" + refs=($(git show-ref|sed 's at .*/@@'|sort -u)) + if in_array "$fragment" "${refs[@]}"; then + # The ref is symbolic, so it may have changed upstream + msg2 "$(gettext 'fetching updates')" + git fetch --all + fi + + # Generate tarball + archive=(git archive --format=tar --prefix="$base/" + "$fragment" -o "$tarball") + msg2 "$(gettext 'trying to generate tarball')" + if "${archive[@]}" 2>/dev/null; then + # success + msg2 "$(gettext 'success')" + exit 0 + else + # the relevent commit is probably newer than we have + msg2 "$(gettext 'failure, forcing an update')" + git fetch --all + msg2 "$(gettext 'generating tarball')" + "${archive[@]}" + local s=$? + exit $? + fi +} + +download_svn() { + msg "$(gettext 'Getting source from Subversion')" + parse_authority "$authority" + send_url="${scheme#svn+}://${sock}${path}" + args=('--config-dir' "$basedir/svn/.config") + if [ -n "$user" ]; then + args+=('--username' "$user") + fi + if [ -n "$pass" ]; then + args+=('--password' "$pass") + fi + if [ -n "$fragment" ]; then + args+=('-r' "$fragment") + fi + + # Where to checkout the code to + dir="$basedir/svn/$send_url#$fragment" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + svn co "${args[@]}" "$send_url" "$dir" + else + cd "$dir" + svn up "${args[@]}" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + find "$dir.tmp$$/$base" -name .svn -exec rm -rf '{}' + + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +download_bzr() { + # TODO finish bzr support + error "$(gettext 'Bazaar not implemented')" + exit 1 + + send_url="${url%%#*}" + if [ "$scheme" != 'bzr+ssh' ]; then + send_url="${send_url#bzr+}" + fi + + dir="$basedir/bzr/$send_url" + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial checkout')" + mkdir -p "$dir" + bzr checkout "$send_url" "$dir" + fi + + cd "$dir" + bzr update + + # Generate tarball +} + +download_hg() { + msg "$(gettext 'Getting source from Mercurial')" + send_url="${url#hg+}" + hg clone -U "$send_url" + dir="$basedir/hg/${send_url%%#*}" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + hg clone "$send_url" "$dir" + else + cd "$dir" + hg update -r "$fragment" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + rm -rf "$dir.tmp$$/$base/.hg" + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +main() { + url=$1 + tarball=$(readlink -m "$2") + + base="$(echo "$tarball"|sed -r 's at .*/(.*)\.tar(\..*)?$@\1@')" + basedir="${tarball%/*}/vcs-cache" + + if ( in_array '-h' "$@" || in_array '--help' "$@" ); then + usage + exit 0 + fi + if [ "$#" -ne 2 ]; then + usage + exit 1 + fi + + msg "$(printf "$(gettext "Saving '%s' to '%s'")" "$url" "$tarball")" + + parse_url "$url" + case "$scheme" in + cvs) download_cvs;; + git+*|git) download_git;; + svn+*) download_svn;; + bzr) download_bzr;; + hg+*) download_hg;; + *) + error "$(gettext 'Unable to match a VCS with scheme'): $scheme" + exit 1;; + esac +} + +main "$@" >&2 -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:33:14 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:33:14 -0400 Subject: [Dev] [PATCH 2/3] makepkg: do glob expansion in DLAGENTS maps In-Reply-To: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> References: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345915995-31160-3-git-send-email-LukeShu@sbcglobal.net> This allows vcs-* schemes to be used for vcsget. Signed-off-by: Luke Shumaker --- scripts/makepkg.sh.in | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b30e9d0..f2a5008 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -394,16 +394,25 @@ source_has_signatures() { return 1 } +string_matches_glob() { + local string=$1 + local glob=$2 + case "$string" in + $glob) return 0;; + esac + return 1 +} + get_downloadclient() { # $1 = URL with valid protocol prefix local url=$1 local proto="${url%%://*}" - # loop through DOWNLOAD_AGENTS variable looking for protocol + # loop through DLAGENTS (download agents) variable looking for protocol local i for i in "${DLAGENTS[@]}"; do local handler="${i%%::*}" - if [[ $proto = "$handler" ]]; then + if string_matches_glob "$proto" "$handler"; then local agent="${i##*::}" break fi -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:33:15 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:33:15 -0400 Subject: [Dev] [PATCH 3/3] makepkg.conf: add vcsget DLAGENTS In-Reply-To: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> References: <1345915995-31160-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345915995-31160-4-git-send-email-LukeShu@sbcglobal.net> Signed-off-by: Luke Shumaker --- etc/makepkg.conf.in | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dcec6f5..7d19dfb 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -12,7 +12,13 @@ DLAGENTS=('ftp::/usr/bin/curl -qfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o 'http::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'https::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') + 'scp::/usr/bin/scp -C %u %o' + 'cvs::/usr/bin/vcsget %u %o' + 'git::/usr/bin/vcsget %u %o' + 'git+*::/usr/bin/vcsget %u %o' + 'svn+*::/usr/bin/vcsget %u %o' + 'bzr::/usr/bin/vcsget %u %o' + 'hg+*::/usr/bin/vcsget %u %o') # Other common tools: # /usr/bin/snarf -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:36:41 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:36:41 -0400 Subject: [Dev] [PATCH 0/3] makepkg: Alternate implementation of VCS URLs in sources array. Message-ID: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> A while ago I started working on a derivative of makepkg to support having 'git://...' type urls in the sources=() array. When preparing to file this patch, I did a `git rebase`, and noticed that Allan McRae began working on a similar feature. Our implementations are in many ways similar. Hopefully mine will be useful. My implementation makes minimal changes to makepkg itself (only adding blob expansion to DLAGENTS, allowing for things like "git+*::""). Instead I added a `vcsget` tool which generates a tarball from the VCS repo, in a very similar manner to the way Allan's implementation does so within makepkg. It looks as if Allan's download_*() functions are more verbose than mine about what failed when there is an error. His svn and hg handlers are likely more robust--though my git is pretty solid. I also have a half-written handler for for bzr. An advantage of my design is that it does allow for integrity checks of VCS packages, rather than inserting 'SKIP' into the md5sums array. This is very important to the derivative distribution Parabola. (However, the 'SKIP' option is still valuable for URLs that track a branch) Happy hacking, ~ Luke Shumaker Luke Shumaker (3): Add a `vcsget` tool to download source from VCS repositories. makepkg: do glob expansion in DLAGENTS maps makepkg.conf: add vcsget DLAGENTS etc/makepkg.conf.in | 8 +- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/makepkg.sh.in | 13 ++- scripts/vcsget.sh.in | 294 ++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 316 insertions(+), 4 deletions(-) create mode 100644 scripts/vcsget.sh.in -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:36:42 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:36:42 -0400 Subject: [Dev] [PATCH 1/3] Add a `vcsget` tool to download source from VCS repositories. In-Reply-To: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> References: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345916204-3459-2-git-send-email-LukeShu@sbcglobal.net> It would support * cvs * git * svn * bzr * hg But cvs support is yet to be written, and bzr support is half-way written. Git is the only mode that I'm thoroughly confident in, the others need testing. Signed-off-by: Luke Shumaker --- scripts/.gitignore | 1 + scripts/Makefile.am | 4 +- scripts/vcsget.sh.in | 294 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 scripts/vcsget.sh.in diff --git a/scripts/.gitignore b/scripts/.gitignore index 9e403bf..e59551e 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -6,3 +6,4 @@ pkgdelta repo-add repo-elephant repo-remove +vcsget diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 29c81aa..8e6d037 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -14,7 +14,8 @@ OURSCRIPTS = \ pacman-key \ pacman-optimize \ pkgdelta \ - repo-add + repo-add \ + vcsget EXTRA_DIST = \ makepkg.sh.in \ @@ -23,6 +24,7 @@ EXTRA_DIST = \ pacman-optimize.sh.in \ pkgdelta.sh.in \ repo-add.sh.in \ + vcsget.sh.in \ $(LIBRARY) LIBRARY = \ diff --git a/scripts/vcsget.sh.in b/scripts/vcsget.sh.in new file mode 100644 index 0000000..f8f6e4b --- /dev/null +++ b/scripts/vcsget.sh.in @@ -0,0 +1,294 @@ +#!/bin/bash +# +# vcsget - downloader agent that generates tarballs from VCS repos +# @configure_input@ +# +# Copyright (c) 2012 Luke Shumaker +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# gettext initialization +export TEXTDOMAIN='pacman-scripts' +export TEXTDOMAINDIR='@localedir@' + +unset CDPATH + +m4_include(library/output_format.sh) + +## +# usage : in_array( $needle, $haystack ) +# return : 0 - found +# 1 - not found +## +in_array() { + local needle=$1; shift + local item + for item in "$@"; do + [[ $item = "$needle" ]] && return 0 # Found + done + return 1 # Not Found +} + +usage() { + echo "Usage: $0 " + echo "" + echo "Talks with multiple version control systems (VCSs) to create a" + echo "tarball of a specific commit." + echo "" + echo " is a plain, uncompressed tarball. Given the same" + echo "commit, the tarball will always have the same checksum." + echo "" + echo " is the repository, and possibly commit to download and tar." + echo "The following schemes are recognized:" + echo " * cvs://" + echo " * git://" + echo " * git+PROTO://" + echo " * svn+PROTO://" + echo " * bzr://" + echo " * hg+PROTO://" + echo "" + echo "URLs to be consumed by $0 are not always in the format of the" + echo "relevent VCS program, but normalized to proper URLs. Simply, this" + echo "means:" + echo "" + echo " scheme://[authinfo@]host[:port]/path[#fragment]" + echo "" + echo "Where is in the format \"username[:password]\" and" + echo " is the commit ID, branch, or tag to be returned." +} + +## +# Get the column "$2" from the space-delimited string "$1". +# This has the advantage over `set -- $1` and `arr=($1)` that it separates on +# a single space, instead of '\s+', which allows it to have empty columns. +## +get_field() { + local str=$1 + local i=$2 + echo "$str"|cut -d' ' -f $i +} + +## +# Parse a URL into parts, according to RFC 3986, "URI Generic Syntax". +# Sets the variables `scheme`, `authority`, `path`, `query` and `fragment`. +## +parse_url() { + local url=$1 + + # This regex is copied from RFC 3986 + local regex='^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?' + local parts=$(echo "$url"|sed -r "s@$regex@\2 \4 \5 \7 \9@") + + scheme=$( get_field "$parts" 1) + authority=$(get_field "$parts" 2) + path=$( get_field "$parts" 3) + query=$( get_field "$parts" 4) + fragment=$( get_field "$parts" 5) +} + +## +# Parse an authority from a URL. +# Sets the variables `user`, `pass`, `sock`, `host` and `port`. +# `sock` is socket, which is defined as `hostport` in the RFC. +## +parse_authority() { + local authority=$1 + + local regex='^(([^:]*)(:([^@]*))?@)?(([^:]+)(:([0-9]*))?)' + # 12 3 4 56 7 8 + # 1 = userinfo@ + # 2 = user + # 3 = :pass + # 4 = pass + # 5 = hostport + # 6 = host + # 7 = :port + # 8 = port + local parts=$(echo "$authority"|sed -r "s%$regex%\2 \4 \5 \6 \8%") + user=$(get_field "$parts" 1) + pass=$(get_field "$parts" 2) + sock=$(get_field "$parts" 3) + host=$(get_field "$parts" 4) + port=$(get_field "$parts" 5) +} + +download_cvs() { + # TODO + error "$(gettext 'CVS not implemented')" + exit 1 +} + +download_git() { + msg "$(gettext 'Getting source from git')" + : ${fragment:=master} # default to 'master' if not set + send_url="${scheme#git+}://${authority}${path}" + + # Where to checkout the code to + dir="$basedir/git/$send_url" + if [[ "${dir##*.}" != git ]]; then + dir="${dir}/.git" + fi + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial clone')" + mkdir -p "$dir" + git clone --bare "$send_url" "$dir" + fi + + cd "$dir" + refs=($(git show-ref|sed 's at .*/@@'|sort -u)) + if in_array "$fragment" "${refs[@]}"; then + # The ref is symbolic, so it may have changed upstream + msg2 "$(gettext 'fetching updates')" + git fetch --all + fi + + # Generate tarball + archive=(git archive --format=tar --prefix="$base/" + "$fragment" -o "$tarball") + msg2 "$(gettext 'trying to generate tarball')" + if "${archive[@]}" 2>/dev/null; then + # success + msg2 "$(gettext 'success')" + exit 0 + else + # the relevent commit is probably newer than we have + msg2 "$(gettext 'failure, forcing an update')" + git fetch --all + msg2 "$(gettext 'generating tarball')" + "${archive[@]}" + local s=$? + exit $? + fi +} + +download_svn() { + msg "$(gettext 'Getting source from Subversion')" + parse_authority "$authority" + send_url="${scheme#svn+}://${sock}${path}" + args=('--config-dir' "$basedir/svn/.config") + if [ -n "$user" ]; then + args+=('--username' "$user") + fi + if [ -n "$pass" ]; then + args+=('--password' "$pass") + fi + if [ -n "$fragment" ]; then + args+=('-r' "$fragment") + fi + + # Where to checkout the code to + dir="$basedir/svn/$send_url#$fragment" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + svn co "${args[@]}" "$send_url" "$dir" + else + cd "$dir" + svn up "${args[@]}" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + find "$dir.tmp$$/$base" -name .svn -exec rm -rf '{}' + + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +download_bzr() { + # TODO finish bzr support + error "$(gettext 'Bazaar not implemented')" + exit 1 + + send_url="${url%%#*}" + if [ "$scheme" != 'bzr+ssh' ]; then + send_url="${send_url#bzr+}" + fi + + dir="$basedir/bzr/$send_url" + + # Download + if [ ! -d "$dir" ]; then + msg2 "$(gettext 'doing initial checkout')" + mkdir -p "$dir" + bzr checkout "$send_url" "$dir" + fi + + cd "$dir" + bzr update + + # Generate tarball +} + +download_hg() { + msg "$(gettext 'Getting source from Mercurial')" + send_url="${url#hg+}" + hg clone -U "$send_url" + dir="$basedir/hg/${send_url%%#*}" + + # Download + if [ ! -d "$dir" ]; then + mkdir -p "$dir" + hg clone "$send_url" "$dir" + else + cd "$dir" + hg update -r "$fragment" + fi + + # Generate tarball + mkdir "$dir.tmp$$" + cp -r "$dir" "$dir.tmp$$/$base" + rm -rf "$dir.tmp$$/$base/.hg" + cd "$dir.tmp$$" + bsdtar cf "$tarball" "$base" + rm -rf "$dir.tmp$$" +} + +main() { + url=$1 + tarball=$(readlink -m "$2") + + base="$(echo "$tarball"|sed -r 's at .*/(.*)\.tar(\..*)?$@\1@')" + basedir="${tarball%/*}/vcs-cache" + + if ( in_array '-h' "$@" || in_array '--help' "$@" ); then + usage + exit 0 + fi + if [ "$#" -ne 2 ]; then + usage + exit 1 + fi + + msg "$(printf "$(gettext "Saving '%s' to '%s'")" "$url" "$tarball")" + + parse_url "$url" + case "$scheme" in + cvs) download_cvs;; + git+*|git) download_git;; + svn+*) download_svn;; + bzr) download_bzr;; + hg+*) download_hg;; + *) + error "$(gettext 'Unable to match a VCS with scheme'): $scheme" + exit 1;; + esac +} + +main "$@" >&2 -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:36:43 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:36:43 -0400 Subject: [Dev] [PATCH 2/3] makepkg: do glob expansion in DLAGENTS maps In-Reply-To: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> References: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1345916204-3459-3-git-send-email-LukeShu@sbcglobal.net> This allows vcs-* schemes to be used for vcsget. Signed-off-by: Luke Shumaker --- scripts/makepkg.sh.in | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b30e9d0..f2a5008 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -394,16 +394,25 @@ source_has_signatures() { return 1 } +string_matches_glob() { + local string=$1 + local glob=$2 + case "$string" in + $glob) return 0;; + esac + return 1 +} + get_downloadclient() { # $1 = URL with valid protocol prefix local url=$1 local proto="${url%%://*}" - # loop through DOWNLOAD_AGENTS variable looking for protocol + # loop through DLAGENTS (download agents) variable looking for protocol local i for i in "${DLAGENTS[@]}"; do local handler="${i%%::*}" - if [[ $proto = "$handler" ]]; then + if string_matches_glob "$proto" "$handler"; then local agent="${i##*::}" break fi -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:37:35 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:37:35 -0400 Subject: [Dev] [PATCH 2/3] makepkg: do glob expansion in DLAGENTS maps Message-ID: <1345916256-4628-3-git-send-email-LukeShu@sbcglobal.net> This allows vcs-* schemes to be used for vcsget. Signed-off-by: Luke Shumaker --- scripts/makepkg.sh.in | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b30e9d0..f2a5008 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -394,16 +394,25 @@ source_has_signatures() { return 1 } +string_matches_glob() { + local string=$1 + local glob=$2 + case "$string" in + $glob) return 0;; + esac + return 1 +} + get_downloadclient() { # $1 = URL with valid protocol prefix local url=$1 local proto="${url%%://*}" - # loop through DOWNLOAD_AGENTS variable looking for protocol + # loop through DLAGENTS (download agents) variable looking for protocol local i for i in "${DLAGENTS[@]}"; do local handler="${i%%::*}" - if [[ $proto = "$handler" ]]; then + if string_matches_glob "$proto" "$handler"; then local agent="${i##*::}" break fi -- 1.7.12 From LukeShu at sbcglobal.net Sat Aug 25 17:38:16 2012 From: LukeShu at sbcglobal.net (Luke Shumaker) Date: Sat, 25 Aug 2012 13:38:16 -0400 Subject: [Dev] [PATCH 3/3] makepkg.conf: add vcsget DLAGENTS Message-ID: <1345916296-5454-4-git-send-email-LukeShu@sbcglobal.net> Signed-off-by: Luke Shumaker --- etc/makepkg.conf.in | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dcec6f5..7d19dfb 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -12,7 +12,13 @@ DLAGENTS=('ftp::/usr/bin/curl -qfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o 'http::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'https::/usr/bin/curl -qb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') + 'scp::/usr/bin/scp -C %u %o' + 'cvs::/usr/bin/vcsget %u %o' + 'git::/usr/bin/vcsget %u %o' + 'git+*::/usr/bin/vcsget %u %o' + 'svn+*::/usr/bin/vcsget %u %o' + 'bzr::/usr/bin/vcsget %u %o' + 'hg+*::/usr/bin/vcsget %u %o') # Other common tools: # /usr/bin/snarf -- 1.7.12 From lukeshu at sbcglobal.net Sat Aug 25 17:55:47 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Sat, 25 Aug 2012 13:55:47 -0400 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs Message-ID: <87harq26vw.wl%lukeshu@sbcglobal.net> A while back I started working on a feature to add VCS url capabilities to makepkg, based on a whitepaper I published on this list. https://gitorious.org/parabola/pacman/commits/lukeshu-fix Shortly after I began, the upstream pacman devs started working on a similar feature. Their implementation contains good code--ideally I'd replace my download_$VCS() methods with theirs (we even chose the same function names, how 'bout that!). However, the general design of our implementations differ. I made minimal modifications to makepkg itself, and added an external `vcsget` tool that spits out tarballs. I believe that mine is better design. This is not the important part; their design does not allow for checking source integrity--an important requirement for Parabola. So, it'd be awesome if you guys would test my version. It's available as 'pacman-lukeshu' from the [~lukeshu] repo. ~ Luke Shumaker From lukeshu at sbcglobal.net Sat Aug 25 18:02:18 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Sat, 25 Aug 2012 14:02:18 -0400 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs In-Reply-To: <87harq26vw.wl%lukeshu@sbcglobal.net> References: <87harq26vw.wl%lukeshu@sbcglobal.net> Message-ID: <87d32e26l1.wl%lukeshu@sbcglobal.net> I forgot to mention, an example of how to use it is [~lukeshu]/etckeeper Also, sorry for spamming the list with PATCHes, I cc'd the parabola list, and the first 2 times it didn't go through. Happy Hacking! ~ Luke Shumaker From fauno at kiwwwi.com.ar Sat Aug 25 18:09:12 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Sat, 25 Aug 2012 15:09:12 -0300 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs In-Reply-To: <87harq26vw.wl%lukeshu@sbcglobal.net> References: <87harq26vw.wl%lukeshu@sbcglobal.net> Message-ID: <87lih2u9mf.fsf@kiwwwi.com.ar> "Luke T.Shumaker" writes: > A while back I started working on a feature to add VCS url > capabilities to makepkg, based on a whitepaper I published on this > list. > > https://gitorious.org/parabola/pacman/commits/lukeshu-fix \o/ did they reply? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From lukeshu at sbcglobal.net Sat Aug 25 21:57:29 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Sat, 25 Aug 2012 17:57:29 -0400 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs In-Reply-To: <87lih2u9mf.fsf@kiwwwi.com.ar> References: <87harq26vw.wl%lukeshu@sbcglobal.net> <87lih2u9mf.fsf@kiwwwi.com.ar> Message-ID: <87a9xi1vp2.wl%lukeshu@sbcglobal.net> At Sat, 25 Aug 2012 15:09:12 -0300, Nicol?s Reynolds wrote: > > [1 ] > "Luke T.Shumaker" writes: > > > A while back I started working on a feature to add VCS url > > capabilities to makepkg, based on a whitepaper I published on this > > list. > > > > https://gitorious.org/parabola/pacman/commits/lukeshu-fix > > \o/ did they reply? > [2 ] > Yes! Following is a draft of my reply back. Things in capslock are for you to comment on. I didn't remove anything from Dave's email. ~ Luke Shumaker At Sat, 25 Aug 2012 14:43:32 -0400, Dave Reisner wrote: > > On Sat, Aug 25, 2012 at 01:36:41PM -0400, Luke Shumaker wrote: > > A while ago I started working on a derivative of makepkg to support > > having 'git://...' type urls in the sources=() array. When preparing > > to file this patch, I did a `git rebase`, and noticed that Allan McRae > > began working on a similar feature. Our implementations are in many > > ways similar. Hopefully mine will be useful. > > An interesting approach. As you've noticed, I think we're fairly > committed to the implementation that Allan has provided. If you have > specific concerns, maybe we can work to fix those. * Allan's URL parsing has some issues, see point 2 below. This is an easy fix. * Composability. It's not really a "problem", but it seems to me that It's better to create a new general-purpose tool, instead of shoving a bunch of "special case" behavior into makepkg. Allan's implementation could even fairly easily be pulled out of makepkg. * SOURCE TARBALLS--DO THEY WORK? * FAUNO, AM I MISSING SOMETHING? > > > My implementation makes minimal changes to makepkg itself (only adding > > blob expansion to DLAGENTS, allowing for things like > > "git+*::""). Instead I added a `vcsget` tool which generates a tarball > > from the VCS repo, in a very similar manner to the way Allan's > > implementation does so within makepkg. > > I'm not thrilled with the shell I saw in that patch -- there's extensive > use of external commands when simple bash features would have sufficed. I assume you're speaking of my 1. use of cut in get_field 2. use of sed to parse URLs 3. use of sed to parse URL authorities 4. use of readlink to establish the tarball name 5. use of sed to create a basedir name from the tarball name 6. use of '[' with if statements My defense of each: 1. `cut` in get_field --------------------- This was simply the simplest solution that was obviously correct. Another solution would have been to manipulate IFS, which has all kinds of funny side effects and fringe behaviors. That was a weak argument, and as a coder, I would be thrilled to find out if there is a better solution. 2. `sed` to parse URLs ---------------------- I wanted full URL parsing, for a few cases when the translation is less than than straight-forward. For example, including authentication info in svn URLs. Further, I wanted to make sure that *any* URL parsing done would be robust. Given that the regex used was taken straight from the relevent RFC, I knew I could count on it to work, even in fringe cases. For example a "fragment" *should* be able to contain a '#', but Allan's implementation discards anything before the last '#' in the fragment. This is a problem for (at least) git, as tags and branches may contain the symbol. 3. `sed` to parse URL authorities --------------------------------- I believe that a robust (not having the same problems as fragments for URLs) implementation in bash would be non-trivial. 4. `readlink` to establish the tarball name ------------------------------------------- I used `readlink -m` to turn the tarball name into an absolute path. This was not an absolute must, but it allowed me to avoid worrying about keeping track of the current directory. It would be fairly easy to remove this by useing pushd/popd instead of cd. If you would like this changed, I can do that. 5. `sed` to create a basedir name from the tarball name ------------------------------------------------------- I'll admit, this was me getting a little lazy. A pure bash implementation is: base=${tarball##*/} base=${base%%.tar.*} base=${base%.tar} 6. `[` with if statements ------------------------- Can I call stylistic choice on this one? It is trivial to replace this with bash's '[['. > > > It looks as if Allan's download_*() functions are more verbose than > > mine about what failed when there is an error. His svn and hg handlers > > are likely more robust--though my git is pretty solid. I also have > > a half-written handler for for bzr. > > > > An advantage of my design is that it does allow for integrity checks > > of VCS packages, rather than inserting 'SKIP' into the md5sums > > array. This is very important to the derivative distribution Parabola. > > (However, the 'SKIP' option is still valuable for URLs that track a > > branch) > > I don't see this as an advantage so much as a duplication. The backing > VCS takes care of integrity checks. They're only necessary with tarballs > because there is no "builtin" checksum to reference. This is only partially true of the non-distributed VCSs. BUT I FEEL LIKE THATS A LAME REASON. FAUNO, ANY INPUT? Happy hacking, ~ Luke Shumaker From fauno at kiwwwi.com.ar Sat Aug 25 23:19:54 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Sat, 25 Aug 2012 20:19:54 -0300 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs In-Reply-To: <87a9xi1vp2.wl%lukeshu@sbcglobal.net> References: <87harq26vw.wl%lukeshu@sbcglobal.net> <87lih2u9mf.fsf@kiwwwi.com.ar> <87a9xi1vp2.wl%lukeshu@sbcglobal.net> Message-ID: <87fw7atv8l.fsf@kiwwwi.com.ar> "Luke T.Shumaker" writes: > At Sat, 25 Aug 2012 15:09:12 -0300, > Nicol?s Reynolds wrote: >> >> [1 ] >> "Luke T.Shumaker" writes: >> >> > A while back I started working on a feature to add VCS url >> > capabilities to makepkg, based on a whitepaper I published on this >> > list. >> > >> > https://gitorious.org/parabola/pacman/commits/lukeshu-fix >> >> \o/ did they reply? >> [2 ] >> > > Yes! Following is a draft of my reply back. Things in capslock are for > you to comment on. I didn't remove anything from Dave's email. > > ~ Luke Shumaker the main reason to have tarballs is not that we loooove checksums, but that at least GPL2 software requires you to host the corresponding sources for the packages you provide. arch is following this doing makepkg --allsource for GPL packages on ABS, how would they solve it? or won't they allow vcs packges on repos? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From lukeshu at sbcglobal.net Sun Aug 26 02:30:11 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Sat, 25 Aug 2012 22:30:11 -0400 Subject: [Dev] Parabola-friendly implementation of VCS urls for PKGBUILDs In-Reply-To: <87fw7atv8l.fsf@kiwwwi.com.ar> References: <87harq26vw.wl%lukeshu@sbcglobal.net> <87lih2u9mf.fsf@kiwwwi.com.ar> <87a9xi1vp2.wl%lukeshu@sbcglobal.net> <87fw7atv8l.fsf@kiwwwi.com.ar> Message-ID: <874nnq1j2k.wl%lukeshu@sbcglobal.net> At Sat, 25 Aug 2012 20:19:54 -0300, Nicol?s Reynolds wrote: > the main reason to have tarballs is not that we loooove checksums, but > that at least GPL2 software requires you to host the corresponding > sources for the packages you provide. arch is following this doing > makepkg --allsource for GPL packages on ABS, how would they solve it? or > won't they allow vcs packges on repos? Their implementation includes (for the example of git) a bare repository alongside the PKGBUILD when running `makepkg --allsource`. About accessing the network during build()--it doesn't, though it will always try to run `git fetch` before build(), even if unnecessary. But if it doesn't succeed, everything is just jolly, so I don't see an issue with that. I personally think my design is better, but if checksums aren't a must, I see no real problems with their version. ~ Luke Shumaker From jorgean at lavabit.com Mon Aug 27 05:47:44 2012 From: jorgean at lavabit.com (Jorge Araya Navarro) Date: Sun, 26 Aug 2012 23:47:44 -0600 Subject: [Dev] [Fwd: [Assist] Parabola LiveUSB] Message-ID: <1346046464.16792.1.camel@abril.local> Anyone willing to help? --------- Mensaje reenviado -------- De: Stayvoid Reply-to: "Ask here for assistance." Para: assist at lists.parabolagnulinux.org Asunto: [Assist] Parabola LiveUSB Fecha: Mon, 27 Aug 2012 03:53:08 +0400 Hi, I'd like to install Parabola to Yeeloong. But I must change my partition scheme prior to that. That's why I want to boot from an external media. What should I do to make a LiveUSB? Here is what I did: 1. Downloaded this [1] package; 2. Unpacked and copied it to my USB flash drive; 3. Used the following commands to load it: load (usb0,0)/boot/vmlinuz-linux-libre initrd (usb0,0)/boot/initramfs-linux-libre.img g console=tty no_auto_cmd Output: :: Starting udevd... done. :: Running Hook [udev] :: Triggering uevents...done. ERROR: device '' not found. Skipping fsck. ERROR: Unable to find root device ''. You are being dropped to a recovery shell Type 'exit' to try and continue booting sh: can't access tty; job control turned off exit Recovery shell has a limited functionality. For example, it doesn't contain fdisk nor parted. And I haven't found a way to configure the network. But I was able to mount partitions. I think that I can copy fdisk from there, but the whole process looks like an overkill. Am I doing it wrong? Is there a better package for such purpose? [1] http://mtjm.eu/releases/parabola/parabola-mips64el-20120503.tar.bz2 Thanks _______________________________________________ Assist mailing list Assist at lists.parabolagnulinux.org https://lists.parabolagnulinux.org/mailman/listinfo/assist ____________________________________________________________________________________ Find a list of top job sites and job boards all listed on one page to help with your search at Compare.US. http://click.lavabit.com/4y45haw7pb8daauu4mmydgfc6ufuf8c5ec88xqfmcbhnyaqczsoy/ ____________________________________________________________________________________ -- Jorge Araya Navarro Universitario, idealista y pseudo-activista del Software Libre. Siquirres, Lim?n, Costa Rica. http://swt.encyclomundi.net Diaspora*: http://diasp.org/u/shackra identi.ca: http://parlementum.net/sweet Jabber: shackra at jabberes.org Skype: ?De ninguna manera, tras de privativo, te esp?an!. el software privativo en GNU/Linux, al igual que en Windows o en MacOs, te hace un ser no-libre. Deja de enga?arte, ??despierta ahora!!: http://www.gnu.org/distros/free-distros.html http://replicant.us/about/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From lukeshu at sbcglobal.net Mon Aug 27 13:55:57 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Mon, 27 Aug 2012 09:55:57 -0400 Subject: [Dev] [pacman-dev] [PATCH 0/3] makepkg: Alternate implementation of VCS URLs in sources array. In-Reply-To: <20120825184332.GU553@rampage> References: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> <20120825184332.GU553@rampage> Message-ID: <87txvotp5e.wl%lukeshu@sbcglobal.net> At Sat, 25 Aug 2012 14:43:32 -0400, Dave Reisner wrote: > > On Sat, Aug 25, 2012 at 01:36:41PM -0400, Luke Shumaker wrote: > > A while ago I started working on a derivative of makepkg to support > > having 'git://...' type urls in the sources=() array. When preparing > > to file this patch, I did a `git rebase`, and noticed that Allan McRae > > began working on a similar feature. Our implementations are in many > > ways similar. Hopefully mine will be useful. > > An interesting approach. As you've noticed, I think we're fairly > committed to the implementation that Allan has provided. If you have > specific concerns, maybe we can work to fix those. * Allan's URL parsing has some issues, see point 2 below. This is an easy fix. * Composability. It's not really a "problem", but it seems to me that It's better to create a new general-purpose tool, instead of shoving a bunch of "special case" behavior into makepkg. Allan's implementation could even fairly easily be pulled out of makepkg. And really, does it make sense to have these URL schemes hardcoded into makepkg? Why have DLAGENTS at all, if we're going to hard-code schemes into makepkg? Again, I'm not against Allan's imlementation, but I am for moving it out of makepkg. > > My implementation makes minimal changes to makepkg itself (only adding > > blob expansion to DLAGENTS, allowing for things like > > "git+*::""). Instead I added a `vcsget` tool which generates a tarball > > from the VCS repo, in a very similar manner to the way Allan's > > implementation does so within makepkg. > > I'm not thrilled with the shell I saw in that patch -- there's extensive > use of external commands when simple bash features would have sufficed. I assume you're speaking of my 1. use of cut in get_field 2. use of sed to parse URLs 3. use of sed to parse URL authorities 4. use of readlink to establish the tarball name 5. use of sed to create a basedir name from the tarball name 6. use of '[' with if statements My defense of each: 1. `cut` in get_field --------------------- This was simply the simplest solution that was obviously correct. Another solution would have been to manipulate IFS, which has all kinds of funny side effects and fringe behaviors. That was a weak argument, and as a coder, I would be thrilled to find out if there is a better solution. 2. `sed` to parse URLs ---------------------- I wanted full URL parsing, for a few cases when the translation is less than than straight-forward. For example, including authentication info in svn URLs. Further, I wanted to make sure that *any* URL parsing done would be robust. Given that the regex used was taken straight from the relevent RFC, I knew I could count on it to work, even in fringe cases. For example a "fragment" *should* be able to contain a '#', but Allan's implementation discards anything before the last '#' in the fragment. This is a problem for (at least) git, as tags and branches may contain the symbol. 3. `sed` to parse URL authorities --------------------------------- I believe that a robust (not having the same problems as fragments for URLs) implementation in bash would be non-trivial. 4. `readlink` to establish the tarball name ------------------------------------------- I used `readlink -m` to turn the tarball name into an absolute path. This was not an absolute must, but it allowed me to avoid worrying about keeping track of the current directory. It would be fairly easy to remove this by useing pushd/popd instead of cd. If you would like this changed, I can do that. 5. `sed` to create a basedir name from the tarball name ------------------------------------------------------- I'll admit, this was me getting a little lazy. A pure bash implementation is: base=${tarball##*/} base=${base%%.tar.*} base=${base%.tar} 6. `[` with if statements ------------------------- Can I call stylistic choice on this one? It is trivial to replace this with bash's '[['. > > > It looks as if Allan's download_*() functions are more verbose than > > mine about what failed when there is an error. His svn and hg handlers > > are likely more robust--though my git is pretty solid. I also have > > a half-written handler for for bzr. > > > > An advantage of my design is that it does allow for integrity checks > > of VCS packages, rather than inserting 'SKIP' into the md5sums > > array. This is very important to the derivative distribution Parabola. > > (However, the 'SKIP' option is still valuable for URLs that track a > > branch) > > I don't see this as an advantage so much as a duplication. The backing > VCS takes care of integrity checks. They're only necessary with tarballs > because there is no "builtin" checksum to reference. This is only partially true of the non-distributed VCSs. There was concern from other Parabola developers about being able to use the checksum to unambiguously be able to confirm that a source tree is the correct version, but Allan's implementation seems acceptable on that front. Happy hacking, ~ Luke Shumaker From fauno at kiwwwi.com.ar Mon Aug 27 22:34:46 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Mon, 27 Aug 2012 19:34:46 -0300 Subject: [Dev] certs renewed Message-ID: <87ipc4q7zt.fsf@kiwwwi.com.ar> i renewed the certs for parabola.nu, *.parabolagnulinux.org and parabolagnulinux.org, they are valid until february -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From fauno at kiwwwi.com.ar Thu Aug 30 16:03:46 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Thu, 30 Aug 2012 13:03:46 -0300 Subject: [Dev] parabola inquiry Message-ID: <87txvkmknx.fsf@kiwwwi.com.ar> > Dear Sir > > I would like to install Parabola and become an active supporter. I > found your Distribution on Free Software Foundation (FSF). I'm trying > to connect with the IRC but having difficulty. I need to ask questions > regarding installation and if you support software that allows me to > use video editing and a Digital Audio Workstation like Adour. > > Thank you > Wally Lepore NJ USA Wally, hi, yes we have video/audio editing software and we even support a libre version of the real time kernel. if the software you look it's not on repos we can work on that, packaging is simple. we only require for it to be free software, but you know this already. have you tried freenode's web irc? http://webchat.freenode.net/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From fauno at kiwwwi.com.ar Thu Aug 30 17:49:46 2012 From: fauno at kiwwwi.com.ar (=?utf-8?Q?Nicol=C3=A1s?= Reynolds) Date: Thu, 30 Aug 2012 14:49:46 -0300 Subject: [Dev] parabola inquiry In-Reply-To: <87txvkmknx.fsf@kiwwwi.com.ar> Message-ID: <87oblsmfr9.fsf@kiwwwi.com.ar> Wally, please reply directly to me or subscribe to the mailing list, otherwise others won't be able to read you. I'm replying your questions below :) > Hi Nicolas, > > Yes I did receive your IRC channel # from a helpful support member of > gNewSense. I've been looking for a good distro for music productopn > applications as well as everyday computer use. This field is amazing > with way too many choices. I was interested in your distro as it was > recommended by FSF. http://www.gnu.org/distros/free-distros.html > > I have tried unsuccessfully to install Ubuntu on my Dual Pentium III > system. Tried multiple fixes that did not work. Just gave up and began > to explore the other choices out there. I was initially directed to > Ubuntu from a friend but knew little about GNU/Linux. After days of > research I am well grounded (or at least getting better) in > understanding the whole GNU/Linux thing. The choices are amazing! > > A few questions please: > > #1 Why is your distro based on Arch? Most of us were Arch users before starting hacking on Parabola, we liked it technically but were troubled with its lack of software freedom awareness. > #2 Can Parabola run just about any GNU/Linux based free software? The package system is quite simple so it's fairly easy to contribute new packages. > #3 How is Parabola different from a Debian based distro? Parabola doesn't do anything for you, the whole system is in your hands to build! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From jorgean at lavabit.com Fri Aug 31 07:20:49 2012 From: jorgean at lavabit.com (Jorge Araya Navarro) Date: Fri, 31 Aug 2012 01:20:49 -0600 Subject: [Dev] [PATCH 0/3] makepkg: Alternate implementation of VCS URLs in sources array. In-Reply-To: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> References: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> Message-ID: <1346397649.3962.5.camel@abril.local> El s?b, 25-08-2012 a las 13:36 -0400, Luke Shumaker escribi?: > A while ago I started working on a derivative of makepkg to support > having 'git://...' type urls in the sources=() array. When preparing > to file this patch, I did a `git rebase`, and noticed that Allan McRae > began working on a similar feature. Our implementations are in many > ways similar. Hopefully mine will be useful. > > My implementation makes minimal changes to makepkg itself (only adding > blob expansion to DLAGENTS, allowing for things like > "git+*::""). Instead I added a `vcsget` tool which generates a tarball > from the VCS repo, in a very similar manner to the way Allan's > implementation does so within makepkg. > > It looks as if Allan's download_*() functions are more verbose than > mine about what failed when there is an error. His svn and hg handlers > are likely more robust--though my git is pretty solid. I also have > a half-written handler for for bzr. > > An advantage of my design is that it does allow for integrity checks > of VCS packages, rather than inserting 'SKIP' into the md5sums > array. This is very important to the derivative distribution Parabola. > (However, the 'SKIP' option is still valuable for URLs that track a > branch) > > Happy hacking, > ~ Luke Shumaker > > Luke Shumaker (3): > Add a `vcsget` tool to download source from VCS repositories. > makepkg: do glob expansion in DLAGENTS maps > makepkg.conf: add vcsget DLAGENTS > > etc/makepkg.conf.in | 8 +- > scripts/.gitignore | 1 + > scripts/Makefile.am | 4 +- > scripts/makepkg.sh.in | 13 ++- > scripts/vcsget.sh.in | 294 ++++++++++++++++++++++++++++++++++++++++++++++++++ > 5 files changed, 316 insertions(+), 4 deletions(-) > create mode 100644 scripts/vcsget.sh.in > Hi! Emulatorman and his brrrrrrother Coadde have had discovered a way to compile packages for 32bits architecture in 64bits computers more details here https://wiki.parabolagnulinux.org/Package_maintainer_guide#32_bits_Chroot_on_64_bits_machine however, both Emulatorman and myself though that would be wonderful and more easy have a option to switch between 64 and 32 bits architecture, what's your opinion about this feature, Arch devs are willing to implement that? If not, may you could implement it? -- Jorge Araya Navarro Universitario, idealista y pseudo-activista del Software Libre. Siquirres, Lim?n, Costa Rica. http://swt.encyclomundi.net Diaspora*: http://diasp.org/u/shackra identi.ca: http://parlementum.net/sweet Jabber: shackra at jabberes.org Skype: ?De ninguna manera, tras de privativo, te esp?an!. el software privativo en GNU/Linux, al igual que en Windows o en MacOs, te hace un ser no-libre. Deja de enga?arte, ??despierta ahora!!: http://www.gnu.org/distros/free-distros.html http://replicant.us/about/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From lukeshu at sbcglobal.net Fri Aug 31 15:22:18 2012 From: lukeshu at sbcglobal.net (Luke T.Shumaker) Date: Fri, 31 Aug 2012 11:22:18 -0400 Subject: [Dev] [makepkg] 32 bit chroot on 64 bit machine In-Reply-To: <1346397649.3962.5.camel@abril.local> References: <1345916204-3459-1-git-send-email-LukeShu@sbcglobal.net> <1346397649.3962.5.camel@abril.local> Message-ID: <87ehmnulw5.wl%lukeshu@sbcglobal.net> At Fri, 31 Aug 2012 01:20:49 -0600, Jorge Araya Navarro wrote: > Hi! > > Emulatorman and his brrrrrrother Coadde have had discovered a way to > compile packages for 32bits architecture in 64bits computers more > details here > https://wiki.parabolagnulinux.org/Package_maintainer_guide#32_bits_Chroot_on_64_bits_machine > however, both Emulatorman and myself though that would be wonderful > and more easy have a option to switch between 64 and 32 bits > architecture, what's your opinion about this feature, Arch devs are > willing to implement that? If not, may you could implement it? This should be implemented in libretools, not makepkg. Right now libre* supports [-d chrootdir] and [-n chrootname] to allow for easily switching between multiple chroots $ sudo libremakepkg -n i686 If anything, I think that the only change that should be made is to libremkchroot, to allow it to easily load preset configuration for another arch. Arch's devtools package supports this. PS: I am the author of libremkchroot Happy hacking, ~ Luke Shumaker From illusion.of.life92 at gmail.com Sun Aug 5 15:30:32 2012 From: illusion.of.life92 at gmail.com (illusionoflife) Date: Sun, 05 Aug 2012 15:30:32 -0000 Subject: [Dev] Dropbox is unconflicting. Message-ID: <1480999.7shaPuz7mh@miracle> Hello. I just switched from Arch with installed dropbox proprietary daemon to my freedom -- to parabola GNU/Linux. I noticed, that your-freedom do not conflict with dropbox-daemon. I think it is a bug. -- Best regards, illusionoflife Contact me on illusion.of.life92 at gmail.com. Please, read rfc1855, if did not already. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From coriroc at gmail.com Wed Aug 22 01:37:39 2012 From: coriroc at gmail.com (Aerok) Date: Wed, 22 Aug 2012 01:37:39 -0000 Subject: [Dev] New message Message-ID: ?TATTOO KOREA + We only use Authentic Needles & Ink...Unlike other shops that use fake ink & needles... +The Full-Sleeve event is back!!! Consultation Starts Today !!! Known as the Best Tattoo parlor in Korea is having The Best Event of all Tattoo events! +Tattoo Korea is giving out the Best Quality Full-Sleeves Simply put it!! It doesn?t matter what the design is. It could be detail as your brain!! Your design could be the size of the Hulk!!! All of our best international artists are going to be working on this event. +The event is from September 7th to September 28th. This event is only for 22 days, for the first 30 people!! Reservations are open, starting today!! <<<<<<<<<<< Sign up now!! >>>>>>>>>>>> ***Black & Grey ~ Males = 1.5mil.won, ~ Female = 1mil.won ***Color ~ Males = 2 mil.won, ~ Female = 1.5 mil.won ***For our Payment Plans, contact for details +You will Not get this deal anywhere else in Korea All Tattoo Lovers in Korea!! Don?t Miss out this event!!! +To make an Appointment we need... 1) Full Name 2) Date & time you would like to come in 3) Phone # Email: coriroc at msn.com Phone: +82 10 8998 1911 ask for Aerok Our website www.tattoo-korea.asia Go to 'Location' for the map to our shop We are the one, and only, TATTOO KOREA!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallylepore at gmail.com Thu Aug 30 17:17:40 2012 From: wallylepore at gmail.com (Wally Lepore) Date: Thu, 30 Aug 2012 13:17:40 -0400 Subject: [Dev] parabola inquiry In-Reply-To: <87txvkmknx.fsf@kiwwwi.com.ar> References: <87txvkmknx.fsf@kiwwwi.com.ar> Message-ID: Hi Nicolas, Yes I did receive your IRC channel # from a helpful support member of gNewSense. I've been looking for a good distro for music productopn applications as well as everyday computer use. This field is amazing with way too many choices. I was interested in your distro as it was recommended by FSF. http://www.gnu.org/distros/free-distros.html I have tried unsuccessfully to install Ubuntu on my Dual Pentium III system. Tried multiple fixes that did not work. Just gave up and began to explore the other choices out there. I was initially directed to Ubuntu from a friend but knew little about GNU/Linux. After days of research I am well grounded (or at least getting better) in understanding the whole GNU/Linux thing. The choices are amazing! A few questions please: #1 Why is your distro based on Arch? #2 Can Parabola run just about any GNU/Linux based free software? #3 How is Parabola different from a Debian based distro? Thank you Wally Thu, Aug 30, 2012 at 12:03 PM, Nicol?s Reynolds wrote: > > Dear Sir > > > > I would like to install Parabola and become an active supporter. I > > found your Distribution on Free Software Foundation (FSF). I'm trying > > to connect with the IRC but having difficulty. I need to ask questions > > regarding installation and if you support software that allows me to > > use video editing and a Digital Audio Workstation like Adour. > > > > Thank you > > Wally Lepore NJ USA > > Wally, hi, yes we have video/audio editing software and we even support > a libre version of the real time kernel. if the software you look it's > not on repos we can work on that, packaging is simple. we only require > for it to be free software, but you know this already. > > have you tried freenode's web irc? http://webchat.freenode.net/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: