Debian Bug report logs - #477942
git storage backend for chroots

Toggle useless messages

Message #3 received at submit@bugs.debian.org (full text, mbox, reply):

From: martin f krafft <madduck@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: git storage backend for chroots

Date: Sat, 26 Apr 2008 00:23:01 +0400

[Message part 1 (text/plain, inline)]

Package: schroot
Version: 1.1.6-1
Severity: wishlist

Please find attached the files 05git and a diff to 00check, which
together should enable Git to be used as chroot backend. The
rationale is that

- Git is itself a filesystem, really.
- Because it snapshots the tree, it gives the same functionality as
  copy-on-write.
- If you make a change in a temporary session which you'd actually
  like to keep, Git allows you to create the appropriate patch
  (unless the change was binary).
- Git's in-place branches could be really cool for testing stuff in
  temporary chroots.
- Git's branches allow multiple flavours of a chroot to be served
  from the same repository
- Git's store is highly efficient. While operations take about the
  same time as with tgz files, the packed repository takes up less
  space than the tgz. For instance, a buildd-flavour chroot of sid
  on i386 creates a 109M tgz file, but the .git object store is 104M
  in size. a tbz2 file is smaller, but operations take significantly
  longer than with Git (bzip2 is slow).
    Obviously, when a chroot is updated a couple of times, the
  history is kept and the size increases. While the history might be
  a feature, I have yet to find a way to properly prune it.
  Nevertheless, operation speeds are not affected.
- ...

When using type=git, the configuration must also specify
repository=/path/to/repo.git or repository=/path/to/repo/.git,
depending on whether the source is a bare repo or not.

The repository is set up so that the object store of the source is
used and thus the copying is minimal. Since existing objects cannot
be changed with Git, only new ones created, this is effectively
copy-on-write and safe. Even if source and destination are on the
same filesystem, Git does not use hardlinks to prevent involuntary
changes to the source.

git-core is currently not installed into the checkout. I am
currently ambivalent of whether this should happen as part of the
setup, or whether Git-enabled chroots just expect the admin to
install git-core into the source chroot. It's *not* needed in the
chroot for normal operation, but might be needed for -source
handling (see below).

Simple use of the repo already works, but -source handling does not
work at all yet. Right now, -source chroots set CHROOT_FILE_REPACK
(I'm abusing the file type, see below) and that's not handled at all
in 05git, simply because it's not the way things should happen.

Instead, 05git should check whether the source repository is bare or
not.

- for bare repositories, it should clone the repository just as
  before, but on close, it should push all commits (from outside the
  chroot, or else the source repository is unreachable). It remains
  to be seen whether it makes sense to prevent schroot from leaving
  the chroot if the tree has uncommitted changes (see separate bug).

- for non-bare repositories, it can chroot straight into the
  worktree of the repository. In this case, uncommitted changes in
  the tree should prevent schroot from leaving the chroot (see
  separate bug).

19devices, which is also attached, creates all devices, since those
cannot be stored in Git. This takes quite a while and could be
optimised by using a tarball with the devices. 19devices includes
a similarly ugly hack to work around the aforementioned type
limitation (it uses type=file and the .git/config file).

There's a problem related to empy directories, which Git does not
checkout. Until mount mkdir's mountpoints before trying to mount
onto them, 05git creates a number of standard directories, but not
all. I may have to figure out a way to recreate all empty
directories found in the source, which might be possible with
a simple find call.

Right now, schroot also checks for the chroot type in the binary
(see other bug) and thus I am using the following at the top of
05file:

  # hackery until I can add my own types easily
  if [ "${CHROOT_FILE#*/.git/}" = "config" ]; then
    CHROOT_TYPE=git CHROOT_REPOSITORY="${CHROOT_FILE%/.git/*}" \
      exec ${0%/*}/05git "$@"
  fi

this allows me to set file=../path/to/repo/.git/config and have the
git method invoked.

Note that 05git and 19devices are GPLv2 explicitly. I don't approve
of the (philosophy behind) GPLv3 and thus prefer to stay with v2. If
that's a problem, let me know and we can surely find a way.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.24-1+scoflowctrl.1-686 (SMP w/1 CPU core)
Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages schroot depends on:
ii  libboost-program-options1.34. 1.34.1-11  program options library for C++
ii  libboost-regex1.34.1          1.34.1-11  regular expression library for C++
ii  libc6                         2.7-10     GNU C Library: Shared libraries
ii  libgcc1                       1:4.3.0-3  GCC support library
ii  liblockdev1                   1.0.3-1.2  Run-time shared library for lockin
ii  libpam0g                      0.99.7.1-6 Pluggable Authentication Modules l
ii  libstdc++6                    4.3.0-3    The GNU Standard C++ Library v3
ii  libuuid1                      1.40.8-2   universally unique id library
ii  schroot-common                1.1.6-1    common files for schroot

schroot recommends no packages.

-- no debconf information

-- 
 .''`.   martin f. krafft <madduck@debian.org>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems

[00check.diff (text/x-diff, attachment)]

[05git (text/plain, attachment)]

[19devices (text/plain, attachment)]

[digital_signature_gpg.asc (application/pgp-signature, inline)]

Message #6 received at 477942@bugs.debian.org (full text, mbox, reply):

From: martin f krafft <madduck@debian.org>

To: 477942@bugs.debian.org

Subject: problem with users and permissions

Date: Sun, 27 Apr 2008 00:25:30 +0400

[Message part 1 (text/plain, inline)]

I forgot that Git does not store permissions apart from the owner's
+x bit, and thus, a Git-managed chroot will not have any
ug+s,o+t bits set, nor will it restore groups. /home and /tmp appear
to be alright, but that's because they're bind-mounted.

madduck@lapse:~$ ls /etc/shadow -l
-rw-r--r-- 1 root root 1028 Apr 26 20:22 /etc/shadow

In particular, /etc/shadow will be world-readable in the chroot.
Since it's bind-mounted or copied from the host system, my proposed
Git backend is useless on multiuser systems until I figure out a way
to restore permissions sensibly.

One idea might be to specify an ACL file created with getfacl -R and
applied with setfacl, but that still might leave
/var/lib/schroot/mount/*/etc/shadow exposed until the chroot is
fully unpacked and the ACL scriptlet run.

I still find the Git backend interesting, but it's not ready for
deployment until I figure out these issues. It might thus be a good
idea to leave this bug open and "unfixed" for the time being.

-- 
 .''`.   martin f. krafft <madduck@debian.org>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems

[digital_signature_gpg.asc (application/pgp-signature, inline)]

Message #13 received at 477942@bugs.debian.org (full text, mbox, reply):

From: Adam Baxter <voltagex@voltagex.org>

To: 477942@bugs.debian.org

Date: Sun, 2 Sep 2012 18:54:12 +1000

Permissions could probably be maintained by a dotfile somewhere in the
chroot, then a git hook to restore permissions when needed.

Message #18 received at 477942@bugs.debian.org (full text, mbox, reply):

From: Roger Leigh <rleigh@codelibre.net>

To: Adam Baxter <voltagex@voltagex.org>, 477942@bugs.debian.org

Subject: Re: [buildd-tools-devel] Bug#477942:

Date: Sun, 2 Sep 2012 10:13:57 +0100

On Sun, Sep 02, 2012 at 06:54:12PM +1000, Adam Baxter wrote:
> Permissions could probably be maintained by a dotfile somewhere in the
> chroot, then a git hook to restore permissions when needed.

With schroot 1.6, we have a "custom" chroot type and user-definable
keys which would enable anyone who wishes to work on this to do so
without any need to hack on the core code.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800

Message #23 received at 477942-done@bugs.debian.org (full text, mbox, reply):

From: Roger Leigh <rleigh@codelibre.net>

To: Adam Baxter <voltagex@voltagex.org>, 477942-done@bugs.debian.org

Subject: Re: [buildd-tools-devel] Bug#477942: Bug#477942:

Date: Sat, 4 Jan 2014 16:01:30 +0000

On Sun, Sep 02, 2012 at 10:13:57AM +0100, Roger Leigh wrote:
> On Sun, Sep 02, 2012 at 06:54:12PM +1000, Adam Baxter wrote:
> > Permissions could probably be maintained by a dotfile somewhere in the
> > chroot, then a git hook to restore permissions when needed.
> 
> With schroot 1.6, we have a "custom" chroot type and user-definable
> keys which would enable anyone who wishes to work on this to do so
> without any need to hack on the core code.

This has been open for nearly six years, and it's not looking particularly
realistic for this to be implemented anytime soon.  I'm closing this for
now as a result.  Alternatives such as btrfs-snapshot will offer much
better performance when snapshotting at this point in time compared with
cloning, though git obviously would offer a lot more in terms of retaining
the change history.  However, this isn't really something which most
current use cases would benefit greatly from.

Please feel free to reopen if a patch to implement the functionality
becomes available, or git gains the ability to handle real filesystems
natively.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800

Send a report that this bug log contains spam.