Debian Bug report logs - #498274
Coordination of testing migrations in Emdebian

Package: buildd.emdebian.org; Maintainer for buildd.emdebian.org is Debian Embedded Team <debian-embedded@lists.debian.org>;

Reported by: Neil Williams <codehelp@debian.org>

Date: Mon, 8 Sep 2008 16:21:01 UTC

Severity: minor

Tags: confirmed, help

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org:
Bug#498274; Package emdebian-tools. Full text and rfc822 format available.

Acknowledgement sent to Neil Williams <codehelp@debian.org>:
New Bug report received and forwarded. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Neil Williams <codehelp@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: /usr/bin/emdebcheck: emdebian-qa: emdebcheck needs to support testing migration checks
Date: Mon, 08 Sep 2008 17:18:41 +0100
Package: emdebian-tools
Version: 1.4.6
Severity: minor
File: /usr/bin/emdebcheck

emdebian-qa is a new package, currently available separately in Emdebian
but not uploaded to Debian due to the Lenny freeze. The scripts within
emdebian-qa are available in emdebian-tools 1.4.3.

Some future version of emdebcheck will need to be able to work alongside
'emtargetcmp -m', a new mode for emtargetcmp in emdebian-qa 1.4.6 that
tries to identify packages needing assistance to migrate into Emdebian
testing, prior to the release of Emdebian stable as Emdebian 1.0.

emtargetcmp -m does a fair bit of the work but the final checks before
copying the package using reprepro need to have a dependency check
against the dependencies of the Emdebian packages. emtargetcmp -m can do
this *after* the migration but that isn't really the point.

emdebcheck is capable of this extension, it just means that I need to
make it happen and hence this bug report to ensure I don't miss it.
Patches are welcome.
:-)

The theory would be that emdebcheck would check the Packages.gz from the
Emdebian testing target repository, substituting the data from the
package seeking to migrate as if the migration had already happened.
This mode would need to identify all binaries built from the relevant
source and check each. emtargetcmp -m itself might need to become aware
of other architectures once those become supported in Emdebian
repositories.



-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages emdebian-qa depends on:
ii  apt-cross                    0.12.3      retrieve, build and install librar
ii  dpkg-cross                   2.3.1       tools for cross compiling Debian p
ii  edos-debcheck                1.0-8       Checks whether dependencies of Deb
ii  libemdebian-tools-perl       1.4.6       emdebian support library
ii  libtext-formattable-perl     1.01-2      Format text tables
ii  lintian                      1.24.4      Debian package checker
ii  perl                         5.10.0-13   Larry Wall's Practical Extraction 
ii  whiptail                     0.52.2-11.3 Displays user-friendly dialog boxe
ii  zenity                       2.22.1-2    Display graphical dialog boxes fro

Versions of packages emdebian-qa recommends:
ii  dput                          0.9.2.33   Debian package upload tool
ii  emdebian-rootfs               1.4.6      emdebian root filesystem support

Versions of packages emdebian-qa suggests:
ii  deb-gview                     0.2.2      GNOME viewer for .deb package file
ii  rsync                         3.0.3-2    fast remote file copy program (lik
ii  trickle                       1.07-5     user-space bandwidth shaper

-- no debconf information




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#498274; Package emdebian-tools. (Sun, 12 Oct 2008 16:27:13 GMT) Full text and rfc822 format available.

Acknowledgement sent to Neil Williams <codehelp@debian.org>:
Extra info received and forwarded to list. (Sun, 12 Oct 2008 16:27:13 GMT) Full text and rfc822 format available.

Message #10 received at 498274@bugs.debian.org (full text, mbox):

From: Neil Williams <codehelp@debian.org>
To: 498274@bugs.debian.org
Subject: methodology problems in testing migration checks
Date: Sun, 12 Oct 2008 17:23:52 +0100
[Message part 1 (text/plain, inline)]
emdebian-qa: emdebcheck needs to support testing migration checks
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=498274

The theory would be that emdebcheck would check the Packages.gz from the
Emdebian testing target repository, substituting the data from the
package seeking to migrate as if the migration had already happened.
This mode would need to identify all binaries built from the relevant
source and check each. emtargetcmp -m itself might need to become aware
of other architectures once those become supported in Emdebian
repositories.

Not sure how to proceed for this one. The problems are:

1. emdebcheck needs data from a .changes file - i.e. a build machine or
the full list of .debs/.tdebs and must be able to locate and process the
actual files (either a build machine or the repository pool).
2. the testing migration check needs to either be run on the server or
the data from 'emtargetcmp -m' needs to be moderated according to the
data from the testing check. #2 can't be coded until #1 is decided.

One idea is to run the testing check at the same time as the check prior
to an upload to unstable by extending emrecent. The only problem is the
time delay: package removals during the time that the package is waiting
to migrate would break the subsequent migration but as removals are a
manual task, this might be acceptable.

A second issue is the TDeb migrations - the same testing migration
checks needs to be done for the locale repository. embritney (the new
server-side script to handle migrations) can handle the existing
migrations and it should be relatively simple to migrate the TDeb at the
same time, as long as embritney is able to work out which packages have
TDebs. That task requires access to the .changes file (or at least the
package build directory where the TDeb .changes file can be located).
That requires some coordination between emdebcheck (which knows the
location of the files) and emtargetcmp (which knows the cache data from
Debian) to pass on data to embritney (which knows neither of these
things). One option is for emtargetcmp to query the locale repository
and match up the source package names, skipping the need to coordinate
with emdebcheck or a build machine entirely.

Running the whole thing on the server is possible (after all, the server
has all the .debs) but involves having various packages like
libcache-apt-perl at versions later than the Lenny freeze. Once Lenny is
released, I'll see about backporting all these packages anyway (probably
via the Emdebian repositories) because emdebian-qa was originally
envisaged as suitable for "server-side" operations as well as buildd
side.

I want to get the dpkg-cross -> dpkg migration complete ASAP after Lenny
though and this complicates things with apt-cross and libcache-apt-perl.
(See my other email to the list).
http://lists.debian.org/debian-embedded/2008/10/msg00015.html

A final problem:
Emdebian lacks support for "release-critical" bug detection/blocks - RC
bugs in Debian will block Debian migration and therefore Emdebian
migration but bugs against buildd.emdebian.org cannot be RC at this time
as issues within the Emdebian packages cannot delay the main Debian
release.

I'm open to ideas.

1. Maybe we can persuade Debian and the scripts that generate the list
of RC bugs that buildd.emdebian.org is ignored for the Debian RC count?
I can see that causing confusion unless the same principle is applied to
bugs.debian.org in which case we might as well use tags (or usertags) to
specify what is RC to us. To use these tags, we need a fix for usertags
in bugs.debian.org as I triggered an infinite loop in the b.d.o code
last time I tried to set such a configuration.

2. Delay the solution to this until Squeeze and Emdebian 2.0 where all
these issues can be resolved "properly". This increases the amount of
manual work needed to get Emdebian 1.0 out but not by a large amount
because Debian is frozen. Once Lenny is released, keeping Emdebian
testing in line with Debian testing will become very difficult until Sid
settles down a bit. In some ways, scripts developed during the freeze
could break in very strange ways when faced with the chaos of the
post-release frenzy in Sid.

As this is a "policy" thing rather than simply a coding change, I'm
thinking of reassigning this bug to buildd.emdebian.org.

-- 


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/


[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#498274; Package emdebian-tools. (Wed, 28 Jan 2009 11:24:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Neil Williams <codehelp@debian.org>:
Extra info received and forwarded to list. (Wed, 28 Jan 2009 11:24:02 GMT) Full text and rfc822 format available.

Message #15 received at 498274@bugs.debian.org (full text, mbox):

From: Neil Williams <codehelp@debian.org>
To: 498274@bugs.debian.org
Cc: control@bugs.debian.org
Subject: Reassigning
Date: Wed, 28 Jan 2009 11:22:12 +0000
[Message part 1 (text/plain, inline)]
reassign 498274 buildd.emdebian.org
notfound 498274 1.4.6
tag 498274 + confirmed
tag 498274 + help
retitle 498274 Coordination of testing migrations in Emdebian
thanks

All Emdebian flavours need to coordinate migrations into testing -
preferably together with migrations of the toolchain packages and the
tools themselves. i.e. the debian/, emdebian/, grip/ and locale/
repositories should migrate all binaries from the relevant source
package(s) at the same time and maintain installability for each. If a
particular binary is not available, Emdebian needs to have guidelines
on whether the entire migration is paused or certain flavours are
allowed to migrate ahead of the rest.

1. filter/ updates via a daily cron task, directly from Debian.
2. grip/ updates a short time later - after the packages are built on
the server.
3. locale/ needs better coordination between Grip and Crush but
reprepro handles the current duplication without problems. Generally,
locale/ will be updated by the Grip builds quicker than the Crush
builds.
4. emdebian/ (which probably should be renamed/symlinked as crush/)
migrations are not currently automated, although information necessary
to perform the migration is collated daily with a 24hr delay. Packages
that fail to cross-build after an update in Debian will take longer to
fix but most fixes are done within the 10day testing migration delay.
There probably needs to be a system to alert the mailing list when a
package consistently fails to cross-build.
5. debian/ (i.e. toolchains) take the longest to build and fix when the
build fails but as long as all toolchains remain installable, it
probably isn't necessary to delay the migration of other flavours
simply because the toolchain is delayed.
5.1. Tools (also in debian/) like emdebian-tools and apt-cross are
managed via Debian and Britney - releases only build up in debian/
during a release freeze, as now. Migrations within debian/ must avoid
migrating packages other than those necessary for the toolchains.

The 'filter/' repository maintained by emdebian-grip-server can provide
all the data we need about migrations within Debian - support already
exists such that when the chosen mirror updates, the filter is updated
by a cron task and the Packages.gz file in the filter repository is
updated by reprepro. This removes the need to listen to the email
announcements from Britney or devise other ways to monitor Debian
testing.

This is a Policy thing for Emdebian - it will be supported via scripts
but the implementation needs to be bespoke. Debian::Packages::Compare
will be the main agent, tying the TDebs to the source packages via the
Source: field of the TDeb.

Of course, this problem would go away if there was a way to get these
repositories into the main Debian archive. :-)

Alternatively, if the real 'britney' script becomes available as a
regularly updated package in Debian, it could be possible to use that
with reprepro.

This could be a good sub-project for someone else to develop. It will
be essential when Crush starts to support more architectures. If you
want to work on it, just claim this bug report and keep it (and
therefore the mailing list) regularly updated.

-- 


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

[Message part 2 (application/pgp-signature, inline)]

Bug reassigned from package `emdebian-tools' to `buildd.emdebian.org'. Request was from Neil Williams <codehelp@debian.org> to control@bugs.debian.org. (Wed, 28 Jan 2009 11:24:03 GMT) Full text and rfc822 format available.

Bug no longer marked as found in version 1.4.6. Request was from Neil Williams <codehelp@debian.org> to control@bugs.debian.org. (Wed, 28 Jan 2009 11:24:04 GMT) Full text and rfc822 format available.

Tags added: confirmed Request was from Neil Williams <codehelp@debian.org> to control@bugs.debian.org. (Wed, 28 Jan 2009 11:24:05 GMT) Full text and rfc822 format available.

Tags added: help Request was from Neil Williams <codehelp@debian.org> to control@bugs.debian.org. (Wed, 28 Jan 2009 11:24:05 GMT) Full text and rfc822 format available.

Changed Bug title to `Coordination of testing migrations in Emdebian' from `/usr/bin/emdebcheck: emdebian-qa: emdebcheck needs to support testing migration checks'. Request was from Neil Williams <codehelp@debian.org> to control@bugs.debian.org. (Wed, 28 Jan 2009 11:24:06 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 12:34:16 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.