Debian Bug report logs - #120418
Policy for code using CPU extensions

version graph

Package: debian-policy; Maintainer for debian-policy is Debian Policy List <debian-policy@lists.debian.org>; Source for debian-policy is src:debian-policy.

Reported by: <camm@enhanced.com>

Date: Tue, 20 Nov 2001 23:18:06 UTC

Severity: wishlist

Found in version 3.1.1.1

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#120418; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to <camm@enhanced.com>:
New Bug report received and forwarded. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: <camm@enhanced.com>
To: submit@bugs.debian.org
Subject: cc: Michel LESPINASSE <walken@zoy.org>,Robert Stone <talby@trap.mtview.ca.us>,Eduard Bloch <edi@gmx.de>,Adam C Powell IV <hazelsct@mit.edu>,Dirk Eddelbuettel <edd@debian.org>,bates@debian.org,bcollins@debian.org,camm
Date: Tue, 20 Nov 2001 18:13:03 -0500
Subject: debian-policy: Policy proposal: Cpu extension code
Package: debian-policy
Version: 3.1.1.1
Severity: wishlist

Greetings!  I posted this proposal to debian-policy for discussion in
June, but am only now formally requesting its inclusion in the
policy.  I've taken the liberty of including recommendations from
various submitters.

The goal of this proposal is to provide a mechanism whereby packages
can safely make use of cpu instruction set extensions
(e.g. sse1,sse2,3dnow,3dnowext, etc.)  In this scheme, packages can
install on any machine of the general architecture, and extension
instructions  only get executed on appropriately capable cpus, even if
/usr is served up over NFS.

Proposal outline:

1) All packages providing programs and/or libraries which make use of
   CPU instruction set extensions (CISEs) must provide identical functionality
   on all machines of the given general architecture.  One way to
   achieve this is for the code to be protected by runtime probes of
   the cpu, followed by appropriate branching to correctly executable
   sections.  Alternatively, packages can separate routines using
   CISEs into shared libraries. This section of the policy details the
   machanism whereby ldso will load appropriate versions of such
   libraries according to the capabilities of the runtime cpu.

2) Libraries containing CSIEs should be placed in a designated subdir
   of /usr/lib specialized for code requiring the cpu extension in
   question.  All such directories should be agreed upon and sanctioned
   by the policy committee.  As a suggestion, the directory might be
   named after the cpu capability (as reported in /proc/cpuinfo) required
   to run the code, (e.g. /usr/lib/{sse,sse2,3dnow,3dnowext} on i386,
   /usr/lib/ev5 on alpha, and perhaps /usr/lib/sparc64 on sparc, etc.)

3) All functionality provided by this code should also be provided by
   a binary-equivalent generic shared lib of the same name and soname
   in /usr/lib.

4) Ideally, ldso should be smart enough to search the special
   directories first given the running cpu characteristics.  Barring
   this, a system script, update-ld.so.conf, should be created, which
   packages can call at install/remove time, to add/remove the
   specialized paths appropriate to the running cpu to /etc/ld.so.conf
   and then run ldconfig.  Optionally (and
   preferably), this script could also be run at boot time to catch
   cpu-upgrades and/or kernel upgrades (SSE requires kernel support).
   Paths need not be added, and can be removed, if no libraries exist in
   the directory.  The package providing this script should be in the
   'base' section if invoked at boot time.  Otherwise, packages using
   this functionality should depend upon the update-ld.so.conf package.

5) For this scheme to work, it is necessary to ensure that ldso loads 
   libraries in the directories specified in /etc/ld.so.conf in
   preference to identically named versions in /lib or /usr/lib, as is
   currently the case with Debian's ldso.  It is part of this policy
   that such behavior remain standard on Debian systems.

The Debian atlas packages currently use this scheme.  I believe that
at least gmp can currently make use of it as well.  I've provided a
sample update-ld.so.conf below.  If this proposal is accepted, I'd be
happy to package it or something similar.  

Thanks for your work on Debian!

-- 
Camm Maguire			     			camm@enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

=============================================================================
sample update-ld.so.conf, can doubtlessly be improved
=============================================================================
#!/bin/bash

set -e

FILE=/etc/ld.so.conf

usage() {
	echo "Usage: $0 [ --enable <ext> ] [ --disable <ext> ] [ --auto ]"
	exit 1
}

do_ldso() {

	local j=$@ jj=$@ p q qp qm i k zi pp tf

	tf=$(tempfile -m 644)
	( 	while read p; do
			q=$(basename $p)
			qp="+$q"
			qm="_$q"
			k=""
			for i in $j ; do
				if [ $i = $qp ] ; then
					k=p
					break
				fi
				if [ $i = $qm ] ; then
					k=m
					break
				fi
			done
			case $k in
			p)	pp=""
				zi=""
				for i in $jj; do
					if [ -z $zi ] && echo $i | grep -q "^+" ; then 
						echo /usr/lib/${i#+}
					else
						pp="$pp $i"
					fi
					if [ $i = $qp ] ; then zi=t ; fi
				done
				jj=$pp
			;;
			m) ;;
			*)      echo $p;;
			esac

		done < $FILE 
		for i in $jj; do
			if echo $i | grep -q "^+" ; then 
				echo /usr/lib/${i#+}
			fi
		done 
	) >$tf
	mv $tf $FILE
	ldconfig		

}

cpu_probe_i386() {

	local fl i res kv j

	if ! [ -r /proc/cpuinfo ] ; then
		echo This script needs the proc filesystem to work
		exit 1
	fi
	fl=$(grep ^flags /proc/cpuinfo)
	kv=$(uname -a | cut -f3 -d\  | cut -f2 -d.)
#
#  Names changed in 2.4.x
#  Listed possibilities must be in order of preference here
#	
	case $kv in
		2) ss="26 xmm 3dnowext 3dnow";;
		*) ss="sse2 sse 3dnowext 3dnow";;
	esac
	res=""
	for i in $ss; do
		j=${i/xmm/sse}
		j=${j/26/sse2}
		if echo $fl | grep -q "\<$i\>" ; then
			res="$res +$j"
		else
			res="$res _$j"
		fi
	done

	echo $res

}

cpu_probe_alpha() {

	local fl i res j

	if ! [ -r /proc/cpuinfo ] ; then
		echo This script needs the proc filesystem to work
		exit 1
	fi
	fl=$(grep ^"cpu model" /proc/cpuinfo)
	res=""
	for i in EV7 EV6 EV5; do
		j=${i/EV7/ev5}
		j=${j/EV6/ev5}
		if grep -q "\<$i\>" $fl ; then
			res="$res +$j"
		else
			res="$res -$j"
		fi
	done

	echo $res

}

cpu_probe_sparc() {

	local fl i res

	if ! [ -r /proc/cpuinfo ] ; then
		echo This script needs the proc filesystem to work
		exit 1
	fi
	fl=$(grep ^"cpu" /proc/cpuinfo)
	res=""
	for i in $fl; do
		case $i in
			UltraSparc) res="$res sparc64";;
			*);;
		esac
	done

	echo $res

}

mmatch() {

	local k t j i

	k=$1
	t=""
	shift
	j=$@
	for i in $j; do
		if [ $k = ${i#?} ] ; then
			t=$i;
		fi
	done
	echo $t

}

off() {

	local k j t i

	k=$1
	shift
	j=$@
	t=""
	for i in $j ; do
		if [ $k = ${i#?} ] ; then
			t="$t _$k"
		else
			t="$t $i"
		fi
	done
	echo $t

}

on() {

	local k j t i

	k=$1
	shift
	j=$@
	t=""
	for i in $j ; do
		if [ $k = ${i#?} ] ; then
			t="$t +$k"
		else
			t="$t $i"
		fi
	done
	echo $t

}



AUTO=""
EN=""
DIS=""
EXT=""

if [ $# -lt 1 ] ; then
	usage;
fi

case $1 in
--auto)    AUTO=t; if [ $# -ne 1 ] ; then usage; fi;;
--enable)  EN=t;   if [ $# -ne 2 ] ; then usage; fi; EXT=$2;;		
--disable) DIS=t;  if [ $# -ne 2 ] ; then usage; fi; EXT=$2;;		
*) usage;;
esac

ARCH=$(dpkg --print-architecture)
case $ARCH in
i386)  cpu=$(cpu_probe_i386);;
alpha) cpu=$(cpu_probe_alpha);;
sparc) cpu=$(cpu_probe_sparc);;
*)     echo "There are no cpu extensions supported for this architecture ($ARCH)";
       exit 0;;
esac

if [ "$AUTO" = "" ] ; then

	if [ "$(mmatch $EXT $cpu)" != "+$EXT" ] ; then
		echo "Cpu extension $EXT is not supported on this architecture ($ARCH)"
		exit 1
	fi

	ldso=""
	while read p ; do
		q=$(basename $p)
		ldso="$ldso +$q"
	done < $FILE

	final=""
	for i in $cpu ; do
		t="_${i#?}"
		for j in $ldso ; do
			if [ $i = $j ] ; then
				t=$j
			fi
		done
		final="$final $t"
	done

	if [ "$EN" != "" ] ; then
		final=$(on $EXT $final)
	else
		final=$(off $EXT $final)
	fi

fi

do_ldso $final 

exit 0

=============================================================================

-- System Information
Debian Release: 2.2
Kernel Version: Linux intech19 2.2.19-gen #1 SMP Fri May 25 15:14:23 EDT 2001 i686 unknown




Changed Bug title. Request was from Julian Gilbey <J.D.Gilbey@qmul.ac.uk> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#120418; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Camm Maguire <camm@enhanced.com>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #12 received at 120418@bugs.debian.org (full text, mbox):

From: Camm Maguire <camm@enhanced.com>
To: Michel LESPINASSE <walken@zoy.org>
Cc: Dirk Eddelbuettel <edd@debian.org>, Robert Stone <talby@trap.mtview.ca.us>, Eduard Bloch <edi@gmx.de>, Adam C Powell IV <hazelsct@mit.edu>, bates@debian.org, bcollins@debian.org,120418@bugs.debian.org
Subject: Re: Policy proposal: Cpu extension code
Date: 27 Nov 2001 09:16:11 -0500
Greetings!  I agree as well.  Lets add 6) to the policy proposal.
Thanks for your input!

Michel LESPINASSE <walken@zoy.org> writes:

> Camm,
> 
> I still like your proposal a lot, but I'd like to add this to it:
> 
> 6) Libraries containing CISEs are exempted from the usual requirement
>    to build all other libraries with -fPIC. If the package maintainer
>    decides to take advantage of this on architectures that do not
>    require -fPIC, it is still his/her responsibility to make sure that
>    the library will correctly build on all currently-supported debian
>    architectures, including those that do require -fPIC for correct
>    runtime behaviour. As a suggestion, one possible way to do this is
>    to use libtool with the --without-pic option. One other possibility
>    is to have the library's build system use some special-case for
>    architectures known to not require the -fPIC option, and default to
>    -fPIC for the other ones.
> 
> The rationale for this proposed addition is that -fPIC adds a
> noticeable overhead on some architectures (about 5-10% on
> processor-intensive code on x86, due to the %ebx register being
> unuseable in -fPIC code). For most libraries it still makes most sense
> to use -fPIC so that the code can actually be shared between all users
> of this library, but for some of the CISE libraries (possibly very
> thight, hand-optimized modules for each subarchitecture), that are
> expected to be very CPU intensive, the tradeof is different. I think
> we should leave some freedom to individual package maintainers to make
> the right decision on a case by case basis.
> 
> Also in the case of CISE libraries, I suppose that most maintainers
> will need subarchitecture-specific build rules anyway, to enable the
> various CISEs, so the cost of adding special-case -fPIC handling is
> not very big actually.
> 
> On Mon, Nov 26, 2001 at 07:24:29PM -0500, Camm Maguire wrote:
> > Greetings!  I posted this proposal to debian-policy for discussion in
> > June, but am only now formally requesting its inclusion in the
> > policy.  I've taken the liberty of including recommendations from
> > various submitters.
> > 
> > The goal of this proposal is to provide a mechanism whereby packages
> > can safely make use of cpu instruction set extensions
> > (e.g. sse1,sse2,3dnow,3dnowext, etc.)  In this scheme, packages can
> > install on any machine of the general architecture, and extension
> > instructions  only get executed on appropriately capable cpus, even if
> > /usr is served up over NFS.
> > 
> > Proposal outline:
> > 
> > 1) All packages providing programs and/or libraries which make use of
> >    CPU instruction set extensions (CISEs) must provide identical
> >    functionality on all machines of the given general architecture.
> >    One way to achieve this is for the code to be protected by runtime
> >    probes of the cpu, followed by appropriate branching to correctly
> >    executable sections.  Alternatively, packages can separate routines
> >    using CISEs into shared libraries. This section of the policy
> >    details the machanism whereby ldso will load appropriate versions
> >    of such libraries according to the capabilities of the runtime cpu.
> > 
> > 2) Libraries containing CSIEs should be placed in a designated subdir
> >    of /usr/lib specialized for code requiring the cpu extension in
> >    question.  All such directories should be agreed upon and sanctioned
> >    by the policy committee.  As a suggestion, the directory might be
> >    named after the cpu capability (as reported in /proc/cpuinfo) required
> >    to run the code, (e.g. /usr/lib/{sse,sse2,3dnow,3dnowext} on i386,
> >    /usr/lib/ev5 on alpha, and perhaps /usr/lib/sparc64 on sparc, etc.)
> > 
> > 3) All functionality provided by this code should also be provided by
> >    a binary-equivalent generic shared lib of the same name and soname
> >    in /usr/lib.
> > 
> > 4) Ideally, ldso should be smart enough to search the special
> >    directories first given the running cpu characteristics.  Barring
> >    this, a system script, update-ld.so.conf, should be created, which
> >    packages can call at install/remove time, to add/remove the
> >    specialized paths appropriate to the running cpu to /etc/ld.so.conf
> >    and then run ldconfig.  Optionally (and preferably), this script
> >    could also be run at boot time to catch cpu-upgrades and/or kernel
> >    upgrades (SSE requires kernel support).  Paths need not be added,
> >    and can be removed, if no libraries exist in the directory.  The
> >    package providing this script should be in the 'base' section if
> >    invoked at boot time.  Otherwise, packages using this functionality
> >    should depend upon the update-ld.so.conf package.
> > 
> > 5) For this scheme to work, it is necessary to ensure that ldso loads 
> >    libraries in the directories specified in /etc/ld.so.conf in
> >    preference to identically named versions in /lib or /usr/lib, as is
> >    currently the case with Debian's ldso.  It is part of this policy
> >    that such behavior remain standard on Debian systems.
> > 
> > The Debian atlas packages currently use this scheme.  I believe that
> > at least gmp can currently make use of it as well.  I've provided a
> > sample update-ld.so.conf below.  If this proposal is accepted, I'd be
> > happy to package it or something similar.  
> 
> -- 
> Michel "Walken" LESPINASSE
> Is this the best that god can do ? Then I'm not impressed.
> 
> 

-- 
Camm Maguire			     			camm@enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah



Information forwarded to debian-policy@packages.qa.debian.org:
Bug#120418; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Eduard Bloch <edi@gmx.de>:
Extra info received and filed, but not forwarded. Copy sent to debian-policy@packages.qa.debian.org. Full text and rfc822 format available.

Message #17 received at 120418-quiet@bugs.debian.org (full text, mbox):

From: Eduard Bloch <edi@gmx.de>
To: Wichert Akkerman <wichert@wiggy.net>
Cc: Michael Banck <mbanck@debian.org>, 120418-quiet@bugs.debian.org, apt@packages.debian.org
Subject: Re: Bug#77324: dpkg wish: conditional dependencies
Date: Fri, 10 Jan 2003 13:00:53 +0100
#include <hallo.h>
* Wichert Akkerman [Fri, Jan 10 2003, 12:47:19PM]:
> Previously Eduard Bloch wrote:
> > NOTE: This is not another flamewar about CPU optimised packages, there
> > are packages where the optimisation makes sense and Camms method
> > (#120418) is already used, successfully.
> 
> We have already indicated we want to use another method to do that.

Where? I do not remember a such decission, nor can I find a consens from
a productive discussion, nor do you provide any hints in the last message.
So what's up?

Gruss/Regards,
Eduard.
-- 
Microsoft: Where do you want to go today?
Linux: Where do you want to go tomorrow?
BSD: Are you guys coming or what?



Changed Bug title to `Policy for code using CPU extensions' from `[PROPOSAL] CPU extension code'. Request was from Russ Allbery <rra@debian.org> to control@bugs.debian.org. (Mon, 17 Mar 2008 05:24:11 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Policy List <debian-policy@lists.debian.org>:
Bug#120418; Package debian-policy. Full text and rfc822 format available.

Acknowledgement sent to Clint Adams <schizo@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Policy List <debian-policy@lists.debian.org>. Full text and rfc822 format available.

Message #24 received at 120418@bugs.debian.org (full text, mbox):

From: Clint Adams <schizo@debian.org>
To: camm@enhanced.com, 120418@bugs.debian.org
Subject: CISE and -fPIC
Date: Thu, 5 Jun 2008 19:04:13 +0000
Camm,

Would simply documenting the current glibc behavior with regard to
hwcap be sufficient to address your policy proposal?




Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 19:04:28 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.