Debian Bug report logs - #621500
gem2deb: Gems with external .so and LOAD_PATH issues

version graph

Package: gem2deb; Maintainer for gem2deb is Debian Ruby Extras Maintainers <pkg-ruby-extras-maintainers@lists.alioth.debian.org>; Source for gem2deb is src:gem2deb.

Reported by: David Greaves <david@dgreaves.com>

Date: Thu, 7 Apr 2011 14:06:29 UTC

Severity: important

Found in version 0.2.1-1

Done: Antonio Terceiro <terceiro@softwarelivre.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Bug#621500; Package gem2deb. (Thu, 07 Apr 2011 14:06:37 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Greaves <david@dgreaves.com>:
New Bug report received and forwarded. Copy sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>. (Thu, 07 Apr 2011 14:07:07 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: David Greaves <david@dgreaves.com>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: gem2deb: Gems with external .so and LOAD_PATH issues
Date: Thu, 07 Apr 2011 12:55:20 +0100
Package: gem2deb
Version: 0.2.1-1
Severity: important


Note this is a personally built gem2deb but that shouldn't be a factor here.

The yajl-ruby gem has yajl.rb and yajl.so

I think gem2deb is responsible for putting the extension into a
consistent place; however, since the names are the same I had to patch
my packaging to change the main yajl.rb from:
 require 'yajl/yajl'
to
 require 'yajl.so'

see https://github.com/lbt/yajl-ruby/blob/debian/debian/patches/0001-debian-changes-0.8.1-1.patch

In gem2deb 0.1 it would install to:

/usr/lib/ruby/vendor_ruby/1.8/yajl.rb
/usr/lib/ruby/vendor_ruby/1.8/i486-linux/yajl.so

in 0.2 it installs to:
/usr/lib/ruby/vendor_ruby/yajl.rb
/usr/lib/ruby/vendor_ruby/1.8/i486-linux/yajl.so

Which wouldn't be a problem ... but that

$LOAD_PATH is ["/usr/local/lib/site_ruby/1.8", "/usr/local/lib/site_ruby/1.8/i486-linux", "/usr/local/lib/site_ruby/1.8/i386-linux", "/usr/local/lib/site_ruby", "/usr/lib/ruby/vendor_ruby/1.8", "/usr/lib/ruby/vendor_ruby/1.8/i486-linux", "/usr/lib/ruby/vendor_ruby", "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i486-linux", "/usr/lib/ruby/1.8/i386-linux", "."]

Note that 
  /usr/lib/ruby/vendor_ruby/1.8
precedes
  /usr/lib/ruby/vendor_ruby/1.8/i486-linux

but 
  /usr/lib/ruby/1.8
supercedes it

The result of which is that the initial 'require yajl' now pulls in
the yajl.so which succeeds but of course many methods are "just
missing". (This was not easy to find...)

It would be awfully nice if upstream could call the extension
yajl_ext.so but I wonder if it makes more sense to have all extension
PATH_LOAD entries follow all ruby PATH_LOAD entries by default.


-- System Information:
Debian Release: 6.0.1
  APT prefers stable
  APT policy: (990, 'stable')
Architecture: i386 (x86_64)

Kernel: Linux 2.6.37-trunk-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages gem2deb depends on:
ii  build-essential         11.5             Informational list of build-essent
ii  debhelper               8.0.0            helper programs for debian/rules
ii  devscripts              2.10.69+squeeze1 scripts to make the life of a Debi
ii  perl                    5.10.1-17        Larry Wall's Practical Extraction 
ii  ruby1.8                 1.8.7.302-2      Interpreter of object-oriented scr
ii  ruby1.8-dev             1.8.7.302-2      Header files for compiling extensi
ii  ruby1.9.1               1.9.2.0-2        Interpreter of object-oriented scr
ii  ruby1.9.1-dev           1.9.2.0-2        Header files for compiling extensi
ii  rubygems1.8             1.3.7-3          package management framework for R

gem2deb recommends no packages.

gem2deb suggests no packages.

-- no debconf information




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#621500; Package gem2deb. (Sun, 10 Apr 2011 04:53:51 GMT) Full text and rfc822 format available.

Acknowledgement sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Extra info received and forwarded to list. (Sun, 10 Apr 2011 04:53:56 GMT) Full text and rfc822 format available.

Message #10 received at 621500@bugs.debian.org (full text, mbox):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>
To: David Greaves <david@dgreaves.com>, 621500@bugs.debian.org
Subject: Re: Bug#621500: gem2deb: Gems with external .so and LOAD_PATH issues
Date: Sat, 9 Apr 2011 13:10:53 +0200
On 07/04/11 at 12:55 +0100, David Greaves wrote:
> Package: gem2deb
> Version: 0.2.1-1
> Severity: important
> 
> 
> Note this is a personally built gem2deb but that shouldn't be a factor here.
> 
> The yajl-ruby gem has yajl.rb and yajl.so
> 
> I think gem2deb is responsible for putting the extension into a
> consistent place; however, since the names are the same I had to patch
> my packaging to change the main yajl.rb from:
>  require 'yajl/yajl'
> to
>  require 'yajl.so'
> 
> see https://github.com/lbt/yajl-ruby/blob/debian/debian/patches/0001-debian-changes-0.8.1-1.patch
> 
> In gem2deb 0.1 it would install to:
> 
> /usr/lib/ruby/vendor_ruby/1.8/yajl.rb
> /usr/lib/ruby/vendor_ruby/1.8/i486-linux/yajl.so
> 
> in 0.2 it installs to:
> /usr/lib/ruby/vendor_ruby/yajl.rb
> /usr/lib/ruby/vendor_ruby/1.8/i486-linux/yajl.so
> 
> Which wouldn't be a problem ... but that
> 
> $LOAD_PATH is ["/usr/local/lib/site_ruby/1.8", "/usr/local/lib/site_ruby/1.8/i486-linux", "/usr/local/lib/site_ruby/1.8/i386-linux", "/usr/local/lib/site_ruby", "/usr/lib/ruby/vendor_ruby/1.8", "/usr/lib/ruby/vendor_ruby/1.8/i486-linux", "/usr/lib/ruby/vendor_ruby", "/usr/lib/ruby/1.8", "/usr/lib/ruby/1.8/i486-linux", "/usr/lib/ruby/1.8/i386-linux", "."]
> 
> Note that 
>   /usr/lib/ruby/vendor_ruby/1.8
> precedes
>   /usr/lib/ruby/vendor_ruby/1.8/i486-linux
> 
> but 
>   /usr/lib/ruby/1.8
> supercedes it
> 
> The result of which is that the initial 'require yajl' now pulls in
> the yajl.so which succeeds but of course many methods are "just
> missing". (This was not easy to find...)
> 
> It would be awfully nice if upstream could call the extension
> yajl_ext.so but I wonder if it makes more sense to have all extension
> PATH_LOAD entries follow all ruby PATH_LOAD entries by default.

I think that the correct fix is to install the extension to 
/usr/lib/ruby/vendor_ruby/1.8/i486-linux/yagl/yajl.so
But I'm not sure what's the good way to do that...

Would it make sense to simply move the .so file after the build?

- Lucas




Information forwarded to debian-bugs-dist@lists.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Bug#621500; Package gem2deb. (Wed, 20 Apr 2011 00:21:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bryan McLellan <btm@loftninjas.org>:
Extra info received and forwarded to list. Copy sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>. (Wed, 20 Apr 2011 00:21:03 GMT) Full text and rfc822 format available.

Message #15 received at 621500@bugs.debian.org (full text, mbox):

From: Bryan McLellan <btm@loftninjas.org>
To: 621500@bugs.debian.org
Subject: override file for extension path
Date: Tue, 19 Apr 2011 17:19:19 -0700
I hacked a config file in to specify an override file for where to
place the extension. I'm not sure if there are any other packages with
this issue or not, so it is hard to gauge the appropriate fix. With
this change to gem2deb, adding a debian/extension_path.overrides file
that contains 'yajl' produces a good yajl-ruby deb using gem2deb for
me.

Note that somewhere along the line the vendor_ruby/VERSION/ARCH
directory included in the $LOAD_PATH for i386 changed, which caused me
a little trouble debugging.

/usr/lib/ruby/vendor_ruby/1.8/i486-linux (lucid)
/usr/lib/ruby/vendor_ruby/1.8/i686-linux (maverick)

Bryan

https://github.com/btm/gem2deb/tree/621500
https://github.com/btm/gem2deb/commit/e133ee81c27176889cb3cacf19e0f95ee27b1607

commit e133ee81c27176889cb3cacf19e0f95ee27b1607
Author: Bryan McLellan <btm@opscode.com>
Date:   Tue Apr 19 18:39:09 2011 +0000

    621500: add override for extension path subdirectories

diff --git a/lib/gem2deb/extension_builder.rb b/lib/gem2deb/extension_builder.rb
index bd09e6b..1938541 100644
--- a/lib/gem2deb/extension_builder.rb
+++ b/lib/gem2deb/extension_builder.rb
@@ -57,7 +57,12 @@ module Gem2Deb
           exit(1)
         end
       begin
-        target = File.expand_path(File.join('debian', @package,
RbConfig::CONFIG['vendorarchdir']))
+        if File::exists?('debian/extension_path.overrides')
+          extension_path = File.new('debian/extension_path.overrides').gets
+          target = File.expand_path(File.join('debian', @package,
RbConfig::CONFIG['vendorarchdir'], extension_path))
+        else
+          target = File.expand_path(File.join('debian', @package,
RbConfig::CONFIG['vendorarchdir']))
+        end
         Dir.chdir(directory) do
           rubygems_builder.build(extension, '.', target, results)
           puts results




Reply sent to Antonio Terceiro <terceiro@softwarelivre.org>:
You have taken responsibility. (Wed, 20 Apr 2011 04:00:03 GMT) Full text and rfc822 format available.

Notification sent to David Greaves <david@dgreaves.com>:
Bug acknowledged by developer. (Wed, 20 Apr 2011 04:00:03 GMT) Full text and rfc822 format available.

Message #20 received at 621500-done@bugs.debian.org (full text, mbox):

From: Antonio Terceiro <terceiro@softwarelivre.org>
To: Bryan McLellan <btm@loftninjas.org>, 621500-done@bugs.debian.org
Subject: Re: Bug#621500: override file for extension path
Date: Tue, 19 Apr 2011 20:49:36 -0700
[Message part 1 (text/plain, inline)]
Bryan McLellan escreveu isso aí:
> I hacked a config file in to specify an override file for where to
> place the extension. I'm not sure if there are any other packages with
> this issue or not, so it is hard to gauge the appropriate fix. With
> this change to gem2deb, adding a debian/extension_path.overrides file
> that contains 'yajl' produces a good yajl-ruby deb using gem2deb for
> me.
> 
> Note that somewhere along the line the vendor_ruby/VERSION/ARCH
> directory included in the $LOAD_PATH for i386 changed, which caused me
> a little trouble debugging.
> 
> /usr/lib/ruby/vendor_ruby/1.8/i486-linux (lucid)
> /usr/lib/ruby/vendor_ruby/1.8/i686-linux (maverick)

I dont't think we should hack gem2deb to cope with a broken upstream
buildsystem:

terceiro@morere:/tmp/yajl-ruby/ext/yajl (master)$ make install DESTDIR=/tmp/bli
mkdir -p /tmp/bli/usr/local/lib/site_ruby/1.8/i486-linux
/usr/bin/install -c -m 0755 yajl.so
/tmp/bli/usr/local/lib/site_ruby/1.8/i486-linux
terceiro@morere:/tmp/yajl-ruby/ext/yajl (master)$ find /tmp/bli/
/tmp/bli/
/tmp/bli/usr
/tmp/bli/usr/local
/tmp/bli/usr/local/lib
/tmp/bli/usr/local/lib/site_ruby
/tmp/bli/usr/local/lib/site_ruby/1.8
/tmp/bli/usr/local/lib/site_ruby/1.8/i486-linux
/tmp/bli/usr/local/lib/site_ruby/1.8/i486-linux/yajl.so

so yajl.so is supposed to be in a directory called 'yajl', but
upstream's `make install` does not put it there.

It only works with rubygems because rubygems does not call make install,
it just add whatever paths the gem author says to the $LOAD_PATH.

mkmf actually documents the correct way of putting extentions in
subdirectories:

# Generates the Makefile for your extension, passing along any options and
# preprocessor constants that you may have generated through other methods.
#
# The +target+ name should correspond the name of the global function name
# defined within your C extension, minus the 'Init_'.  For example, if your
# C extension is defined as 'Init_foo', then your target would simply be 'foo'.
#
# If any '/' characters are present in the target name, only the last name
# is interpreted as the target name, and the rest are considered toplevel
# directory names, and the generated Makefile will be altered accordingly to
# follow that directory structure.
#
# For example, if you pass 'test/foo' as a target name, your extension will
# be installed under the 'test' directory.  This means that in order to
# load the file within a Ruby program later, that directory structure will
# have to be followed, e.g. "require 'test/foo'".
#
# The +srcprefix+ should be used when your source files are not in the same
# directory as your build script. This will not only eliminate the need for
# you to manually copy the source files into the same directory as your build
# script, but it also sets the proper +target_prefix+ in the generated
# Makefile.
#
# Setting the +target_prefix+ will, in turn, install the generated binary in
# a directory under your Config::CONFIG['sitearchdir'] that mimics your local
# filesystem when you run 'make install'.
#
# For example, given the following file tree:
#
#    ext/
#       extconf.rb
#       test/
#          foo.c
#
# And given the following code:
#
#    create_makefile('test/foo', 'test')
#
# That will set the +target_prefix+ in the generated Makefile to 'test'. That,
# in turn, will create the following file tree when installed via the
# 'make install' command:
#
#    /path/to/ruby/sitearchdir/test/foo.so
#
# It is recommended that you use this approach to generate your makefiles,
# instead of copying files around manually, because some third party
# libraries may depend on the +target_prefix+ being set properly.
#
# The +srcprefix+ argument can be used to override the default source
# directory, i.e. the current directory . It is included as part of the VPATH
# and added to the list of INCFLAGS.
#
def create_makefile(target, srcprefix = nil)
[...]

Actually, with the attached patch, yajl-ruby installs to correct place:

terceiro@morere:/tmp/yajl-ruby/ext (master)$ make install DESTDIR=/tmp/flock
mkdir -p /tmp/flock/usr/local/lib/site_ruby/1.8/i486-linux/yajl
/usr/bin/install -c -m 0755 yajl.so /tmp/flock/usr/local/lib/site_ruby/1.8/i486-linux/yajl
terceiro@morere:/tmp/yajl-ruby/ext (master)$ find /tmp/flock/
/tmp/flock/
/tmp/flock/usr
/tmp/flock/usr/local
/tmp/flock/usr/local/lib
/tmp/flock/usr/local/lib/site_ruby
/tmp/flock/usr/local/lib/site_ruby/1.8
/tmp/flock/usr/local/lib/site_ruby/1.8/i486-linux
/tmp/flock/usr/local/lib/site_ruby/1.8/i486-linux/yajl
/tmp/flock/usr/local/lib/site_ruby/1.8/i486-linux/yajl/yajl.so
terceiro@morere:/tmp/yajl-ruby/ext (master)$ 

-- 
Antonio Terceiro <terceiro@softwarelivre.org>
http://softwarelivre.org/terceiro


[0001-Fix-native-extention-installation-path.patch (text/x-diff, attachment)]
[signature.asc (application/pgp-signature, inline)]

Message #21 received at 621500-done@bugs.debian.org (full text, mbox):

From: Antonio Terceiro <terceiro@softwarelivre.org>
To: Bryan McLellan <btm@loftninjas.org>, 621500-done@bugs.debian.org
Subject: Re: Bug#621500: override file for extension path
Date: Tue, 19 Apr 2011 21:22:42 -0700
[Message part 1 (text/plain, inline)]
Antonio Terceiro escreveu isso aí:
> Actually, with the attached patch, yajl-ruby installs to correct place:

This other patch fixes the problem in a even better (an unintrusive)
way.

-- 
Antonio Terceiro <terceiro@softwarelivre.org>
http://softwarelivre.org/terceiro


[0001-Fix-installation-path-of-native-extension.patch (text/x-diff, attachment)]
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Bug#621500; Package gem2deb. (Wed, 20 Apr 2011 17:27:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bryan McLellan <btm@loftninjas.org>:
Extra info received and forwarded to list. Copy sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>. (Wed, 20 Apr 2011 17:27:05 GMT) Full text and rfc822 format available.

Message #26 received at 621500@bugs.debian.org (full text, mbox):

From: Bryan McLellan <btm@loftninjas.org>
To: Antonio Terceiro <terceiro@softwarelivre.org>
Cc: 621500 <621500@bugs.debian.org>
Subject: Re: Bug#621500: override file for extension path
Date: Wed, 20 Apr 2011 10:25:43 -0700
On Tue, Apr 19, 2011 at 9:22 PM, Antonio Terceiro
<terceiro@softwarelivre.org> wrote:
> This other patch fixes the problem in a even better (an unintrusive)
> way.

Is this a problem that is unique to ruby-yajl, or one that may come up
for other packages?

It's possible we can get upstream to rename the extension, but I
wonder if this is something that gem2deb should know how to deal with.

Bryan




Information forwarded to debian-bugs-dist@lists.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Bug#621500; Package gem2deb. (Wed, 20 Apr 2011 18:00:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Antonio Terceiro <terceiro@softwarelivre.org>:
Extra info received and forwarded to list. Copy sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>. (Wed, 20 Apr 2011 18:00:03 GMT) Full text and rfc822 format available.

Message #31 received at 621500@bugs.debian.org (full text, mbox):

From: Antonio Terceiro <terceiro@softwarelivre.org>
To: Bryan McLellan <btm@loftninjas.org>
Cc: 621500 <621500@bugs.debian.org>
Subject: Re: Bug#621500: override file for extension path
Date: Wed, 20 Apr 2011 10:49:14 -0700
[Message part 1 (text/plain, inline)]
Bryan McLellan escreveu isso aí:
> On Tue, Apr 19, 2011 at 9:22 PM, Antonio Terceiro
> <terceiro@softwarelivre.org> wrote:
> > This other patch fixes the problem in a even better (an unintrusive)
> > way.
> 
> Is this a problem that is unique to ruby-yajl, or one that may come up
> for other packages?

This problem will happen with any upstream packages that do not build
correctly, and is not specific to Ruby of gem2deb. In such cases, we
must help upstream to fix their build system, and not add workarounds to
our tools.

> It's possible we can get upstream to rename the extension, but I
> wonder if this is something that gem2deb should know how to deal with.

Please read the last patch I sent. It is a one-line change, and does not
require renaming the extension at all. I can't imagine a reason why
upstream would reject it.

-- 
Antonio Terceiro <terceiro@softwarelivre.org>
http://softwarelivre.org/terceiro


[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Lucas Nussbaum <lucas@lucas-nussbaum.net>:
Bug#621500; Package gem2deb. (Wed, 20 Apr 2011 18:42:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Bryan McLellan <btm@loftninjas.org>:
Extra info received and forwarded to list. Copy sent to Lucas Nussbaum <lucas@lucas-nussbaum.net>. (Wed, 20 Apr 2011 18:42:05 GMT) Full text and rfc822 format available.

Message #36 received at 621500@bugs.debian.org (full text, mbox):

From: Bryan McLellan <btm@loftninjas.org>
To: Antonio Terceiro <terceiro@softwarelivre.org>
Cc: 621500 <621500@bugs.debian.org>
Subject: Re: Bug#621500: override file for extension path
Date: Wed, 20 Apr 2011 11:38:46 -0700
On Wed, Apr 20, 2011 at 10:49 AM, Antonio Terceiro
<terceiro@softwarelivre.org> wrote:
> This problem will happen with any upstream packages that do not build
> correctly, and is not specific to Ruby of gem2deb. In such cases, we
> must help upstream to fix their build system, and not add workarounds to
> our tools.

This is reasonable to me, I was just wondering if there was a pattern
that we could workaround that could save us work.

> Please read the last patch I sent. It is a one-line change, and does not
> require renaming the extension at all. I can't imagine a reason why
> upstream would reject it.

I did, and did some testing around it as well. It works for me. I'm
somewhat curious about the original reporters issue where it sounds
like the wrong extension may have been chosen(?) In all of my testing
the extension simply wasn't found.

I've filed the bug upstream with your patch.

https://github.com/brianmario/yajl-ruby/issues/56

Bryan




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Thu, 19 May 2011 07:31:49 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sat Apr 19 12:45:38 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.