Debian Bug report logs - #774213
zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Kirill Smelkov <kirr@nexedi.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Tue, 30 Dec 2014 14:32:17 +0400

Package: zope2.13
Version: 2.13.22-1
Severity: grave
Justification: renders package unusable

Dear Maintainer,

With zope2.13 I've tried to create a (user) instance and start it, but a
`SystemError: dynamic module not initialized properly` is raised while
zopectl tries to import zope.security._proxy .

The bug could be demonstrated in essence this way:

    $ /usr/lib/zope2.13/bin/python 
    
    >>> import zope.security._proxy
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
        from zope.security.management import checkPermission
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
        from zope.security.checker import CheckerPublic
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
        from zope.security._proxy import _Proxy as Proxy, getChecker
    SystemError: dynamic module not initialized properly


Full log demonstrating the problem is attached.

Thanks,
Kirill

---- 8< ----
$ whoami 
kirr            # non-root

$ pwd
/home/kirr

$ rm -rf zope   # make sure there are no user instances

$ dzhandle -z2.13 make-instance -m all ktest -u user:passwd
$ dzhandle -z2.13 list-instances 
ktest                2.13   addon-mode=all addon-technique=tree-linked userfile=inituser purged

$ ls zope/instance/zope2.13/
ktest

$ dzhandle -z2.13 zopectl ktest start
Traceback (most recent call last):
  File "/usr/lib/zope2.13/bin/python", line 78, in <module>
    execfile(__file__)
  File "/usr/lib/zope2.13/bin/zopectl", line 6, in <module>
    Zope2.Startup.zopectl.run()
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/Zope2/Startup/zopectl.py", line 484, in run
    exitstatus = main()
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/Zope2/Startup/zopectl.py", line 440, in main
    options.realize(args)
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/Zope2/Startup/zopectl.py", line 158, in realize
    ZDOptions.realize(self, *args, **kw)
  File "/usr/lib/zope2.13/lib/python/zdaemon-2.0.7.egg/zdaemon/zdoptions.py", line 290, in realize
    self.load_schema()
  File "/usr/lib/zope2.13/lib/python/zdaemon-2.0.7.egg/zdaemon/zdoptions.py", line 338, in load_schema
    self.schema = ZConfig.loadSchema(self.schemafile)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/loader.py", line 33, in loadSchema
    return SchemaLoader().loadURL(url)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/loader.py", line 67, in loadURL
    return self.loadResource(r)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/loader.py", line 195, in loadResource
    schema = ZConfig.schema.parseResource(resource, self)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/schema.py", line 27, in parseResource
    xml.sax.parse(resource.file, parser)
  File "/usr/lib/python2.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 210, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 304, in start_element
    self._cont_handler.startElement(name, AttributesImpl(attrs))
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/schema.py", line 103, in startElement
    getattr(self, "start_" + name)(attrs)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/schema.py", line 306, in start_import
    src = self._loader.schemaComponentSource(pkg, file)
  File "/usr/lib/zope2.13/lib/python/ZConfig-2.9.1.egg/ZConfig/loader.py", line 212, in schemaComponentSource
    __import__(package)
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/ZServer/__init__.py", line 41, in <module>
    from HTTPServer import zhttp_server, zhttp_handler
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/ZServer/HTTPServer.py", line 44, in <module>
    from HTTPResponse import make_response
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/ZServer/HTTPResponse.py", line 29, in <module>
    from ZPublisher.HTTPResponse import HTTPResponse # XXX WTF?
  File "/usr/lib/zope2.13/lib/python/Zope2-2.13.22.egg/ZPublisher/__init__.py", line 20, in <module>
    from zExceptions import NotFound, BadRequest, InternalError, Forbidden
  File "/usr/lib/zope2.13/lib/python/zExceptions-2.13.0.egg/zExceptions/__init__.py", line 24, in <module>
    from zope.publisher.interfaces import INotFound
  File "/usr/lib/zope2.13/lib/python/zope.publisher-3.12.6.egg/zope/publisher/interfaces/__init__.py", line 26, in <module>
    from zope.security.interfaces import IParticipation
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
    from zope.security.management import checkPermission
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
    from zope.security.checker import CheckerPublic
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
    from zope.security._proxy import _Proxy as Proxy, getChecker
SystemError: dynamic module not initialized properly
---- 8< ----




-- System Information:
Debian Release: 8.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)

Versions of packages zope2.13 depends on:
ii  debconf [debconf-2.0]  1.5.55
ii  libc6                  2.19-13
ii  lsb-base               4.1+Debian13+nmu1
ii  python-docutils        0.12+dfsg-1
ii  python-mechanize       1:0.2.5-3
ii  python-pkg-resources   5.5.1-1
ii  python-tz              2012c+dfsg-0.1
ii  python2.7              2.7.8-11
pn  python:any             <none>
ii  zope-common            0.5.54

zope2.13 recommends no packages.

Versions of packages zope2.13 suggests:
ii  python [python-profiler]  2.7.8-2
pn  python-unit               <none>

-- no debconf information

Message #10 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: pkg-zope-developers@lists.alioth.debian.org

Cc: Kirill Smelkov <kirr@nexedi.com>, 774213@bugs.debian.org

Subject: Re: Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Wed, 7 Jan 2015 13:08:33 -0500

On Dec 30, 2014, at 02:32 PM, Kirill Smelkov wrote:

>Package: zope2.13
>Version: 2.13.22-1
>Severity: grave
>Justification: renders package unusable
>
>With zope2.13 I've tried to create a (user) instance and start it, but a
>`SystemError: dynamic module not initialized properly` is raised while
>zopectl tries to import zope.security._proxy .

I've tried both recipes on two different sid machines and I am unable to
reproduce the problem.  Importing zope.security._proxy succeeds for me, as
does the dzhandle [...] start command.

Perhaps it's a local problem?  Try re-installing zope.security?

Cheers,
-Barry

Message #17 received at 774213-done@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 774213-done@bugs.debian.org

Date: Thu, 8 Jan 2015 10:14:55 -0500

[Message part 1 (text/plain, inline)]

As described in Message #10, I cannot reproduce this, so I am closing this
issue.  If you re-open it, please provide some additional information to help
diagnose and reproduce the problem.

[Message part 2 (application/pgp-signature, inline)]

Message #22 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Kirill Smelkov <kirr@nexedi.com>

To: Barry Warsaw <barry@debian.org>

Cc: pkg-zope-developers@lists.alioth.debian.org, 774213@bugs.debian.org, control@bugs.debian.org

Subject: Re: Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Fri, 9 Jan 2015 23:29:26 +0300

reopen 774213
thanks

On Wed, Jan 07, 2015 at 01:08:33PM -0500, Barry Warsaw wrote:
> On Dec 30, 2014, at 02:32 PM, Kirill Smelkov wrote:
> 
> >Package: zope2.13
> >Version: 2.13.22-1
> >Severity: grave
> >Justification: renders package unusable
> >
> >With zope2.13 I've tried to create a (user) instance and start it, but a
> >`SystemError: dynamic module not initialized properly` is raised while
> >zopectl tries to import zope.security._proxy .
> 
> I've tried both recipes on two different sid machines and I am unable to
> reproduce the problem.  Importing zope.security._proxy succeeds for me, as
> does the dzhandle [...] start command.
> 
> Perhaps it's a local problem?  Try re-installing zope.security?

After reinstalling zope.security (= reinstalling zope2.13 because that
mega-package contains all the dependencies) and finding nothing changed,
I think I've found the cause - the bug relates to other python-zope.*
packages being installed or not:

If there is no python-zope.* packages installed we have:

    $ /usr/lib/zope2.13/bin/python 
    >>> import zope.security._proxy
    >>> # imported ok

But if we install python-zope.proxy (or thus any package which depends
on it, e.g. python-zodb) we'll have:

    $ sudo aptitude install -y python-zope.proxy
    $ /usr/lib/zope2.13/bin/python
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
        from zope.security.management import checkPermission
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
        from zope.security.checker import CheckerPublic
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
        from zope.security._proxy import _Proxy as Proxy, getChecker
    SystemError: dynamic module not initialized properly

and to me this is reproducible 100% - I've tried to remove/install
python-zope.proxy several times and each time the import outcome was
consistent - ok, if no python-zope.proxy installed, SystemError if
installed.

So it looks like the bug relates to zope2.13 being packaged not the
Debian-way - e.g. it have all the dependencies inside it (zope.proxy,
zodb, etc...) instead of using system packages and that shipped
libraries somehow are not 100% isolated from the usual environment.

Thanks,
Kirill

Message #29 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: 774213@bugs.debian.org

Date: Mon, 12 Jan 2015 15:40:36 -0500

Something else must still be going on.  I have the following zope packages
installed:

% aptitude search zope | grep ^i
i   python-zope.component           - Zope Component Architecture               
i   python-zope.configuration       - Zope Configuration Markup Language (ZCML) 
i   python-zope.deprecation         - Zope Deprecation Infrastructure           
i   python-zope.event               - Very basic event publishing system        
i   python-zope.exceptions          - Zope exceptions for Python                
i   python-zope.i18nmessageid       - Message Identifiers for internationalizati
i A python-zope.interface           - Interfaces for Python                     
i   python-zope.location            - Tools for working with object locations   
i   python-zope.proxy               - Generic transparent proxies for Python    
i   python-zope.schema              - zope.interface extension for defining data
i   python-zope.security            - Zope Security Framework                   
i   python-zope.testing             - Zope testing helpers                      
i   python-zope.testrunner          - Flexible test runner with layer support fo
i   python3-zope.component          - Zope Component Architecture               
i   python3-zope.configuration      - Zope Configuration Markup Language (ZCML) 
i   python3-zope.deprecation        - Zope Deprecation Infrastructure           
i   python3-zope.event              - Very basic event publishing system        
i   python3-zope.exceptions         - Zope exceptions for Python 3              
i   python3-zope.i18nmessageid      - Message Identifiers for internationalizati
i   python3-zope.interface          - Interfaces for Python3                    
i   python3-zope.location           - Tools for working with object locations   
i   python3-zope.proxy              - Generic transparent proxies for Python    
i   python3-zope.schema             - zope.interface extension for defining data
i   python3-zope.testing            - Zope testing helpers for Python 3         
i   python3-zope.testrunner         - Flexible test runner with layer support fo
i A zope-common                     - common settings and scripts for Zope insta
i   zope2.13                        - Open Source Web Application Server

so definitely python-zope.security, python-zope.proxy, and zope2.13.

and yet...

% /usr/lib/zope2.13/bin/python 

>>> import zope.security._proxy
>>> zope.security._proxy.__file__
'/usr/lib/python2.7/dist-packages/zope/security/_proxy.x86_64-linux-gnu.so'

Message #34 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Kirill Smelkov <kirr@nexedi.com>

To: Barry Warsaw <barry@debian.org>

Cc: 774213@bugs.debian.org

Subject: Re: your mail

Date: Tue, 13 Jan 2015 16:16:09 +0300

On Mon, Jan 12, 2015 at 03:40:36PM -0500, Barry Warsaw wrote:
> Something else must still be going on.  I have the following zope packages
> installed:
> 
> % aptitude search zope | grep ^i
> i   python-zope.component           - Zope Component Architecture               
> i   python-zope.configuration       - Zope Configuration Markup Language (ZCML) 
> i   python-zope.deprecation         - Zope Deprecation Infrastructure           
> i   python-zope.event               - Very basic event publishing system        
> i   python-zope.exceptions          - Zope exceptions for Python                
> i   python-zope.i18nmessageid       - Message Identifiers for internationalizati
> i A python-zope.interface           - Interfaces for Python                     
> i   python-zope.location            - Tools for working with object locations   
> i   python-zope.proxy               - Generic transparent proxies for Python    
> i   python-zope.schema              - zope.interface extension for defining data
> i   python-zope.security            - Zope Security Framework                   
> i   python-zope.testing             - Zope testing helpers                      
> i   python-zope.testrunner          - Flexible test runner with layer support fo
> i   python3-zope.component          - Zope Component Architecture               
> i   python3-zope.configuration      - Zope Configuration Markup Language (ZCML) 
> i   python3-zope.deprecation        - Zope Deprecation Infrastructure           
> i   python3-zope.event              - Very basic event publishing system        
> i   python3-zope.exceptions         - Zope exceptions for Python 3              
> i   python3-zope.i18nmessageid      - Message Identifiers for internationalizati
> i   python3-zope.interface          - Interfaces for Python3                    
> i   python3-zope.location           - Tools for working with object locations   
> i   python3-zope.proxy              - Generic transparent proxies for Python    
> i   python3-zope.schema             - zope.interface extension for defining data
> i   python3-zope.testing            - Zope testing helpers for Python 3         
> i   python3-zope.testrunner         - Flexible test runner with layer support fo
> i A zope-common                     - common settings and scripts for Zope insta
> i   zope2.13                        - Open Source Web Application Server
> 
> so definitely python-zope.security, python-zope.proxy, and zope2.13.
                ^^^^^^^^^^^^^^^^^^^^


> and yet...
> 
> % /usr/lib/zope2.13/bin/python 
> 
> >>> import zope.security._proxy
> >>> zope.security._proxy.__file__
> '/usr/lib/python2.7/dist-packages/zope/security/_proxy.x86_64-linux-gnu.so'

I think I'm starting to understand: I do _not_ have python-zope.security
installed (because nothing depends on it including zope2.13).

    clean@teco:~$ dpkg -l |grep zope
    ii  python-zope.interface                 4.1.1-3.1
    ii  python-zope.proxy                     4.1.4-2
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

If I install python-zope.security, then import goes ok:

    clean@teco:~$ dpkg -l |grep zope
    ii  python-zope.component                 4.2.1-2
    ii  python-zope.event                     4.0.3-2
    ii  python-zope.i18nmessageid             4.0.3-1
    ii  python-zope.interface                 4.1.1-3.1
    ii  python-zope.location                  4.0.3-1
    ii  python-zope.proxy                     4.1.4-2
    ii  python-zope.schema                    4.4.2-1
    ii  python-zope.security                  4.0.1-1
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

    clean@teco:~$ /usr/lib/zope2.13/bin/python 

    >>> import zope.security._proxy
    >>> zope.security._proxy.__file__
    '/usr/lib/python2.7/dist-packages/zope/security/_proxy.x86_64-linux-gnu.so'

but the imported module is NOT from zope2.13 package, as it should be:

    clean@teco:~$ dpkg -L zope2.13 |grep security |grep proxy.so
    /usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/_proxy.so


after purging python-zope.security import fails, but in a new way:

    clean@teco:~$ dpkg -l |grep zope
    ii  python-zope.interface                 4.1.1-3.1
    ii  python-zope.proxy                     4.1.4-2
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

    clean@teco:~$ /usr/lib/zope2.13/bin/python 
    
    >>> import zope.security._proxy
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    ImportError: No module named security._proxy

and if I remove and then install zope2.13 (with python-zope.security
still being removed) it is still the same:

    clean@teco:~$ dpkg -l |grep zope
    ii  python-zope.interface                 4.1.1-3.1
    ii  python-zope.proxy                     4.1.4-2
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

    clean@teco:~$ /usr/lib/zope2.13/bin/python 
    
    >>> import zope.security._proxy
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    ImportError: No module named security._proxy

and if then I remove python-zope.proxy:

    clean@teco:~$ dpkg -l |grep zope
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

    clean@teco:~$ /usr/lib/zope2.13/bin/python 

    >>> import zope.security._proxy
    >>> zope.security._proxy.__file__
    '/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/_proxy.so'
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    NOTE - different from /usr/lib/python2.7/dist-packages/zope/security/_proxy.x86_64-linux-gnu.so'

and if I install python-zope.proxy again:

    clean@teco:~$ dpkg -l |grep zope
    ii  python-zope.interface                 4.1.1-3.1
    ii  python-zope.proxy                     4.1.4-2
    ii  zope-common                           0.5.54
    ii  zope2.13                              2.13.22-1

    clean@teco:~$ /usr/lib/zope2.13/bin/python 
    
    >>> import zope.security._proxy
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
        from zope.security.management import checkPermission
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
        from zope.security.checker import CheckerPublic
      File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
        from zope.security._proxy import _Proxy as Proxy, getChecker
    SystemError: dynamic module not initialized properly


oops.

Conclusion: not enough isolation leads to either flaky behaviour (i.e.
zope.security imports, but version is  4.0.1-1 instead of expected by
zope2.13 3.7.4) or import failing at all.

And there is some hidden state which depends on packages installation
order etc...

So sad,
Kirill

Message #39 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Jonathan Wiltshire <jmw@debian.org>

To: Kirill Smelkov <kirr@nexedi.com>, 774213@bugs.debian.org

Subject: Re: Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Sun, 18 Jan 2015 17:41:10 +0000

[Message part 1 (text/plain, inline)]

Control: severity -1 normal
Control: tag -1 - unreproducible

Hi,

On Tue, Dec 30, 2014 at 02:32:17PM +0400, Kirill Smelkov wrote:
> With zope2.13 I've tried to create a (user) instance and start it, but a
> `SystemError: dynamic module not initialized properly` is raised while
> zopectl tries to import zope.security._proxy .
> 
> The bug could be demonstrated in essence this way:
> 
>     $ /usr/lib/zope2.13/bin/python 
>     
>     >>> import zope.security._proxy
>     Traceback (most recent call last):
>       File "<console>", line 1, in <module>
>       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
>         from zope.security.management import checkPermission
>       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
>         from zope.security.checker import CheckerPublic
>       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
>         from zope.security._proxy import _Proxy as Proxy, getChecker
>     SystemError: dynamic module not initialized properly

We have been working on this as part of a BSP and have some findings.

The problem appears to be when zope2.13, and python-zope.proxy are
installed but NOT python-zope.security. In that situation:

import zope.security
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
    from zope.security.management import checkPermission
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
    from zope.security.checker import CheckerPublic
  File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
    from zope.security._proxy import _Proxy as Proxy, getChecker
SystemError: dynamic module not initialized properly

It seems that zope.security gets loaded from the bundled copy, and
zope.proxy from the system. This explodes loudly.

If both or neither external packages are installed, everything is fine.

The reporter got into this situation by installing python-zope.security,
which has a dependency on python-zope.proxy, and then removing
python-zope.security. python-zope.proxy therefore stays around, causing the
SystemError.

Downgrading since it all works fine if both extensions are bundled or both
extensions are external, and 'apt install python-zope.security' does the
right thing.

I notice that the external packages are not the same versions as their
bundled counterparts, so perhaps an interface changed at some point or a
rebuild is needed or something.


-- 
Jonathan Wiltshire                                      jmw@debian.org
Debian Developer                         http://people.debian.org/~jmw

4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC  74C3 5394 479D D352 4C51

[signature.asc (application/pgp-signature, inline)]

Message #48 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Kirill Smelkov <kirr@nexedi.com>

To: Jonathan Wiltshire <jmw@debian.org>

Cc: 774213@bugs.debian.org

Subject: Re: Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Sun, 18 Jan 2015 23:48:57 +0300

On Sun, Jan 18, 2015 at 05:41:10PM +0000, Jonathan Wiltshire wrote:
> Control: severity -1 normal
> Control: tag -1 - unreproducible
> 
> Hi,
> 
> On Tue, Dec 30, 2014 at 02:32:17PM +0400, Kirill Smelkov wrote:
> > With zope2.13 I've tried to create a (user) instance and start it, but a
> > `SystemError: dynamic module not initialized properly` is raised while
> > zopectl tries to import zope.security._proxy .
> > 
> > The bug could be demonstrated in essence this way:
> > 
> >     $ /usr/lib/zope2.13/bin/python 
> >     
> >     >>> import zope.security._proxy
> >     Traceback (most recent call last):
> >       File "<console>", line 1, in <module>
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
> >         from zope.security.management import checkPermission
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
> >         from zope.security.checker import CheckerPublic
> >       File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
> >         from zope.security._proxy import _Proxy as Proxy, getChecker
> >     SystemError: dynamic module not initialized properly
> 
> We have been working on this as part of a BSP and have some findings.
> 
> The problem appears to be when zope2.13, and python-zope.proxy are
> installed but NOT python-zope.security. In that situation:
> 
> import zope.security
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/__init__.py", line 19, in <module>
>     from zope.security.management import checkPermission
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/management.py", line 23, in <module>
>     from zope.security.checker import CheckerPublic
>   File "/usr/lib/zope2.13/lib/python/zope.security-3.7.4.egg/zope/security/checker.py", line 46, in <module>
>     from zope.security._proxy import _Proxy as Proxy, getChecker
> SystemError: dynamic module not initialized properly
> 
> It seems that zope.security gets loaded from the bundled copy, and
> zope.proxy from the system. This explodes loudly.
> 
> If both or neither external packages are installed, everything is fine.

Thanks for looking into this. I've also digged a bit previously and my
findings were posted in the bugtracker:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774213#34

In short (quoting from the detailing mail):

"""
Conclusion: not enough isolation leads to either flaky behaviour (i.e.
zope.security imports, but version is  4.0.1-1 instead of expected by
zope2.13 3.7.4) or import failing at all.

And there is some hidden state which depends on packages installation
order etc...
"""

> The reporter got into this situation by installing python-zope.security,
> which has a dependency on python-zope.proxy, and then removing
> python-zope.security. python-zope.proxy therefore stays around, causing the
> SystemError.

_No_, I did _not_ install python-zope.security at all - initially I've
just installed zope2.13 and also python-zodb which I needed for other
tasks and python-zodb depends on python-zope.proxy.

So I did _not_ remove python-zope.security or did anything lower than
apt-get install or remove and I did _not_ used anything like
--force-depends.

> Downgrading since it all works fine if both extensions are bundled or both
> extensions are external, and 'apt install python-zope.security' does the
> right thing.
  ^^^^^^^^^^^

> I notice that the external packages are not the same versions as their
                                          ^^^^^^^^^^^^^^^^^^^^^

> bundled counterparts, so perhaps an interface changed at some point or a
> rebuild is needed or something.

exactly.

Is it really the "right thing" that different version of packages get
imported? To me this whole "let's bundle everything" approach is
only justified because each package then could specify which version of
dependencies to use _exactly_. And if not we should just drop that
shipped-libs and use system-wide packages right away.

And again, as noted in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774213#34
there is not enough isolation between system-wide and shipped packages
which leads to `what gets imported is not deterministic`; and that "not
deterministic" also depends on packages installation / removal order.

I strongly suspect the effect is not limited only to zope.proxy and
zope.security and affects the whole import mechanism...

Thanks,
Kirill

Message #53 received at 774213@bugs.debian.org (full text, mbox, reply):

From: Barry Warsaw <barry@debian.org>

To: pkg-zope-developers@lists.alioth.debian.org

Cc: Kirill Smelkov <kirr@nexedi.com>, 774213@bugs.debian.org

Subject: Re: Bug#774213: zope2.13: import zope.security._proxy -> SystemError: dynamic module not initialized properly

Date: Sun, 18 Jan 2015 16:56:41 -0500

On Jan 18, 2015, at 11:48 PM, Kirill Smelkov wrote:

>To me this whole "let's bundle everything" approach is only justified because
>each package then could specify which version of dependencies to use
>_exactly_.

I don't believe Debian packages are allowed to bundle packages that are also
available in the archive.  We've been fighting this bundling ("vendorizing")
issue in several other packages and often have to go through lots of hoops to
unbundled (or "devendorize").  Look at python-pip for a particularly painful
example of this.

I certainly understand the motivation of upstreams for wanting to bundle.  But
IMO integration is job of distro packagers, so we try to do it correctly.  I
really don't know the details of zope bundling (and really don't want to ;).

Send a report that this bug log contains spam.