Debian Bug report logs - #800501
SystemError on missing files in /var/lib/apt/lists

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Sebastien Bacher <seb128@ubuntu.com>

To: submit@bugs.debian.org

Subject: SystemError on missing files in /var/lib/apt/lists

Date: Wed, 30 Sep 2015 10:12:52 +0200

Package: python-apt
Version: 1.1.0~alpha3

Unsure when/how that is triggered but oneconf seems to hit that often in
Ubuntu

example

"Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/oneconf/packagesethandler.py",
line 63, in update
    newpkg_list = self.distro.compute_local_packagelist()
  File "/usr/lib/python3/dist-packages/oneconf/distributor/Ubuntu.py",
line 42, in compute_local_packagelist
    with apt.Cache() as apt_cache:
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 113, in __init__
    self.open(progress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 166, in open
    self._records = apt_pkg.PackageRecords(self._cache)
SystemError: E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_restricted_binary-i386_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_multiverse_binary-i386_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_universe_binary-i386_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_main_binary-i386_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_restricted_binary-amd64_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_multiverse_binary-amd64_Packages
- open (2: No such file or directory), E:Could not open file
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_universe_binary-amd64_Packages
- open (2: No such file or directory)"

Message #10 received at 800501@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Sebastien Bacher <seb128@ubuntu.com>, 800501@bugs.debian.org

Subject: Re: Bug#800501: SystemError on missing files in /var/lib/apt/lists

Date: Wed, 30 Sep 2015 10:20:39 +0200

Control: tag -1 unreproducible moreinfo

On Wed, Sep 30, 2015 at 10:12:52AM +0200, Sebastien Bacher wrote:
> Package: python-apt
> Version: 1.1.0~alpha3
> 
> Unsure when/how that is triggered but oneconf seems to hit that often in
> Ubuntu
> 
> example
> 
> "Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/oneconf/packagesethandler.py",
> line 63, in update
>     newpkg_list = self.distro.compute_local_packagelist()
>   File "/usr/lib/python3/dist-packages/oneconf/distributor/Ubuntu.py",
> line 42, in compute_local_packagelist
>     with apt.Cache() as apt_cache:
>   File "/usr/lib/python3/dist-packages/apt/cache.py", line 113, in __init__
>     self.open(progress)
>   File "/usr/lib/python3/dist-packages/apt/cache.py", line 166, in open
>     self._records = apt_pkg.PackageRecords(self._cache)
> SystemError: E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_restricted_binary-i386_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_multiverse_binary-i386_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_universe_binary-i386_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_main_binary-i386_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_restricted_binary-amd64_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_multiverse_binary-amd64_Packages
> - open (2: No such file or directory), E:Could not open file
> /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_wily_universe_binary-amd64_Packages
> - open (2: No such file or directory)"

You're probably opened the cache without holding a lock, and
an update ran in parallel. That's really the only possible
explanation I could think of.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Be friendly, do not top-post, and follow RFC 1855 "Netiquette".
    - If you don't I might ignore you.

Message #17 received at 800501@bugs.debian.org (full text, mbox, reply):

From: Sebastien Bacher <seb128@ubuntu.com>

To: 800501@bugs.debian.org, Julian Andres Klode <jak@debian.org>

Subject: Re: Bug#800501: SystemError on missing files in /var/lib/apt/lists

Date: Wed, 30 Sep 2015 10:59:26 +0200

> You're probably opened the cache without holding a lock, and
> an update ran in parallel. That's really the only possible
> explanation I could think of.

Is there a way to hold a lock as a normal user process? (oneconf is an
user session service)
If not is there a standard way to detect that something is changing the
cache under your feet and to deal with that?

Cheers,
Sebastien Bacher

Message #22 received at 800501@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Sebastien Bacher <seb128@ubuntu.com>

Cc: 800501@bugs.debian.org

Subject: Re: Bug#800501: SystemError on missing files in /var/lib/apt/lists

Date: Wed, 30 Sep 2015 11:04:01 +0200

On Wed, Sep 30, 2015 at 10:59:26AM +0200, Sebastien Bacher wrote:
> > You're probably opened the cache without holding a lock, and
> > an update ran in parallel. That's really the only possible
> > explanation I could think of.
> 
> Is there a way to hold a lock as a normal user process? (oneconf is an
> user session service)
> If not is there a standard way to detect that something is changing the
> cache under your feet and to deal with that?

Use inotify on the cache directory? But in any case, this is problematic
and could probably be fixed by adding some new interfaces to APT (so we
can open fds when opening the cache, and fail if the files are not
present there - and if they are present, pass those fds to the
PackageRecords - this would make things race-free).

I'll look more closely at this later.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Be friendly, do not top-post, and follow RFC 1855 "Netiquette".
    - If you don't I might ignore you.

Send a report that this bug log contains spam.