Debian Bug report logs - #518093
Very slow, high CPU when syncing after thousands of remote deletions.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Trent W. Buck" <trentbuck@gmail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Very slow, high CPU when syncing after thousands of remote deletions.

Date: Wed, 04 Mar 2009 14:34:36 +1100

Package: offlineimap
Version: 6.0.3
Severity: normal

About once a year, I connect to imap.gmail.com with mutt and delete
lots of mail -- duplicates, anything archived on a mailing list, spam,
etc.

I did this just now, and ended up deleting around 20,000 messages.
When I run offlineimap, it happily prints that it will delete stuff
from "[...]) in Maildir[all], LocalStatus[all]".  And indeed, almost
immediately I can see with du that the maildir folder has gone from
290MB to 100MB.  Admirably efficient.

But offlineimap doesn't stop.  So far it has been running for
twenty-five minutes, and in that time has used up 13 minutes of CPU
(i.e. about 50% CPU utilization).

Clearly something extremely suboptimal is going on here.  If I had to
guess, I would suspect the problem is that offlineimap is repeatedly
opening its ~/.offlineimap metadata, deleting a single message, then
closing those files again.

This problem may also be exacerbated by the fact that my nonvolatile
storage is an SSD, which (I hear) has slow write speeds compared to a
traditional HDD.  The disk activity light is blinking at around 2Hz.

-- System Information:
Debian Release: 5.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages offlineimap depends on:
ii  python                        2.5.4-2    An interactive high-level object-o
ii  python-support                0.8.7      automated rebuilding support for P

offlineimap recommends no packages.

Versions of packages offlineimap suggests:
pn  python-kerberos               <none>     (no description available)

-- no debconf information

Message #10 received at 518093@bugs.debian.org (full text, mbox, reply):

From: John Goerzen <jgoerzen@complete.org>

To: "Trent W. Buck" <trentbuck@gmail.com>, 518093@bugs.debian.org

Subject: Re: Bug#518093: Very slow, high CPU when syncing after thousands of remote deletions.

Date: Wed, 04 Mar 2009 10:44:05 -0600

Trent W. Buck wrote:
> Package: offlineimap
> Version: 6.0.3
> Severity: normal
> 
> About once a year, I connect to imap.gmail.com with mutt and delete
> lots of mail -- duplicates, anything archived on a mailing list, spam,
> etc.
> 
> I did this just now, and ended up deleting around 20,000 messages.
> When I run offlineimap, it happily prints that it will delete stuff
> from "[...]) in Maildir[all], LocalStatus[all]".  And indeed, almost
> immediately I can see with du that the maildir folder has gone from
> 290MB to 100MB.  Admirably efficient.
> 
> But offlineimap doesn't stop.  So far it has been running for
> twenty-five minutes, and in that time has used up 13 minutes of CPU
> (i.e. about 50% CPU utilization).
> 
> Clearly something extremely suboptimal is going on here.  If I had to
> guess, I would suspect the problem is that offlineimap is repeatedly
> opening its ~/.offlineimap metadata, deleting a single message, then
> closing those files again.
> 
> This problem may also be exacerbated by the fact that my nonvolatile
> storage is an SSD, which (I hear) has slow write speeds compared to a
> traditional HDD.  The disk activity light is blinking at around 2Hz.

OfflineIMAP flushes the cache after each write.  This will likely be
extremely slow on SSD.  It's known to be slow in general.  Some have
proposed SQLite for the cache, but the patches posted to date haven't
been suitable for integration into OfflineIMAP yet.

In the meantime, you can experiment with setting doautosave=0 in
offlineimap/folder/LocalStatus.py.  However, this hasn't been tested, so
I have no idea if it will dramatically improve performance or wipe out
all your mail.

-- John

> 
> -- System Information:
> Debian Release: 5.0
>   APT prefers testing
>   APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
> Architecture: i386 (i686)
> 
> Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
> Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> 
> Versions of packages offlineimap depends on:
> ii  python                        2.5.4-2    An interactive high-level object-o
> ii  python-support                0.8.7      automated rebuilding support for P
> 
> offlineimap recommends no packages.
> 
> Versions of packages offlineimap suggests:
> pn  python-kerberos               <none>     (no description available)
> 
> -- no debconf information
> 
> 
> 

Message #15 received at 518093@bugs.debian.org (full text, mbox, reply):

From: Sebastian Spaeth <Sebastian@SSpaeth.de>

To: "Trent W. Buck" <trentbuck@gmail.com>, 518093@bugs.debian.org, control@bugs.debian.org

Subject: OfflineImap just got more efficient

Date: Sat, 07 May 2011 20:52:51 +0200

tag 518093 fixed-upstream

Just to let you know that upstream has fixed this in git and a solution
will be available in offlineimap 6.3.4. An optional cache backend based
on sqlite rather than plaintext files (which will be fully written out after
every single change) has gone in and can be configured to be used with

[Account foo]
status_backend = sqlite

This should solve quite some performance problems with big Maildirs.

Sebastian

Message #22 received at 518093-close@bugs.debian.org (full text, mbox, reply):

From: John Francesco Ferlito <johnf@inodes.org>

To: 518093-close@bugs.debian.org

Subject: Bug#518093: fixed in offlineimap 6.3.4-1

Date: Tue, 23 Aug 2011 14:51:00 +0000

Source: offlineimap
Source-Version: 6.3.4-1

We believe that the bug you reported is fixed in the latest version of
offlineimap, which is due to be installed in the Debian FTP archive:

offlineimap_6.3.4-1.debian.tar.gz
  to main/o/offlineimap/offlineimap_6.3.4-1.debian.tar.gz
offlineimap_6.3.4-1.dsc
  to main/o/offlineimap/offlineimap_6.3.4-1.dsc
offlineimap_6.3.4-1_all.deb
  to main/o/offlineimap/offlineimap_6.3.4-1_all.deb
offlineimap_6.3.4.orig.tar.gz
  to main/o/offlineimap/offlineimap_6.3.4.orig.tar.gz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 518093@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
John Francesco Ferlito <johnf@inodes.org> (supplier of updated offlineimap package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Tue, 23 Aug 2011 22:50:13 +1000
Source: offlineimap
Binary: offlineimap
Architecture: source all
Version: 6.3.4-1
Distribution: unstable
Urgency: low
Maintainer: John Francesco Ferlito <johnf@inodes.org>
Changed-By: John Francesco Ferlito <johnf@inodes.org>
Description: 
 offlineimap - IMAP/Maildir synchronization and reader support
Closes: 190262 203889 470679 489309 518093 618264 627004 629502 629515 630447 637716
Changes: 
 offlineimap (6.3.4-1) unstable; urgency=low
 .
   * New upstream release.
     - re-syncs some of my large folders (Closes: #629515).
     - uses invalid date notation in SEARCH request (Closes: #630447).
     - Fails to upload emails with some locales (Closes: #637716).
     - leftover reference to old interface name Noninteractive.Basic (Closes: #629502).
     - Improvements to man page (Closes: #470679).
     - Maximum message size (Closes: #190262).
     - does not handle network downtime gracefully (Closes: #203889).
     - Very slow, high CPU when syncing after thousands of remote
       deletions (Closes: #518093).
     - TypeError: sigterm_handler() takes exactly 3 arguments (Closes: #618264).
     - ask for password once, then autorefresh in background (Closes: #489309).
   * debian/patches - Added
     - 0001-Fix-manpage-build-failures-from-inconsistent-heading.patch
   * debian/changelog
     - Removed excess whitespace
   * debian/control
     - point VCS- fields at debian repo not upstream (Closes: #627004).
Checksums-Sha1: 
 37b8a7a0703c90ef189200f8fd01391189987e6f 1237 offlineimap_6.3.4-1.dsc
 2ba3d1145a3e36d0e81ba0eea73bd4c5b48631e6 137313 offlineimap_6.3.4.orig.tar.gz
 68803fb87de2880b0b3abae376fd35bd09c7b0c4 20109 offlineimap_6.3.4-1.debian.tar.gz
 af483ca7b96955f0fdf2e22cabdb5b52b3840228 143720 offlineimap_6.3.4-1_all.deb
Checksums-Sha256: 
 27d14e9bf06701df44e32691f6f79573aa0b7fbbc9229b7623c717cf20465d88 1237 offlineimap_6.3.4-1.dsc
 a2ac69a58605338e971658e930e095e4c2611383837a9378b88c0749cd374333 137313 offlineimap_6.3.4.orig.tar.gz
 e0397827e69ad22188b9d617811973543b4dfffb5d92998265efb3b565e0f012 20109 offlineimap_6.3.4-1.debian.tar.gz
 a9eec127dd069489aa18c3a95ecdfe186a17c795a7657d56c067b72be5860c86 143720 offlineimap_6.3.4-1_all.deb
Files: 
 5563ed0af09596c93954744158f728e6 1237 mail optional offlineimap_6.3.4-1.dsc
 84c42088ffcc7092473fef76e17c082b 137313 mail optional offlineimap_6.3.4.orig.tar.gz
 d5615aae6677e9b323ff48f83113234f 20109 mail optional offlineimap_6.3.4-1.debian.tar.gz
 07bd3556613421df2f36ea6f7334fc76 143720 mail optional offlineimap_6.3.4-1_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk5TurEACgkQYcdJscd4KNTvFACgjEf9KfbggJcZfkdxWDb7D+AJ
B6AAn21W8XLaO2+HwwZN5YTfVPKFXkfJ
=wipK
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.