Debian Bug report logs -
#1015301
gitlab: Upgrade to 15.0.4 fails because BackgroundMigrations could not finalize
Reported by: Maximilian Stein <m@steiny.biz>
Date: Tue, 19 Jul 2022 09:57:01 UTC
Severity: important
Merged with 1015302
Found in versions 15.0.4+ds1-1~fto11+2, gitlab/15.0.4+ds1-1
Fixed in version 13.4.7-2+rm
Done: Debian FTP Masters <ftpmaster@ftp-master.debian.org>
Reply or subscribe to this bug.
Toggle useless messages
Report forwarded
to debian-bugs-dist@lists.debian.org, Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>:
Bug#1015301; Package gitlab.
(Tue, 19 Jul 2022 09:57:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Maximilian Stein <m@steiny.biz>:
New Bug report received and forwarded. Copy sent to Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>.
(Tue, 19 Jul 2022 09:57:03 GMT) (full text, mbox, link).
Message #5 received at submit@bugs.debian.org (full text, mbox, reply):
Package: gitlab
Version: 15.0.4+ds1-1~fto11+2
Severity: important
Dear Maintainer,
I just tried to upgrade to Gitlab 15.0.4 on my two instances. While I
had no issues in my smaller instance, my bigger one had some issues.
First, postinst failed with the message
/etc/systemd/system/gitaly.service.d/override.conf already exist
I moved the file away and the upgrade could continue. At the end of
the upgrade, gitaly did not start anymore, so I restored the file (it
actually contains the setting of the user running gitaly).
Then, however, I stumbled upon a much more serious issue: The database
migration could not finish as there were pending background jobs that
failed to finalize:
gitlab_production database is not empty, skipping gitlab setup
Attention: used pure ruby version of MurmurHash3
/usr/share/gitlab/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/usr/share/gitlab/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
/usr/share/gitlab/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
/usr/share/gitlab/lib/gitlab.rb:48: warning: previous definition of VERSION was here
/usr/share/gitlab/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/usr/share/gitlab/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
/usr/share/gitlab/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/usr/share/gitlab/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
== 20220213103859 RemoveIntegrationsType: migrating ===========================
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
…
I did some research, and found somebody else having a similar issue
[1]. Unfortunately, manually running the background jobs [2] did not
work either as they continued to fail. In the database I identified
the following stuck background migrations [3]:
gitlab_production=# select id,status,job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
id | status | job_class_name | table_name | column_name | job_arguments
----+--------+--------------------------------------------------+--------------+-------------+---------------
17 | 4 | BackfillNamespaceIdForNamespaceRoute | routes | id | []
19 | 4 | BackfillMemberNamespaceForGroupMembers | members | id | []
20 | 4 | MigratePersonalNamespaceProjectMaintainerToOwner | members | id | []
23 | 4 | BackfillGroupFeatures | namespaces | id | [10000]
15 | 4 | BackfillIntegrationsTypeNew | integrations | id | []
16 | 4 | BackfillUserNamespace | namespaces | id | []
18 | 4 | BackfillIssueSearchData | issues | id | []
I then proceeded to simply change the status of the jobs to 3 in the
database as proposed in the issue mentioned above [1]. I could then
finish the upgrade normally with `apt upgrade`.
Afterwards, I simply undid the database change (i.e., reverted the
status of the failed background migrations to 4) and then restarted
the migrations in the web UI.
As far as I can tell, everything seems normal now. The seven
background migration jobs are still pending, but I can continue to use
Gitlab normally.
Do you need any more information on the matter?
Thanks for your investigation into the issue!
Best,
Maximilian
[1]: https://gitlab.com/gitlab-org/gitlab/-/issues/340193
[2]: https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#manually-finishing-a-batched-background-migration
[3]: https://docs.gitlab.com/ee/update/index.html#batched-background-migrations
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>:
Bug#1015301; Package gitlab.
(Tue, 19 Jul 2022 11:36:09 GMT) (full text, mbox, link).
Acknowledgement sent
to Maximilian Stein <m@steiny.biz>:
Extra info received and forwarded to list. Copy sent to Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>.
(Tue, 19 Jul 2022 11:36:09 GMT) (full text, mbox, link).
Message #10 received at 1015301@bugs.debian.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
Update on the background migrations:
All except one background migrations finished successfully, leaving only
"BackfillIntegrationsTypeNew: integrations
<https://<redacted>/admin/background_migrations/15>" stuck at failed.
This one is exactly the migration mentioned in [1], so I guess this one
might cause trouble in general
I then tried to migrate manually with `gitlab-rake db:migrate:redo
VERSION=20210727113447`. This actually worked, however, I still have the
Background Migration job in the admin page. So I guess I simply set the
job's state on success as I don't experience any issues.
[Message part 2 (text/html, inline)]
Marked as found in versions gitlab/15.0.4+ds1-1.
Request was from Antoine Le Gonidec <debian.bugs@dotslashplay.it>
to control@bugs.debian.org.
(Tue, 19 Jul 2022 17:15:02 GMT) (full text, mbox, link).
Merged 1015301 1015302
Request was from Antoine Le Gonidec <debian.bugs@dotslashplay.it>
to control@bugs.debian.org.
(Tue, 19 Jul 2022 17:15:03 GMT) (full text, mbox, link).
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>:
Bug#1015301; Package gitlab.
(Tue, 19 Jul 2022 17:33:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Antoine Le Gonidec <debian.bugs@dotslashplay.it>:
Extra info received and forwarded to list. Copy sent to Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>.
(Tue, 19 Jul 2022 17:33:03 GMT) (full text, mbox, link).
Message #19 received at 1015301@bugs.debian.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
I could work around the failure to execute the last remaining migration using the attached patch.
This patch is provided only for helping in diagnosing the underlying issue, I would not suggest applying it as-is.
[0001-Work-around-background-migration-failure.patch (text/x-diff, attachment)]
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>:
Bug#1015301; Package gitlab.
(Wed, 20 Jul 2022 06:27:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Maximilian Stein <m@steiny.biz>:
Extra info received and forwarded to list. Copy sent to Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>.
(Wed, 20 Jul 2022 06:27:02 GMT) (full text, mbox, link).
Message #24 received at 1015301@bugs.debian.org (full text, mbox, reply):
On Tue, 19 Jul 2022 19:22:02 +0200 Antoine Le Gonidec
<debian.bugs@dotslashplay.it> wrote:
> I could work around the failure to execute the last remaining
migration using the attached patch.
>
> This patch is provided only for helping in diagnosing the underlying
issue, I would not suggest applying it as-is.
Thanks for the patch!
Unfortunately, the jobs still fails for me. I actually get the message:
PG::UndefinedColumn: ERROR: Column integrations.type does not exist
LINE 47: AND integrations.type = mapping.old_type ^
So I guess my database is now to new for this migration and I can only
manually set it to "successful"…
Best,
Maximilian
Reply sent
to Debian FTP Masters <ftpmaster@ftp-master.debian.org>:
You have taken responsibility.
(Sun, 19 Feb 2023 01:06:12 GMT) (full text, mbox, link).
Notification sent
to Maximilian Stein <m@steiny.biz>:
Bug acknowledged by developer.
(Sun, 19 Feb 2023 01:06:12 GMT) (full text, mbox, link).
Message #29 received at 1015301-done@bugs.debian.org (full text, mbox, reply):
Version: 13.4.7-2+rm
Dear submitter,
as the package gitlab has just been removed from the Debian archive
unstable we hereby close the associated bug reports. We are sorry
that we couldn't deal with your issue properly.
For details on the removal, please see https://bugs.debian.org/1031519
The version of this package that was in Debian prior to this removal
can still be found using https://snapshot.debian.org/.
Please note that the changes have been done on the master archive and
will not propagate to any mirrors until the next dinstall run at the
earliest.
This message was generated automatically; if you believe that there is
a problem with it please contact the archive administrators by mailing
ftpmaster@ftp-master.debian.org.
Debian distribution maintenance software
pp.
Scott Kitterman (the ftpmaster behind the curtain)
Reply sent
to Debian FTP Masters <ftpmaster@ftp-master.debian.org>:
You have taken responsibility.
(Sun, 19 Feb 2023 01:06:13 GMT) (full text, mbox, link).
Notification sent
to Patrick Matthäi <patrick@linux-dev.org>:
Bug acknowledged by developer.
(Sun, 19 Feb 2023 01:06:13 GMT) (full text, mbox, link).
Send a report that this bug log contains spam.
Debian bug tracking system administrator <owner@bugs.debian.org>.
Last modified:
Fri Jan 12 10:07:03 2024;
Machine Name:
buxtehude
Debian Bug tracking system
Debbugs is free software and licensed under the terms of the GNU
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.
Copyright © 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson,
2005-2017 Don Armstrong, and many other contributors.