Debian Bug report logs - #886998
intel-microcode: regressions on Haswell, Broadwell (crashes/reboots) and Kaby Lake (sleep-to-ram)

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Henrique de Moraes Holschuh <hmh@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: intel-microcode: regressions on Haswell, Broadwell (crashes/reboots) and Kaby Lake (sleep-to-ram)

Date: Fri, 12 Jan 2018 09:39:01 -0200

Package: intel-microcode
Version: 3.20180108.1
Severity: grave

According to this:
https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/

and this far more helpful information:
https://pcsupport.lenovo.com/br/en/product_security/ps500151

A subset of systems with the recent Haswell, Haswell/E, Broadwell,
Broadwell/E/DE, and Kaby Lake microcode updates have shown regressions
of either the "crash" sort, or of the "doesn't wake up from C3 sleep
(sleep-to-ram) properly" sort.

The Lenovo advisory implies that the regressions won't trigger always/on
every system.  It is unclear at this time if they would only possibly
trigger when the new IBRS/IBPB Spectre mitigation functionality is
active (hint: Microsoft Windows 10 already activates it, but Debian
kernels don't enable it as we are still waiting for the upstream changes
to Linux, gcc and clang to stabilize).

To put it in simple terms: we don't know at the present time if Debian
systems would be affected by these microcode regressions right now, or
if they would start to be affected after the microcode-based Spectre
mitigation support is enabled in the kernels, or if they would be
affected at all.

Some of these possibly problematic microcode updates are indeed present
in Intel's 20180108 Linux microcode release.  I will pause the
deployment of the 20180108 update in Debian: it will be restricted to
unstable (and possibly to testing, if the packages already migrated by
the time this bug report is active) until we either get more information
or a new set of microcode updates.

Updated packages removing the subset of problematic updates are not
going to be produced until at least next monday/tuesday for two reasons:

1. we don't have an exact list of signatures that are possibly
   affected at this time.  Removing all updates is equivalent to just
   rolling back the package to the previous version.

2. there is a reasonably high chance of a new Intel microcode update
   release in the next few days, which would most probably either revert
   or fix the problematic microcode updates.

Should you face issues with the new microcode (note: test it with an
older kernel as well, since the current crop of new kernels are *ALSO*
causing boot and resume-from-sleep issues that are completely unrelated
to the microcode updates), please send a note to this bug report, with
the output of /proc/cpuinfo.

Previous versions of the packages are available here:
http://snapshot.debian.org/package/intel-microcode/

-- 
  Henrique Holschuh

Message #10 received at 886998@bugs.debian.org (full text, mbox, reply):

From: Henrique de Moraes Holschuh <hmh@debian.org>

To: 886998@bugs.debian.org

Subject: if you downgrade, do it to the 20171117 release, NOT 20171215

Date: Fri, 12 Jan 2018 10:20:23 -0200

On Fri, 12 Jan 2018, Henrique de Moraes Holschuh wrote:
> Should you face issues with the new microcode (note: test it with an
> older kernel as well, since the current crop of new kernels are *ALSO*
> causing boot and resume-from-sleep issues that are completely unrelated
> to the microcode updates), please send a note to this bug report, with
> the output of /proc/cpuinfo.
> 
> Previous versions of the packages are available here:
> http://snapshot.debian.org/package/intel-microcode/

And the version to pick when downgrading is 20171117.

DO NOT use 20171215 if you are trying to avoid possible regressions.

-- 
  Henrique Holschuh

Message #15 received at 886998@bugs.debian.org (full text, mbox, reply):

From: Henrique de Moraes Holschuh <hmh@debian.org>

To: 886998@bugs.debian.org

Subject: List of sigs for microcode with regressions

Date: Sat, 13 Jan 2018 21:33:24 -0200

The VMWare KB52345 article at:
https://kb.vmware.com/s/article/52345

Includes an _incomplete_ list of signatures for the microcode updates
that are problematic in release 20180108:

Processor name
(non-exhaustive)             signature   stepping name

Intel Xeon E3-1200-v3,
Intel i3-4300,
Intel i5-4500-TE,
Intel i7-4700-EQ:            0x000306C3  C0

Intel Xeon E5-1600-v3,
Intel Xeon E5-2400-v3,
Intel Xeon E5-2600-v3,
Intel Xeon E5-4600-v3:       0x000306F2  C0/C1, M0/M1, R1/R2

Intel Xeon E7-8800/4800-v3:  0x000306F4  E0

Intel Xeon E3-1200-v4:       0x00040671  G0

ntel Xeon E5-1600-v4,
Intel Xeon E5-2600-v4,
Intel Xeon E5-4600-v4:       0x000406F1  B0/M0/R0

Intel Xeon E7-8800/4800-v4:  0x000406F1  B0/M0/R0
Intel Xeon D-1500:           0x00050663  V2

This list is only relevant for server processors.  Desktop/mobile
processors were not included, and we cannot assume anything about
updates for such processors that were deployed in release 20180108.

-- 
  Henrique Holschuh

Message #20 received at 886998@bugs.debian.org (full text, mbox, reply):

From: Henrique de Moraes Holschuh <hmh@debian.org>

To: 886998@bugs.debian.org

Subject: regressions also possible on Skylake

Date: Thu, 18 Jan 2018 22:50:23 -0200

Updated information from Intel:
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088

---8<---

Recommendations: 

Status

    Intel has made significant progress in our investigation into the
customer reboot sightings that we confirmed publicly last week
    Intel has reproduced these issues internally and has developed a
test method that allows us to do so in a predictable manner
    Initial sightings were reported on Broadwell and Haswell based
platforms in some configurations. During due diligence we determined
that similar behavior occurs on other products including Ivy Bridge,
Sandy Bridge, Skylake, and Kaby Lake based platforms in some
configurations
    We are working toward root cause
    While our root cause analysis continues, we will start making beta
microcode updates available to OEMs, Cloud service providers, system
manufacturers and Software vendors next week for internal evaluation
purposes
    In all cases, the existing and any new beta microcode updates
continue to provide protection against the exploit (CVE-2017-5715) also
known as “Spectre Variant 2”
    Variants 1 (Spectre) and Variant 3 (Meltdown) continue to be
mitigated through system software changes from operating system and
virtual machine vendors
    As we gather feedback from our customers we will continue to provide
updates that improve upon performance and usability

Intel recommendations to OEMs, Cloud service providers, system
manufacturers and software vendors

    Intel recommends that these partners maintain availability of
existing microcode updates already released to end users. Intel does not
recommend pulling back any updates already made available to end users
    NEW - Intel recommends that these partners, at their discretion,
continue development and release of updates with existing microcode to
provide protection against these exploits, understanding that the
current versions may introduce issues such as reboot in some
configurations
    NEW - We further recommend that OEMs, Cloud service providers,
system manufacturers and software vendors begin evaluation of Intel beta
microcode update releases in anticipation of definitive root cause and
subsequent production releases suitable for end users

---8<---


As such, current plans are to _not_ distributed updated microcode
packages to Debian stable users, until a new batch of microcode updates
are released by Intel.

"Beta" microcode updates will be initially uploaded to experimental (if
such updates are made available to Debian): do *not* assume that beta
updates "can't be worse than the current ones".

-- 
  Henrique Holschuh

Message #27 received at 886998-close@bugs.debian.org (full text, mbox, reply):

From: Henrique de Moraes Holschuh <hmh@debian.org>

To: 886998-close@bugs.debian.org

Subject: Bug#886998: fixed in intel-microcode 3.20180108.1+really20171117.1

Date: Tue, 23 Jan 2018 01:19:24 +0000

Source: intel-microcode
Source-Version: 3.20180108.1+really20171117.1

We believe that the bug you reported is fixed in the latest version of
intel-microcode, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 886998@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Henrique de Moraes Holschuh <hmh@debian.org> (supplier of updated intel-microcode package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Mon, 22 Jan 2018 23:01:59 -0200
Source: intel-microcode
Binary: intel-microcode
Architecture: source amd64
Version: 3.20180108.1+really20171117.1
Distribution: unstable
Urgency: critical
Maintainer: Henrique de Moraes Holschuh <hmh@debian.org>
Changed-By: Henrique de Moraes Holschuh <hmh@debian.org>
Description:
 intel-microcode - Processor microcode firmware for Intel CPUs
Closes: 886998
Changes:
 intel-microcode (3.20180108.1+really20171117.1) unstable; urgency=critical
 .
   * Revert to release 20171117, as per Intel instructions issued to
     the public in 2018-01-22 (closes: #886998)
   * This effectively removes IBRS/IBPB/STIPB microcode support for
     Spectre variant 2 mitigation.
Checksums-Sha1:
 2ecfe6036525207dea082973949a538fb027af0d 1865 intel-microcode_3.20180108.1+really20171117.1.dsc
 0dae4e35e418c31f0f8918dc156e959720f95e21 1801480 intel-microcode_3.20180108.1+really20171117.1.tar.xz
 bdc109d548b74f7aea64ae1f17703f2f1ffbeb61 5665 intel-microcode_3.20180108.1+really20171117.1_amd64.buildinfo
 210553661c42faae406f4edfedacdf0c804b0e7e 1270444 intel-microcode_3.20180108.1+really20171117.1_amd64.deb
Checksums-Sha256:
 a7df931b02fa66df34106c1d4ff96b39921309b07fac2452b3112eeb29b87bcc 1865 intel-microcode_3.20180108.1+really20171117.1.dsc
 376c988faedbee9ddfdfed6073fba65fbd3504a7fab4c3fe1d7e141e8e417126 1801480 intel-microcode_3.20180108.1+really20171117.1.tar.xz
 04040e9355771e3f9c75a995107f9938f9e6fd304ea41ba351eddc77cbead848 5665 intel-microcode_3.20180108.1+really20171117.1_amd64.buildinfo
 57a22d3dd3379149fc550f371874bfa864f6d2f264665379e8fc8c85bd90e6f0 1270444 intel-microcode_3.20180108.1+really20171117.1_amd64.deb
Files:
 04360f77e2fe481ecce632f214cea4b8 1865 non-free/admin standard intel-microcode_3.20180108.1+really20171117.1.dsc
 c5d6395ae7719dabd161739b6aa21cef 1801480 non-free/admin standard intel-microcode_3.20180108.1+really20171117.1.tar.xz
 2a6ca1e54e22d5e16fae1f908edf4918 5665 non-free/admin standard intel-microcode_3.20180108.1+really20171117.1_amd64.buildinfo
 d99a8480e9d1405749b5b1ae4bc5c232 1270444 non-free/admin standard intel-microcode_3.20180108.1+really20171117.1_amd64.deb

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEq7GuO+cOOhDUp+2l/hG/posVjpgFAlpmifIACgkQ/hG/posV
jpgfPw/8CMVJPItsxGqebnznKCmN5UXvxkSiGTpivUyzZoik8Nmh6259ukF1iIvv
kYHnFoxTpYNbWMrCBD9KnCAxIARmru1G1jK1K6kytxHroYseBKgE2fsvcSsqcN5/
lctedHZj0cwNqhRN2p8ROuw9Di3/ZukcyyxICAj0W8CyYHfdNfBpXw1hnJ0gp0Bx
TDjZ4DUhScLQ9W6nOr3YkiAB2VSIDM2w8TAPgrtQnuNUYyHQmnajuaI/1U7tjsrB
Huj/rS0lUzkoLzm7yp9OPpEOmdKcUtKEcyk63v5oiqf4eNCeg1yGg0QPr6ET/p49
J2IOYBAopUja60CmmF1fveXAuaBwPvvibvOZ+GNmt52E4ybnvs2MK5Bd4KwRhA82
ZMks7lP8Dr56Zl0GsDwKBfVhgvVAlXNOyFhzDJnQlhpRCcWKlVTI1b5yDnoO4+vm
C6YqBf/4Jh1YmPI6og8lKVtcU0jUxVKSnxTNOYfJz2T2qb3yvfMmS8ZV9PbUBpa5
nwLWJcLGVQYlOiNEhFaP8oV2dI4Ip7plHjT6m6SorhCSisz4B6nPcpDyNhFZ2Mue
SaRhicv+gtCMLruvY4UYt9JYlzo7dQUAcnANZjaNu7l6nF5jZxB7FCA9WqmcaTbd
3uleOMd2VGAO5uN4evnuaWRA1bbizlN9fkgve/T4FKnyCEEXkyQ=
=fP+B
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.