Debian Bug report logs - #923880
iptables -m tos --tos mask value is wrong

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: ssh: IPQoS defaults change interacts badly with iptables -m tos

Date: Wed, 6 Mar 2019 18:15:41 +0100

Package: openssh-client
Version: 1:7.8p1-1
Control: clone -1 -2
Control: reassign -2 iptables
Control: retitle -2 iptables -m tos --tos mask value is wrong
Control: affects -1 + iptables
Control: affects -2 openssh-client

In openssh/1:7.8p1-1, the default for IPQoS changed from

    IPQoS lowdelay throughput

to

    IPQoS af21 cs1

Good reasons for this change are given in
https://lists.gt.net/openssh/commits/71079.

Now since the old ssh used TOS values, matching them with iptables
naturally involed the tos module. Your match for bulk traffic would
usually look like this:

    iptables -m tos --tos Maximize-Throughput ...

Unfortunately, that becomes "08x/0x3f". That interacts badly with DSCP
class af21. IPTOS_DSCP_AF21 is valued 0x48. The Maximize-Throuput match
now matches interactive traffic. This is very bad.

What I don't understand is why this happens though. The 0x3f mask used
by iptables here is supposed to exclude the ECN bits. DSCP is supposed
to coexist with ECN, so it shouldn't be setting any ECN bits. Why would
it match interactive traffic as bulk then? <netinet/ip.h>, which defines
IPTOS_DSCP_AF21 as 0x48, also defines IPTOS_ECN_MASK as 0x3. This
suggests that iptables' ECN mask is wrong. It should be using 0xfc
rather than 0x3f.

Unfortunately, this is deployed now and ssh's new default breaks users
of -m tos (that matched ssh's old default) now. Thus I suggest reverting
the IPQoS change until iptables has been fixed.

And fixing iptables is going to be "interesting". It also defines --tos
Minimize-Cost, which happens to be bit 6 (RFC 1349). Bit 6 and 7 are ECN
bits though. So offering Minimize-Cost with an ECN mask simply won't
work. I guess the best thing we can do here is acknowledge that TOS and
ECN don't work well together. Indeed the relevant RFCs define bit 7 as
"must be zero". This suggests changing the mask to 0xff is in order.

For ssh, I recommend temporarily reverting to the old default to give
iptables some time.

Helmut

Message #20 received at 923880@bugs.debian.org (full text, mbox, reply):

From: Peter Lebbing <peter@digitalbrains.com>

To: 923880@bugs.debian.org

Subject: Re: ssh: IPQoS defaults change interacts badly with iptables -m tos

Date: Thu, 8 Aug 2019 10:35:19 +0200

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, 6 Mar 2019 18:15:41 +0100 Helmut Grohne <helmut@subdivi.de> wrote:
> This suggests that iptables' ECN mask is wrong. It should be using
> 0xfc rather than 0x3f.

Yes, I'm convinced the mask is wrong. However, fixing that would change
the behaviour of already deployed firewalls. It's a thorny situation.

Here's why I conclude the mask is wrong.

Let's take RFC 1349[1], which added the Minimize-Cost bit to ToS. First
off, realise that this conflicts with ECN, that's just part of the can
of worms that is ToS, and it complicates a solution to this issue even
more.

RFC 1349 chapter 3 defines the Type of Service Octet as:

The Type of Service octet consists of three fields:

                0     1     2     3     4     5     6     7
             +-----+-----+-----+-----+-----+-----+-----+-----+
             |                 |                       |     |
             |   PRECEDENCE    |          TOS          | MBZ |
             |                 |                       |     |
             +-----+-----+-----+-----+-----+-----+-----+-----+

Note how the bit endianness is different than, for example, usual x86
diagrams. I think this is where the wrong mask stems from: the author of
the wrong mask was accustomed to diagrams with a different bit
endianness and ended up confused.

Now let's look at chapter 4 which defines this TOS field from bits 3
through 6:

                    1000   --   minimize delay
                    0100   --   maximize throughput
                    0010   --   maximize reliability
                    0001   --   minimize monetary cost
                    0000   --   normal service

And let's compare that to:

# iptables -m tos --help
iptables v1.6.0
[...]
tos match options:
[!] --tos value[/mask]    Match Type of Service/Priority field value
[!] --tos symbol          Match TOS field (IPv4 only) by symbol
                          Accepted symbolic names for value are:
                          (0x10) 16 Minimize-Delay
                          (0x08)  8 Maximize-Throughput
                          (0x04)  4 Maximize-Reliability
                          (0x02)  2 Minimize-Cost
                          (0x00)  0 Normal-Service

Take a good look at these hexadecimals corresponding to the symbolic
names. They match the byte from RFC 1349 only if you flip the
bit-endianness such that the least significant bit is on the right
(Minimize-Cost has the lowest numerical value). Note that these
hexadecimals are correct; it is only the mask that is wrong.

This is on stretch/oldstable, but the help is no different on
buster/stable. I'll continue with a stretch system, though.

We can make it more concrete. Let's create an iptables rule with
numerical values that matches DSCP CS6, which corresponds to IP
Precendence 6, numerical value 0xC0, where in the terms of RFC 1349 bits
0 and 1 are set in the PRECEDENCE portion of the ToS octet.

# iptables -I INPUT -m tos --tos 0xc0 -j NFLOG --nflog-group 2

Ping it:

$ ping -Q 0xc0 -c 1 10.0.1.1

And take a look at the packet in the PCAP log file of that nflog, with
Wireshark:

Internet Protocol Version 4, Src: 10.0.1.133, Dst: 10.0.1.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 84
    Identification: 0x5029 (20521)
    Flags: 0x4000, Don't fragment
    Time to live: 64
    Protocol: ICMP (1)
    Header checksum: 0xd33a [validation disabled]
    [Header checksum status: Unverified]
    Source: 10.0.1.133
    Destination: 10.0.1.1

This proves:

- - That -m tos --tos 0xc0 matches a packet that has 0xc0 in the DS Field
  (because this is the only rule in the firewall logging to that nflog
  group)

- - That 0xc0 means DSCP CS6, because I believe Wireshark's analysis, it's
  been correct in different instances of looking at the DSCP field with
  packets generated by several systems.

So that means that the mask 0x3f is a mistake.

But changing the mask to 0xfc will make -m tos --tos Minimize-Cost break
because that is actually one of the ECN bits. I tested it and --tos
0x02/0xfc predictably did not match ping -Q 0x02. Changing the mask to
0xff causes less breakage, but still changes the behaviour on existing
deployments... :-(

Perhaps the best solution is to deprecate the symbolic --tos arguments,
urging everyone to exclusively use the numerical format. Put this in
NEWS so people hopefully notice, perhaps the Release Notes. And then
maybe someday DSCP and ECN will be less broken. Firewalls currently
using symbolic --tos arguments already misqualify ECN and IP Precedence
as well, it's not just DSCP.

HTH,

Peter.

[1] <https://tools.ietf.org/html/rfc1349>

- -- 
I use the GNU Privacy Guard (GnuPG) in combination with Enigmail.
You can send me encrypted mail if you want some privacy.
My key is available at <http://digitalbrains.com/2012/openpgp-key-peter>
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEZQCNwiCq4qJXTWzVlp4Bj95s3KEFAl1L3sYACgkQlp4Bj95s
3KFaAggAiffHqD+4Y7xTxyfXJbJSn0TN5cxd+42ffnbgkqlNKonzkxWEr0Hd9EOA
ab9aDzSPGpjeFy1Hzuj9z0SrBUp30zWL3WqRxJTifxkIg9AXrWn9xuG1VgH4t1T+
HTpSrjn/Y5NsaUOBdzkWvumoIOY7NQqHxh2mLOPIg9AGJ4XVGbn5PEi2YHM2OVok
9QaB5wPMeNcVeUv4719vMFkZ+VoycremY7F23OMSBiY+vbQI6AntBOf3sJLDb2qQ
Vjoh7EvR2F28+w+DyK77vqAwbMa/+VGgF+ld5BDCxv0w8h/Q434MdkcTJ8CefSNA
4cx1+SP6nSbOuPLirzhez1VANayTsA==
=eTRc
-----END PGP SIGNATURE-----

Message #25 received at 923880@bugs.debian.org (full text, mbox, reply):

From: Peter Lebbing <peter@digitalbrains.com>

To: 923880@bugs.debian.org

Subject: Re: ssh: IPQoS defaults change interacts badly with iptables -m tos

Date: Thu, 8 Aug 2019 11:28:08 +0200

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> We can make it more concrete. Let's create an iptables rule with
> numerical values that matches DSCP CS6, which corresponds to IP
> Precendence 6, numerical value 0xC0, where in the terms of RFC 1349 bits
> 0 and 1 are set in the PRECEDENCE portion of the ToS octet.

I think I should emphasise the point that DSCP CS6 is the same as IP
Precedence 6 *by design*, it's not a coincidence. So the experiment
proves that IP Precedence 6 is encoded as 0xc0, which is what proves the
layout of the octet.

Peter.

- -- 
I use the GNU Privacy Guard (GnuPG) in combination with Enigmail.
You can send me encrypted mail if you want some privacy.
My key is available at <http://digitalbrains.com/2012/openpgp-key-peter>
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEZQCNwiCq4qJXTWzVlp4Bj95s3KEFAl1L6ycACgkQlp4Bj95s
3KHnVgf/a7pav4qO9gYI0rMXi2GsIAtjxWKHaJheph8YXP3vP1LE1v+vOStQB+zC
MPFN1IcUElo0+8ozrQ2RQsus3f4KxJIGPVRSXNAWnprlcEF3NRvVAZBkIjMYSWX9
se/v3BnExMk+IU9sUhDw1nH8KFYosa84zbxbT1buoGxbUzEoXZyezBkEB3CSqqtB
kn4mSZEoWrkahYsW71bFW7IEfuhCfh7GGq4VmO3aoLxGjIqhjrQgI6Cu+j3XUn3G
Sz32dfH5QGgDhsNBIHjlZPr2zlUAhVPJRv/+kdtgXVQiNAQUmFbHLEzoMeGlk4LP
Q6J6xy9ZOK66q1UpIC2cygqdx4WbCA==
=zRTY
-----END PGP SIGNATURE-----

Message #30 received at 923880@bugs.debian.org (full text, mbox, reply):

From: Donovan Baarda <abo@minkirri.apana.org.au>

To: 923880@bugs.debian.org

Subject: Fwd: ssh: IPQoS defaults change interacts badly with iptables -m tos

Date: Mon, 4 May 2020 13:24:29 +1000

[Message part 1 (text/plain, inline)]

G'day,

TLDR; I think the current openssh TOS settings of AF21 and CS1 are a poor
match for the old TOS bit fields and are causing confusion. IMHO AF22 and
AF11 would be a better backwards-compatible choice. I think the
iptables use of "08x/0x3f" for Maximize-Throughput is only wrong in that
the mask 0x3f includes 1 bit of the DSCP "Precedence" field.

I sent a message to 923879@bugs.debian.org which bounced because it's
closed/archived, and found this bug as probably the most relevant open bug.

Reading this bug it looks to me like RFC 1349 confused Peter Lebbing by
using less-common big-endian bit numbering for their bit labels in the TOS
field and not making that very clear. Everything else I've seen has the
precedence bits in the Most-Significant-Bits of the octet including these;

http://www.rhyshaden.com/ipdgram.htm
https://my.ezycloud.com.au/knowledgebase/201204284/Quality-of-Service-QoS----DSCP-TOS-CoS-Precedence-Conversion-Chart.html

This is the convention ssh is currently using when it sets AF21 as 0x48 and
CS1 ax 0x20, and this appears to be what IP tables is also using.

This means I believe that iptables use of "08x/0x3f"
for Maximize-Throughput is only wrong in that the mask 0x3f includes 1 bit
of the DSCP "Precedence" field. I think the correct mask should be 0x1f,
though maybe there are reasons why you'd want to explicitly clear the LSB
of the DSCP Precedence field that I'm unaware of.

I stumbled upon https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879
trying to figure out problems with my QoS firewall setup and a recent
Debian ssh which is using AF21 and CS1 for interactive and bulk traffic
respectively. This is the new ssh upstream default, and after speaking to
one of the ssh developers (dtucker) it seems TOS standards are a mess and
everyone they've spoken to have given them different advice. There are now
more than two RFCs and 100k of text per TOS header bit. The choice of AF21
and CS1 was taken as an attempt to pick the least-wrong option, and no-one
has been able to clearly justify anything better.

The reason AF21 is being classified as "Maximize-Throughput" is because
AF21 sets the (old?) Maximize-Throughput bit, not the Minimize-Delay bit,
despite the various DSCP docs describing AF21 as "interactive" (actually,
all of AF2[123] is documented as for "low latency traffic"). Note CS1
doesn't set any of the old Delay/Throughput/Reliablity/Cost TOS bits so is
likely classified as "normal" traffic, not "bulk" by most older QoS
systems. See the following chart for details;

https://my.ezycloud.com.au/knowledgebase/201204284/Quality-of-Service-QoS----DSCP-TOS-CoS-Precedence-Conversion-Chart.html

It appears from the settings that AF21 was intended for important
high-volume traffic. It looks like AF22 was intended for important
low-latency traffic (it sets the Minimize-Delay bit) and is probably a
better fit for ssh interactive traffic. Note that the ssh devs probably
chose AF21 over AF22 because the DSCP docs say AF22 should be dropped
before AF21, implying AF21 is "more important" than AF22;

https://en.wikipedia.org/wiki/Differentiated_services#Assured_Forwarding

However, I think the DSCP standards structure the drop-order this way to
reflect that they are actually equally important (as indicated by the
precedence setting of "Immediate), but high-volume traffic is more likely
to need the packets during congestion. Note that AF23 which sets both
Minimize-Delay AND Maximize-Throughput has an even higher drop probability,
probably because that's the price you pay for asking for both high volume
AND low latency.

The use of CS1 for ssh bulk traffic is also problematic, since it doesn't
set any of the old TOS bits and thus looks "normal" to anything only
looking at the old Delay/Throughput/Reliablity/Cost TOS bits. The
equivalent DSCP "Priority" precedence that does set the Maximize-Throughput
bit is AF11, which does seem like a better fit than CS1. Another option
would be to set the Minimize-Cost bit, but that bit is not used by DSCP so
there is no DSCP class with that bit set.

So personally I'm now using the following setting in both
/etc/ssh/ssh_config and /etc/ssh/sshd_config to override the defaults.

IPQoS af22 af11

Note the client/server settings affect the traffic in each direction
independently; a server without this setting will send packets with
tos=0x48 (AKA AF21) to a client sending packets with tos=0x50 (AKA AF22)
that does have these settings.

My only problem now is it seems at least one old Debian ssh package
(1:6.7p1-5+deb8u8) and/or the VPS I'm using for one of my machines doesn't
seem to honor the IPQoS setting and is just using tos=0x0... but that's
mostly unrelated.


--
abo@minkirri.apana.org.au

[Message part 2 (text/html, inline)]

Send a report that this bug log contains spam.