Debian Bug report logs - #842040
Please add https support

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Marga Manterola <marga@google.com>

To: "submit@bugs.debian.org" <submit@bugs.debian.org>

Subject: Please add https support

Date: Tue, 25 Oct 2016 13:17:23 +0000

[Message part 1 (text/plain, inline)]

Package: debian-installer
Severity: normal

The installer currently doesn't support downloading packages from https
mirrors, because busybox's wget doesn't support https.

The wget package has been changed to provide a udeb, compiled against the
libssl udeb, so that it provides https support.

The change needed in d-i is to add the wget udeb and it's dependencies
(libcrypto1.0.2, libssl1.0.2 and libpcre3) to the initrd.

Regarding total size (as this was mentioned when I discussed this in the
past), this is the size of the needed udebs:
-rw-r----- 1 marga marga 878072 Sep 26 20:17
libcrypto1.0.2-udeb_1.0.2j-1_amd64.udeb
-rw-r----- 1 marga marga 153852 Aug 19 12:59 libpcre3-udeb_8.39-2_amd64.udeb
-rw-r----- 1 marga marga 141318 Sep 26 20:17
libssl1.0.2-udeb_1.0.2j-1_amd64.udeb
-rw-r----- 1 marga marga 147960 Sep 26 15:19 wget-udeb_1.18-4_amd64.udeb

And this is the size difference of the old initrd and the new one:
-rw-r----- 1 marga eng 22580707 Oct 25 02:07 initrd.gz
-rw-r----- 1 marga eng 23381570 Oct 25 15:11 newinitrd.gz

So, approximately 780k extra for the initrd image (3.5% increase)

Thanks!
-- 
Cheers,
Marga

[Message part 2 (text/html, inline)]

Message #10 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Roger Shimizu <rogershimizu@gmail.com>

To: Marga Manterola <marga@google.com>, Martin Michlmayr <tbm@cyrius.com>

Cc: 842040@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Wed, 26 Oct 2016 00:59:55 +0900

On Tue, Oct 25, 2016 at 10:17 PM, Marga Manterola <marga@google.com> wrote:
> Package: debian-installer
> Severity: normal
>
> So, approximately 780k extra for the initrd image (3.5% increase)

I'm not sure whether any libs already is included in the d-i image, if
not, adding 780k extra would definitely affect armel/orion5x qnap d-i
initrd image.

So I append Martin, the porter of armel/orion5x qnap, to CC list.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Message #15 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Marga Manterola <marga@google.com>, 842040@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Wed, 26 Oct 2016 08:36:03 +0200

[Message part 1 (text/plain, inline)]

On 10/25/2016 03:17 PM, Marga Manterola wrote:
> The installer currently doesn't support downloading packages from https
> mirrors, because busybox's wget doesn't support https.
> 
> The wget package has been changed to provide a udeb, compiled against
> the libssl udeb, so that it provides https support.
> 
> The change needed in d-i is to add the wget udeb and it's dependencies
> (libcrypto1.0.2, libssl1.0.2 and libpcre3) to the initrd.

Which I guess boils down to adding wget-udeb to the installer's
pkg-lists/base because I think all flavors and all architectures should
have the same feature set.

> Regarding total size (as this was mentioned when I discussed this in the
> past), this is the size of the needed udebs:
> -rw-r----- 1 marga marga 878072 Sep 26 20:17
> libcrypto1.0.2-udeb_1.0.2j-1_amd64.udeb
> -rw-r----- 1 marga marga 153852 Aug 19 12:59 libpcre3-udeb_8.39-2_amd64.udeb
> -rw-r----- 1 marga marga 141318 Sep 26 20:17
> libssl1.0.2-udeb_1.0.2j-1_amd64.udeb
> -rw-r----- 1 marga marga 147960 Sep 26 15:19 wget-udeb_1.18-4_amd64.udeb

libcrypto is already in there. So we end up with just libpcre3-udeb,
libssl1.0.2-udeb and wget-udeb.

> And this is the size difference of the old initrd and the new one:
> -rw-r----- 1 marga eng 22580707 Oct 25 02:07 initrd.gz
> -rw-r----- 1 marga eng 23381570 Oct 25 15:11 newinitrd.gz
> 
> So, approximately 780k extra for the initrd image (3.5% increase)

I'm not sure how you got this number (from a d-i rebuild?), but I end up
with 22580517B (~same) to 23106785B (2.3% increase) when rebuilding
amd64 netboot from d-i git. Of course it's way less percentage-wise for
the default amd64 netboot-gtk (which has a 44696935B initrd right now).

So at least size-wise this shouldn't be very controversial. Adding
wget-udeb to pkg-lists/base ends up with this:

$ lsinitrd dest/netboot/debian-installer/amd64/initrd.gz | grep wget
-rwxr-xr-x   1 root     root       409016 Sep 26 15:11 usr/bin/wget

So that seems to have the desired result. I did not try out the
resulting installer, though.

Kind regards
Philipp Kern

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Martin Michlmayr <tbm@cyrius.com>

To: Roger Shimizu <rogershimizu@gmail.com>

Cc: Marga Manterola <marga@google.com>, Philipp Kern <pkern@debian.org>, 842040@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Wed, 9 Nov 2016 20:45:39 -0800

* Roger Shimizu <rogershimizu@gmail.com> [2016-10-26 00:59]:
> > So, approximately 780k extra for the initrd image (3.5% increase)
> 
> I'm not sure whether any libs already is included in the d-i image, if
> not, adding 780k extra would definitely affect armel/orion5x qnap d-i
> initrd image.
> 
> So I append Martin, the porter of armel/orion5x qnap, to CC list.

Thanks for the CC.  I just added wget-udeb and it adds 345 KB,
which breaks the orion5x-qnap image.  However, this image is really
quite a special case and I don't want to block https support because
of it.  I can always exclude wget-udeb from this particular image.

-- 
Martin Michlmayr
http://www.cyrius.com/

Message #25 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Jose R R <Jose.r.r@metztli-it.com>

To: Marga Manterola <marga@google.com>, 842040@bugs.debian.org

Cc: ReiserFS Development List <reiserfs-devel@vger.kernel.org>

Subject: Re: Bug#842040: Please add https support

Date: Thu, 10 Nov 2016 01:14:33 -0800

Niltze [Hello]!

On Tue, Oct 25, 2016 at 6:17 AM, Marga Manterola <marga@google.com> wrote:
> Package: debian-installer
> Severity: normal
>
> The installer currently doesn't support downloading packages from https
> mirrors, because busybox's wget doesn't support https.

In order to add SSL support to BusyBox wget, itself to be used wih
ssl_helper, I used matrixssl-3-4-2-open.tgz
< http://www.matrixssl.org/ >

to build my custom BusyBox udeb for my Reiser4-enabled Debian-Installer
< https://sf.net/projects/debian-reiser4/ >

No idea if suggestion fulfills bug need - just my 2 cents ;-)

>
> The wget package has been changed to provide a udeb, compiled against the
> libssl udeb, so that it provides https support.
>
> The change needed in d-i is to add the wget udeb and it's dependencies
> (libcrypto1.0.2, libssl1.0.2 and libpcre3) to the initrd.
>
> Regarding total size (as this was mentioned when I discussed this in the
> past), this is the size of the needed udebs:
> -rw-r----- 1 marga marga 878072 Sep 26 20:17
> libcrypto1.0.2-udeb_1.0.2j-1_amd64.udeb
> -rw-r----- 1 marga marga 153852 Aug 19 12:59 libpcre3-udeb_8.39-2_amd64.udeb
> -rw-r----- 1 marga marga 141318 Sep 26 20:17
> libssl1.0.2-udeb_1.0.2j-1_amd64.udeb
> -rw-r----- 1 marga marga 147960 Sep 26 15:19 wget-udeb_1.18-4_amd64.udeb
>
> And this is the size difference of the old initrd and the new one:
> -rw-r----- 1 marga eng 22580707 Oct 25 02:07 initrd.gz
> -rw-r----- 1 marga eng 23381570 Oct 25 15:11 newinitrd.gz
>
> So, approximately 780k extra for the initrd image (3.5% increase)
>
> Thanks!
> --
> Cheers,
> Marga


Best Professional Regards

-- 
Jose R R
http://metztli.it
---------------------------------------------------------------------------------------------
Try at no charge http://b2evolution.net for http://OpenShift.com PaaS
---------------------------------------------------------------------------------------------
from our GitHub http://Nepohualtzintzin.com repository. Cloud the easy way!
---------------------------------------------------------------------------------------------

Message #30 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Josh Triplett <josh@joshtriplett.org>

To: 842040@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Sat, 12 Nov 2016 09:16:54 -0800

On Thu, 10 Nov 2016 01:14:33 -0800 Jose R R <Jose.r.r@metztli-it.com> wrote:
> On Tue, Oct 25, 2016 at 6:17 AM, Marga Manterola <marga@google.com> wrote:
> > Package: debian-installer
> > Severity: normal
> >
> > The installer currently doesn't support downloading packages from https
> > mirrors, because busybox's wget doesn't support https.
> 
> In order to add SSL support to BusyBox wget, itself to be used wih
> ssl_helper, I used matrixssl-3-4-2-open.tgz
> < http://www.matrixssl.org/ >
> 
> to build my custom BusyBox udeb for my Reiser4-enabled Debian-Installer
> < https://sf.net/projects/debian-reiser4/ >
> 
> No idea if suggestion fulfills bug need - just my 2 cents ;-)

Can you provide a link for your patches to busybox wget to add SSL
support?  That sounds like it'd substantially decrease size compared to
including GNU wget and supporting libraries.

- Josh Triplett

Message #35 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Josh Triplett <josh@joshtriplett.org>, 842040@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Fri, 18 Nov 2016 17:18:38 +0100

[Message part 1 (text/plain, inline)]

On 12.11.2016 18:16, Josh Triplett wrote:
> On Thu, 10 Nov 2016 01:14:33 -0800 Jose R R <Jose.r.r@metztli-it.com> wrote:
>> On Tue, Oct 25, 2016 at 6:17 AM, Marga Manterola <marga@google.com> wrote:
>>> Package: debian-installer
>>> Severity: normal
>>>
>>> The installer currently doesn't support downloading packages from https
>>> mirrors, because busybox's wget doesn't support https.
>>
>> In order to add SSL support to BusyBox wget, itself to be used wih
>> ssl_helper, I used matrixssl-3-4-2-open.tgz
>> < http://www.matrixssl.org/ >
>>
>> to build my custom BusyBox udeb for my Reiser4-enabled Debian-Installer
>> < https://sf.net/projects/debian-reiser4/ >
>>
>> No idea if suggestion fulfills bug need - just my 2 cents ;-)
> 
> Can you provide a link for your patches to busybox wget to add SSL
> support?  That sounds like it'd substantially decrease size compared to
> including GNU wget and supporting libraries.

Unfortunately matrixssl isn't even in Debian at this point.

Kind regards
Philipp Kern

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Martin Michlmayr <tbm@cyrius.com>, 842040@bugs.debian.org, Roger Shimizu <rogershimizu@gmail.com>

Cc: Marga Manterola <marga@google.com>, Philipp Kern <pkern@debian.org>

Subject: Re: Bug#842040: Please add https support

Date: Fri, 18 Nov 2016 17:19:12 +0100

[Message part 1 (text/plain, inline)]

On 10.11.2016 05:45, Martin Michlmayr wrote:
> * Roger Shimizu <rogershimizu@gmail.com> [2016-10-26 00:59]:
>>> So, approximately 780k extra for the initrd image (3.5% increase)
>>
>> I'm not sure whether any libs already is included in the d-i image, if
>> not, adding 780k extra would definitely affect armel/orion5x qnap d-i
>> initrd image.
>>
>> So I append Martin, the porter of armel/orion5x qnap, to CC list.
> 
> Thanks for the CC.  I just added wget-udeb and it adds 345 KB,
> which breaks the orion5x-qnap image.  However, this image is really
> quite a special case and I don't want to block https support because
> of it.  I can always exclude wget-udeb from this particular image.

So how do we move forward here? Exclude wget-udeb from the orion5x-qnap
image and otherwise include it by default?

Kind regards and thanks
Philipp Kern

[signature.asc (application/pgp-signature, attachment)]

Message #45 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Martin Michlmayr <tbm@cyrius.com>

To: Philipp Kern <pkern@debian.org>

Cc: 842040@bugs.debian.org, Roger Shimizu <rogershimizu@gmail.com>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Fri, 18 Nov 2016 10:22:51 -0800

* Philipp Kern <pkern@debian.org> [2016-11-18 17:19]:
> > Thanks for the CC.  I just added wget-udeb and it adds 345 KB,
> > which breaks the orion5x-qnap image.  However, this image is really
> > quite a special case and I don't want to block https support because
> > of it.  I can always exclude wget-udeb from this particular image.
> 
> So how do we move forward here? Exclude wget-udeb from the orion5x-qnap
> image and otherwise include it by default?

That should work.

-- 
Martin Michlmayr
http://www.cyrius.com/

Message #50 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Jose R R <Jose.r.r@metztli-it.com>

To: Philipp Kern <pkern@debian.org>, 842040@bugs.debian.org

Cc: Josh Triplett <josh@joshtriplett.org>

Subject: Re: Bug#842040: Please add https support

Date: Fri, 18 Nov 2016 21:54:33 -0800

On Fri, Nov 18, 2016 at 8:18 AM, Philipp Kern <pkern@debian.org> wrote:
> On 12.11.2016 18:16, Josh Triplett wrote:
>> On Thu, 10 Nov 2016 01:14:33 -0800 Jose R R <Jose.r.r@metztli-it.com> wrote:
>>> On Tue, Oct 25, 2016 at 6:17 AM, Marga Manterola <marga@google.com> wrote:
>>>> Package: debian-installer
>>>> Severity: normal
>>>>
>>>> The installer currently doesn't support downloading packages from https
>>>> mirrors, because busybox's wget doesn't support https.
>>>
>>> In order to add SSL support to BusyBox wget, itself to be used wih
>>> ssl_helper, I used matrixssl-3-4-2-open.tgz
>>> < http://www.matrixssl.org/ >
>>>
>>> to build my custom BusyBox udeb for my Reiser4-enabled Debian-Installer
>>> < https://sf.net/projects/debian-reiser4/ >
>>>
>>> No idea if suggestion fulfills bug need - just my 2 cents ;-)
>>
>> Can you provide a link for your patches to busybox wget to add SSL
>> support?  That sounds like it'd substantially decrease size compared to
>> including GNU wget and supporting libraries.
>
> Unfortunately matrixssl isn't even in Debian at this point.
>
The official Debian maintainer(s) busybox source lacks a directory
that is upstream in the busybox official source.
For instance, for upstream busybox-1.25.1, the relevant path to *that*
directory is:

busybox-1.25.1/networking/ssl_helper

then,
If we untar matrixssl-3-4-2-open.tgz , we get directory

matrixssl-3-4-2-open/

copying ssl_helper directory into matrixssl-3-4-2-open/., we build
first matrixssl:

make

After successful build:

cd ssl_helper

./ssl_helper.sh  ## to compile and link the helper, thus

ssl_helper.c generates ssl_helper executable (and ssl_helper.o ? ;-)

To provide SSL-enabled busybox wget you only need the executable ssl_helper

hence,

Prior to building your busybox udeb

create a directory inside your 'official' Debian maintainers busybox
untarred source

mkdir --verbose -p _install/usr/bin

copy your generated executable ssl_helper into above directory, i.e.,

_install/usr/bin/ssl_helper   ##<== executable should be this and here

From this dirty hack, *any* Debian-helper-generated busybox UDEB
should contain busybox SSL-enabled wget

Sorry, I am busy - but will try to create a patch for official Debian
source from git -- though I am not an official Debian maintainer.

sample from my reiser4 Debian-Installer (d-i):
< https://metztli.it/readOnlyEphemeral/busybox-udeb_1.22.0-19.1_amd64.udeb.SHA256SUM
># Text contents
< https://metztli.it/readOnlyEphemeral/busybox-udeb_1.22.0-19.1_amd64.udeb >

If above udeb link is downloaded locally, it may be expanded to analyze as:
dpkg -X busybox-udeb_1.22.0-19.1_amd64.udeb  tempDir

Happy hacking!


Best Professional Regards.

-- 
Jose R R
http://metztli.it
---------------------------------------------------------------------------------------------
Download Debian-Reiser4 for AMD64 https://sf.net/projects/debian-reiser4/
---------------------------------------------------------------------------------------------
Try at no charge http://b2evolution.net for http://OpenShift.com PaaS
---------------------------------------------------------------------------------------------
from our GitHub http://Nepohualtzintzin.com repository. Cloud the easy way!
---------------------------------------------------------------------------------------------

Message #55 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Jose R R <Jose.r.r@metztli-it.com>, 842040@bugs.debian.org

Cc: Philipp Kern <pkern@debian.org>, Josh Triplett <josh@joshtriplett.org>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 05:46:27 +0100

[Message part 1 (text/plain, inline)]

Jose R R <Jose.r.r@metztli-it.com> (2016-11-18):
> The official Debian maintainer(s) busybox source lacks a directory
> that is upstream in the busybox official source.

I think the main issue here is that busybox in Debian lacks a
maintainer.

The other is that matrixssl isn't packaged in Debian, as already
pointed out.


KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #60 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Philipp Kern <pkern@debian.org>, 842040@bugs.debian.org

Cc: Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 05:52:14 +0100

[Message part 1 (text/plain, inline)]

Hi all,

Philipp Kern <pkern@debian.org> (2016-10-26):
> Which I guess boils down to adding wget-udeb to the installer's
> pkg-lists/base because I think all flavors and all architectures
> should have the same feature set.

That would seem fair to me.

> I'm not sure how you got this number (from a d-i rebuild?), but I end
> up with 22580517B (~same) to 23106785B (2.3% increase) when rebuilding
> amd64 netboot from d-i git. Of course it's way less percentage-wise
> for the default amd64 netboot-gtk (which has a 44696935B initrd right
> now).

I think Marga mentioned on IRC she was unpacking/repacking the initrd
manually.

> So at least size-wise this shouldn't be very controversial. Adding
> wget-udeb to pkg-lists/base ends up with this:
> 
> $ lsinitrd dest/netboot/debian-installer/amd64/initrd.gz | grep wget
> -rwxr-xr-x   1 root     root       409016 Sep 26 15:11 usr/bin/wget
> 
> So that seems to have the desired result. I did not try out the
> resulting installer, though.

Well, I think this is a crucial issue: what use case(s) are you trying
to fix? “We want https” isn't clear to me.

Besides wget supporting https, is there any work needed on the retriever
side? What about trust chains, do you have any bundled list of trusted
CAs? Do you want to be able to rebuild d-i with a specific trusted CA,
and trust none by default?


KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #65 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Cyril Brulebois <kibi@debian.org>, 842040@bugs.debian.org

Cc: Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 11:34:13 +0100

On 20.11.2016 05:52, Cyril Brulebois wrote:
> Well, I think this is a crucial issue: what use case(s) are you trying
> to fix? “We want https” isn't clear to me.

After d-i has installed the system, we use HTTPS with client
certificates - using apt-transport-https. The use case there is
authentication and be allowed to fetch packages from any network,
including the Internet. During d-i we unfortunately still have to rely
on network trust, where we run against the company policy of not having
unencrypted services. Plus we'd need to have various non-HTTPS endpoints
(packages, configuration, images[1]) in addition to the HTTPS ones we
already have, which complicates maintenance. You'd think that we aren't
the only ones who'd host configuration behind a HTTPS server, though[2].
That we also serve all of the packages through HTTPS is just a byproduct.

> Besides wget supporting https, is there any work needed on the retriever
> side? What about trust chains, do you have any bundled list of trusted
> CAs? Do you want to be able to rebuild d-i with a specific trusted CA,
> and trust none by default?

I can say what works for us: adding another cpio archive to the netboot
that contains files in /etc/ssl/certs (PEM files plus the result of
c_rehash). You can pass multiple initrds to the kernel and it will
unpack them one by one, which easily allows to add more files and
overwrite existing ones (but not to remove files, AFAIK). It's not
really much worse than other bits of configuration, like preseeds.
Embedding another binary like wget and not just scripts, however, is
more tricky (getting dependencies right, fighting against mklibs
removing symbols - which I guess was... fixed).

But you are absolutely correct in for this to be universally useful,
we'd also need a ca-certificates-udeb. I can take a look at that but I
somewhat fear that it won't be that much smaller than the regular one
(maybe ~150k udeb size).

Kind regards and thanks
Philipp Kern

[1] We extended d-i to download image files of system installs.
[2] Thinking of preseed/url across the Internet. I used to need that for
s390x installs.

Message #70 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Philipp Kern <pkern@debian.org>

Cc: 842040@bugs.debian.org, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 11:45:05 +0100

[Message part 1 (text/plain, inline)]

Philipp Kern <pkern@debian.org> (2016-11-20):
> On 20.11.2016 05:52, Cyril Brulebois wrote:
> > Well, I think this is a crucial issue: what use case(s) are you trying
> > to fix? “We want https” isn't clear to me.
> 
> After d-i has installed the system, we use HTTPS with client
> certificates - using apt-transport-https. The use case there is
> authentication and be allowed to fetch packages from any network,
> including the Internet. During d-i we unfortunately still have to rely
> on network trust, where we run against the company policy of not having
> unencrypted services. Plus we'd need to have various non-HTTPS endpoints
> (packages, configuration, images[1]) in addition to the HTTPS ones we
> already have, which complicates maintenance. You'd think that we aren't
> the only ones who'd host configuration behind a HTTPS server, though[2].
> That we also serve all of the packages through HTTPS is just a byproduct.
> 
> > Besides wget supporting https, is there any work needed on the retriever
> > side? What about trust chains, do you have any bundled list of trusted
> > CAs? Do you want to be able to rebuild d-i with a specific trusted CA,
> > and trust none by default?
> 
> I can say what works for us: adding another cpio archive to the netboot
> that contains files in /etc/ssl/certs (PEM files plus the result of
> c_rehash). You can pass multiple initrds to the kernel and it will
> unpack them one by one, which easily allows to add more files and
> overwrite existing ones (but not to remove files, AFAIK). It's not
> really much worse than other bits of configuration, like preseeds.
> Embedding another binary like wget and not just scripts, however, is
> more tricky (getting dependencies right, fighting against mklibs
> removing symbols - which I guess was... fixed).
> 
> But you are absolutely correct in for this to be universally useful,
> we'd also need a ca-certificates-udeb. I can take a look at that but I
> somewhat fear that it won't be that much smaller than the regular one
> (maybe ~150k udeb size).

If you're going to need another cpio archive with PEM files, can't you
just add the needed bits (wget & libraries) for https there?

Adding packages for every single image just so that Google people can
append a cpio archive with some CAs doesn't look too reasonable to me:
you need to do extra work on your end anyway, and everybody pays that
price without getting any advantage…


KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #75 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Cyril Brulebois <kibi@debian.org>

Cc: 842040@bugs.debian.org, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 11:52:09 +0100

On 20.11.2016 11:45, Cyril Brulebois wrote:
>> But you are absolutely correct in for this to be universally useful,
>> we'd also need a ca-certificates-udeb. I can take a look at that but I
>> somewhat fear that it won't be that much smaller than the regular one
>> (maybe ~150k udeb size).
> 
> If you're going to need another cpio archive with PEM files, can't you
> just add the needed bits (wget & libraries) for https there?
> 
> Adding packages for every single image just so that Google people can
> append a cpio archive with some CAs doesn't look too reasonable to me:
> you need to do extra work on your end anyway, and everybody pays that
> price without getting any advantage…

Well, I said why adding wget plus somehow determining the required
libraries is harder than just adding some static content.[1] We also
wouldn't need to do the PEM cpio dance if ca-certificates-udeb would be
part of the image. We don't need to add an internal CA or something like
that.

I understand the bit about paying the price, which is why I tried to
address that in my reply as well. Recent discussions on -devel showed
that there's a general interest in HTTPS enablement.

Kind regards
Philipp Kern

[1] Unless we rebuild d-i, which we could do.

Message #80 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Julien Cristau <jcristau@debian.org>

To: Philipp Kern <pkern@debian.org>, 842040@bugs.debian.org

Cc: Cyril Brulebois <kibi@debian.org>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 12:10:43 +0100

On Sun, Nov 20, 2016 at 11:52:09 +0100, Philipp Kern wrote:

> On 20.11.2016 11:45, Cyril Brulebois wrote:
> >> But you are absolutely correct in for this to be universally useful,
> >> we'd also need a ca-certificates-udeb. I can take a look at that but I
> >> somewhat fear that it won't be that much smaller than the regular one
> >> (maybe ~150k udeb size).
> > 
> > If you're going to need another cpio archive with PEM files, can't you
> > just add the needed bits (wget & libraries) for https there?
> > 
> > Adding packages for every single image just so that Google people can
> > append a cpio archive with some CAs doesn't look too reasonable to me:
> > you need to do extra work on your end anyway, and everybody pays that
> > price without getting any advantage…
> 
> Well, I said why adding wget plus somehow determining the required
> libraries is harder than just adding some static content.[1] We also
> wouldn't need to do the PEM cpio dance if ca-certificates-udeb would be
> part of the image. We don't need to add an internal CA or something like
> that.
> 
I think until there's a ca-certificates-udeb, adding wget for https in
all images isn't reasonable, vs google rebuilding d-i with added wget
and the PEM bits you need.  I guess ca-certificates-udeb would need some
way to preseed a list of trusted CAs.

Cheers,
Julien

Message #85 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Julien Cristau <jcristau@debian.org>

Cc: 842040@bugs.debian.org, Cyril Brulebois <kibi@debian.org>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Sun, 20 Nov 2016 16:49:57 +0100

On 2016-11-20 12:10, Julien Cristau wrote:
> I think until there's a ca-certificates-udeb, adding wget for https in
> all images isn't reasonable, vs google rebuilding d-i with added wget
> and the PEM bits you need.  I guess ca-certificates-udeb would need 
> some
> way to preseed a list of trusted CAs.

I have no problem working on that. Given that today ca-certificates only 
contains Mozilla's set I don't think it's necessarily required to 
provide that preseeding option (i.e. it could be added by someone who 
cared enough later).

The problem with rebuilding d-i is that you can't really do it from the 
source package in unstable, you need to do it from the VCS.

It boils down to the question if it's useful beyond just us. :)

Kind regards
Philipp Kern

Message #92 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Rick Thomas <rbthomas@pobox.com>

To: Martin Michlmayr <tbm@cyrius.com>, 842040@bugs.debian.org

Cc: Philipp Kern <pkern@debian.org>, Roger Shimizu <rogershimizu@gmail.com>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Thu, 24 Nov 2016 21:28:25 -0800

On Nov 18, 2016, at 10:22 AM, Martin Michlmayr <tbm@cyrius.com> wrote:

> * Philipp Kern <pkern@debian.org> [2016-11-18 17:19]:
>>> Thanks for the CC.  I just added wget-udeb and it adds 345 KB,
>>> which breaks the orion5x-qnap image.  However, this image is really
>>> quite a special case and I don't want to block https support because
>>> of it.  I can always exclude wget-udeb from this particular image.
>> 
>> So how do we move forward here? Exclude wget-udeb from the orion5x-qnap
>> image and otherwise include it by default?
> 
> That should work.

Are there other machines that have equally sever size restrictions?

Rick

Message #97 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Martin Michlmayr <tbm@cyrius.com>

To: Rick Thomas <rbthomas@pobox.com>

Cc: 842040@bugs.debian.org, Philipp Kern <pkern@debian.org>, Roger Shimizu <rogershimizu@gmail.com>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Fri, 25 Nov 2016 11:13:57 -0800

* Rick Thomas <rbthomas@pobox.com> [2016-11-24 21:28]:
> >> So how do we move forward here? Exclude wget-udeb from the orion5x-qnap
> >> image and otherwise include it by default?
> > 
> > That should work.
> 
> Are there other machines that have equally sever size restrictions?

I don't think so.
-- 
Martin Michlmayr
http://www.cyrius.com/

Message #102 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <atomo64@gmail.com>

To: Philipp Kern <pkern@debian.org>

Cc: Julien Cristau <jcristau@debian.org>, 842040@bugs.debian.org, Cyril Brulebois <kibi@debian.org>, Marga Manterola <marga@google.com>, 839707@bugs.debian.org, 839707-submitter@bugs.debian.org

Subject: Re: Bug#842040: Please add https support

Date: Fri, 25 Nov 2016 20:28:13 +0100

Hi,

On Sunday, 20 November 2016 16:49:57 CET Philipp Kern wrote:
> On 2016-11-20 12:10, Julien Cristau wrote:
> > I think until there's a ca-certificates-udeb, adding wget for https in
> > all images isn't reasonable, vs google rebuilding d-i with added wget
> > and the PEM bits you need.  I guess ca-certificates-udeb would need
> > some way to preseed a list of trusted CAs.
[...]
> The problem with rebuilding d-i is that you can't really do it from the
> source package in unstable, you need to do it from the VCS.
> 
> It boils down to the question if it's useful beyond just us. :)

FWIW, at work we've also had the need of https (and ftps) support in d-i for 
retrieving preseeds and some other files plus uploading a few logs.

Given the need of ftps we've switched from the then-proposed wget-udeb to a 
curl-based one (#839707). It is more flexible and future-proof, all in all.

As for the certificates, we don't use ca-certificates at all, we use a $company 
CA.

The above is just a part of what we end up injecting into d-i. So even though 
adding something like the curl udebs would come handy, at this point we still 
need to build a custom media.

Just my two cents, and not on behalf of my employer.

Cheers,
-- 
Raphael Geissert

Message #107 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: 842040@bugs.debian.org, Cyril Brulebois <kibi@debian.org>, Martin Michlmayr <tbm@cyrius.com>

Cc: Julien Cristau <jcristau@debian.org>, Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Thu, 8 Dec 2016 09:18:37 +0100

[Message part 1 (text/plain, inline)]

On 11/20/2016 12:10 PM, Julien Cristau wrote:
> I think until there's a ca-certificates-udeb, adding wget for https in
> all images isn't reasonable, vs google rebuilding d-i with added wget
> and the PEM bits you need.  I guess ca-certificates-udeb would need some
> way to preseed a list of trusted CAs.

I just tried it out with the following patch to the base package list:

diff --git a/build/pkg-lists/base b/build/pkg-lists/base
index 3da0e4c..6f1d955 100644
--- a/build/pkg-lists/base
+++ b/build/pkg-lists/base
@@ -25,3 +25,6 @@ ca-certificates-udeb

 libkmod2-udeb [linux]
 kldutils-udeb [kfreebsd]
+
+wget-udeb
+ca-certificates-udeb

choose-mirror does not ask for the protocol by default, as the question
is priority medium. I did my installation by passing priority=medium on
the command-line, but you could as well preseed the protocol to https I
think. In that case it does not show a list of mirrors (because
Mirrorlist does not list https capabilities), but works just fine with
deb.debian.org, which points to Cloudfront for HTTPS support. d-i
component load worked, debootstrap worked and the resulting chroot had
apt-transport-https and a sources.list pointing to
https://deb.debian.org. The security archive was added without https,
but that's unavoidable at this point given that it does not actually
support it.

As for not breaking orion5x images, I suppose the following could do the
trick:

diff --git a/build/pkg-lists/netboot/armel/orion5x.cfg
b/build/pkg-lists/netboot/armel/orion5x.cfg
index 9fc7584..c0c8b83 100644
--- a/build/pkg-lists/netboot/armel/orion5x.cfg
+++ b/build/pkg-lists/netboot/armel/orion5x.cfg
@@ -1,2 +1,6 @@
 # To control the LED and beeper on Buffalo devices
 micro-evtd-udeb
+
+# Do not include HTTPS support to keep the image small.
+wget-udeb -
+ca-certificates-udeb -

However this is untested on armel because abel died on me when I tried
to set up my chroot and debian-installer does not support
cross-compilation. I tried out the same through amd64.cfg, overriding
base and it worked for me.

So I suppose this should be ok to commit and push?

Kind regards and thanks
Philipp Kern

[signature.asc (application/pgp-signature, attachment)]

Message #112 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Julien Cristau <jcristau@debian.org>

To: Philipp Kern <pkern@debian.org>, 842040@bugs.debian.org, Cyril Brulebois <kibi@debian.org>, Martin Michlmayr <tbm@cyrius.com>

Cc: Marga Manterola <marga@google.com>

Subject: Re: Bug#842040: Please add https support

Date: Thu, 8 Dec 2016 09:58:05 -1000

On 12/07/2016 10:18 PM, Philipp Kern wrote:
> choose-mirror does not ask for the protocol by default, as the question
> is priority medium. I did my installation by passing priority=medium on
> the command-line, but you could as well preseed the protocol to https I
> think. In that case it does not show a list of mirrors (because
> Mirrorlist does not list https capabilities), but works just fine with
> deb.debian.org, which points to Cloudfront for HTTPS support. d-i
> component load worked, debootstrap worked and the resulting chroot had
> apt-transport-https and a sources.list pointing to
> https://deb.debian.org. The security archive was added without https,
> but that's unavoidable at this point given that it does not actually
> support it.

We could use deb.debian.org as default security mirror, I guess.  Or add
a https vhost on our security mirrors.  Definitely not a blocker for
landing this though.

[...]
> So I suppose this should be ok to commit and push?
> 
No objection from me, at least.  Thanks for doing this work.

Cheers,
Julien

Message #119 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Josh Triplett <josh@joshtriplett.org>

To: 842040@bugs.debian.org, Philipp Kern <pkern@debian.org>

Subject: https support: pending?

Date: Tue, 10 Jan 2017 00:23:23 -0800

Bug 842040 got tagged pending on 2016-12-10.  Did the support get
committed?  I'd love to see this available in stretch, particularly for
https preseed support.

- Josh Triplett

Message #124 received at 842040@bugs.debian.org (full text, mbox, reply):

From: Josh Triplett <josh@joshtriplett.org>

To: 842040@bugs.debian.org, Philipp Kern <pkern@debian.org>

Subject: Re: https support: pending?

Date: Tue, 10 Jan 2017 00:56:20 -0800

On Tue, Jan 10, 2017 at 12:23:23AM -0800, Josh Triplett wrote:
> Bug 842040 got tagged pending on 2016-12-10.  Did the support get
> committed?  I'd love to see this available in stretch, particularly for
> https preseed support.

Just did some digging, and it looks like this works in the latest d-i
daily snapshot.

Thanks to everyone who worked on it!

- Josh Triplett

Message #129 received at 842040-close@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: 842040-close@bugs.debian.org

Subject: Bug#842040: fixed in debian-installer 20170112

Date: Thu, 12 Jan 2017 11:18:39 +0000

Source: debian-installer
Source-Version: 20170112

We believe that the bug you reported is fixed in the latest version of
debian-installer, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 842040@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Cyril Brulebois <kibi@debian.org> (supplier of updated debian-installer package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Thu, 12 Jan 2017 11:08:53 +0100
Source: debian-installer
Binary: debian-installer
Architecture: source
Version: 20170112
Distribution: unstable
Urgency: medium
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Cyril Brulebois <kibi@debian.org>
Description:
 debian-installer - Debian Installer documentation
Closes: 837926 842040 843916
Changes:
 debian-installer (20170112) unstable; urgency=medium
 .
   [ Cyril Brulebois ]
   * Bump Linux kernel version from 4.7.0-1 to 4.8.0-1.
   * Add “set -e;” before while loops in a few arm* config files
     to make sure errors are reported (Closes: #843916).
   * Revert the following workaround after changes in grub2 2.02~beta3:
     config/kfreebsd.cfg: Put xorriso's -J flag after “--” to work around
     grub-mkrescue's change in option handling (See: #741656).
   * Exclude libssl1.1-udeb as well for qnap, wget will likely depend on
     it later on.
   * Update translation-status for the release.
 .
   [ Adam Conrad ]
   * build/util/efi-image: Fix up FAT filesystem size calculation:
     - We appear to need 32k headroom, not 24k; this was masked
       by rounding errors and blind luck until grub grew 1.5kB.
     - Break out the magic headroom number into 32+31 to match
       the comment and make the arithmetic more obvious to read.
     - Round file sizes up, not down, so we don't break in future.
 .
   [ Martin Michlmayr ]
   * Disable u-boot images for OpenRD because they've been removed
     (see #837629).
   * Exclude wpasupplicant-udeb on orion5x-qnap because it brings in
     libcrypto1.1-udeb.
 .
   [ Aurélien COUDERC ]
   * Update theme to Soft Waves by Juliette Belin.
 .
   [ Christian Perrier ]
   * Replace fonts-lklug-sinhala by fonts-noto-hinted-udeb for Sinhala.
     Closes: #837926
 .
   [ Cyril Brulebois ]
   * Bump Linux kernel version from 4.8.0-1 to 4.8.0-2.
 .
   [ Philipp Kern ]
   * Add https support to the installer. (Closes: #842040)
 .
   [ Philip Hands ]
   * pkg-lists/gtk-common: ttf-dejavu-udeb is now called fonts-dejavu-udeb
 .
   [ Samuel Thibault ]
   * Bump gnumach kernel version.
Checksums-Sha1:
 7fc8a9653714d550ba6d792c4fe359fd602f5b03 3553 debian-installer_20170112.dsc
 51c9676cb7e42140b1d0988e8916462ea6d9c668 1393377 debian-installer_20170112.tar.gz
Checksums-Sha256:
 c40acceb088dff41a723224a53d5f981f3363a77f00cbc35dd8aafd97fca0171 3553 debian-installer_20170112.dsc
 885092a9771f822b442ee4f783728687cc6e84226bb8c82b53a544e4d89556ea 1393377 debian-installer_20170112.tar.gz
Files:
 154503fa11ecef6df78d568118c003c3 3553 devel optional debian-installer_20170112.dsc
 2989549cf35d1ef394ce2d771253c830 1393377 devel optional debian-installer_20170112.tar.gz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJYd2GyAAoJEP+RSvDCs1UgSQAP/2MpjOHQ8hHd0RMk/DyHEmQV
kkIVHAdql+nJh/+pgcICmbHgIUYPUa926/ZfzinmIcOGod/J+4gXmlpdWiSfZqlK
GyZpu5yBHOPgxkqZxV0tEyEQNchHHnCZPuqu73Dk1KM89J15yVijbW1v7oaxKH3Z
FktHu67KdwxZDzdyd6soXxg86qsDqKlCadifKEmMcxQhmatVPqHIChttEVH7Hf74
3B6iJkO0Q7/12fH2l3Px+5KF9yGWXAmYYsRGRda8Fw5aw/iSWYbqgom1wskbSbME
ISE/dJmhTMUItFe7qhPTwYcZ6TqYp9IH7KaTNgwtJI+sY6ylcY0Xcru3Isg0ADOY
NJ6jst1Fm0eSwC57v1aHj+KGHCjgQjtXE9BWbD8QiZ3PeezBJ++eK0hrTTkWlUVU
gLr/tykSj2Nwck0YLU9FCbfqwDcROueBQypn8uF1FPA1XgQ6zTwK175Pbj2xRgbp
DisScwX1s+rVcvwnh4HKReXjAU/T+8r5SyyKdffTeIyNpvZp5qewsQao8ymsIE+X
5JmR0d+G4/TszD6/twCH6qwz6FV4JvubDyurAHcw4QVDXdKc7424O0iCcUF03e0o
P7J1QqiTKzczgQGEg0t3GndJKlIIuMe1UV2FUMPtWvcqF4LxpfeQt91h/9D1+fVJ
i58IuL32Gv9Uvz3A2+9K
=Nt7Q
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.