Debian Bug report logs - #813313
[planet-venus] planet-venus fails on SNI enabled websites

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ingo Juergensmann <ij@2013.bluespice.org>

To: submit@bugs.debian.org

Subject: [planet-venus] planet-venus fails on SNI enabled websites

Date: Sun, 31 Jan 2016 14:45:22 +0100

Package: planet-venus
Version: 0~git9de2109-3
Severity: normal

--- Please enter the report below this line. ---

Apparently planet-venus fails to fetch feeds from SNI enabled hosts.
After migrating my blog from HTTP to HTTPS with letsencrypt.org
certificate, I noticed that Planet Debian fails to include my blog
postings.

When following https://wiki.debian.org/PlanetDebian "Testing changes to
the config/templates" I noticed this error:

ij@debian:~/planet$ grep blog.windfluechter log/planet.log.0
INFO:planet.runner:Fetching
https://blog.windfluechter.net/taxonomy/term/2/feed via 5
ERROR:planet.runner:HttpLib2Error: Server presented certificate that
does not match host blog.windfluechter.net: {'subjectAltName': (('DNS',
'abi94oesede.de'), ('DNS', 'www.abi94oesede.de')), 'notBefore': u'Jan 26
18:05:00 2016 GMT', 'caIssuers':
(u'http://cert.int-x1.letsencrypt.org/',), 'OCSP':
(u'http://ocsp.int-x1.letsencrypt.org/',), 'serialNumber':
u'01839A051BF9D2873C0A3BAA9FD0227C54D1', 'notAfter': 'Apr 25 18:05:00
2016 GMT', 'version': 3L, 'subject': ((('commonName',
u'abi94oesede.de'),),), 'issuer': ((('countryName', u'US'),),
(('organizationName', u"Let's Encrypt"),), (('commonName', u"Let's
Encrypt Authority X1"),))} via 5
ERROR:planet.runner:Error 500 while updating feed
https://blog.windfluechter.net/taxonomy/term/2/feed

When directly using my feed link in a browser everything works without
any problems and the correct certificate is loaded.

Grepping through my planet.log reveals that I might not be the only one
experiencing this kind of problems:

ij@debian:~/planet$ grep "Error 500" log/planet.log.0 | grep -c https
9

PS: I've tested this on Debian stable, but another DD has been
confirming that this happens on unstable as well.

--- System information. ---
Architecture: amd64
Kernel:       Linux 4.3.0-1-amd64

Debian Release: stretch/sid
  500 unstable        www.deb-multimedia.org
  500 unstable        ftp.de.debian.org

--- Package information. ---
Package's Depends field is empty.

Package's Recommends field is empty.

Package's Suggests field is empty.




-- 
Ciao...            //      Fon: 0381-2744150
      Ingo       \X/       http://blog.windfluechter.net
Please don't share this address with Facebook or Google!
gpg pubkey: http://www.juergensmann.de/ij_public_key.asc

Message #10 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 813313@bugs.debian.org

Subject: raising severity to important

Date: Tue, 9 Feb 2016 15:24:03 +0100

[Message part 1 (text/plain, inline)]

control: severity -1 important

Hi,

according to https://blog.windfluechter.net/content/blog/2016/02/09/1719-
letsencrypt-when-your-blog-entries-dont-show-planet-debian there are 18 blogs 
not showing up on planet.debian.org and the usage of https is thankfully 
rising further, thus making the impact of this bug bigger all the time, thus 
raising the severity.


cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 813313@bugs.debian.org (full text, mbox, reply):

From: anarcat <anarcat@debian.org>

To: Holger Levsen <holger@layer-acht.org>

Cc: 813313@bugs.debian.org

Subject: Re: Bug#813313: raising severity to important

Date: Tue, 9 Feb 2016 09:33:51 -0500

[Message part 1 (text/plain, inline)]

On Tue, Feb 09, 2016 at 03:24:03PM +0100, Holger Levsen wrote:
> control: severity -1 important

This is typical of Python apps that depend only on urllib and so on. I
have had the same problem in the sopel IRC bot and solved it by
switching to using the Requests library:

https://github.com/sopel-irc/sopel/pull/988

requests is so simple to use that the change makes the code actually
easier to read and shorter, not to mention faster..

a.

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Jakub Wilk <jwilk@debian.org>

To: 813313@bugs.debian.org

Cc: anarcat <anarcat@debian.org>, Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#813313: [planet-venus] planet-venus fails on SNI enabled websites

Date: Wed, 10 Feb 2016 01:05:59 +0100

* anarcat <anarcat@debian.org>, 2016-02-09, 09:33:
>This is typical of Python apps that depend only on urllib and so on.

Not really. Python's stdlib supports SNI since 2.7.9, which was the 
first version that enabled certificate verification by default, and 
what's in jessie.

Here the culrprit is httplib2:

$ python -c 'import urllib; urllib.urlopen("https://blog.windfluechter.net/").read()' && echo yay
yay

$ python -c 'import httplib2; httplib2.Http().request("https://blog.windfluechter.net/")'
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1607, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1349, in _request
   (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1271, in _conn_request
   conn.connect()
 File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1046, in connect
   'host %s: %s' % (hostname, cert), hostname, cert)
httplib2.CertificateHostnameMismatch: Server presented certificate that does not match host blog.windfluechter.net: {'subjectAltName': (('DNS', 'abi94oesede.de'), ('DNS', 'www.abi94oesede.de')), 'notBefore': u'Jan 26 18:05:00 2016 GMT', 'caIssuers': (u'http://cert.int-x1.letsencrypt.org/',), 'OCSP': (u'http://ocsp.int-x1.letsencrypt.org/',), 'serialNumber': u'01839A051BF9D2873C0A3BAA9FD0227C54D1', 'notAfter': 'Apr 25 18:05:00 2016 GMT', 'version': 3L, 'subject': ((('commonName', u'abi94oesede.de'),),), 'issuer': ((('countryName', u'US'),), (('organizationName', u"Let's Encrypt"),), (('commonName', u"Let's Encrypt Authority X1"),))}

-- 
Jakub Wilk

Message #25 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Andreas Metzler <ametzler@bebt.de>

To: Ingo Juergensmann <ij@2013.bluespice.org>, 813313@bugs.debian.org

Subject: Re: Bug#813313: [planet-venus] planet-venus fails on SNI enabled websites

Date: Wed, 10 Feb 2016 19:11:39 +0100

On 2016-01-31 Ingo Juergensmann <ij@2013.bluespice.org> wrote:
> Package: planet-venus
> Version: 0~git9de2109-3
> Severity: normal

> Apparently planet-venus fails to fetch feeds from SNI enabled hosts.
> After migrating my blog from HTTP to HTTPS with letsencrypt.org
> certificate, I noticed that Planet Debian fails to include my blog
> postings.
[...]
> ERROR:planet.runner:Error 500 while updating feed
> https://blog.windfluechter.net/taxonomy/term/2/feed

> When directly using my feed link in a browser everything works without
> any problems and the correct certificate is loaded.
[...]

Hello,

Just out of interest: Am I looking wrong or is blog.windfluechter.net
making strange use of SNI, having a single SNI that is identical with
the CN?

ametzler@argenau:/tmp$ openssl x509 -text < log.windfluechter | grep -E -A1 'Subject:|Subject Alternative Name:'
        Subject: CN=blog.windfluechter.net
        Subject Public Key Info:
--
            X509v3 Subject Alternative Name:
                DNS:blog.windfluechter.net


I have recently also switched to letsencrypt (but have not re-pointed
the planet feed yet).
ametzler@argenau:/tmp$ openssl x509 -text < log.bebt | grep -E -A1 'Subject:|Subject Alternative Name:'
        Subject: CN=vsrv21575.customer.vlinux.de
        Subject Public Key Info:
--
            X509v3 Subject Alternative Name:
                DNS:kraftwerk-au.bebt.de, DNS:vsrv21575.customer.vlinux.de, DNS:www.bebt.de
The simple test case posted by Jakub Wilk[1] succeeds here.

cu Andreas

[1] python -c 'import httplib2; httplib2.Http().request("https://www.bebt.de/")'
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Message #30 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Andreas Metzler <ametzler@bebt.de>, 813313@bugs.debian.org, Ingo Juergensmann <ij@2013.bluespice.org>, 813313@bugs.debian.org

Subject: Re: Bug#813313: [planet-venus] planet-venus fails on SNI enabled websites

Date: Wed, 10 Feb 2016 13:27:32 -0500

On 2016-02-10 13:11:39, Andreas Metzler wrote:
> Just out of interest: Am I looking wrong or is blog.windfluechter.net
> making strange use of SNI, having a single SNI that is identical with
> the CN?

That doesn't seem so strange to me...

Furthermore, the canonical test host for SNI makes httplib fail as well:

$ python -c 'import httplib2; httplib2.Http().request("https://sni.velox.ch/")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1592, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1334, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1256, in _conn_request
    conn.connect()
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1031, in connect
    'host %s: %s' % (hostname, cert), hostname, cert)
httplib2.CertificateHostnameMismatch: Server presented certificate that does not match host sni.velox.ch: {'crlDistributionPoints': (u'http://crl.quovadisglobal.com/qvsslg2.crl',), 'subjectAltName': (('DNS', 'alice.sni.velox.ch'), ('DNS', 'carol.sni.velox.ch')), 'notBefore': u'Apr 21 17:30:43 2014 GMT', 'caIssuers': (u'http://trust.quovadisglobal.com/qvsslg2.crt',), 'OCSP': (u'http://ocsp.quovadisglobal.com',), 'serialNumber': u'398C82B54E24FA61DB9CF244AACDEFD21A0544E2', 'notAfter': 'Apr 21 17:30:42 2017 GMT', 'version': 3L, 'subject': ((('countryName', u'CH'),), (('stateOrProvinceName', u'Zuerich'),), (('localityName', u'Zuerich'),), (('organizationName', u'Kaspar Brand'),), (('commonName', u'alice.sni.velox.ch'),)), 'issuer': ((('countryName', u'BM'),), (('organizationName', u'QuoVadis Limited'),), (('commonName', u'QuoVadis Global SSL ICA G2'),))}

That, at the very least, should be fixed.

a.
-- 
The problem is not a lack of highly educated workers, the problem is a
lack of highly educated workers willing to work for the minimum wage or
lower in the U.S. Costs are driving outsourcing, not the quality of
American schools.       - Scott Kirwin, IT Professionals Association

Message #35 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Andreas Metzler <ametzler@bebt.de>

To: Ingo Juergensmann <ij@2013.bluespice.org>, 813313@bugs.debian.org

Subject: Re: Bug#813313: [planet-venus] planet-venus fails on SNI enabled websites

Date: Wed, 10 Feb 2016 19:35:32 +0100

On 2016-02-10 Andreas Metzler <ametzler@bebt.de> wrote:
[...]

Nevermind, abbreviation thinko. You wrote SNI and I read SAN.

Sorry for the noise.
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Message #40 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Ingo Jürgensmann <ij@2013.bluespice.org>

To: 813313@bugs.debian.org

Subject: Re: Bug#813313: [planet-venus] planet-venus fails on SNI enabled websites

Date: Sat, 12 Mar 2016 17:24:53 +0100

[Message part 1 (text/plain, inline)]

Am 10.02.2016 um 19:35 schrieb Andreas Metzler <ametzler@bebt.de>:
> On 2016-02-10 Andreas Metzler <ametzler@bebt.de> wrote:
> [...]
> Nevermind, abbreviation thinko. You wrote SNI and I read SAN.
> Sorry for the noise.

I’ve just tested with the current version of planet-venus in unstable and the problem still exist.
When doing a "grep -i "certificate that does not match" log/planet.log.0 -c“ I get now 21 instead of 9 affected sites.

Is this a problem of planet-venus or of the Python HTTPlib? Is there anything I can do as non-coding user (i.e. testing)?

--
Ciao...          //        http://blog.windfluechter.net
      Ingo     \X/     XMPP: ij@jabber.windfluechter.net

gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[signature.asc (application/pgp-signature, attachment)]

Message #45 received at 813313@bugs.debian.org (full text, mbox, reply):

From: anarcat <anarcat@orangeseeds.org>

To: Jakub Wilk <jwilk@debian.org>, 813313@bugs.debian.org

Cc: anarcat <anarcat@debian.org>, Holger Levsen <holger@layer-acht.org>, planet@debian.org

Subject: Re: Bug#813313: [planet-venus] fails on SNI enabled websites

Date: Fri, 13 May 2016 16:19:19 -0400

[Message part 1 (text/plain, inline)]

Control: forwarded -1 https://github.com/jcgregorio/httplib2/issues/233 

On Wed, Feb 10, 2016 at 01:05:59AM +0100, Jakub Wilk wrote:
> * anarcat <anarcat@debian.org>, 2016-02-09, 09:33:
> >This is typical of Python apps that depend only on urllib and so on.
> 
> Not really. Python's stdlib supports SNI since 2.7.9, which was the first
> version that enabled certificate verification by default, and what's in
> jessie.
> 
> Here the culrprit is httplib2: [...]

You're right of course. Here's the upstream bug:

https://github.com/jcgregorio/httplib2/issues/233

httplib2 folks say that the problem doesn't appear in Python3, so maybe
Venus could run in Py3k? I actually doubt it, considering that software
hasn't been updated since well, 2010 or so...

The number of failed hosts keeps on growing in the meantime:

$ grep -c 'Server presented certificate that does not match host' planet.log.0
22

Note that those are not all let's encrypt certs, but they are probably
mostly SNI.

Those wishing to reproduce the issue can follow the instructions on the
wiki page here:

https://wiki.debian.org/PlanetDebian#test

Or simply run the bootstrap script:

http://anonscm.debian.org/viewvc/planet-debian/trunk/planet-bootstrap.sh?view=co&content-type=text%2Fplain

The faulty code in planet venus itself is in planet/spider.py, around
line 300, in the httpThread function. It *looks* like it could be fairly
easy to convert it to requests, as the API usage is fairly superficial.

A.

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 813313@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@orangeseeds.org>

To: Jakub Wilk <jwilk@debian.org>, 813313@bugs.debian.org

Subject: Re: Bug#813313: [planet-venus] fails on SNI enabled websites

Date: Fri, 13 May 2016 16:42:58 -0400

> The faulty code in planet venus itself is in planet/spider.py, around
> line 300, in the httpThread function. It *looks* like it could be fairly
> easy to convert it to requests, as the API usage is fairly superficial.

Well, "easy"... There's only a few calls to httplib itself, but it makes
heavy use of the API it turns out. It uses the resp object extensively,
and even uses the cache. So it will need to be cleaned up and use
cachecontrol:

https://github.com/ionrock/cachecontrol

... which is not in stable...

Hopefully, httplib2 will just fix this?

a.
-- 
Marijuana grows naturally on the planet.
Mushrooms grows naturally on the planet.
Don't you think making nature against the law is a bit... unnatural?
                        - Bill Hicks

Message #61 received at 813313@bugs.debian.org (full text, mbox, reply):

From: David North <david-debian-bugs@dnorth.net>

To: 813313@bugs.debian.org

Date: Fri, 9 Aug 2019 22:19:57 +0100

For the record, it is possible to work around this with "sudo pip
install --upgrade httplib2"

Message #66 received at 813313-done@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 553218-done@bugs.debian.org,553219-done@bugs.debian.org,568218-done@bugs.debian.org,641915-done@bugs.debian.org,641916-done@bugs.debian.org,668358-done@bugs.debian.org,668388-done@bugs.debian.org,706401-done@bugs.debian.org,736915-done@bugs.debian.org,758858-done@bugs.debian.org,761814-done@bugs.debian.org,780010-done@bugs.debian.org,804747-done@bugs.debian.org,813313-done@bugs.debian.org,824348-done@bugs.debian.org,848123-done@bugs.debian.org,893157-done@bugs.debian.org,910026-done@bugs.debian.org,917914-done@bugs.debian.org,934931-done@bugs.debian.org,937298-done@bugs.debian.org,

Cc: planet-venus@packages.debian.org

Subject: Bug#940982: Removed package(s) from unstable

Date: Mon, 30 Sep 2019 04:50:46 +0000

Version: 0~git9de2109-4.2+rm

Dear submitter,

as the package planet-venus has just been removed from the Debian archive
unstable we hereby close the associated bug reports.  We are sorry
that we couldn't deal with your issue properly.

For details on the removal, please see https://bugs.debian.org/940982

The version of this package that was in Debian prior to this removal
can still be found using http://snapshot.debian.org/.

This message was generated automatically; if you believe that there is
a problem with it please contact the archive administrators by mailing
ftpmaster@ftp-master.debian.org.

Debian distribution maintenance software
pp.
Scott Kitterman (the ftpmaster behind the curtain)

Send a report that this bug log contains spam.