Debian Bug report logs - #955268
udd watch: "429 too many requests" from GitHub

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Rebecca N. Palmer" <rebecca_palmer@zoho.com>

To: submit@bugs.debian.org

Subject: udd watch: "429 too many requests" from GitHub

Date: Sat, 28 Mar 2020 23:00:51 +0000

Package: qa.debian.org
User: qa.debian.org@packages.debian.org
Usertags: udd

Some packages have a dash in the watch column of 
qa.debian.org/developer.php and this error message on tracker.debian.org:

uscan had problems while searching for a new upstream version:

In watchfile debian/watch, reading webpage
  https://github.com/statsmodels/statsmodels/tags failed: 429 too many 
requests

Which packages have this error seems to vary over time, but it seems to 
be common (~50-90% of GitHub upstreams - e.g. see 
https://qa.debian.org/developer.php?email=pkg-go-maintainers%40lists.alioth.debian.org 
but note that a dash can also mean "no debian/watch" or another error).

I don't know if this was triggered by GitHub introducing/lowering their 
limit, by a bug on our end making more requests than we intended to, or 
by our generally increasing size.  (Currently 9490 GitHub-hosted 
packages each checked every 3 days, according to 
https://salsa.debian.org/qa/udd/-/blob/master/rimporters/upstream.rb#L174 
 and codesearch github\.com path:debian/watch.)

Message #10 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 00:28:38 +0100

On Sat, Mar 28, 2020 at 11:00:51PM +0000, Rebecca N. Palmer wrote:
> In watchfile debian/watch, reading webpage
>   https://github.com/statsmodels/statsmodels/tags failed: 429 too many
> requests
> 
> Which packages have this error seems to vary over time, but it seems to be
> common (~50-90% of GitHub upstreams - e.g. see https://qa.debian.org/developer.php?email=pkg-go-maintainers%40lists.alioth.debian.org
> but note that a dash can also mean "no debian/watch" or another error).
> 
> I don't know if this was triggered by GitHub introducing/lowering their
> limit, by a bug on our end making more requests than we intended to, or by
> our generally increasing size.  (Currently 9490 GitHub-hosted packages each
> checked every 3 days

IIRC with an access token the limit is 5000 queries per hour.  Without a
token, much, much less.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Message #15 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@debian.org>

To: Adam Borowski <kilobyte@angband.pl>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 16:39:01 +0200

Hi,

On 29/03/20 at 00:28 +0100, Adam Borowski wrote:
> On Sat, Mar 28, 2020 at 11:00:51PM +0000, Rebecca N. Palmer wrote:
> > In watchfile debian/watch, reading webpage
> >   https://github.com/statsmodels/statsmodels/tags failed: 429 too many
> > requests
> > 
> > Which packages have this error seems to vary over time, but it seems to be
> > common (~50-90% of GitHub upstreams - e.g. see https://qa.debian.org/developer.php?email=pkg-go-maintainers%40lists.alioth.debian.org
> > but note that a dash can also mean "no debian/watch" or another error).
> > 
> > I don't know if this was triggered by GitHub introducing/lowering their
> > limit, by a bug on our end making more requests than we intended to, or by
> > our generally increasing size.  (Currently 9490 GitHub-hosted packages each
> > checked every 3 days
> 
> IIRC with an access token the limit is 5000 queries per hour.  Without a
> token, much, much less.

The UDD code calls uscan. Is there a way for uscan to use an access
token?

Lucas

Message #20 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 17:31:33 +0200

Le 29/03/2020 à 16:39, Lucas Nussbaum a écrit :
> Hi,
> 
> On 29/03/20 at 00:28 +0100, Adam Borowski wrote:
>> On Sat, Mar 28, 2020 at 11:00:51PM +0000, Rebecca N. Palmer wrote:
>>> In watchfile debian/watch, reading webpage
>>>   https://github.com/statsmodels/statsmodels/tags failed: 429 too many
>>> requests
>>>
>>> Which packages have this error seems to vary over time, but it seems to be
>>> common (~50-90% of GitHub upstreams - e.g. see https://qa.debian.org/developer.php?email=pkg-go-maintainers%40lists.alioth.debian.org
>>> but note that a dash can also mean "no debian/watch" or another error).
>>>
>>> I don't know if this was triggered by GitHub introducing/lowering their
>>> limit, by a bug on our end making more requests than we intended to, or by
>>> our generally increasing size.  (Currently 9490 GitHub-hosted packages each
>>> checked every 3 days
>>
>> IIRC with an access token the limit is 5000 queries per hour.  Without a
>> token, much, much less.
> 
> The UDD code calls uscan. Is there a way for uscan to use an access
> token?
> 
> Lucas

Hi,

not for now, but I can add this feature. Just to insert an additional
header ?

Message #25 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 17:53:11 +0200

[Message part 1 (text/plain, inline)]

Le 29/03/2020 à 17:31, Xavier a écrit :
> Le 29/03/2020 à 16:39, Lucas Nussbaum a écrit :
>> Hi,
>>
>> On 29/03/20 at 00:28 +0100, Adam Borowski wrote:
>>> On Sat, Mar 28, 2020 at 11:00:51PM +0000, Rebecca N. Palmer wrote:
>>>> In watchfile debian/watch, reading webpage
>>>>   https://github.com/statsmodels/statsmodels/tags failed: 429 too many
>>>> requests
>>>>
>>>> Which packages have this error seems to vary over time, but it seems to be
>>>> common (~50-90% of GitHub upstreams - e.g. see https://qa.debian.org/developer.php?email=pkg-go-maintainers%40lists.alioth.debian.org
>>>> but note that a dash can also mean "no debian/watch" or another error).
>>>>
>>>> I don't know if this was triggered by GitHub introducing/lowering their
>>>> limit, by a bug on our end making more requests than we intended to, or by
>>>> our generally increasing size.  (Currently 9490 GitHub-hosted packages each
>>>> checked every 3 days
>>>
>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
>>> token, much, much less.
>>
>> The UDD code calls uscan. Is there a way for uscan to use an access
>> token?
>>
>> Lucas
> 
> Hi,
> 
> not for now, but I can add this feature. Just to insert an additional
> header ?


This little diff adds --http-header option. To use it:

uscan --http-header=Access-Token=qwertyuiop

Sounds good for you ?

[uscan-http-headers.diff (text/x-patch, attachment)]

Message #30 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: Lucas Nussbaum <lucas@debian.org>

Cc: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 19:36:51 +0200

On Sun, Mar 29, 2020 at 04:39:01PM +0200, Lucas Nussbaum wrote:
> On 29/03/20 at 00:28 +0100, Adam Borowski wrote:
> > IIRC with an access token the limit is 5000 queries per hour.  Without a
> > token, much, much less.
> 
> The UDD code calls uscan. Is there a way for uscan to use an access
> token?

First, some longer-winded way:
One of snippets I use is:

TOKEN="kilobyte:cafebabe00000000deadbeef00000000baadf00d"
org=joeblows
p=hotbabe
for (( i=1; i<=$max; i++ )); do
    if [ -e ./$i ];then T="-z ./$i";else T="";fi
    curl -u "$TOKEN" $T -sS -o "$i" "https://api.github.com/repos/$org/$p/issues/$i"
done

You want timestamping -- as, if the file is already there, it doesn't count
against the limit.  But not all parts of GitHub bear timestamps; for those,
"sleep 1" is nice as 3600/h < 5000/h.


Back to uscan:
The watch file includes only the URL, so there's no need to parse a command.
So the hack in uscan would be just:
    if the URL is github.com and $ENV{GITHUB_TOKEN}, auth as that

Yeah, it would be site-specific code which is philosophically a wrong thing
to do, but meh.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Message #35 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Adam Borowski <kilobyte@angband.pl>

To: Xavier <yadd@debian.org>, 955268@bugs.debian.org

Cc: Lucas Nussbaum <lucas@debian.org>

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sun, 29 Mar 2020 23:52:56 +0200

On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
> >>> IIRC with an access token the limit is 5000 queries per hour.  Without a
> >>> token, much, much less.
> >>
> >> The UDD code calls uscan. Is there a way for uscan to use an access
> >> token?
> > 
> > not for now, but I can add this feature. Just to insert an additional
> > header ?
> 
> This little diff adds --http-header option. To use it:
> 
> uscan --http-header=Access-Token=qwertyuiop
> 
> Sounds good for you ?

Tokens hardly ever are portable between sites, thus I guess they'd need to
be used only for matching URLs.  I'm not sure if udd already parses target
servers or calls uscan as-is, letting it do all the work.

Second issue is, tokens are (initially) an user-"password" pair that
requires further processing.  For example, GitHub uses Basic http auth
(https://developer.github.com/v3/auth/).  If I read the RFC
(https://www.ietf.org/rfc/rfc2617.txt) correctly, Basic auth gets mangled
into a header and is sent as-is; no idea if other token-using webservices
use fancier auth schemes.  Even if just Basic is used, it might be simpler
to not require the user how to do the mangling.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Message #40 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@debian.org>

To: Adam Borowski <kilobyte@angband.pl>

Cc: Xavier <yadd@debian.org>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Mon, 30 Mar 2020 08:25:35 +0200

On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
> > >>> IIRC with an access token the limit is 5000 queries per hour.  Without a
> > >>> token, much, much less.
> > >>
> > >> The UDD code calls uscan. Is there a way for uscan to use an access
> > >> token?
> > > 
> > > not for now, but I can add this feature. Just to insert an additional
> > > header ?
> > 
> > This little diff adds --http-header option. To use it:
> > 
> > uscan --http-header=Access-Token=qwertyuiop
> > 
> > Sounds good for you ?

Yes, thanks!

> Tokens hardly ever are portable between sites, thus I guess they'd need to
> be used only for matching URLs.  I'm not sure if udd already parses target
> servers or calls uscan as-is, letting it do all the work.
> 
> Second issue is, tokens are (initially) an user-"password" pair that
> requires further processing.  For example, GitHub uses Basic http auth
> (https://developer.github.com/v3/auth/).  If I read the RFC
> (https://www.ietf.org/rfc/rfc2617.txt) correctly, Basic auth gets mangled
> into a header and is sent as-is; no idea if other token-using webservices
> use fancier auth schemes.  Even if just Basic is used, it might be simpler
> to not require the user how to do the mangling.

In the UDD case, I think that's enough to fix the present issue.

I'm not sure something more general should be done?

Lucas

Message #45 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>

Cc: Adam Borowski <kilobyte@angband.pl>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Mon, 30 Mar 2020 08:49:33 +0200

Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
> On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
>> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
>>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
>>>>>> token, much, much less.
>>>>>
>>>>> The UDD code calls uscan. Is there a way for uscan to use an access
>>>>> token?
>>>>
>>>> not for now, but I can add this feature. Just to insert an additional
>>>> header ?
>>>
>>> This little diff adds --http-header option. To use it:
>>>
>>> uscan --http-header=Access-Token=qwertyuiop
>>>
>>> Sounds good for you ?
> 
> Yes, thanks!

Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181

Cheers,
Xavier

Message #48 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer <josch@debian.org>

To: 955268@bugs.debian.org

Cc: lucas@debian.org, kilobyte@angband.pl, yadd@debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 15:06:11 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Mon, 30 Mar 2020 08:49:33 +0200 Xavier <yadd@debian.org> wrote:
> Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
> > On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
> >> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
> >>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
> >>>>>> token, much, much less.
> >>>>>
> >>>>> The UDD code calls uscan. Is there a way for uscan to use an access
> >>>>> token?
> >>>>
> >>>> not for now, but I can add this feature. Just to insert an additional
> >>>> header ?
> >>>
> >>> This little diff adds --http-header option. To use it:
> >>>
> >>> uscan --http-header=Access-Token=qwertyuiop
> >>>
> >>> Sounds good for you ?
> > 
> > Yes, thanks!
> 
> Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181

since there is only little progress on that merge request, could we have an
intermediate solution? For example by severely reducing the rate with which
uscan is run on github.com? Even if this means that there is only a check done
once a week, that's better than the current situation which already persists
for over a month.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #53 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@debian.org>

To: Johannes Schauer <josch@debian.org>

Cc: 955268@bugs.debian.org, kilobyte@angband.pl, yadd@debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 15:57:18 +0200

On 24/04/20 at 15:06 +0200, Johannes Schauer wrote:
> Hi,
> 
> On Mon, 30 Mar 2020 08:49:33 +0200 Xavier <yadd@debian.org> wrote:
> > Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
> > > On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
> > >> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
> > >>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
> > >>>>>> token, much, much less.
> > >>>>>
> > >>>>> The UDD code calls uscan. Is there a way for uscan to use an access
> > >>>>> token?
> > >>>>
> > >>>> not for now, but I can add this feature. Just to insert an additional
> > >>>> header ?
> > >>>
> > >>> This little diff adds --http-header option. To use it:
> > >>>
> > >>> uscan --http-header=Access-Token=qwertyuiop
> > >>>
> > >>> Sounds good for you ?
> > > 
> > > Yes, thanks!
> > 
> > Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181
> 
> since there is only little progress on that merge request, could we have an
> intermediate solution? For example by severely reducing the rate with which
> uscan is run on github.com? Even if this means that there is only a check done
> once a week, that's better than the current situation which already persists
> for over a month.

I think that a suitable workaround would be to have a local uscan copy
in UDD that can add the header, and then modify UDD to use it for
github.

Anybody with access to UDD can do it. If nobody does, I'll try to do it
when I run into Debian time, which is scarse those days.

Lucas

Message #58 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>

Cc: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 16:07:10 +0200

Le 24/04/2020 à 15:57, Lucas Nussbaum a écrit :
> On 24/04/20 at 15:06 +0200, Johannes Schauer wrote:
>> Hi,
>>
>> On Mon, 30 Mar 2020 08:49:33 +0200 Xavier <yadd@debian.org> wrote:
>>> Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
>>>> On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
>>>>> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
>>>>>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
>>>>>>>>> token, much, much less.
>>>>>>>>
>>>>>>>> The UDD code calls uscan. Is there a way for uscan to use an access
>>>>>>>> token?
>>>>>>>
>>>>>>> not for now, but I can add this feature. Just to insert an additional
>>>>>>> header ?
>>>>>>
>>>>>> This little diff adds --http-header option. To use it:
>>>>>>
>>>>>> uscan --http-header=Access-Token=qwertyuiop
>>>>>>
>>>>>> Sounds good for you ?
>>>>
>>>> Yes, thanks!
>>>
>>> Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181
>>
>> since there is only little progress on that merge request, could we have an
>> intermediate solution? For example by severely reducing the rate with which
>> uscan is run on github.com? Even if this means that there is only a check done
>> once a week, that's better than the current situation which already persists
>> for over a month.
> 
> I think that a suitable workaround would be to have a local uscan copy
> in UDD that can add the header, and then modify UDD to use it for
> github.
> 
> Anybody with access to UDD can do it. If nobody does, I'll try to do it
> when I run into Debian time, which is scarse those days.

Hi,

Prefer to compile a 2.20.3~alpha of devscripts to avoid problems with
Perl path (modified code is in /usr/share/perl5/Devscripts/Uscan/)

To use it (with last changes:
  uscan --http-header=https://github.com@GitHubToken=value

Token will be sent only to URL starting with https://github.com. Then
this option can be set by default. Use --verbose to see what happens

Cheers,
Xavier

Message #63 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@debian.org>

To: Xavier <yadd@debian.org>

Cc: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 16:11:08 +0200

On 24/04/20 at 16:07 +0200, Xavier wrote:
> Le 24/04/2020 à 15:57, Lucas Nussbaum a écrit :
> > On 24/04/20 at 15:06 +0200, Johannes Schauer wrote:
> >> Hi,
> >>
> >> On Mon, 30 Mar 2020 08:49:33 +0200 Xavier <yadd@debian.org> wrote:
> >>> Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
> >>>> On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
> >>>>> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
> >>>>>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
> >>>>>>>>> token, much, much less.
> >>>>>>>>
> >>>>>>>> The UDD code calls uscan. Is there a way for uscan to use an access
> >>>>>>>> token?
> >>>>>>>
> >>>>>>> not for now, but I can add this feature. Just to insert an additional
> >>>>>>> header ?
> >>>>>>
> >>>>>> This little diff adds --http-header option. To use it:
> >>>>>>
> >>>>>> uscan --http-header=Access-Token=qwertyuiop
> >>>>>>
> >>>>>> Sounds good for you ?
> >>>>
> >>>> Yes, thanks!
> >>>
> >>> Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181
> >>
> >> since there is only little progress on that merge request, could we have an
> >> intermediate solution? For example by severely reducing the rate with which
> >> uscan is run on github.com? Even if this means that there is only a check done
> >> once a week, that's better than the current situation which already persists
> >> for over a month.
> > 
> > I think that a suitable workaround would be to have a local uscan copy
> > in UDD that can add the header, and then modify UDD to use it for
> > github.
> > 
> > Anybody with access to UDD can do it. If nobody does, I'll try to do it
> > when I run into Debian time, which is scarse those days.
> 
> Hi,
> 
> Prefer to compile a 2.20.3~alpha of devscripts to avoid problems with
> Perl path (modified code is in /usr/share/perl5/Devscripts/Uscan/)
> 
> To use it (with last changes:
>   uscan --http-header=https://github.com@GitHubToken=value
> 
> Token will be sent only to URL starting with https://github.com. Then
> this option can be set by default. Use --verbose to see what happens

Unfortunately I need to do a manual install anyway, I don't have root
access on the UDD machine and thus cannot install a .deb.

Lucas

Message #68 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>

Cc: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 16:13:03 +0200

Le 24/04/2020 à 16:11, Lucas Nussbaum a écrit :
> On 24/04/20 at 16:07 +0200, Xavier wrote:
>> Le 24/04/2020 à 15:57, Lucas Nussbaum a écrit :
>>> On 24/04/20 at 15:06 +0200, Johannes Schauer wrote:
>>>> Hi,
>>>>
>>>> On Mon, 30 Mar 2020 08:49:33 +0200 Xavier <yadd@debian.org> wrote:
>>>>> Le 30/03/2020 à 08:25, Lucas Nussbaum a écrit :
>>>>>> On 29/03/20 at 23:52 +0200, Adam Borowski wrote:
>>>>>>> On Sun, Mar 29, 2020 at 05:53:11PM +0200, Xavier wrote:
>>>>>>>>>>> IIRC with an access token the limit is 5000 queries per hour.  Without a
>>>>>>>>>>> token, much, much less.
>>>>>>>>>>
>>>>>>>>>> The UDD code calls uscan. Is there a way for uscan to use an access
>>>>>>>>>> token?
>>>>>>>>>
>>>>>>>>> not for now, but I can add this feature. Just to insert an additional
>>>>>>>>> header ?
>>>>>>>>
>>>>>>>> This little diff adds --http-header option. To use it:
>>>>>>>>
>>>>>>>> uscan --http-header=Access-Token=qwertyuiop
>>>>>>>>
>>>>>>>> Sounds good for you ?
>>>>>>
>>>>>> Yes, thanks!
>>>>>
>>>>> Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181
>>>>
>>>> since there is only little progress on that merge request, could we have an
>>>> intermediate solution? For example by severely reducing the rate with which
>>>> uscan is run on github.com? Even if this means that there is only a check done
>>>> once a week, that's better than the current situation which already persists
>>>> for over a month.
>>>
>>> I think that a suitable workaround would be to have a local uscan copy
>>> in UDD that can add the header, and then modify UDD to use it for
>>> github.
>>>
>>> Anybody with access to UDD can do it. If nobody does, I'll try to do it
>>> when I run into Debian time, which is scarse those days.
>>
>> Hi,
>>
>> Prefer to compile a 2.20.3~alpha of devscripts to avoid problems with
>> Perl path (modified code is in /usr/share/perl5/Devscripts/Uscan/)
>>
>> To use it (with last changes:
>>   uscan --http-header=https://github.com@GitHubToken=value
>>
>> Token will be sent only to URL starting with https://github.com. Then
>> this option can be set by default. Use --verbose to see what happens
> 
> Unfortunately I need to do a manual install anyway, I don't have root
> access on the UDD machine and thus cannot install a .deb.
> 
> Lucas

Then use PERL5LIB to set the good Perl path

Message #71 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Lucas Nussbaum <lucas@debian.org>, 955268@bugs.debian.org

Cc: Johannes Schauer <josch@debian.org>, kilobyte@angband.pl, yadd@debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 24 Apr 2020 22:53:13 +0200

[Message part 1 (text/plain, inline)]

On Fri, Apr 24, 2020 at 03:57:18PM +0200, Lucas Nussbaum wrote:
> > > Done: https://salsa.debian.org/debian/devscripts/-/merge_requests/181
> > 
> > since there is only little progress on that merge request, could we have an
> > intermediate solution? For example by severely reducing the rate with which
> > uscan is run on github.com? Even if this means that there is only a check done
> > once a week, that's better than the current situation which already persists
> > for over a month.
> 
> I think that a suitable workaround would be to have a local uscan copy
> in UDD that can add the header, and then modify UDD to use it for
> github.
> 
> Anybody with access to UDD can do it. If nobody does, I'll try to do it
> when I run into Debian time, which is scarse those days.

I did a review yesterday, and I saw yadd provided the changes I
required.  I'll try to do a release tomorrow, so I think it would be
best if you just hold on a bit more.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
More about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #76 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Lucas Nussbaum <lucas@debian.org>, 955268@bugs.debian.org

Cc: Xavier <yadd@debian.org>

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sat, 25 Apr 2020 00:34:45 +0000

On Fri, Apr 24, 2020 at 2:15 PM Lucas Nussbaum wrote:

> Unfortunately I need to do a manual install anyway, I don't have root
> access on the UDD machine and thus cannot install a .deb.

Another option would be to ask DSA to install the backport once it
reaches buster-backports.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Message #81 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Paul Wise <pabs@debian.org>, 955268@bugs.debian.org

Cc: Lucas Nussbaum <lucas@debian.org>, Xavier <yadd@debian.org>

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sat, 25 Apr 2020 09:47:23 +0200

[Message part 1 (text/plain, inline)]

On Sat, 25 Apr 2020, 2:39 am Paul Wise, <pabs@debian.org> wrote:

> On Fri, Apr 24, 2020 at 2:15 PM Lucas Nussbaum wrote:
>
> > Unfortunately I need to do a manual install anyway, I don't have root
> > access on the UDD machine and thus cannot install a .deb.
>
> Another option would be to ask DSA to install the backport once it
> reaches buster-backports.
>

I believe we are already running the backport version, and if we are not we
should change that already, regardless of this particular issue.

[Message part 2 (text/html, inline)]

Message #86 received at 955268-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 955268-close@bugs.debian.org

Subject: Bug#955268: fixed in devscripts 2.20.3

Date: Sat, 25 Apr 2020 19:33:31 +0000

Source: devscripts
Source-Version: 2.20.3
Done: Mattia Rizzolo <mattia@debian.org>

We believe that the bug you reported is fixed in the latest version of
devscripts, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 955268@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated devscripts package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sat, 25 Apr 2020 21:15:48 +0200
Source: devscripts
Architecture: source
Version: 2.20.3
Distribution: unstable
Urgency: medium
Maintainer: Devscripts Maintainers <devscripts@packages.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Closes: 835498 843913 951568 953281 955268 955280 955298 956498 958350
Changes:
 devscripts (2.20.3) unstable; urgency=medium
 .
   [ Mattia Rizzolo ]
   * uupdate:
     + Apply patch from Unit 193, adding full stops to the generated changelog.
   * uscan:
     + Apply patch from Unit 193, adding --queit to the `git clone` calls.
   * checkbashisms:
     + Improve check for `command` to properly detect options other than -p
       also when -p is not the first option.  Closes: #835498
       Thanks to Eero Vuojolahti <eero@vuojolahti.com> for the initial patch.
     + After Policy v4.1.5, the POSIX standard for shell script is
       POSIX.1-2017.  Recognize `command -v` and `command -V` as valid.
   * dget:
     + Apply patch from Bilal Muhammad to add support for scp:// and sftp://
       protocols.  Closes: #956498
   * tests:
     + test_package_lifecycle:
       - Require dpkg-dev >= 1.19.1, use Rules-Requires-Root in the test package
         and therefore drop the fakeroot Build-Depends.
 .
   [ Andrius Merkys ]
   * uscan:
     + Fix svn mode with HTTP URLs.  Closes: #951568; MR: !173
 .
   [ Lars Kruse ]
   * checkbashisms:
     + Remove check for "sleep" (not a shell builtin).  Closes: #843913; MR: !176
 .
   [ Paul Wise ]
   * bts:
     + Sync the known tags with bugs.debian.org.
 .
   [ Xavier Guimard ]
   * Reformat perl scripts with the new perltidy 20200110.
 .
   [ Holger Levsen ]
   * debrebuild:
     + Stop using the reproducible-builds.org apt repo.  Closes: #955280
     + Switch to use deb.debian.org as the base apt repo.  Closes: #955298
     + Add support for Debian 12 bookworm.
 .
   [ Antonio Terceiro ]
   * debrepro:
     + Run command before second build via `sh -c`.
     + Run command before second build in the source tree copy, to avoid
       modifying the original tree.
     + Add a new --timeout option.
 .
   [ Simon McVittie ]
   * mass-bug:
     + Add a test.  MR: !186
     + Add a new --control option, to be able to add arbitrary
       Control: pseudo-headers.  MR: !183
   * tests:
     + test_package_lifecycle:
       + Fix test with debhelper 13.  Closes: #958350; MR: 185
 .
   [ Jiří Paleček ]
   * uscan:
     + Fix operation with simple git branches (rather than tags).  MR: !175
   * tests:
     + test_package_lifecycle:
       - Fix test failures under non-EN locales.  Closes: #953281
 .
   [ Alex Murray ]
   * hardening-check:
     + Change to a more reliable to way of detecting control flow protection,
       and add a new --nocfprotection flag to disable the check.  MR: !184
 .
   [ Xavier Guimard ]
   * uscan:
     + Fix DEHS <target> output for components.  MR: !179
     + Fix --download-version with grouped packages.  MR: !156
     + Add "checksum" target to reduce grouped version size.  MR: !156
     + Add --http-header option.  Closes: #955268; MR: !181
   * Update French translation.
   * salsa:
     + Fix error when trying to display an error with "update_repo".
     + Enable "enable_ssl_verification" KGB option by default since GitLab sets
       it
     + During webhook configuration, don't fail when --no-fail is set and
       GitLab fails
Checksums-Sha1:
 71d85e56127821138756bdf11494d5601c789227 3161 devscripts_2.20.3.dsc
 48c57dd6866295d5c1968bf58f5cc528d73f5fbb 859612 devscripts_2.20.3.tar.xz
 10e915422269b338d0f5f7a41a8322acbba9d47f 12270 devscripts_2.20.3_amd64.buildinfo
Checksums-Sha256:
 94c1b4db9ea590513c48fbd3fe93a83cc390c8b7091e8a6ab5c41160805c319a 3161 devscripts_2.20.3.dsc
 84ca5d2d14517d98dafdb7ce27a2905f1ad07b85c5fea4825f2266648b87e4be 859612 devscripts_2.20.3.tar.xz
 5e76af3109f97244c85d39f8e736a1439c7760445ab2df5d4bcbd6091f51d4bf 12270 devscripts_2.20.3_amd64.buildinfo
Files:
 bd6d83d3dff62b562c82c7518b8f4e08 3161 devel optional devscripts_2.20.3.dsc
 82b073ac2405a4bd675efcf21d6c3cf1 859612 devel optional devscripts_2.20.3.tar.xz
 9cb6df31d50b385330a4647eed357fe6 12270 devel optional devscripts_2.20.3_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAl6kjdEACgkQCBa54Yx2
K62bgQ//QgIZDXQ24Ev61MgE1mpCCFPFilxR+5mYCHuicrqmZXJUgOcT6YiiBdWZ
NcNOGAk4rSxrG4atAkDkDe5YQ1gMQkGqrRKLqMdTMzt/1M5i9mmXQoYhyC7qRDbP
/b4Lefo5ICU1FbfWcaWSsQ2Cu4V1ah2RyaA6OlmP+AdzpQ0sJssBeVBhhXf6++fB
im8TYA6R1yRwupVwxDW/bto0b+OQgG4rER/alabqtV69A2oBN8Us+Zon1sNqdRIE
49MDx0GPWTjWgKTlyQ6dPkRPf/TKSiUQ2KP8j5ymvqoqgeMQQCKdusndIpt005ci
DaHkTmzI+8mAj1qXN/+kysL8CniptgBDsg+uvJVGC92gPL4wxzUg221AfdIk8HlZ
FxrPENrdPCome80q1k9tUQuPmvmTuRJfmMz3KT7b69Ddp+a81lFtg1E3mZarV8BX
r+G4sI88jrGg/p+7tlt7+zH+57Pw/XD9ONNGidspMlmHE/Z8x8QVyrMfFIVzBXsc
lJLZUGFgG0UWt4/rKZ+2Zq59nQUaaogK4epWqtvBz6DlB5pJQpLI+GJaYQpnLu0L
e5R74lhNQMJYqcIjUDSUGgsRrzptoUr1nOuL493Z6399RM5nfm7EpKUKnhrqG8Mm
OWnKP+fkriJDcKiN+tUO851Hu+fCa7aCwwt/7C8x1uAzm0uJMwQ=
=90y6
-----END PGP SIGNATURE-----

Message #97 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@debian.org>

To: Mattia Rizzolo <mattia@debian.org>

Cc: 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Fri, 1 May 2020 22:40:37 +0200

Hi

Unfortunately, the fix in devscripts 2.20.3 doesn't work, because watch
files don't use the GitHub API, but rather URLs such as
https://github.com/osallou/cassiopee-c/tags

So different rate limiting rules apply, and authenticating using HTTP
Basic Auth doesn't work.

I ended up implementing a different workaround: when the upstream
importer hits an error 429 for github, it then skips github for the
remaining of the run.

It means that we won't add junk about software hosted on github, but
that they might get tested less frequently.

Let's see if this fixes this problem... To count packages that still
show 429 errors, use:
select count(*) from upstream where watch_file ~ 'github' and warnings ~ '429';
At this point, we are down to 3919, compared to 5200+ before I deployed
the fix.

Lucas

Message #102 received at 955268@bugs.debian.org (full text, mbox, reply):

From: Xavier <yadd@debian.org>

To: Lucas Nussbaum <lucas@debian.org>, 955268@bugs.debian.org

Subject: Re: Bug#955268: udd watch: "429 too many requests" from GitHub

Date: Sat, 2 May 2020 09:55:00 +0200

Le 01/05/2020 à 22:40, Lucas Nussbaum a écrit :
> Hi
> 
> Unfortunately, the fix in devscripts 2.20.3 doesn't work, because watch
> files don't use the GitHub API, but rather URLs such as
> https://github.com/osallou/cassiopee-c/tags
> 
> So different rate limiting rules apply, and authenticating using HTTP
> Basic Auth doesn't work.
> 
> I ended up implementing a different workaround: when the upstream
> importer hits an error 429 for github, it then skips github for the
> remaining of the run.
> 
> It means that we won't add junk about software hosted on github, but
> that they might get tested less frequently.
> 
> Let's see if this fixes this problem... To count packages that still
> show 429 errors, use:
> select count(*) from upstream where watch_file ~ 'github' and warnings ~ '429';
> At this point, we are down to 3919, compared to 5200+ before I deployed
> the fix.
> 
> Lucas

Hi,

another solution is to apply "429" recommendations as proposed here:
https://salsa.debian.org/debian/devscripts/-/merge_requests/187/diffs

But it may have some side effects.

Send a report that this bug log contains spam.