Debian Bug report logs - #861333
r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Johannes Ranke <johannes.ranke@jrwb.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Thu, 27 Apr 2017 15:45:40 +0200

Package: r-base
Version: 3.4.0-1
Severity: normal

With current R, R packages built for Debian before the upload of R 
3.3.3.20170413-1 
on 14 April that use .C or .Fortran do no work properly, because the functions 
calling .C or .Fortran do not find the compiled objects.

Example packages are r-cran-spatial and r-cran-kernsmooth. 

An example session in a fresh Debian sid chroot:

> library(spatial)
> example(surf.gls)

srf.gl> library(MASS)  # for eqscplot

srf.gl> data(topo, package="MASS")

srf.gl> topo.kr <- surf.gls(2, expcov, topo, d=0.7)
Error in surf.gls(2, expcov, topo, d = 0.7) : object 'VR_frset' not found


The relevant NEWS entry from R is

    * Packages which register native routines for .C or .Fortran need
      to be re-installed for this version (unless installed with
      R-devel SVN revision r72375 or later).

Packages compiled locally can simply be rebuilt using

  update.packages(lib.loc="/usr/local/lib/R/site-library", checkBuilt=TRUE)

However the packages provided by Debian packages are installed in a directory
only writable by privileged users.

Cheers,

Johannes

-- System Information:
Debian Release: 9.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64
 (x86_64)

Kernel: Linux 4.9.0-0.bpo.2-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /usr/bin/dash
Init: unable to detect

Versions of packages r-base depends on:
ii  r-base-core    3.4.0-1
ii  r-recommended  3.4.0-1

Versions of packages r-base recommends:
ii  r-base-html  3.4.0-1
ii  r-doc-html   3.4.0-1

Versions of packages r-base suggests:
pn  ess                     <none>
pn  r-doc-info | r-doc-pdf  <none>

-- no debconf information

Message #10 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Don Armstrong <don@debian.org>

To: 861333@bugs.debian.org

Subject: API transition of R packages

Date: Thu, 27 Apr 2017 12:49:23 -0500

Control: severity -1 serious

Do we know if this issue may also mean that any packages built with this
new version are incompatible with older R versions? [I'm thinking so,
but my ABI-fu is not super strong.]

If so, we'll need to make sure that they depend on at least this R
version.

We also may need to populate a breaks with all of those packages which
have the older version.


As a side note, it's really important not to start transitions like this
when we're in a freeze; this upload of R 3.4 should have been made to
experimental, not unstable. [Not that I can really point too many
fingers; I accidentally uploaded a new release of scowl to unstable
which I meant to target at experimental the other day...]



-- 
Don Armstrong                      https://www.donarmstrong.com

a friend will help you move
a best friend will help you move bodies
but if you have to move your best friend's body
you're on your own
 -- a softer world #242
    http://www.asofterworld.com/index.php?id=242

Message #17 received at submit@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Johannes Ranke <johannes.ranke@jrwb.de>, 861333@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Thu, 27 Apr 2017 13:21:42 -0500

On 27 April 2017 at 15:45, Johannes Ranke wrote:
| Package: r-base
| Version: 3.4.0-1
| Severity: normal
| 
| With current R, R packages built for Debian before the upload of R 
| 3.3.3.20170413-1 
| on 14 April that use .C or .Fortran do no work properly, because the functions 
| calling .C or .Fortran do not find the compiled objects.
|
| Example packages are r-cran-spatial and r-cran-kernsmooth. 
| 
| An example session in a fresh Debian sid chroot:
| 
| > library(spatial)
| > example(surf.gls)
| 
| srf.gl> library(MASS)  # for eqscplot
| 
| srf.gl> data(topo, package="MASS")
| 
| srf.gl> topo.kr <- surf.gls(2, expcov, topo, d=0.7)
| Error in surf.gls(2, expcov, topo, d = 0.7) : object 'VR_frset' not found
| 
| 
| The relevant NEWS entry from R is
| 
|     * Packages which register native routines for .C or .Fortran need
|       to be re-installed for this version (unless installed with
|       R-devel SVN revision r72375 or later).

To a very first approximation, just over 1/4 of all CRAN packages contain
compiled code.  The ratio will be higher for the r-cran-* subset as we skew
towards bigger / more well-known packages.  Per the script I posted to
r-sig-debian about 1/3 of packages still use the .C() and .Fortran()
interfaces to compiled code (rather than the now-recommended .Call().

| Packages compiled locally can simply be rebuilt using
| 
|   update.packages(lib.loc="/usr/local/lib/R/site-library", checkBuilt=TRUE)
| 
| However the packages provided by Debian packages are installed in a directory
| only writable by privileged users.

That's irrelevant. You also need to be "privileged" to install a .deb package.

What is more relevant is that the /usr/local/lib/R/site-library path comes
before the package path -- so any user can always 'shadow' a potentially
borked package with a local installation from CRAN (which may be more current
too).

That is a feature.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #27 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Don Armstrong <don@debian.org>, 861333@bugs.debian.org

Subject: Re: Bug#861333: API transition of R packages

Date: Thu, 27 Apr 2017 13:26:29 -0500

On 27 April 2017 at 12:49, Don Armstrong wrote:
| Control: severity -1 serious
| 
| Do we know if this issue may also mean that any packages built with this
| new version are incompatible with older R versions? [I'm thinking so,
| but my ABI-fu is not super strong.]

I don't know, and I tend not to run dated r-base-core packages.  
 
| If so, we'll need to make sure that they depend on at least this R
| version.

Is that what debian/control ensures?  Ie from one of my most recent uploads:

  edd@max:~$ dpkg -f /var/cache/pbuilder/result/r-cran-foreign_0.8.68-1_amd64.deb | grep Depends
  Depends: libc6 (>= 2.14), r-base-core (>= 3.4.0-1), r-api-3
  edd@max:~$

Pretty much ensure you cannot use this with R 3.3.* or older.  
 
| We also may need to populate a breaks with all of those packages which
| have the older version.
| 
| As a side note, it's really important not to start transitions like this
| when we're in a freeze; this upload of R 3.4 should have been made to
| experimental, not unstable. [Not that I can really point too many
| fingers; I accidentally uploaded a new release of scowl to unstable
| which I meant to target at experimental the other day...]

I uploaded one beta build to experimental.  Approximately nobody uses those.

The 'blocking' mechanism really works. R 3.4.0 will not seep into testing.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #32 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Don Armstrong <don@debian.org>

To: 861333@bugs.debian.org

Subject: Re: Bug#861333: API transition of R packages

Date: Thu, 27 Apr 2017 11:37:34 -0700

On Thu, 27 Apr 2017, Dirk Eddelbuettel wrote:
> I don't know, and I tend not to run dated r-base-core packages.

I'll try to check this out later.

> Is that what debian/control ensures?

Cool; I didn't check to see whether the substitution variable had been
updated.

> I uploaded one beta build to experimental. Approximately nobody uses
> those.

Yeah, that's always a problem.

> The 'blocking' mechanism really works. R 3.4.0 will not seep into
> testing.

The problem isn't that R won't enter testing, but that any package which
builds against R has to be rebuilt using the R in testing and uploaded
to testing-proposed-updates.

That's a pretty painful thing to have to do. [Luckily, R is leaf enough
that there aren't too many RC bugs in R packages, so we should be OK.]

-- 
Don Armstrong                      https://www.donarmstrong.com

Mozart tells us what it's like to be human, Beethoven tells us what
it's like to be Beethoven, and Bach tells us what it's like to be the
universe.
 -- Douglas Adams

Message #37 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Don Armstrong <don@debian.org>, 861333@bugs.debian.org, edd@debian.org

Subject: Re: Bug#861333: API transition of R packages

Date: Thu, 27 Apr 2017 13:57:02 -0500

On 27 April 2017 at 11:37, Don Armstrong wrote:
| On Thu, 27 Apr 2017, Dirk Eddelbuettel wrote:
| > I don't know, and I tend not to run dated r-base-core packages.
| 
| I'll try to check this out later.

Thanks!
 
| > Is that what debian/control ensures?
| 
| Cool; I didn't check to see whether the substitution variable had been
| updated.

I feel like we have had the substitution of R (>= 'currentBuildVersion') for
a decade.
 
| > I uploaded one beta build to experimental. Approximately nobody uses
| > those.
| 
| Yeah, that's always a problem.
| 
| > The 'blocking' mechanism really works. R 3.4.0 will not seep into
| > testing.
| 
| The problem isn't that R won't enter testing, but that any package which
| builds against R has to be rebuilt using the R in testing and uploaded
| to testing-proposed-updates.

Yes, I hear you on that one.  
 
| That's a pretty painful thing to have to do. [Luckily, R is leaf enough
| that there aren't too many RC bugs in R packages, so we should be OK.]

Fingers crossed :)

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #42 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Don Armstrong <don@debian.org>

To: 861333@bugs.debian.org

Subject: Re: Bug#861333: API transition of R packages

Date: Thu, 27 Apr 2017 12:02:23 -0700

On Thu, 27 Apr 2017, Dirk Eddelbuettel wrote:
> I feel like we have had the substitution of R (>= 'currentBuildVersion') for
> a decade.

I didn't realize that it was the current build version; I just assumed
it was updated manually.


-- 
Don Armstrong                      https://www.donarmstrong.com

Those who begin coercive elimination of dissent soon find themselves
exterminating dissenters. Compulsory unification of opinion achieves
only the unanimity of the graveyard.
 -- Justice Roberts in 319 U.S. 624 (1943)

Message #47 received at submit@bugs.debian.org (full text, mbox, reply):

From: Johannes Ranke <johannes.ranke@jrwb.de>

To: Dirk Eddelbuettel <edd@debian.org>, 861333@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Fri, 28 Apr 2017 00:31:25 +0200

> | Packages compiled locally can simply be rebuilt using
> | 
> |   update.packages(lib.loc="/usr/local/lib/R/site-library",
> |   checkBuilt=TRUE)
> | 
> | However the packages provided by Debian packages are installed in a
> | directory only writable by privileged users.
> 
> That's irrelevant. You also need to be "privileged" to install a .deb
> package.

Not quite irrelevant, as it was recommended on r-help to Göran, who first 
reported this for Debian, to just use

   update.packages(checkBuilt=TRUE)

which tries to reinstall also the packages in /usr/lib/R/site-library, which 
should be left to the Debian package management.

Message #57 received at submit@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Johannes Ranke <johannes.ranke@jrwb.de>

Cc: Dirk Eddelbuettel <edd@debian.org>, 861333@bugs.debian.org, Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Thu, 27 Apr 2017 18:57:15 -0500

On 28 April 2017 at 00:31, Johannes Ranke wrote:
| > | Packages compiled locally can simply be rebuilt using
| > | 
| > |   update.packages(lib.loc="/usr/local/lib/R/site-library",
| > |   checkBuilt=TRUE)
| > | 
| > | However the packages provided by Debian packages are installed in a
| > | directory only writable by privileged users.
| > 
| > That's irrelevant. You also need to be "privileged" to install a .deb
| > package.
| 
| Not quite irrelevant, as it was recommended on r-help to Göran, who first 
| reported this for Debian, to just use
| 
| update.packages(checkBuilt=TRUE)
| 
| which tries to reinstall also the packages in /usr/lib/R/site-library, which 
| should be left to the Debian package management.

We can split hairs as to whether it is irrelevant, plain wrong or uninformed.

It ignores that users should never write where packages write!  So on a
Debian or Ubuntu system you should NEVER EVER run

   update.packages(checkBuilt=TRUE)

because it would mess up your package install and local install.

What you could do, and which is currently running on my at-work workstation is

   update.packages(lib.loc="/usr/local/lib/R/site-library", ask=FALSE, checkBuilt=TRUE)

which updates you local packages.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #67 received at submit@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Dirk Eddelbuettel <edd@debian.org>

Cc: Johannes Ranke <johannes.ranke@jrwb.de>, 861333@bugs.debian.org, Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Sat, 29 Apr 2017 09:50:01 -0500

Here is a follow-up, and for kicks with some R scripting.

It requires sources to have directory names like 'class-7.3-14', ie (CRAN)
package name followed by (CRAN) version.


Shell script:

-----------------------------------------------------------------------------
#!/bin/bash

newfunc() {
    ## source directories are all named foo-1.2.3
    dirs=$(find . -maxdepth 1 -type d -name \*-\* | sort)

    for d in ${dirs}; do
        if test -d ${d}/src; then
            if grep -q -r \\.Fortran\( ${d}/R; then
                fortran=TRUE
            else 
                fortran=FALSE
            fi
            if grep -q -r \\.C\( ${d}/R; then
                ccall=TRUE
            else
                ccall=FALSE
            fi
            if grep -q "Priority: recommended" ${d}/DESCRIPTION; then
                recommended=TRUE
            else
                recommended=FALSE
            fi
            if [[ "${fortran}" == "TRUE" || "${ccall}" == "TRUE" ]]; then
                pretty=$(basename ${d})
                echo "${pretty} ${fortran} ${ccall} ${recommended}"
            fi
        fi
    done
}

newfunc
-----------------------------------------------------------------------------

We can call this from R to be able to tabulate the columns

R script
-----------------------------------------------------------------------------
dat <- read.table(pipe("./checkCandFortranCall.sh"),
                  col.names=c("package", "dotFortran", "dotC", "recommended"))
library(data.table)
setDT(dat)

sapply(dat[, -1], table)  # counts by type

dat[recommended==TRUE, ]

dat[recommended==FALSE, ]
-----------------------------------------------------------------------------

which on my box gives

      dotFortran dotC recommended
FALSE         28   13          37
TRUE          20   35          11

indicating a fair number (still around a third of my r-cran-* packages).

I think I will cover these by hand now:

               package dotFortran  dotC recommended
 1:       class-7.3-14      FALSE  TRUE        TRUE
 2:      cluster-2.0.6       TRUE  TRUE        TRUE
 3:     foreign-0.8.68      FALSE  TRUE        TRUE
 4: KernSmooth-2.23-15       TRUE FALSE        TRUE
 5:        MASS-7.3-47      FALSE  TRUE        TRUE
 6:      Matrix-1.2-10      FALSE  TRUE        TRUE
 7:        mgcv-1.8-17      FALSE  TRUE        TRUE
 8:       nlme-3.1.131       TRUE  TRUE        TRUE
 9:        nnet-7.3-12      FALSE  TRUE        TRUE
10:     spatial-7.3-11      FALSE  TRUE        TRUE
11:    survival-2.41-3      FALSE  TRUE        TRUE

and we should binNMU the remaining ones satisfying

  binary: any                           (ie applicable for Debian binNMUs)
  has either .C() or .Fortran()

I have access to a fully expanded directory of CRAN sources of if we start we
the reverse depends I can refine the script above to get a set of packages
for which we can then set up binNMUs.

One caveat: the shell script does not (yet?) filter out those packages that
already had a post R 3.3.3 update which includes eg several from the set above.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #77 received at 861333-close@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: 861333-close@bugs.debian.org

Subject: Bug#861333: fixed in quantlib-swig 1.9-2

Date: Sat, 29 Apr 2017 18:05:39 +0000

Source: quantlib-swig
Source-Version: 1.9-2

We believe that the bug you reported is fixed in the latest version of
quantlib-swig, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 861333@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Dirk Eddelbuettel <edd@debian.org> (supplier of updated quantlib-swig package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 29 Apr 2017 11:52:14 -0500
Source: quantlib-swig
Binary: quantlib-python
Architecture: source amd64
Version: 1.9-2
Distribution: unstable
Urgency: medium
Maintainer: Dirk Eddelbuettel <edd@debian.org>
Changed-By: Dirk Eddelbuettel <edd@debian.org>
Description:
 quantlib-python - Python bindings for the Quantlib Quantitative Finance library
Closes: 861333
Changes:
 quantlib-swig (1.9-2) unstable; urgency=medium
 .
   * debian/rules: Uncomment call to dh_md5sums		(Closes: #861333)
Checksums-Sha1:
 ea4dc796e8e831bd1d9febc98825e207bf1e1a82 1819 quantlib-swig_1.9-2.dsc
 7c5f107c3ab82bd55ec8546edb9b1cc11fe03c2e 10053 quantlib-swig_1.9-2.diff.gz
 2ff508f49ca8521d9f1bacb6aee5d3167c93d959 14635162 quantlib-python-dbgsym_1.9-2_amd64.deb
 73bcaecbb80e3be9f636a65ddd751c0d7fa06d16 2058792 quantlib-python_1.9-2_amd64.deb
 2594c6dfdf30ff4de9c8946b638e06f97e468b59 6097 quantlib-swig_1.9-2_amd64.buildinfo
Checksums-Sha256:
 e5d7bf90ba10d6caa252fa9d7e58f803e9171b4046e90b7f6bf7963c57bb81d9 1819 quantlib-swig_1.9-2.dsc
 cffddabf7bd9693bdf9dbf102474b5893faa289f7589ef0e342d00bd8094b6ce 10053 quantlib-swig_1.9-2.diff.gz
 376487359cdc0623dfcf88b24aa15bfcd8c66af8f506ffcb2c69aaa36726212e 14635162 quantlib-python-dbgsym_1.9-2_amd64.deb
 34ae2ff532b68d505b224a6e534b59c47dd3dd774e3c221d03615240c5320d9a 2058792 quantlib-python_1.9-2_amd64.deb
 3c4190cb3d13273e41f31ab7825a4a1be7e9c8d7b02d9e8ebcd8b831fc3cf42b 6097 quantlib-swig_1.9-2_amd64.buildinfo
Files:
 b15763be2503465b09e7f9bf2e7a9f15 1819 interpreters optional quantlib-swig_1.9-2.dsc
 c1dc68c736f326a86399ee996e3c4c7a 10053 interpreters optional quantlib-swig_1.9-2.diff.gz
 3f4c8135e4c74c8a75785fa319058c72 14635162 debug extra quantlib-python-dbgsym_1.9-2_amd64.deb
 32605a3086c62914503bc80473ad0caf 2058792 python optional quantlib-python_1.9-2_amd64.deb
 51f40458fa68e43096cd4bb00e61fa50 6097 interpreters optional quantlib-swig_1.9-2_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIVAwUBWQTSrKFIn+KrmaIaAQgmsQ/+PGyijDgQVoEV38Gceaw06rLJy8O/W6xA
mfN+dY3QrSvmupcR3uuG6JZR2x8RcCkQXNs7gPdZ/jQkBN1Co4E8DZVPxcNMYeJo
LAhKWXlFZuTEBQ5aReULcq4t92MgGpYtqqwTp0U8V13vFx3FC6ez6pYN+UHlUJa3
2Qczkmm0A1y2dFfIaBfY9XzVOsf/CSaB2xHk8HOTakdsWtWJvKMP/VP9FDSBIMyG
PZ0aeTvWlgQykCzDpkEBt25q+Mt0JEfIryNfPO1PdF9+ezPfMU3+cM9tEr+2otlh
9MMv3QaJ4uDmoZ1P1cY8l6RigGjVcgdTJK3EgrUji9aRr71u3hB2gWXo5wNCRcXL
+hr7z8QuaZTCeQd+FGbVtkhDUlqCxpBd5wzFlOV4yLLHCGCnLwPZeNZoa1mKm6na
KyiIV5tmKjXo5jdpPk9EON76sWNFXYcdYZZwlhIYIdu41sQrnv8bfUfDlLrvdjni
Ow1Hyvfumfgzjj5YrljuZUZvWmO2uj+NNkEMDEM+ptqUU2YSaYy+G1Owql2STWXW
R6ULgIy0uGgX54mj9Pcw8B2zkXAYdi0g5tOkqEKhKpsIyYKDqV1rL64eB23uY3RL
gE8N0l8Ix3VjNhZlgVPA5CP9hEzFCXm3xvLeOGPNajkiA4yoRLw+UPR7AaSy0W6N
F/jjfbQzwbs=
=9c2d
-----END PGP SIGNATURE-----

Message #86 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Charles Plessy <plessy@debian.org>

To: Dirk Eddelbuettel <edd@debian.org>, 861333@bugs.debian.org

Cc: Johannes Ranke <johannes.ranke@jrwb.de>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Mon, 1 May 2017 14:53:49 +0900

Le Sat, Apr 29, 2017 at 09:50:01AM -0500, Dirk Eddelbuettel a écrit :
> 
> I think I will cover these by hand now:
> 
>                package dotFortran  dotC recommended
>  1:       class-7.3-14      FALSE  TRUE        TRUE
>  2:      cluster-2.0.6       TRUE  TRUE        TRUE
>  3:     foreign-0.8.68      FALSE  TRUE        TRUE
>  4: KernSmooth-2.23-15       TRUE FALSE        TRUE
>  5:        MASS-7.3-47      FALSE  TRUE        TRUE
>  6:      Matrix-1.2-10      FALSE  TRUE        TRUE
>  7:        mgcv-1.8-17      FALSE  TRUE        TRUE
>  8:       nlme-3.1.131       TRUE  TRUE        TRUE
>  9:        nnet-7.3-12      FALSE  TRUE        TRUE
> 10:     spatial-7.3-11      FALSE  TRUE        TRUE
> 11:    survival-2.41-3      FALSE  TRUE        TRUE
> 
> and we should binNMU the remaining ones satisfying
> 
>   binary: any                           (ie applicable for Debian binNMUs)
>   has either .C() or .Fortran()
> 
> I have access to a fully expanded directory of CRAN sources of if we start we
> the reverse depends I can refine the script above to get a set of packages
> for which we can then set up binNMUs.
> 
> One caveat: the shell script does not (yet?) filter out those packages that
> already had a post R 3.3.3 update which includes eg several from the set above.

Hi Dirk,

this is very neat to pinpoint which package needs to be rebuilt.  Related to
this I had one more reminder from a member of the Release team that r-base
should not be co-installable with the packages that it breaks.  This is related
to the support of partial upgrades that I was mentioning earlier.

At this point I see 3 options:

 - For each rebuild, insert a "Breaks" relationship in r-base's control file;

 - Increment r-api-3 to r-api-4 (or r-api-3.4, etc.) in order to not have to
   maintain a long list of "Breaks" declarations.  In that case, we have to
   rebuild everything.

 - Just rebuild what has to be rebuilt, and do not support partial upgrades,
   which is what has been done until now.

Not supporting partial upgrades puts the maintainers of the r-cran and r-bioc
packages between the hammer and the anvil.  This said, I think that we have
made constant progresses over the years, so I do not feel shy saying "not yet"
to the Release team again if needed.

Have a nice day,

Charles

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

Message #95 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Johannes Ranke <johannes.ranke@jrwb.de>

To: Charles Plessy <plessy@debian.org>, 861333@bugs.debian.org

Cc: Dirk Eddelbuettel <edd@debian.org>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Thu, 04 May 2017 12:44:41 +0200

Am Montag, 1. Mai 2017, 14:53:49 schrieb Charles Plessy:

...

> At this point I see 3 options:
> 
>  - For each rebuild, insert a "Breaks" relationship in r-base's control
> file;

This is the solution favoured by me as the maintainer of the backports on CRAN 
(I know, this is the Debian BTS, but nonetheless), as it would just cause 
rebuilds/reinstalls of the packages really affected, assuming that we manage to 
have a versioned Breaks relationship for those packages (e.g. r-cran-spatial 
<= xy).

>  - Increment r-api-3 to r-api-4 (or r-api-3.4, etc.) in order to not have to
> maintain a long list of "Breaks" declarations.  In that case, we have to
> rebuild everything.

Would be OK for me, but seems to cause a lot of work for r-cran-* and r-bioc* 
maintainers

>  - Just rebuild what has to be rebuilt, and do not support partial upgrades,
> which is what has been done until now.

In this case, I would create a new repository on CRAN (again, I know that this 
is not really Debians business), so people would consciously install R 3.4.0 
and not be surprised by packages suddenly failing to find their objects.
 
> Not supporting partial upgrades puts the maintainers of the r-cran and
> r-bioc packages between the hammer and the anvil.

I do not understand this sentence.

> This said, I think that
> we have made constant progresses over the years, so I do not feel shy
> saying "not yet" to the Release team again if needed.

Message #104 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Charles Plessy <plessy@debian.org>

Cc: Dirk Eddelbuettel <edd@debian.org>, 861333@bugs.debian.org, Johannes Ranke <johannes.ranke@jrwb.de>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Sun, 25 Jun 2017 17:46:37 -0500

Hi Charles,

On 1 May 2017 at 14:53, Charles Plessy wrote:
| Le Sat, Apr 29, 2017 at 09:50:01AM -0500, Dirk Eddelbuettel a écrit :
| > 
| > I think I will cover these by hand now:
| > 
| >                package dotFortran  dotC recommended
| >  1:       class-7.3-14      FALSE  TRUE        TRUE
| >  2:      cluster-2.0.6       TRUE  TRUE        TRUE
| >  3:     foreign-0.8.68      FALSE  TRUE        TRUE
| >  4: KernSmooth-2.23-15       TRUE FALSE        TRUE
| >  5:        MASS-7.3-47      FALSE  TRUE        TRUE
| >  6:      Matrix-1.2-10      FALSE  TRUE        TRUE
| >  7:        mgcv-1.8-17      FALSE  TRUE        TRUE
| >  8:       nlme-3.1.131       TRUE  TRUE        TRUE
| >  9:        nnet-7.3-12      FALSE  TRUE        TRUE
| > 10:     spatial-7.3-11      FALSE  TRUE        TRUE
| > 11:    survival-2.41-3      FALSE  TRUE        TRUE
| > 
| > and we should binNMU the remaining ones satisfying
| > 
| >   binary: any                           (ie applicable for Debian binNMUs)
| >   has either .C() or .Fortran()
| > 
| > I have access to a fully expanded directory of CRAN sources of if we start we
| > the reverse depends I can refine the script above to get a set of packages
| > for which we can then set up binNMUs.
| > 
| > One caveat: the shell script does not (yet?) filter out those packages that
| > already had a post R 3.3.3 update which includes eg several from the set above.
| 
| Hi Dirk,
| 
| this is very neat to pinpoint which package needs to be rebuilt.  Related to

Let's recap: R Core told us in the release notes that with R 3.4.0, packages
containing .C() or .Fortran() calls need rebuilds with R 3.4.0 (or its release
candidate). Older builds will work for their R code, but not the compiled
code; hence the rebuilds.

This means the we have to find

  i)   the subset of packages having compiled code and
  ii)  among those having .C() / .Fortran and not .Call() and
  iii) those not yet having been rebuilt under R 3.4.0 (or its release candidate)

And I have more / better code now, still based on the RcppAPT package (and
its code on GitHub rather than CRAN). See the directory

  https://github.com/eddelbuettel/rcppapt/tree/master/inst/scripts/debian-packages

In a nutshell, based on analysis I now ran twice, we

- have 514 reverse dependencies of r-base-core in Debian

- of which 487 match the '^r-' regexp suitable for package

- of which 241 have themselves a reverse dependency on libc6, ie have src/

- of which 149 have not yet been rebuilt

- of which 69 actually have .C() / .Fortran() calls

For that last step I grep'ed actual sources (as I have shell access to a
machine at CRAN central in Vienna which has all packages sources).

The list of the 69 packages follows (as printed by data.table)

                     package           Package  Version
 1:              r-cran-ade4              ade4    1.7-6
 2:          r-cran-adegenet          adegenet    2.0.1
 3:          r-cran-adephylo          adephylo   1.1-10
 4:            r-cran-amelia            Amelia    1.7.4
 5:               r-cran-ape               ape      4.1
 6:            r-cran-bayesm            bayesm    3.0-2
 7:            r-cran-bitops            bitops    1.0-6
 8:     r-cran-blockmodeling     blockmodeling    0.1.8
 9:           r-cran-boolnet           BoolNet    2.1.3
10:             r-cran-brglm             brglm    0.5-9
11:             r-cran-caret             caret   6.0-76
12:           r-cran-catools           caTools   1.17.1
13:            r-cran-cmprsk            cmprsk    2.2-7
14:              r-cran-coin              coin    1.2-0
15:          r-cran-contfrac          contfrac   1.1-10
16:        r-cran-data.table        data.table   1.10.4
17:              r-cran-deal              deal   1.2-37
18:            r-cran-deldir            deldir   0.1-14
19:           r-cran-desolve           deSolve     1.14
20:       r-cran-dosefinding       DoseFinding   0.9-15
21:               r-cran-eco               eco    3.1-7
22:               r-cran-erm               eRm   0.15-7
23:               r-cran-etm               etm    0.6-2
24:               r-cran-evd               evd    2.3-2
25:              r-cran-expm              expm  0.999-2
26:            r-cran-fields            fields      9.0
27:               r-cran-gam               gam   1.14-4
28:           r-cran-genabel           GenABEL    1.8-0
29:            r-cran-glmnet            glmnet   2.0-10
30:           r-cran-goftest           goftest    1.1-1
31:               r-cran-gsl               gsl 1.9-10.3
32:       r-cran-haplo.stats       haplo.stats    1.7.7
33:            r-cran-hexbin            hexbin   1.27.1
34:            r-cran-igraph            igraph    1.0.1
35:               r-cran-lhs               lhs     0.14
36:         r-cran-logspline         logspline    2.1.9
37:           r-cran-mapproj           mapproj    1.2-5
38:              r-cran-maps              maps    3.2.0
39:          r-cran-maptools          maptools    0.9-2
40:              r-cran-mcmc              mcmc    0.9-5
41:          r-cran-mcmcpack          MCMCpack    1.4-0
42:          r-cran-mixtools          mixtools    1.1.0
43:           r-cran-mlbench           mlbench    2.1-1
44:               r-cran-mnp               MNP    3.0-1
45:               r-cran-msm               msm    1.6.4
46:             r-cran-ncdf4             ncdf4     1.16
47:              r-cran-nnls              nnls      1.4
48:          r-cran-pbivnorm          pbivnorm    0.6.0
49:          r-cran-phangorn          phangorn    2.2.0
50:         r-cran-phylobase         phylobase    0.8.4
51:           r-cran-polycub           polyCub    0.6.0
52:         r-cran-princurve         princurve   1.1-12
53:              r-cran-pscl              pscl    1.4.9
54:               r-cran-qtl               qtl   1.41-6
55:      r-cran-randomfields      RandomFields   3.1.50
56: r-cran-randomfieldsutils RandomFieldsUtils   0.3.25
57:      r-cran-randomforest      randomForest   4.6-12
58:      r-cran-raschsampler      RaschSampler    0.8-8
59:             r-cran-rcurl             RCurl 1.95-4.8
60:                r-cran-sp                sp    1.2-4
61:              r-cran-spam              spam    1.4-0
62:          r-cran-spatstat          spatstat   1.51-0
63:               r-cran-spc               spc    0.5.3
64:             r-cran-spdep             spdep   0.6-13
65:      r-cran-surveillance      surveillance   1.13.1
66:               r-cran-tgp               tgp   2.4-14
67:        r-cran-tikzdevice        tikzDevice   0.10-1
68:             r-cran-vegan             vegan    2.4-3
69:              r-cran-vgam              VGAM    1.0-3
                     package           Package  Version

I'll be traveling to some workshops and conferences over the next two weeks
so it will be harder for me to re-run this.

But the code is generic, and I left comments in there. Anybody with access to
Debian unstable and R can run this (having to install r-cran-rcpp and
r-cran-data.table, and then having to install RcppAPT from source which is
quick). 

| this I had one more reminder from a member of the Release team that r-base
| should not be co-installable with the packages that it breaks.  This is related
| to the support of partial upgrades that I was mentioning earlier.
| 
| At this point I see 3 options:
| 
|  - For each rebuild, insert a "Breaks" relationship in r-base's control file;
| 
|  - Increment r-api-3 to r-api-4 (or r-api-3.4, etc.) in order to not have to
|    maintain a long list of "Breaks" declarations.  In that case, we have to
|    rebuild everything.
| 
|  - Just rebuild what has to be rebuilt, and do not support partial upgrades,
|    which is what has been done until now.

I still think that is _perfectly_ adequate as it reduces _487 potential_
builds down to _69 actual_ ones.

A few dozen binNMUs seems fine, a few hundred including hundreds of false
positives less so.
 
| Not supporting partial upgrades puts the maintainers of the r-cran and r-bioc
| packages between the hammer and the anvil.  This said, I think that we have
| made constant progresses over the years, so I do not feel shy saying "not yet"
| to the Release team again if needed.

We can actually analyse this data and find the packages needing rebuilds, and
that course of actions strikes me as the most appropriate one.

Cheers, Dirk


| Have a nice day,
| 
| Charles
| 
| -- 
| Charles Plessy
| Debian Med packaging team,
| http://www.debian.org/devel/debian-med
| Tsurumi, Kanagawa, Japan

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #109 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Dirk Eddelbuettel <edd@debian.org>

Cc: Charles Plessy <plessy@debian.org>, 861333@bugs.debian.org, Johannes Ranke <johannes.ranke@jrwb.de>

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Sun, 16 Jul 2017 13:10:51 -0500

Just a quick follow-up to say that I finally had some time to go over this
again (twice, in fact) and to also include BioC and 'other' packages.

A full write-up in the (GitHub) sources of package RcppAPT and also available
as a rendered vignette at

   http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html

I have used the information therein to file the required binnmu request as
bug report #868558.

Cheers,  Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #114 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Johannes Ranke <johannes.ranke@jrwb.de>

To: Dirk Eddelbuettel <edd@debian.org>

Cc: Charles Plessy <plessy@debian.org>, 861333@bugs.debian.org

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Tue, 18 Jul 2017 11:55:56 +0200

Nice. Amazing work. So buster should be covered then.

Now (correct me if I am wrong) if we could adapt your scripts to create 
versioned Breaks: relationships with these packages, this would open the 
possibility to create backports for stretch-backports and jessie-backports-
sloppy, taking advantage of the Debian infrastructure for builds on all 
architectures.

Side note: Regarding backports on CRAN, I have chosen to create a separate 
repository for R >= 3.4.0, so people should be conscious about the issue for R 
package debs when they install it.

Message #119 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Johannes Ranke <johannes.ranke@jrwb.de>

Cc: Dirk Eddelbuettel <edd@debian.org>, Charles Plessy <plessy@debian.org>, 861333@bugs.debian.org

Subject: Re: Bug#861333: r-base: R packages uploaded to Debian before 14 April 2017 that use .C or .Fortran fail to find objects

Date: Tue, 18 Jul 2017 10:33:00 -0500

On 18 July 2017 at 11:55, Johannes Ranke wrote:
| Nice. Amazing work. So buster should be covered then.

Thanks!

I also pushed the code to CRAN as RcppAPT 0.0.4 just to mark a snapshot.
 
| Now (correct me if I am wrong) if we could adapt your scripts to create 
| versioned Breaks: relationships with these packages, this would open the 
| possibility to create backports for stretch-backports and jessie-backports-
| sloppy, taking advantage of the Debian infrastructure for builds on all 
| architectures.

Yes. RcppAPT just reads whatever cached data apt has -- will work on stable
too (unless you get unlucky and libapt-pkg-dev is slightly different which we
could probably address).

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #120 received at 861684-close@bugs.debian.org (full text, mbox, reply):

From: Andreas Tille <tille@debian.org>

To: 861684-close@bugs.debian.org

Subject: Bug#861684: fixed in r-cran-randomfields 3.1.50-1

Date: Thu, 24 Aug 2017 21:10:38 +0000

Source: r-cran-randomfields
Source-Version: 3.1.50-1

We believe that the bug you reported is fixed in the latest version of
r-cran-randomfields, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 861684@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Andreas Tille <tille@debian.org> (supplier of updated r-cran-randomfields package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Thu, 24 Aug 2017 22:24:04 +0200
Source: r-cran-randomfields
Binary: r-cran-randomfields
Architecture: source
Version: 3.1.50-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Changed-By: Andreas Tille <tille@debian.org>
Description:
 r-cran-randomfields - GNU R simulation and analysis of random fields
Closes: 861684
Changes:
 r-cran-randomfields (3.1.50-1) unstable; urgency=medium
 .
   * New upstream version
     Closes: #861684
   * Standards-Version: 4.1.0 (no changes needed)
Checksums-Sha1:
 fb7b90a32be942fd5a8bf178f7eb8fde2a71f963 2252 r-cran-randomfields_3.1.50-1.dsc
 cb0049236f41cb72dd6a1df9c65a2c7efd2a5328 1783992 r-cran-randomfields_3.1.50.orig.tar.gz
 0eb64575f967afac94690779071aa2d5d96f1c0a 2760 r-cran-randomfields_3.1.50-1.debian.tar.xz
 77395838d23b8a798adf0a00a52f92e25680c925 15021 r-cran-randomfields_3.1.50-1_source.buildinfo
Checksums-Sha256:
 d78a6affc1d32596bccd983ae0368976a576fca4ce6efee7d33f8a5598ff8e63 2252 r-cran-randomfields_3.1.50-1.dsc
 2d6a07c3a716ce20f9c685deb59e8fcc64fd52c8a50b0f04baf451b6b928e848 1783992 r-cran-randomfields_3.1.50.orig.tar.gz
 f27c27f734be4deae9a22fee7fb05564afb32a1c342e8091f5e9138075799321 2760 r-cran-randomfields_3.1.50-1.debian.tar.xz
 10338029b938ce4ec7c8376d343be0c4f90c3cc4cdae160e751a0782a297b035 15021 r-cran-randomfields_3.1.50-1_source.buildinfo
Files:
 c56b857029efea333159decf4de4841e 2252 gnu-r optional r-cran-randomfields_3.1.50-1.dsc
 fd91aea76365427c0ba3b25fb3af43a6 1783992 gnu-r optional r-cran-randomfields_3.1.50.orig.tar.gz
 b3d1e4f67315f695fffe8c90a19b4165 2760 gnu-r optional r-cran-randomfields_3.1.50-1.debian.tar.xz
 b1cb6ecd2c6976f5efbe00f66eb6ccbb 15021 gnu-r optional r-cran-randomfields_3.1.50-1_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJCBAEBCAAsFiEE8fAHMgoDVUHwpmPKV4oElNHGRtEFAlmfOQwOHHRpbGxlYUBy
a2kuZGUACgkQV4oElNHGRtG1WQ//QKdZ89621KwebRHvPOI5hrh1SWfzSa6PHSQ6
mThkIgQthQu5PwcgI1ceAUEqejud6ikTelfcAFo7ZJ9omVm3b5Cl78f5bvartqlB
2WbVXI8E9APDQKybDukSmAe+x3oqLrBcZ03tHuU4ZuT44ImuSjzQVdz1FzVGEYPN
cRCklKHKFXu9d2CgK6uivBcA7eDCkeoV8aN1cz/BUGKm216GtjjphU7vn7Vw7T1V
M8I7zDv+ixV0PRypbzZ/LzYJtanpe5at3ydz0D0fy29Fxs1t61KNoKFckvHXbJH1
puViMNAyo98zmD4RYWC0ApJwzKgKlRwsiUThAShUDYMj94He8Bz3gKQjn7EP745V
VdhFAp6CcIPj7uvDoW60p8Ae0APExlMKpIlaufVAe9gFxGJu6ysplWM6jaiYvVSO
DGvD2AvMndm9+b6ISshUG3s6XjF9ISCyDDtIuj0EUF92h2ynqOtMtAObu2SBvTdG
z3v3e3glq7lEkYfImX4l5KbXKMHNjrj7RjTsI5ht1kP2+PoiP/ZPhDre97LbT6Cz
Kdbs0VwT3C2YbiuvOl63cWJN2Jn+vsjr2q3O1axEpu+6A5X4Lhs9Iu7pWjb3Il+M
xITRPq+d7aIJUbwAg1DhawuO5r3hjlK087CvQJwvKnMF6ZblF4O9lp07RBiCTMT3
v2qrq9o=
=OGXf
-----END PGP SIGNATURE-----

Message #129 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Andreas Tille <tille@debian.org>

To: Dirk Eddelbuettel <edd@debian.org>

Cc: Debian Science List <debian-science@lists.debian.org>, Chris Lawrence <lawrencc@debian.org>, 861333@bugs.debian.org

Subject: Re: Why does r-cran-rcppgsl not migrate to testing?

Date: Thu, 7 Sep 2017 21:20:18 +0200

Hi Dirk,

On Wed, Sep 06, 2017 at 04:22:33PM -0500, Dirk Eddelbuettel wrote:
> On 6 September 2017 at 22:48, Andreas Tille wrote:
> | [Chris, I took the freedom to move r-cran-mcmc to Debian Science
> |  team since I had the impression that this would generally OK for
> |  you.]
> | 
> | Hi Dirk,
> | 
> | On Wed, Sep 06, 2017 at 10:17:04AM -0500, Dirk Eddelbuettel wrote:
> | > 
> | > | Is there any list of affected packages?
> | > 
> | > I gave this URL about half a dozen times:
> | > 
> | >   http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html
> | 
> | Well, you know from own experience that not all information is reaching
> | the target audience.  It might have helped to address Debian Science and
> | Debian Med team to make some more noise.
> |  
> | > It contains the list, as well as a way to compute it.

Any chance to recompute the list just in case somebody else has also
upgraded a package?  It would be nice if the list would have a timestamp
of creation.

> | The list is not fully up to date.  I recently uploaded a new version of
> | r-cran-randomfields which remains inside the list.  I need to admit a
> | shorter page which points directly to tasks to do which is up to date
> | would be more motivating to lend a helping hand.
> | 
> | I just uploaded
> | 
> |    r-cran-spdep
> |    r-cran-gam
> |    r-cran-mcmc

I uploaded as well:

	r-cran-data.table
	r-cran-vegan
	r-cran-bayesm
	r-cran-expm
	r-cran-phangorn
	r-cran-maptools
	r-cran-caret
	r-cran-goftest
	r-cran-igraph
	r-cran-maps
	r-cran-eco
	r-cran-randomfields
	r-bioc-genefilter

For those who want to help but have no idea how to do a team upload:

	debcheckout --user <user_name_on_alioth> --git-track '*' <package>
	cd <package>
	dch --team
	# do at least a Standards-Version: 4.1.0
	# even better
	uscan --verbose
	# upgrade to new version
	# commit + push your changes

Any Debian developer has commit permissions to Debian Med / Debian
Science repositories.  Other users need to ask for team membership.
It would be fine if you submit for instance

	git format-patch <your_first_commit>

and attach these patches to a sponsoring request bug.

In case a package is not yet in VCS you can do the following:

	gbp import-dscs --debsnap --pristine-tar <package>
	cd <package>
	# ask maintainer whether it is OK to move the package
	# into Debian Science team maintenance.  If yes,
	# add Vcs Fields and the Debian Science maintainer list
	# as Maintainer, keeping the former Maintainer as Uploader
	# do changes as above
	# to inject your freshly created repository you can use
	svn checkout svn://anonscm.debian.org/debian-med/trunk/helper-scripts /tmp/helper-scripts
	/tmp/helper-scripts/inject-into-alioth-git

This is basically what I did with the packages above and I'll try
to keep on working on this.

> | > | I intend to refresh with new upstream sources anyway in the next couple of days.  May be we can do
> | > | real uploads of most of the packages ourselves? 
> | > 
> | > Please do. Being behind upstream is essentially never a good idea.
> | 
> | I'd prefer if you would leave out at least every second chance to repeat
> | this.  We could talk about this once somebody might pay somebody to
> | follow each and every update of any random R package.
> 
> I may once you start to maintain them -- instead of just hoarding them Take
> but one example: https://packages.debian.org/sid/r-cran-data.table
> 
> Exactly who is served by not updating one of the more widely used to package
> to one of the two releases that happened _this calendar year_ ?

If you would ask a non-polemic question I would consider answering
instead of working on the packages even if you are stealing the topic of
the thread. 

Kind regards

      Andreas.

-- 
http://fam-tille.de

Message #134 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Andreas Tille <tille@debian.org>

Cc: Dirk Eddelbuettel <edd@debian.org>, Debian Science List <debian-science@lists.debian.org>, Chris Lawrence <lawrencc@debian.org>, 861333@bugs.debian.org

Subject: Re: Why does r-cran-rcppgsl not migrate to testing?

Date: Thu, 7 Sep 2017 14:46:04 -0500

Hi Andreas,

On 7 September 2017 at 21:20, Andreas Tille wrote:
| Hi Dirk,
| 
| On Wed, Sep 06, 2017 at 04:22:33PM -0500, Dirk Eddelbuettel wrote:
| > On 6 September 2017 at 22:48, Andreas Tille wrote:
| > | [Chris, I took the freedom to move r-cran-mcmc to Debian Science
| > |  team since I had the impression that this would generally OK for
| > |  you.]
| > | 
| > | Hi Dirk,
| > | 
| > | On Wed, Sep 06, 2017 at 10:17:04AM -0500, Dirk Eddelbuettel wrote:
| > | > 
| > | > | Is there any list of affected packages?
| > | > 
| > | > I gave this URL about half a dozen times:
| > | > 
| > | >   http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html
| > | 
| > | Well, you know from own experience that not all information is reaching
| > | the target audience.  It might have helped to address Debian Science and
| > | Debian Med team to make some more noise.
| > |  
| > | > It contains the list, as well as a way to compute it.
| 
| Any chance to recompute the list just in case somebody else has also
| upgraded a package?  It would be nice if the list would have a timestamp
| of creation.

It should just work -- the write up is after all hanging off the RcppAPT
repo, so just install RcppAPT and then you can query R _and Debian_ from R.
The one step Prof Hornik suggested required CRAN sources to grep, but I think
I in the last iteration I proxied that by just fetching the .tar.gz from
CRAN.  Or at least it could be done.

If you have a question about RcppAPT maybe just open an issue at GitHub.

I'll be traveling this weekend so not sure I'll get to that before next week.
 
| > | The list is not fully up to date.  I recently uploaded a new version of
| > | r-cran-randomfields which remains inside the list.  I need to admit a
| > | shorter page which points directly to tasks to do which is up to date
| > | would be more motivating to lend a helping hand.
| > | 
| > | I just uploaded
| > | 
| > |    r-cran-spdep
| > |    r-cran-gam
| > |    r-cran-mcmc
| 
| I uploaded as well:
| 
| 	r-cran-data.table
| 	r-cran-vegan
| 	r-cran-bayesm
| 	r-cran-expm
| 	r-cran-phangorn
| 	r-cran-maptools
| 	r-cran-caret
| 	r-cran-goftest
| 	r-cran-igraph
| 	r-cran-maps
| 	r-cran-eco
| 	r-cran-randomfields
| 	r-bioc-genefilter

That helps with the open bug report, thank you!  As you know I'd also love to
see them be current. I find with my r-cran-* packages (of which I have
several dozen) that it only takes a couple of minutes each time so I
generally do.

| For those who want to help but have no idea how to do a team upload:
| 
| 	debcheckout --user <user_name_on_alioth> --git-track '*' <package>
| 	cd <package>
| 	dch --team
| 	# do at least a Standards-Version: 4.1.0
| 	# even better
| 	uscan --verbose
| 	# upgrade to new version
| 	# commit + push your changes
| 
| Any Debian developer has commit permissions to Debian Med / Debian
| Science repositories.  Other users need to ask for team membership.
| It would be fine if you submit for instance
| 
| 	git format-patch <your_first_commit>
| 
| and attach these patches to a sponsoring request bug.
| 
| In case a package is not yet in VCS you can do the following:
| 
| 	gbp import-dscs --debsnap --pristine-tar <package>
| 	cd <package>
| 	# ask maintainer whether it is OK to move the package
| 	# into Debian Science team maintenance.  If yes,
| 	# add Vcs Fields and the Debian Science maintainer list
| 	# as Maintainer, keeping the former Maintainer as Uploader
| 	# do changes as above
| 	# to inject your freshly created repository you can use
| 	svn checkout svn://anonscm.debian.org/debian-med/trunk/helper-scripts /tmp/helper-scripts
| 	/tmp/helper-scripts/inject-into-alioth-git
| 
| This is basically what I did with the packages above and I'll try
| to keep on working on this.
|
| > | > | I intend to refresh with new upstream sources anyway in the next couple of days.  May be we can do
| > | > | real uploads of most of the packages ourselves? 
| > | > 
| > | > Please do. Being behind upstream is essentially never a good idea.
| > | 
| > | I'd prefer if you would leave out at least every second chance to repeat
| > | this.  We could talk about this once somebody might pay somebody to
| > | follow each and every update of any random R package.
| > 
| > I may once you start to maintain them -- instead of just hoarding them Take
| > but one example: https://packages.debian.org/sid/r-cran-data.table
| > 
| > Exactly who is served by not updating one of the more widely used to package
| > to one of the two releases that happened _this calendar year_ ?
| 
| If you would ask a non-polemic question I would consider answering
| instead of working on the packages even if you are stealing the topic of
| the thread. 

Sorry, but you started this.  After you had the temerity to ask about this
list when I had posted the same URL probably four or five times already.

And I just don't understand why you get so irritated about it. This is a
simple observable reality visible to everybody who cares to look a the QA
pages for debian-med and debian-science.  Hundreds of packages, generally
well maintained with few critical bugs --- but for several years now also
generally outdated. You can jump up and down and scream at me as much as you
want, but the fix is not in yelling at me for pointing this out.  The fix is
in keeping the packages current.  As we do with most other Debian packages.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #139 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Andreas Tille <tille@debian.org>

To: Dirk Eddelbuettel <edd@debian.org>

Cc: Debian Science List <debian-science@lists.debian.org>, Chris Lawrence <lawrencc@debian.org>, 861333@bugs.debian.org

Subject: Re: Why does r-cran-rcppgsl not migrate to testing?

Date: Fri, 8 Sep 2017 22:47:52 +0200

Hi Dirk,

On Thu, Sep 07, 2017 at 02:46:04PM -0500, Dirk Eddelbuettel wrote:
> 
> On 7 September 2017 at 21:20, Andreas Tille wrote:
> | Hi Dirk,
> | 
> | On Wed, Sep 06, 2017 at 04:22:33PM -0500, Dirk Eddelbuettel wrote:
> | > On 6 September 2017 at 22:48, Andreas Tille wrote:
> | > | On Wed, Sep 06, 2017 at 10:17:04AM -0500, Dirk Eddelbuettel wrote:
> | > | > 
> | > | > | Is there any list of affected packages?
> | > | > 
> | > | > I gave this URL about half a dozen times:
> | > | > 
> | > | >   http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html
> | > | 
> | > | Well, you know from own experience that not all information is reaching
> | > | the target audience.  It might have helped to address Debian Science and
> | > | Debian Med team to make some more noise.
> | > |  
> | > | > It contains the list, as well as a way to compute it.
> | 
> | Any chance to recompute the list just in case somebody else has also
> | upgraded a package?  It would be nice if the list would have a timestamp
> | of creation.
> 
> It should just work -- the write up is after all hanging off the RcppAPT
> repo, so just install RcppAPT and then you can query R _and Debian_ from R.
> The one step Prof Hornik suggested required CRAN sources to grep, but I think
> I in the last iteration I proxied that by just fetching the .tar.gz from
> CRAN.  Or at least it could be done.
> 
> If you have a question about RcppAPT maybe just open an issue at GitHub.

Dirk, you misunderstood my query:  I was asking you for upgrading your
list which is now heavily outdated not how I can learn a tool.  You
explained how often you linked to your page - a that frequently linked
page should be up to date to avoid others from trying to pick up work
that is now done.

Please try to understand that if you want to attract people to work on a
common goal with you you should try to make their work as easy as
possible.
 
> I'll be traveling this weekend so not sure I'll get to that before next week.
>  
> | > | The list is not fully up to date.  I recently uploaded a new version of
> | > | r-cran-randomfields which remains inside the list.  I need to admit a
> | > | shorter page which points directly to tasks to do which is up to date
> | > | would be more motivating to lend a helping hand.
> | > | 
> | > | I just uploaded
> | > | 
> | > |    r-cran-spdep
> | > |    r-cran-gam
> | > |    r-cran-mcmc
> | 
> | I uploaded as well:
> | 
> | 	r-cran-data.table
> | 	r-cran-vegan
> | 	r-cran-bayesm
> | 	r-cran-expm
> | 	r-cran-phangorn
> | 	r-cran-maptools
> | 	r-cran-caret
> | 	r-cran-goftest
> | 	r-cran-igraph
> | 	r-cran-maps
> | 	r-cran-eco
> | 	r-cran-randomfields
> | 	r-bioc-genefilter

I worked down the whole list with exception of your own package
r-cran-hdf5 and for two remaining packages I needed to package new
dependencies.  I've pinged #debian-ftp on IRC asking for fast
processing.
 
> That helps with the open bug report, thank you!  As you know I'd also love to
> see them be current. I find with my r-cran-* packages (of which I have
> several dozen) that it only takes a couple of minutes each time so I
> generally do.

I admit that some packages took a bit longer in case of the team hijacks
I did (Chris I keep on assuming that this is OK for you).  I also had to
deal with binary files that should be documented in README.source (I
wished we had a Debian R team clarifying things with ftpmaster in a more
packagers friendly way).

Regarding your response below:  Dirk, I consider my own time to valuable
to correct your offending accusations.  I felt it way better spent by
just fixing the packages.  I have some ideas how we could enhance the
situation but I'm not willing to discuss with you if you always turn to
pointless insulting accusations.

Kind regards

       Andreas.
 
> | For those who want to help but have no idea how to do a team upload:
> | 
> | 	debcheckout --user <user_name_on_alioth> --git-track '*' <package>
> | 	cd <package>
> | 	dch --team
> | 	# do at least a Standards-Version: 4.1.0
> | 	# even better
> | 	uscan --verbose
> | 	# upgrade to new version
> | 	# commit + push your changes
> | 
> | Any Debian developer has commit permissions to Debian Med / Debian
> | Science repositories.  Other users need to ask for team membership.
> | It would be fine if you submit for instance
> | 
> | 	git format-patch <your_first_commit>
> | 
> | and attach these patches to a sponsoring request bug.
> | 
> | In case a package is not yet in VCS you can do the following:
> | 
> | 	gbp import-dscs --debsnap --pristine-tar <package>
> | 	cd <package>
> | 	# ask maintainer whether it is OK to move the package
> | 	# into Debian Science team maintenance.  If yes,
> | 	# add Vcs Fields and the Debian Science maintainer list
> | 	# as Maintainer, keeping the former Maintainer as Uploader
> | 	# do changes as above
> | 	# to inject your freshly created repository you can use
> | 	svn checkout svn://anonscm.debian.org/debian-med/trunk/helper-scripts /tmp/helper-scripts
> | 	/tmp/helper-scripts/inject-into-alioth-git
> | 
> | This is basically what I did with the packages above and I'll try
> | to keep on working on this.
> |
> | > | > | I intend to refresh with new upstream sources anyway in the next couple of days.  May be we can do
> | > | > | real uploads of most of the packages ourselves? 
> | > | > 
> | > | > Please do. Being behind upstream is essentially never a good idea.
> | > | 
> | > | I'd prefer if you would leave out at least every second chance to repeat
> | > | this.  We could talk about this once somebody might pay somebody to
> | > | follow each and every update of any random R package.
> | > 
> | > I may once you start to maintain them -- instead of just hoarding them Take
> | > but one example: https://packages.debian.org/sid/r-cran-data.table
> | > 
> | > Exactly who is served by not updating one of the more widely used to package
> | > to one of the two releases that happened _this calendar year_ ?
> | 
> | If you would ask a non-polemic question I would consider answering
> | instead of working on the packages even if you are stealing the topic of
> | the thread. 
> 
> Sorry, but you started this.  After you had the temerity to ask about this
> list when I had posted the same URL probably four or five times already.
> 
> And I just don't understand why you get so irritated about it. This is a
> simple observable reality visible to everybody who cares to look a the QA
> pages for debian-med and debian-science.  Hundreds of packages, generally
> well maintained with few critical bugs --- but for several years now also
> generally outdated. You can jump up and down and scream at me as much as you
> want, but the fix is not in yelling at me for pointing this out.  The fix is
> in keeping the packages current.  As we do with most other Debian packages.
> 
> Dirk
> 
> -- 
> http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
> 
> 

-- 
http://fam-tille.de

Message #144 received at 861333@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: Andreas Tille <tille@debian.org>

Cc: Dirk Eddelbuettel <edd@debian.org>, Debian Science List <debian-science@lists.debian.org>, Chris Lawrence <lawrencc@debian.org>, 861333@bugs.debian.org

Subject: Re: Why does r-cran-rcppgsl not migrate to testing?

Date: Fri, 8 Sep 2017 16:31:01 -0500

On 8 September 2017 at 22:47, Andreas Tille wrote:
| Hi Dirk,
| 
| On Thu, Sep 07, 2017 at 02:46:04PM -0500, Dirk Eddelbuettel wrote:
| > 
| > On 7 September 2017 at 21:20, Andreas Tille wrote:
| > | Hi Dirk,
| > | 
| > | On Wed, Sep 06, 2017 at 04:22:33PM -0500, Dirk Eddelbuettel wrote:
| > | > On 6 September 2017 at 22:48, Andreas Tille wrote:
| > | > | On Wed, Sep 06, 2017 at 10:17:04AM -0500, Dirk Eddelbuettel wrote:
| > | > | > 
| > | > | > | Is there any list of affected packages?
| > | > | > 
| > | > | > I gave this URL about half a dozen times:
| > | > | > 
| > | > | >   http://eddelbuettel.github.io/rcppapt/binnmuAfterR340.html
| > | > | 
| > | > | Well, you know from own experience that not all information is reaching
| > | > | the target audience.  It might have helped to address Debian Science and
| > | > | Debian Med team to make some more noise.
| > | > |  
| > | > | > It contains the list, as well as a way to compute it.
| > | 
| > | Any chance to recompute the list just in case somebody else has also
| > | upgraded a package?  It would be nice if the list would have a timestamp
| > | of creation.
| > 
| > It should just work -- the write up is after all hanging off the RcppAPT
| > repo, so just install RcppAPT and then you can query R _and Debian_ from R.
| > The one step Prof Hornik suggested required CRAN sources to grep, but I think
| > I in the last iteration I proxied that by just fetching the .tar.gz from
| > CRAN.  Or at least it could be done.
| > 
| > If you have a question about RcppAPT maybe just open an issue at GitHub.
| 
| Dirk, you misunderstood my query:  I was asking you for upgrading your
| list which is now heavily outdated not how I can learn a tool.  You
| explained how often you linked to your page - a that frequently linked
| page should be up to date to avoid others from trying to pick up work
| that is now done.
| 
| Please try to understand that if you want to attract people to work on a
| common goal with you you should try to make their work as easy as
| possible.
|  
| > I'll be traveling this weekend so not sure I'll get to that before next week.
| >  
| > | > | The list is not fully up to date.  I recently uploaded a new version of
| > | > | r-cran-randomfields which remains inside the list.  I need to admit a
| > | > | shorter page which points directly to tasks to do which is up to date
| > | > | would be more motivating to lend a helping hand.
| > | > | 
| > | > | I just uploaded
| > | > | 
| > | > |    r-cran-spdep
| > | > |    r-cran-gam
| > | > |    r-cran-mcmc
| > | 
| > | I uploaded as well:
| > | 
| > | 	r-cran-data.table
| > | 	r-cran-vegan
| > | 	r-cran-bayesm
| > | 	r-cran-expm
| > | 	r-cran-phangorn
| > | 	r-cran-maptools
| > | 	r-cran-caret
| > | 	r-cran-goftest
| > | 	r-cran-igraph
| > | 	r-cran-maps
| > | 	r-cran-eco
| > | 	r-cran-randomfields
| > | 	r-bioc-genefilter
| 
| I worked down the whole list with exception of your own package
| r-cran-hdf5 and for two remaining packages I needed to package new

Thank you for updating the packages. I missed hdf5 as it is "special" --
orphaned upstreamed. I think there are newer hdf5 packages in BioConductor we
should probably try to switch to.

I never listed the remaining ones by maintainer. Mayne I will.

But as I said, traveling -- at my daughter's college and just between giving
two talks.

| dependencies.  I've pinged #debian-ftp on IRC asking for fast
| processing.
|  
| > That helps with the open bug report, thank you!  As you know I'd also love to
| > see them be current. I find with my r-cran-* packages (of which I have
| > several dozen) that it only takes a couple of minutes each time so I
| > generally do.
| 
| I admit that some packages took a bit longer in case of the team hijacks
| I did (Chris I keep on assuming that this is OK for you).  I also had to
| deal with binary files that should be documented in README.source (I
| wished we had a Debian R team clarifying things with ftpmaster in a more
| packagers friendly way).
| 
| Regarding your response below:  Dirk, I consider my own time to valuable
| to correct your offending accusations.  I felt it way better spent by
| just fixing the packages.  I have some ideas how we could enhance the
| situation but I'm not willing to discuss with you if you always turn to
| pointless insulting accusations.

Factually incorrect as I never say anything personal about you.

But I *will* point out poorly maintained packages (e.g. missing months worth
updates) as that is a simple observable fact.

I have put 15+ plus into maintaining R in a timely manner. I cannot recommend
people use r-cran-* packages as many simply stale. I still find this very
upsetting, as you appears to be at the center of this you will hear about it.

Dirk
| 
| Kind regards
| 
|        Andreas.
|  
| > | For those who want to help but have no idea how to do a team upload:
| > | 
| > | 	debcheckout --user <user_name_on_alioth> --git-track '*' <package>
| > | 	cd <package>
| > | 	dch --team
| > | 	# do at least a Standards-Version: 4.1.0
| > | 	# even better
| > | 	uscan --verbose
| > | 	# upgrade to new version
| > | 	# commit + push your changes
| > | 
| > | Any Debian developer has commit permissions to Debian Med / Debian
| > | Science repositories.  Other users need to ask for team membership.
| > | It would be fine if you submit for instance
| > | 
| > | 	git format-patch <your_first_commit>
| > | 
| > | and attach these patches to a sponsoring request bug.
| > | 
| > | In case a package is not yet in VCS you can do the following:
| > | 
| > | 	gbp import-dscs --debsnap --pristine-tar <package>
| > | 	cd <package>
| > | 	# ask maintainer whether it is OK to move the package
| > | 	# into Debian Science team maintenance.  If yes,
| > | 	# add Vcs Fields and the Debian Science maintainer list
| > | 	# as Maintainer, keeping the former Maintainer as Uploader
| > | 	# do changes as above
| > | 	# to inject your freshly created repository you can use
| > | 	svn checkout svn://anonscm.debian.org/debian-med/trunk/helper-scripts /tmp/helper-scripts
| > | 	/tmp/helper-scripts/inject-into-alioth-git
| > | 
| > | This is basically what I did with the packages above and I'll try
| > | to keep on working on this.
| > |
| > | > | > | I intend to refresh with new upstream sources anyway in the next couple of days.  May be we can do
| > | > | > | real uploads of most of the packages ourselves? 
| > | > | > 
| > | > | > Please do. Being behind upstream is essentially never a good idea.
| > | > | 
| > | > | I'd prefer if you would leave out at least every second chance to repeat
| > | > | this.  We could talk about this once somebody might pay somebody to
| > | > | follow each and every update of any random R package.
| > | > 
| > | > I may once you start to maintain them -- instead of just hoarding them Take
| > | > but one example: https://packages.debian.org/sid/r-cran-data.table
| > | > 
| > | > Exactly who is served by not updating one of the more widely used to package
| > | > to one of the two releases that happened _this calendar year_ ?
| > | 
| > | If you would ask a non-polemic question I would consider answering
| > | instead of working on the packages even if you are stealing the topic of
| > | the thread. 
| > 
| > Sorry, but you started this.  After you had the temerity to ask about this
| > list when I had posted the same URL probably four or five times already.
| > 
| > And I just don't understand why you get so irritated about it. This is a
| > simple observable reality visible to everybody who cares to look a the QA
| > pages for debian-med and debian-science.  Hundreds of packages, generally
| > well maintained with few critical bugs --- but for several years now also
| > generally outdated. You can jump up and down and scream at me as much as you
| > want, but the fix is not in yelling at me for pointing this out.  The fix is
| > in keeping the packages current.  As we do with most other Debian packages.
| > 
| > Dirk
| > 
| > -- 
| > http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
| > 
| > 
| 
| -- 
| http://fam-tille.de

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Message #149 received at 861333-done@bugs.debian.org (full text, mbox, reply):

From: Dirk Eddelbuettel <edd@debian.org>

To: 861333-done@bugs.debian.org

Cc: Dirk Eddelbuettel <edd@debian.org>, Andreas Tille <tille@debian.org>

Subject: All affected packages are manually uploaded

Date: Sat, 9 Sep 2017 10:22:25 -0500

Per the email thread in #868558, this bug can also be closed as all packages
needing a rebuild under R 3.4.* have now gotten a rebuild, thanks an
energetic push by Andreas.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Send a report that this bug log contains spam.