Debian Bug report logs - #837992
cloud.debian.org: vagrant libvirt box NFS mount failure on vagrant up

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Emmanuel Kasper <emmanuel@libera.cc>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: cloud.debian.org: vagrant libvirt box NFS mount failure on vagrant up

Date: Fri, 16 Sep 2016 10:17:32 +0200

Package: cloud.debian.org
Severity: normal

The libvirt box has problem with the default shared mode (NFS)

A fresh vagrant up from the atlas box brings:

mount -o vers=3,udp
192.168.121.1:/home/manu/Projects/vagrenvs/jatlavirtdef /vagrant
if command -v /sbin/init && /sbin/init --version | grep upstart; then
  /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=/vagrant
fi
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

this seens to relate to this bug in Jessie:  #775542
NFS exports fail due to rpcbind not starting before nfs-common and
nfs-kernel-server

 <gory technical details>
 the rpc.statd startup depends on portmapper
 portmapper is started by the rpcbind service
 and the rpcbind service is not started early enough if started at all
 </>

I see three possibility fo fix this:
 A) switch to NFSv4 by default via a Vagrant configuration option
NFSv4 does not requires the rpc.statd service to run before mounting a
NFS share
 B) force the rpcbind server to start earlier by patching stuff in /etc
 C) switch to rsync as the default file sharing mecanismus ( or maybe
p9fs) for libvirt boxes. Rsync has also the advantage of not requiring
the root password for each vagrant up.


I would propose to work this bug around in order of preferences via C) or A)

and check if #775542 is fixed in stretch


-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.5.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=fr_FR.utf8, LC_CTYPE=fr_FR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Message #10 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Emmanuel Kasper <emmanuel@libera.cc>

To: 837992@bugs.debian.org

Date: Fri, 16 Sep 2016 10:23:14 +0200

inital bug discussion happpened via email, but we prefer to have this in
the BTS, so move the relevant part of the discussions here


> I see three possibility fo fix this:
>  A) switch to NFSv4 by default via a Vagrant configuration option
> NFSv4 does not requires the rpc.statd service to run before mounting a
> NFS share
>  B) force the rpcbind server to start earlier by patching stuff in /etc
>  C) switch to rsync as the default file sharing mecanismus ( or maybe
> p9fs) for libvirt boxes. Rsync has also the advantage of not requiring
> the root password for each vagrant up.
>
>
> I would propose to work this bug around in order of preferences via C)
or A)

A) seems cleaner.

wrt C):

- I have tried p9fs before and it is not so easy to make it work
  correctly, because you cannot get the uid mapping between host and vm
  correct all the time; it's "easy" to get it working if the share is
  read-only from the PoV of the VM, but not so easy otherwise.

- rsync seems like a terrible regression on how vagrant is supposed to
  work. you expect to change something from the host and it being
  immediately reflected on the VM, but when sharing via rsync you need
  to re-provision the VM to have your changes reflected, which sucks

Message #15 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Marcin Kulisz <debian@kulisz.net>

To: 837992@bugs.debian.org

Date: Fri, 16 Sep 2016 09:46:04 +0100

[Message part 1 (text/plain, inline)]

I personally am not a big vagrant user but I'd opt for the solution A
-- 

|_|0|_|                                                  |
|_|_|0|                  "Panta rei"                     |
|0|0|0|             -------- kuLa --------               |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Emmanuel Kasper <emmanuel@libera.cc>

To: 837992@bugs.debian.org

Subject: nfs problem might be related to rpcbind installation in chroot

Date: Fri, 16 Sep 2016 11:59:01 +0200

Le 16/09/2016 à 10:17, Emmanuel Kasper a écrit :
> Package: cloud.debian.org
> Severity: normal
>
> The libvirt box has problem with the default shared mode (NFS)
>
> A fresh vagrant up from the atlas box brings:
>
> mount -o vers=3,udp
> 192.168.121.1:/home/manu/Projects/vagrenvs/jatlavirtdef /vagrant
> if command -v /sbin/init && /sbin/init --version | grep upstart; then
>   /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=/vagrant
> fi
> mount.nfs: rpc.statd is not running but is required for remote locking.
> mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
> mount.nfs: an incorrect mount option was specified
>
> this seens to relate to this bug in Jessie:  #775542
> NFS exports fail due to rpcbind not starting before nfs-common and
> nfs-kernel-server

further debugging showed me that the problem is not related to #775542
but to rpcbind itself

vagrant base boxes include all packages with standard|important
https://anonscm.debian.org/cgit/cloud/debian-vm-templates.git/tree/helpers/vagrant-setup#n35

this pulls rpcbind which is needed for NFSv3 mounts

now this package installation happens within a chroot with
automatic service startup disabled with
printf '#!/bin/sh\nexit 101\n' > $fs/usr/sbin/policy-rc.d

I don't know how much this plays a role, but the rpcbind service
afterwards _never_ starts on boot in the resulting image
even if you try to reenable it manually via

   systemctl enable rpcbind in the created image

I am now wondering if the problem comes from our script or inoperant
startup sequence from rpcbind

@marcin: since you're a bootstrap-vz expert here, do you know if a
debian image created with boostrap-vz exhibits the same behaviour ?
ie when adding rpc package in a boostrap-vz manifest, is  the service
started on boot (systemctl status rpcbind)

Message #25 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Emmanuel Kasper <emmanuel@libera.cc>

Cc: Antonio Terceiro <terceiro@debian.org>, 837992@bugs.debian.org

Subject: Vagrant NFS bug

Date: Sat, 17 Sep 2016 18:44:40 +0200

[Message part 1 (text/plain, inline)]

>> I see three possibility fo fix this:
>>  A) switch to NFSv4 by default via a Vagrant configuration option
>> NFSv4 does not requires the rpc.statd service to run before mounting a
>> NFS share
>>  B) force the rpcbind server to start earlier by patching stuff in /etc
>>  C) switch to rsync as the default file sharing mecanismus ( or maybe
>> p9fs) for libvirt boxes. Rsync has also the advantage of not requiring
>> the root password for each vagrant up.
>>
>>
>> I would propose to work this bug around in order of preferences via C)
> or A)

> A) seems cleaner.

the attach patch should fix the NFS mounts on startup
it fixes the issue when being applied locally on a Vagrantfile

however I am not confident applying it, since all libvirt boxes I build
hang detecting the IP adress of the libvirt guest

VAGRANT_LOG=debug vagrant up

0x000000023fd700>
DEBUG wait_till_up: Searching for IP for MAC address: 52:54:00:cd:84:16
 INFO interface: info: Waiting for domain to get an IP address...
 INFO interface: info: ==> default: Waiting for domain to get an IP
address...
==> default: Waiting for domain to get an IP address...
 INFO retryable: Retryable exception raised:
#<Fog::Errors::TimeoutError: The specified wait_for timeout (2 seconds)
was exceeded>
 INFO retryable: Retryable exception raised:
#<Fog::Errors::TimeoutError: The specified wait_for timeout (2 seconds)
was exceeded>
 INFO retryable: Retryable exception raised:
#<Fog::Errors::TimeoutError: The specified wait_for timeout (2 seconds)
was exceeded>
 INFO retryable: Retryable exception raised:
#<Fog::Errors::TimeoutError: The specified wait_for timeout (2 seconds)
was exceeded>

and so on ...

if I start some network activity in the guest via virt-manager then
vagrant is able to detect the IP adress of the guest and then go on with
the rest of the vagrant up sequence but you need to this manually
I have no clue here

> wrt C):
> 
> - I have tried p9fs before and it is not so easy to make it work
>   correctly, because you cannot get the uid mapping between host and vm
>   correct all the time; it's "easy" to get it working if the share is
>   read-only from the PoV of the VM, but not so easy otherwise.
> 
> - rsync seems like a terrible regression on how vagrant is supposed to
>   work. you expect to change something from the host and it being
>   immediately reflected on the VM, but when sharing via rsync you need
>   to re-provision the VM to have your changes reflected, which sucks
> 

did you see the vagrant rsync-auto command ?

if you start vagrant rsync-auto in the same directory in another shell
window, vagrant will start a background rsync process who watchers via
inotify the changes to your current dir and push them to the vagrant guest
it is only one way sync though

but yes NFS works better becaure of the two way sync

[virt.patch (text/x-patch, attachment)]

Message #30 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Marcin Kulisz <debian@kulisz.net>

To: debian-cloud@lists.debian.org

Cc: 837992@bugs.debian.org

Subject: Re: Bug#837992: nfs problem might be related to rpcbind installation in chroot

Date: Wed, 21 Sep 2016 13:47:47 +0100

[Message part 1 (text/plain, inline)]

On 2016-09-19 14:46:55, Marcin Kulisz wrote:
> On 2016-09-16 11:59:01, Emmanuel Kasper wrote:
> > @marcin: since you're a bootstrap-vz expert here,
> 
> haha that's a bold statement, I'd say Tiago and Anders are :-)
> 
> > do you know if a
> > debian image created with boostrap-vz exhibits the same behaviour ?
> > ie when adding rpc package in a boostrap-vz manifest, is  the service
> > started on boot (systemctl status rpcbind)
> 
> No idea sorry

I had a quick look into init implementation and looks like it should start, but
...
Bootstrap-vz upstream version (from Anders github) is not working very well with
systemd (yet).

I just uploaded 0.9.10+20160920git to the archive, but for now it's in new (due
to introducing new doc binary) and in it we have patches for Stretch which
should cope nicely with systemd. It also will enable all installed services by
default, unless specified otherwise. I was testing it with EC2 provider but
this part is shared so should work with other providers as well.

I didn't made Stretch support public yet nor created PR for it in github
as I'm still testing it a bit (even if all seams to be working).
-- 

|_|0|_|                                                  |
|_|_|0|                  "Panta rei"                     |
|0|0|0|             -------- kuLa --------               |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3

[signature.asc (application/pgp-signature, inline)]

Message #37 received at 837992-submitter@bugs.debian.org (full text, mbox, reply):

From: Antonio Terceiro <terceiro@debian.org>

To: control@bugs.debian.org

Cc: 837992-submitter@bugs.debian.org

Subject: closing 837992

Date: Mon, 16 Jan 2017 14:33:16 -0200

close 837992 
thanks

boxes now use NFSv4 by default to solve this issue.

Message #42 received at 837992@bugs.debian.org (full text, mbox, reply):

From: Foxx Neo <larrylively@icloud.com>

To: "837992@bugs.debian.org" <837992@bugs.debian.org>

Subject: Bugs in my jailbreak

Date: Mon, 16 Jan 2017 16:48:58 -0500

Sent from my iPhone

Send a report that this bug log contains spam.