Debian Bug report logs - #837049
Please configure zfs_arc_max in postinst

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Lumin <cdluminate@gmail.com>

To: submit <submit@bugs.debian.org>

Subject: Please configure zfs_arc_max in postinst

Date: Thu, 8 Sep 2016 09:59:56 +0000

Package: zfsutils-linux
Version: 0.6.5.7-2
Severity: minor

Hi,

We never know on what machine will the user deploy ZFS, and
it may be expensive to run zfs under the default arc config.
For example, my laptop has 4G memory and nearly half of
it would be gone after zfs being loaded.

With reference to Archlinux/ZFS and Gentoo wiki/ZFS,
I suggest that a little enhancement should be made to postinst --
let the user input a zfs_arc_max value and then put it
into /etc/modprobe.d/zfs.conf. If user input nothing then
nothing will happen.

Maybe it should also be registered as a conffile.

-- 
Best,
Lumin

Message #10 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Craig Sanders <cas@taz.net.au>

To: 837049@bugs.debian.org

Subject: Re: [Pkg-zfsonlinux-devel] Bug#837049: Please configure zfs_arc_max in postinst

Date: Sun, 11 Sep 2016 12:52:04 +1000

On Thu, Sep 08, 2016 at 09:59:56AM +0000, Lumin wrote:
> I suggest that a little enhancement should be made to postinst
> -- let the user input a zfs_arc_max value and then put it into
> /etc/modprobe.d/zfs.conf. If user input nothing then nothing will
> happen.

good idea. at the least, we should include an example in the package.
e.g. I made the following file for my systems, which I copy to various
machines and modify to suit.

$ cat /etc/modprobe.d/zfs.conf 
# Min and Max units are in bytes. So for n GB, multiply n * 1073741824
# (i.e. 1024*1024*1024)

# use minimum 1GB and maximum of 4GB RAM for ZFS ARC
#options zfs zfs_arc_min=1073741824 zfs_arc_max=4294967296

# use minimum 1GB and maximum of 8GB RAM for ZFS ARC
#options zfs zfs_arc_min=1073741824 zfs_arc_max=8589934592

# use minimum 4GB and maximum of 8GB RAM for ZFS ARC
options zfs zfs_arc_min=4294967296 zfs_arc_max=8589934592

in a package, all of these would be commented out, and it would also be
worthwhile putting in well-commented examples of other commonly tweaked
zfs.ko module options.

All of which need better documentation anyway (e.g. what does option
'l2arc_feed_again' actually DO?  'Turbo L2ARC warmup' is all that
modinfo says, and googling for either of those strings doesn't come up
with anything more descriptive in the first few result pages) - most
options are best left at their default values, but I'm sure that there
are several tuning options I could/should use if only I knew what they
did.

> Maybe it should also be registered as a conffile.

definitely.

craig

--
craig sanders <cas@taz.net.au>

Message #15 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Petter Reinholdtsen <pere@hungry.com>

To: Craig Sanders <cas@taz.net.au>, 837049@bugs.debian.org

Subject: Re: [Pkg-zfsonlinux-devel] Bug#837049: Bug#837049: Please configure zfs_arc_max in postinst

Date: Sun, 11 Sep 2016 07:12:49 +0200

[Craig Sanders]
> good idea. at the least, we should include an example in the package.
> e.g. I made the following file for my systems, which I copy to various
> machines and modify to suit.

Make sense, as long as it isn't stored under /etc/ where any chnages to
the examples will trigger a conffile question for those enabling the
option.

>> Maybe it should also be registered as a conffile.
>
> definitely.

We do not want the postinst to modify a conffile.  It will just confuse
during upgrades those that have not seen the content of the conffile
before.

-- 
Happy hacking
Petter Reinholdtsen

Message #20 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Craig Sanders <cas@taz.net.au>

To: 837049@bugs.debian.org

Subject: Re: [Pkg-zfsonlinux-devel] Bug#837049: Bug#837049: Please configure zfs_arc_max in postinst

Date: Sun, 11 Sep 2016 16:58:00 +1000

On Sun, Sep 11, 2016 at 07:12:49AM +0200, Petter Reinholdtsen wrote:
> [Craig Sanders]
> > good idea. at the least, we should include an example in the
> > package. e.g. I made the following file for my systems, which I copy
> > to various machines and modify to suit.
>
> Make sense, as long as it isn't stored under /etc/ where any chnages
> to the examples will trigger a conffile question for those enabling
> the option.

that's how conffiles are supposed to work. they'll get a question only
when the package-supplied conffile is both:

1. different from their version of the cf (if any)
2. different from the .dpkg-dist version of it (if any)

i.e. they'll only be asked about it when the package-provided version is
first added or when it's actually changed, not on every package upgrade.

> >> Maybe it should also be registered as a conffile.
> >
> > definitely.
> 
> We do not want the postinst to modify a conffile.  

in general, postinsts shouldn't modify conffiles unless using debconf.
which also has the advantage of making it pre-seedable for automated
installs.

> It will just confuse during upgrades those that have not seen the
> content of the conffile before.

for those who don't have an /etc/modprobe.d/zfs.conf already, the
package version would just be silently installed (which is why it should
only have commented out lines or, at most, sane defaults)

for those who do have one, they'll get to choose whether to use their
version or the package version, as usual.

this is normal and expected behaviour in debian, so anyone annoyed by
it is going to be annoyed by every other package in debian that has
conffiles.


i don't care much whether it's just an example under /usr/share/doc or
if it's a real conffile in /etc/modprobe.d/ - the important thing from
my POV is providing a well-commented example.

craig

--
craig sanders <cas@taz.net.au>

Message #25 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Lumin <cdluminate@gmail.com>

To: 837049@bugs.debian.org

Cc: Aron Xu <aron@debian.org>, cas@taz.net.au, pere@hungry.com

Subject: A stub package waiting for comment

Date: Sun, 11 Sep 2016 08:11:19 +0000

[Message part 1 (text/plain, inline)]

Hello,

I've created an empty package which just ships config, postinst and postrm,
as attached. Feel free to take any part of it if found useful.
Note, debugging symbols are not stripped from these scripts.

Seriously my scripts don't regard zfs.conf as conffile, and the
following considerations are made:

1. if zfs.conf and zfs_arc_max already exists before config, parse it
and skip the prompt,
   and record its sha1sum as "" to prevent  it from being removed at postrm
2. if zfs.conf doesn't exist on configure, create one and record its sha1sum.
3. on postrm, check the sha1sum, remove zfs.conf if the hashes match.

But obviously with these scripts no comment will be wrote to zfs.conf.

First time writting maintainer scripts, please forgive me for stupid parts.
Comments?

-- 
Best,
Lumin

[zfsconf.tar.gz (application/x-gzip, attachment)]

Message #30 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Lumin <cdluminate@gmail.com>

To: 837049@bugs.debian.org

Cc: Aron Xu <aron@debian.org>

Subject: Preliminary patch

Date: Sun, 11 Sep 2016 08:53:17 +0000

[Message part 1 (text/plain, inline)]

Control: tag -1 +patch

attached.

-- 
Best,
Lumin

[configure_zfs_arc_max.patch (text/x-diff, attachment)]

Message #37 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Petter Reinholdtsen <pere@hungry.com>

To: Craig Sanders <cas@taz.net.au>, 837049@bugs.debian.org

Subject: Re: Bug#837049: Please configure zfs_arc_max in postinst

Date: Mon, 12 Sep 2016 11:03:34 +0200

[Craig Sanders]
> that's how conffiles are supposed to work. they'll get a question only
> when the package-supplied conffile is both:

I suspect you misunderstand me.  The conffile mechanism is designed to
work well with system administrators manually editing files.  Their
manual edits are not automatically replaced and a question is asked
during upgrades allowing the admin to figure out how to handle / merge
the changes into newer versions of the conffile.  In this case it is all
good.

But the conffile mechanism is not designed to work well for people how
do _not_ manually edit the files.  If something else modified a conffile
(say a postinst script or some configuration tool), the user is asked to
look at a file he never saw before and decide what to do with changes he
know nothing about in the file.  In short, as long as the person did not
modify the file himself, the conffile mechanism is far from ideal.  This
is the reason /etc/pam.d/common-* are not conffiles (they are modified
by /usr/sbin/pam-auth-update).

> in general, postinsts shouldn't modify conffiles unless using debconf.
> which also has the advantage of making it pre-seedable for automated
> installs.

The system administrator is just as confused by conffile questions if
the postinst used debconf as when the postinst just modified the
conffile directly.  In general, postinst should avoid modify conffiles,
and the file in question should not be a conffile if the postinst is
modifying it.  Instead one should modify the setup to use several
.d-like directories to make sure configuration can be appended or
replaced without confusing the system administrator with conffile
questions.

Conffile questions only make sense when the system administrator
actually modified the file in question using a text editor.

> for those who don't have an /etc/modprobe.d/zfs.conf already, the
> package version would just be silently installed (which is why it should
> only have commented out lines or, at most, sane defaults)
>
> for those who do have one, they'll get to choose whether to use their
> version or the package version, as usual.
>
> this is normal and expected behaviour in debian, so anyone annoyed by
> it is going to be annoyed by every other package in debian that has
> conffiles.

It is not a question about being annoyed, it is a question of ensuring
industrial rollout of upgrades can be done while avoiding unneeded
confusion among Debian system administrators.  I described a much better
setup at the end of my talk from Debian Conference 2005, recording
available from <URL: https://www.youtube.com/watch?v=zMT2M5BApZM >.

> i don't care much whether it's just an example under /usr/share/doc or
> if it's a real conffile in /etc/modprobe.d/ - the important thing from
> my POV is providing a well-commented example.

Well, it is important to understand the effect of difference.  Package
defaults and examples should go in /usr/, while host specific setup and
overrides for the defaults should go into /etc/.

-- 
Happy hacking
Petter Reinholdtsen

Message #42 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Petter Reinholdtsen <pere@hungry.com>

To: 837049@bugs.debian.org, Craig Sanders <cas@taz.net.au>, Lumin <cdluminate@gmail.com>

Subject: Re: Bug#837049: Please configure zfs_arc_max in postinst

Date: Sun, 12 Mar 2017 08:05:40 +0100

I had a closer look at this proposal, and given that this is a cross
distribution challenge, it seem better to me to change the default
behaviour of zfs, instead of creating a configuration file during
installation on the Linux distributions that remember to do so.

What about modifying the zfs source code and documentation to set the
zfs_arc_max default value based on the total amount of memory in the
machine?  This way fewer people would have to configure it, and those
that already did configure it will keep their override value.
-- 
Happy hacking
Petter Reinholdtsen

Message #47 received at 837049@bugs.debian.org (full text, mbox, reply):

From: Chris Zubrzycki <debian@mid-earth.net>

To: 837049@bugs.debian.org

Subject: arc max

Date: Wed, 19 Dec 2018 12:47:25 -0500

From what I’ve seen, this change should not really be needed anymore. ZFS is much better at release arc memory (disk cache) with 0.7.x and later. As soon as there’s memory pressure zfs will give up that ram. The only issues are when there’s a single large chunk requested like a VM, zfs can’t release the ram fast enough.

Message #52 received at 837049-close@bugs.debian.org (full text, mbox, reply):

From: Mo Zhou <lumin@debian.org>

To: 837049-close@bugs.debian.org

Subject: Re: Please configure zfs_arc_max in postinst

Date: Wed, 29 May 2019 00:57:17 -0700

this bug is no longer useful.

Send a report that this bug log contains spam.