ACLs on a jailed ZFS volume with FreeBSD

While setting up a couple of things on the new monotone server, which is bascially a FreeBSD jail on a physical server kindly provided by Lapo Luchini, I stumbled upon a quite basic use case: I needed to give one user group read and write access to files created and modified by another group.

On a Linux system I’d have simply enabled POSIX ACLs for the volume and call

$ setfacl -m d:user:foo:rwx /path/to/directory
$ setfacl -m d:user:bar:rwx /path/to/directory

and voila, all new files and directories underknees the path would have received the default ACLs with read and write rights for the `foo` and `bar` user.

It turns out that on FreeBSD, which gained zfs ACL support in its 8.1 release recently, the call is quite similar – just that `NFSv4` ACLs are used instead of `POSIX` ACLs, which are a bit more powerful (read more on the topic here):

$ setfacl -m user:foo:rwxp:fd:allow /path/to/directory
$ setfacl -m user:bar:rwxp:fd:allow /path/to/directory

Instead of `d:` for `default` which is used in the `POSIX` version, `fd` is used, which stands for `file_inherit` and `directory_inherit` and is exactly what we need. And while `POSIX` ACLs are add-only, you can either specify `:allow` or `:deny` to also explicitely deny access to a user or usergroup. Finally, you might have seen the little `p` which stands for the `append_data` right. This was needed for me in one case, so I added it here. It is actually one of many more rights beside the well-known `rwx` that `NFSv4` ACLs defines here – if you’re curious, just read more on the aforementioned link.

“So if everything works, what is the point of this blog?” you might ask. Well, unfortunately it did not all work out so nicely. One thing caused me some headaches which is actually dubbed a feature of FreeBSDs NFSv4 ACL implementation – namely that the `umask` (the create mask for new files) is taken into account on file creation, which leads to an unwanted recalculation of the default (inherited) ACLs. So if you have a file foo created by user `u1` and the ACLs should also give write access to user `u2`, then you might end up with something like this:

$ getfacl foo
   # file: foo
   # owner: u1
   # group: users
              user:u1:--x-----------:------:deny
              user:u1:rwxp----------:------:allow
              user:u2:rwxp----------:------:deny
              user:u2:rwxp----------:------:allow
               owner@:--x-----------:------:deny
               owner@:rw-p---A-W-Co-:------:allow
               group@:rwxp----------:------:deny
               group@:--------------:------:allow
            everyone@:rwxp---A-W-Co-:------:deny
            everyone@:------a-R-c--s:------:allow

The file was created with `rw- — —` (i.e. umask 077) which lead to a :deny rule for user `u2` and since deny rules take precendence over allow rules, `u2` gets no access to `foo`. I contacted the author of FreeBSD’s NFSv4 ACL implementation, Edward Tomasz Napierała, and thankfully he drawed a way out of the mess:

Your problem is umask. When you create a file, ACL entries are inherited,
and then the new ACL is modified according to some (pretty complicated)
rules. You probably want to disable that, so that entries are simply
inherited, without further messing and adding deny entries. To do this,
set aclmode=passthrough and aclinherit=passthrough ZFS properties.

The only downside to this approach is that you cannot change these settings inside the jail, but only from the outside. Luckily, the host part is also under our control, so Lapo remounted the jail volume with the new settings, and voilá, the ACLs now work as expected.

You can learn something new every day.