ZFS write performance

[dezza]

I just did a test

gocryptfs v2.3.2 without_openssl; go-fuse v2.3.0; 2023-04-29 go1.20.3 linux/amd64

dd if=/dev/zero of=test bs=128k
^C248108+0 records in
248108+0 records out
32520011776 bytes (33 GB, 30 GiB) copied, 147.568 s, 220 MB/s

fio --name TEST --eta-newline=5s --filename=fio-write.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs
=1 --runtime=60 --group_reporting
WRITE: bw=175MiB/s (184MB/s), 175MiB/s-175MiB/s (184MB/s-184MB/s), io=10.0GiB (10.7GB), run=58451-58451msec
WRITE: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=10.0GiB (10.7GB), run=50653-50653msec
WRITE: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=10.0GiB (10.7GB), run=50464-50464msec
WRITE: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=10.0GiB (10.7GB), run=56287-56287msec
WRITE: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=10.0GiB (10.7GB), run=58918-58918msec
WRITE: bw=191MiB/s (200MB/s), 191MiB/s-191MiB/s (200MB/s-200MB/s), io=10.0GiB (10.7GB), run=53659-53659msec
WRITE: bw=301MiB/s (316MB/s), 301MiB/s-301MiB/s (316MB/s-316MB/s), io=10.0GiB (10.7GB), run=33971-33971msec
WRITE: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=10.0GiB (10.7GB), run=49961-49961msec
WRITE: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=10.0GiB (10.7GB), run=52801-52801msec
WRITE: bw=204MiB/s (213MB/s), 204MiB/s-204MiB/s (213MB/s-213MB/s), io=10.0GiB (10.7GB), run=50294-50294msec

sudo zfs create -o canmount=off -o mountpoint=/srv/crypt -o compression=off -o atime=off -o relatime=off -o xattr=off mpool/CRYPT
sudo zfs create mpool/CRYPT/gocryptfs

Tried with and without -noprealloc

/usr/local/bin/gocryptfs -config /mnt/crypt/gocryptfs/test/gocryptfs.conf -passfile /mnt/crypt/gocryptfs/test/gocryptfs.pass /srv/crypt/gocryptfs/ /mnt/gocryptfs -o -acl -allow_other -noexec

zfs get all mpool/CRYPT/gocryptfs
NAME                   PROPERTY              VALUE                  SOURCE
mpool/CRYPT/gocryptfs  type                  filesystem             -
mpool/CRYPT/gocryptfs  creation              Sat Jun  3 17:41 2023  -
mpool/CRYPT/gocryptfs  used                  32.6G                  -
mpool/CRYPT/gocryptfs  available             16.0T                  -
mpool/CRYPT/gocryptfs  referenced            32.6G                  -
mpool/CRYPT/gocryptfs  compressratio         1.00x                  -
mpool/CRYPT/gocryptfs  mounted               yes                    -
mpool/CRYPT/gocryptfs  quota                 none                   default
mpool/CRYPT/gocryptfs  reservation           none                   default
mpool/CRYPT/gocryptfs  recordsize            128K                   default
mpool/CRYPT/gocryptfs  mountpoint            /srv/crypt/gocryptfs   inherited from mpool/CRYPT
mpool/CRYPT/gocryptfs  sharenfs              off                    default
mpool/CRYPT/gocryptfs  checksum              on                     default
mpool/CRYPT/gocryptfs  compression           off                    inherited from mpool/CRYPT
mpool/CRYPT/gocryptfs  atime                 off                    inherited from mpool/CRYPT
mpool/CRYPT/gocryptfs  devices               on                     default
mpool/CRYPT/gocryptfs  exec                  on                     default
mpool/CRYPT/gocryptfs  setuid                on                     default
mpool/CRYPT/gocryptfs  readonly              off                    default
mpool/CRYPT/gocryptfs  zoned                 off                    default
mpool/CRYPT/gocryptfs  snapdir               hidden                 default
mpool/CRYPT/gocryptfs  aclmode               discard                default
mpool/CRYPT/gocryptfs  aclinherit            restricted             default
mpool/CRYPT/gocryptfs  createtxg             21512                  -
mpool/CRYPT/gocryptfs  canmount              on                     default
mpool/CRYPT/gocryptfs  xattr                 off                    inherited from mpool/CRYPT
mpool/CRYPT/gocryptfs  copies                1                      default
mpool/CRYPT/gocryptfs  version               5                      -
mpool/CRYPT/gocryptfs  utf8only              on                     -
mpool/CRYPT/gocryptfs  normalization         formD                  -
mpool/CRYPT/gocryptfs  casesensitivity       sensitive              -
mpool/CRYPT/gocryptfs  vscan                 off                    default
mpool/CRYPT/gocryptfs  nbmand                off                    default
mpool/CRYPT/gocryptfs  sharesmb              off                    default
mpool/CRYPT/gocryptfs  refquota              none                   default
mpool/CRYPT/gocryptfs  refreservation        none                   default
mpool/CRYPT/gocryptfs  guid                  12935340181850618411   -
mpool/CRYPT/gocryptfs  primarycache          all                    default
mpool/CRYPT/gocryptfs  secondarycache        all                    default
mpool/CRYPT/gocryptfs  usedbysnapshots       0B                     -
mpool/CRYPT/gocryptfs  usedbydataset         32.6G                  -
mpool/CRYPT/gocryptfs  usedbychildren        0B                     -
mpool/CRYPT/gocryptfs  usedbyrefreservation  0B                     -
mpool/CRYPT/gocryptfs  logbias               latency                default
mpool/CRYPT/gocryptfs  objsetid              330                    -
mpool/CRYPT/gocryptfs  dedup                 off                    default
mpool/CRYPT/gocryptfs  mlslabel              none                   default
mpool/CRYPT/gocryptfs  sync                  standard               default
mpool/CRYPT/gocryptfs  dnodesize             auto                   inherited from mpool
mpool/CRYPT/gocryptfs  refcompressratio      1.00x                  -
mpool/CRYPT/gocryptfs  written               32.6G                  -
mpool/CRYPT/gocryptfs  logicalused           32.6G                  -
mpool/CRYPT/gocryptfs  logicalreferenced     32.6G                  -
mpool/CRYPT/gocryptfs  volmode               default                default
mpool/CRYPT/gocryptfs  filesystem_limit      none                   default
mpool/CRYPT/gocryptfs  snapshot_limit        none                   default
mpool/CRYPT/gocryptfs  filesystem_count      none                   default
mpool/CRYPT/gocryptfs  snapshot_count        none                   default
mpool/CRYPT/gocryptfs  snapdev               hidden                 default
mpool/CRYPT/gocryptfs  acltype               posix                  inherited from mpool
mpool/CRYPT/gocryptfs  context               none                   default
mpool/CRYPT/gocryptfs  fscontext             none                   default
mpool/CRYPT/gocryptfs  defcontext            none                   default
mpool/CRYPT/gocryptfs  rootcontext           none                   default
mpool/CRYPT/gocryptfs  relatime              off                    inherited from mpool/CRYPT
mpool/CRYPT/gocryptfs  redundant_metadata    all                    default
mpool/CRYPT/gocryptfs  overlay               on                     default
mpool/CRYPT/gocryptfs  encryption            off                    default
mpool/CRYPT/gocryptfs  keylocation           none                   default
mpool/CRYPT/gocryptfs  keyformat             none                   default
mpool/CRYPT/gocryptfs  pbkdf2iters           0                      default
mpool/CRYPT/gocryptfs  special_small_blocks  0                      default

Got an idea what might be wrong?

Its a mirror of 2x 7200RPM 512/4K X20 EXOS drives. A compressed lz4 dataset (without gocryptfs) does about 700-800MB/s

WRITE: bw=736MiB/s (772MB/s), 736MiB/s-736MiB/s (772MB/s-772MB/s), io=10.0GiB (10.7GB), run=13906-13906msec

[rfjakob]

Your disks should max out at 265 MB/s ( https://www.hardwareluxx.de/index.php/artikel/hardware/storage/57754-seagate-exos-x20-mit-20-tb-im-test-mehr-kapazitaet-durch-mehr-platter.html?start=4 ).

And you get 220 MB/s with gocryptfs. Seems pretty good, no?

[dezza]

You're right it looks fine.

Would there be a way to take usage of the ARC on ZFS with gocryptfs? virtiofsd does it, which is also a FUSE filesystem, but I understand gocryptfs might be special in this case.

[rfjakob]

ARC is just read cache, right? Then gocryptfs uses it, yet. If the file is in ARC you should be only limited by your CPUs decryption speed.

[dezza]

ARC is just read cache, right?

Also write-cache. I believe this is why I see 700-800MB/s on 10GB total writes on plain host and overall initially magnitudes faster writes on all sorts of storage devices, but since it scores about 3500MB/s+ in benchmark I can't see why it shouldn't be able to use it to write faster, it seems like gocryptfs is stable at 200MB/s compared to this initially high write speed while ARC fills up.

Anyways, just something to look into :)