/
s3backer.1.in
1240 lines (1240 loc) · 43 KB
/
s3backer.1.in
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.\" -*- nroff -*-
.\"
.\" s3backer - FUSE-based single file backing store via Amazon S3
.\"
.\" Copyright 2008-2023 Archie L. Cobbs <archie.cobbs@gmail.com>
.\"
.\" This program is free software; you can redistribute it and/or
.\" modify it under the terms of the GNU General Public License
.\" as published by the Free Software Foundation; either version 2
.\" of the License, or (at your option) any later version.
.\"
.\" This program is distributed in the hope that it will be useful,
.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
.\" GNU General Public License for more details.
.\"
.\" You should have received a copy of the GNU General Public License
.\" along with this program; if not, write to the Free Software
.\" Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
.\" 02110-1301, USA.
.\"
.\" In addition, as a special exception, the copyright holders give
.\" permission to link the code of portions of this program with the
.\" OpenSSL library under certain conditions as described in each
.\" individual source file, and distribute linked combinations including
.\" the two.
.\"
.\" You must obey the GNU General Public License in all respects for all
.\" of the code used other than OpenSSL. If you modify file(s) with this
.\" exception, you may extend this exception to your version of the
.\" file(s), but you are not obligated to do so. If you do not wish to do
.\" so, delete this exception statement from your version. If you delete
.\" this exception statement from all source files in the program, then
.\" also delete it here.
.\"
.Dd September 7, 2009
.Dt S3BACKER 1
.Os
.Sh NAME
.Nm s3backer
.Nd FUSE-based single file backing store via Amazon S3
.Sh SYNOPSIS
.Nm s3backer
.Bk -words
.Op options
.Ar bucket[/subdir]
.Ar /mount/point
.Ek
.Pp
.Nm s3backer
.Bk -words
.Fl \-nbd
.Op options
.Ar bucket[/subdir]
.Ar /dev/nbdX
.Ek
.Pp
.Nm s3backer
.Bk -words
.Fl \-test
.Op options
.Ar dir
.Ar /mount/point
.Ek
.Pp
.Nm s3backer
.Bk -words
.Fl \-erase
.Op options
.Ar bucket[/subdir]
.Ek
.Pp
.Nm s3backer
.Bk -words
.Fl \-reset-mounted-flag
.Op options
.Ar bucket[/subdir]
.Ek
.Sh DESCRIPTION
.Nm
is a filesystem that contains a single file backed by the Amazon Simple Storage Service (Amazon S3).
As a filesystem, it is very simple: it provides a single normal file having a fixed size.
Underneath, the file is divided up into blocks, and the content of each block is stored in a unique Amazon S3 object.
In other words, what
.Nm
provides is really more like an S3-backed virtual hard disk device, rather than a filesystem.
.Pp
In typical usage, a `normal' filesystem is mounted on top of the file exported by the
.Nm
filesystem using a loopback mount (or disk image mount on Mac OS X).
.Pp
This arrangement has several benefits compared to more complete S3 filesystem implementations:
.Bl -tag -width xx
.It o
By not attempting to implement a complete filesystem, which is a complex undertaking and difficult to get right,
.Nm
can stay very lightweight and simple.
All of the experience and knowledge about how to properly implement filesystems that already exists can be reused.
.It o
By utilizing existing filesystems, you get full UNIX filesystem semantics.
Subtle bugs or missing functionality relating to hard links, extended attributes, POSIX locking, etc. are avoided.
.It o
The gap between normal filesystem semantics and Amazon S3 ``eventual consistency'' is more easily and simply solved
when one can interpret S3 objects as simple device blocks rather than filesystem objects (see below).
.It o
When storing your data on Amazon S3 servers, which are not under your control, the ability to encrypt and
authenticate data becomes a critical issue.
.Nm
supports secure encryption and authentication.
Alternately, the encryption capability built into the Linux loopback device can be used.
.It o
Since S3 data is accessed over the network, local caching is also very important for performance reasons.
Since
.Nm
presents the equivalent of a virtual hard disk to the kernel, most of the filesystem caching can be done
where it should be: in the kernel, via the kernel's page cache.
However
.Nm
also includes its own internal block cache for increased performance, using asynchronous worker threads
to take advantage of the parallelism inherent in the network.
.El
.Ss Consistency Guarantees
Previously, Amazon S3 made relatively weak guarantees relating to the timing and consistency of reads vs. writes
(collectively known as ``eventual consistency'').
.Nm
includes logic and configuration parameters to work around these limitations, allowing the user to
guarantee consistency to whatever level desired, up to and including 100% detection and avoidance
of incorrect data.
These are:
.Bl -tag -width xx
.It 1.
.Nm
enforces a minimum delay between consecutive PUT or DELETE operations on the same block.
This ensures that Amazon S3 doesn't apply these operations out of order.
.It 2.
.Nm
maintains an internal block MD5 checksum cache, which enables automatic detection and rejection of `stale' blocks returned by GET operations.
.El
.Pp
This logic is configured by the following command line options:
.Fl \-md5CacheSize ,
.Fl \-md5CacheTime ,
and
.Fl \-minWriteDelay .
.Pp
In December of 2020, Amazon announced that S3 is now fully consistent.
Therefore, the MD5 cache is disabled by default in
.Nm
versions after 1.6.3.
Other S3-compatible services may still require it however.
.Pp
To guarantee consistency,
.Fl \-md5CacheTime
and
.Fl \-minWriteDelay
should be greater than the maximum S3 convergence time.
.Ss Zeroed Block Optimization
As a simple optimization,
.Nm
does not store blocks containing all zeroes; instead, they are simply deleted.
Conversely, reads of non-existent blocks will contain all zeroes.
In other words, the backed file is always maximally sparse.
.Pp
As a result, blocks do not need to be created before being used and no special initialization is necessary when creating a new filesystem.
.Pp
.Nm
also uses a simple bitmap to track blocks that are known to be zero on an ongoing basis.
When the
.Fl \-listBlocks
flag is given (recommended),
.Nm
will list all existing blocks at startup to initialize this bitmap.
This especially affects performance with operations that read or write a large number of zero blocks,
such as filesystem initialization,
.Xr fstrim 8 ,
bulk copies, etc.
.Ss File and Block Size Auto-Detection
As a convenience, whenever the first block of the backed file is written,
.Nm
includes as meta-data (in the ``x-amz-meta-s3backer-filesize'' header) the total size of the file.
Along with the size of the block itself, this value can be checked and/or auto-detected later when
the filesystem is remounted, eliminating the need for the
.Fl \-blockSize
or
.Fl \-size
flags to be explicitly provided and avoiding accidental mis-interpretation of an existing filesystem.
.Ss Block Cache
.Nm
includes support for an internal block cache to increase performance.
The block cache cache is completely separate from the MD5 cache which only stores MD5 checksums transiently and whose sole purpose is to
mitigate ``eventual consistency''.
The block cache is a traditional cache containing cached data blocks.
When full, clean blocks are evicted as necessary in LRU order.
.Pp
Reads of cached blocks will return immediately with no network traffic.
Writes to the cache also return immediately and trigger an asynchronous write operation to the network via a separate worker thread.
Because the kernel typically writes blocks through FUSE filesystems one at a time, performing writes asynchronously allows
.Nm
to take advantage of the parallelism inherent in the network, vastly improving write performance.
.Pp
When the cache is full, least recently accessed blocks are evicted first
(but see also the
.Fl \-blockCacheNumProtected
flag).
.Pp
The block cache can be configured to store the cached data in a local file instead of in memory.
This permits larger cache sizes and allows
.Nm
to reload cached data after a restart.
Reloaded data is verified via MD5 checksum with Amazon S3 before reuse.
.Pp
Having said all that, Linux users may want to consider instead using the kernel "bcache" mechanism for local caching of blocks.
.Pp
The block cache is configured by the following command line options:
.Fl \-blockCacheFile ,
.Fl \-blockCacheMaxDirty ,
.Fl \-blockCacheNoVerify ,
.Fl \-blockCacheNumProtected ,
.Fl \-blockCacheSize ,
.Fl \-blockCacheSync ,
.Fl \-blockCacheThreads ,
.Fl \-blockCacheTimeout ,
.Fl \-blockCacheWriteDelay ,
and
.Fl \-blockCacheRecoverDirtyBlocks .
.Ss Read Ahead
.Nm
implements a simple read-ahead algorithm in the block cache.
When a configurable number of blocks are read in order, block cache worker threads are awoken to begin reading subsequent blocks into the block cache.
Read ahead continues as long as the kernel continues reading blocks sequentially.
The kernel typically requests blocks one at a time, so having multiple worker threads already reading the next few blocks
improves read performance by taking advantage of the parallelism inherent in the network.
.Pp
Note that the kernel implements a read ahead algorithm as well; its behavior should be taken into consideration.
By default,
.Nm
passes the
.Fl o Ar max_readahead=0
option to FUSE.
.Pp
Read ahead is configured by the
.Fl \-readAhead
and
.Fl \-readAheadTrigger
command line options.
.Ss Encryption and Authentication
.Nm
supports encryption via the
.Fl \-encrypt ,
.Fl \-password ,
and
.Fl \-passwordFile
flags.
When encryption is enabled, SHA1 HMAC authentication is also automatically enabled, and
.Nm
rejects any blocks that are not properly encrypted and signed.
.Pp
Encrypting at the
.Nm
layer is preferable to encrypting at an upper layer (e.g., at the loopback device layer), because if
the data
.Nm
sees is already encrypted it can't optimize away zeroed blocks or do meaningful compression.
.Ss Compression
.Nm
supports block-level compression, which minimizes transfer time and storage costs.
.Pp
Compression is configured via the
.Fl \-compress
and
.Fl \-compress-level
flags.
Compression is automatically enabled when encryption is enabled.
.Ss Server Side Encryption
.Nm
supports server side encryption via the
.Fl \-sse
flag.
.Ss Read-Only Access
An Amazon S3 account is not required in order to use
.Nm .
The filesystem must already exist and have S3 objects with ACL's configured for public read access
(see
.Fl \-accessType
below);
users should perform the looback mount with the read-only flag (see
.Xr mount 8 )
and provide the
.Fl \-readOnly
flag to
.Nm .
This mode of operation facilitates the creation of public, read-only filesystems.
.Ss Simultaneous Mounts
Although it functions over the network, the
.Nm
filesystem is not a distributed filesystem and does not support simultaneous read/write mounts.
(This is not something you would normally do with a hard-disk partition either.)
As a safety measure,
.Nm
attempts to detect this situation using an 'already mounted' flag
in the data store, and will fail to start if it does.
.Pp
This detection may produce a false positive if a former
.Nm
process was not shutdown cleanly; if so, the
.Fl \-reset-mounted-flag
flag can be used to reset the 'already mounted' flag.
But see also BUGS below.
.Ss Statistics File
.Nm
populates the filesystem with a human-readable statistics file.
Use
.Fl \-statsFilename
to change the name of this file (default `stats').
The statistics can be reset to zero by attempting to remove the file.
.Ss NBD Plugin
On platforms with
.Xr ndbkit 1 ,
.Nm
is also installed as an
.Xr nbdkit 1
plugin.
.Pp
When using the NBD plugin, command line flags like
.Fl \-blockSize=SIZE
should instead be specified as plugin parameters like
.Ar blockSize=SIZE ;
for boolean flags, specify a value of
.Ar true .
The bucket name (with optional subdirectory) must be specified with the
.Ar bucket
parameter.
.Pp
To avoid conflicts with other
.Xr nbdkit 1
flags, any
.Nm
flag can be specified with a
.Ar s3b_
prefix, for example
.Ar s3b_force=true .
.Pp
As an alternative to launching
.Xr nbdkit 1
directly, the
.Fl \-nbd
flag causes
.Nm
to function as a wrapper process for managing an NBD session; see below.
.Ss Logging
In normal operation
.Nm
will log via
.Xr syslog 3 .
When run with the
.Fl d
or
.Fl f
flags,
.Nm
will log to standard error.
.Sh OPTIONS
Each command line flag has two forms, for example
.Fl \-accessFile=FILE
and
.Fl o Ar accessFile=FILE .
Only the first form is shown below.
Either form many be used; both are equivalent.
The second form allows mount options to be specified directly in
.Pa /etc/fstab
and passed seamlessly to
.Nm
by FUSE.
.Bl -tag -width Ds
.It Fl \-accessFile=FILE
Specify a file containing `accessID:accessKey' pairs, one per-line.
Blank lines and lines beginning with a `#' are ignored.
If no
.Fl \-accessKey
or
.Fl \-accessKeyEnv
is specified, this file will be searched for the entry matching the access ID specified via
.Fl \-accessId;
if no
.Fl \-accessId
is specified, the first entry in this file will be used.
Default value is
.Pa $HOME/.s3backer_passwd .
.It Fl \-accessId=ID
Specify Amazon S3 access ID.
Specify an empty string to force no access ID.
If no access ID is specified (and none is found in the access file) then
.Nm
will still function, but only reads of publicly available filesystems will work.
.It Fl \-accessKey=KEY
Specify Amazon S3 access key.
.Pp
To avoid publicizing this secret via the command line, use
.Fl \-accessKeyEnv
or
.Fl \-accessFile
instead of this flag.
.It Fl \-accessKeyEnv=VARNAME
Specify an environment variable containing the Amazon S3 access key.
.It Fl \-accessType=TYPE
Specify the Amazon S3 access privilege ACL type for newly written blocks.
The value must be one of `private', `public-read', `public-read-write', or `authenticated-read'.
Default is `private'.
.It Fl \-accessEC2IAM=ROLE
Download access credentials and security token in JSON document form from
.Bk -words
.Ar http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE
.Ek
every five minutes.
.Pp
This option allows S3 credentials to be provided automatically via the specified IAM role to
.Nm
when running on an Amazon EC2 instance.
.It Fl \-authVersion=TYPE
Specify how to authenticate requests. There are two supported authentication methods:
.Ar aws2
is the original AWS authentication scheme.
.Ar aws4
is the newer, recommended authentication scheme.
.Pp
.Ar aws4
is the default setting starting in version 1.4, and is required for certain non-US regions, while
.Ar aws2
may still be required by some non-Amazon S3 providers.
.It Fl \-baseURL=URL
Specify the base URL, which must end in a forward slash. Default is `http://s3.amazonaws.com/'.
.Pp
Use this flag to specify FIPS and dual-stack endpoints.
.Pp
Note: the region name is used in authentication, so if you include a region name you probably also need to specify it via
.Fl \-region .
.It Fl \-blockCacheFile=FILE
Specify a file in which to store cached data blocks.
Without this flag, the block cache lives entirely in process memory and the cached data disappears when
.Nm
is stopped.
The file will be created if it doesn't exist.
.Pp
Cache files that have been created by previous invocations of
.Nm
are reusable as long as they were created with the same configured block size (if not, startup will fail).
This is true even if
.Nm
was stopped abruptly, e.g., due to a system crash;
however, this guarantee rests on the assumption that the filesystem containing the cache file will not
reorder writes across calls to
.Xr fsync 2 .
.Pp
If an existing cache is used but was created with a different size,
.Nm
will automatically expand or shrink the file at startup.
When shrinking, blocks that don't fit in the new, smaller cache are discarded.
This process also compacts the cache file to the extent possible.
.Pp
By default, only clean cache blocks are recoverable after a restart.
This means a system crash will cause dirty blocks in the cache to be lost (of course, that is the case
with an in-memory cache as well).
.Pp
With the newer cache file format introduced in release 1.5.0, you can recover these dirty blocks by specifying the
.Fl \-blockCacheRecoverDirtyBlocks
option.
This will cause any dirty blocks in the cache file to be made writable again on startup.
If your cache file was created with a prior release of
.Nm
or you do not specify this option, dirty blocks in the cache file are discarded on startup.
The window of this data loss can be limited by
.Fl \-blockCacheWriteDelay .
.Pp
By default, when having reloaded the cache from a cache file,
.Nm
will verify the MD5 checksum of each reloaded block with Amazon S3 before its first use.
This verify operation does not require actually reading the block's data, and therefore is relatively quick.
This guards against the cached data having unknowingly gotten out of sync since the cache file was last used,
a situation that is otherwise impossible for
.Nm
to detect.
.It Fl \-blockCacheMaxDirty=NUM
Specify a limit on the number of dirty blocks in the block cache.
When this limit is reached, subsequent write attempts will block until an existing dirty block
is successfully written (and therefore becomes no longer dirty).
This flag limits the amount of inconsistency there can be with respect to the underlying S3 data store.
.Pp
The default value is zero, which means no limit.
.It Fl \-blockCacheNoVerify
Disable the MD5 verification of blocks loaded from a cache file specified via
.Fl \-blockCacheFile .
Using this flag is dangerous;
use only when you are sure the cached file is uncorrupted and the data it contains is up to date.
.It Fl \-blockCacheSize=SIZE
Specify the block cache size (in number of blocks).
Each entry in the cache will consume approximately block size plus 20 bytes.
A value of zero disables the block cache.
Default value is 1000.
.It Fl \-blockCacheThreads=NUM
Set the size of the thread pool associated with the block cache (if enabled).
This bounds the number of simultaneous writes that can occur to the network.
Default value is 20.
.It Fl \-blockCacheTimeout=MILLIS
Specify the maximum time a clean entry can remain in the block cache before it will be forcibly evicted and its associated memory freed.
A value of zero means there is no timeout; in this case, the number of entries in the block cache will never decrease, eventually reaching
the maximum size configured by
.Fl \-blockCacheSize
and staying there.
Configure a non-zero value if the memory usage of the block cache is a concern.
Default value is zero (no timeout).
.It Fl \-blockCacheWriteDelay=MILLIS
Specify the maximum time a dirty block can remain in the block cache before it must be written out to the network.
Blocks may be written sooner when there is cache pressure.
A value of zero configures a ``write-through'' policy; greater values configure a ``write-back'' policy.
Larger values increase performance when a small number of blocks are accessed repeatedly, at the cost of
greater inconsistency with the underlying S3 data store.
Default value is 250 milliseconds.
.It Fl \-blockCacheSync
Forces synchronous writes in the block cache layer.
Instead of returning immediately and scheduling the actual write to operation happen later,
write requests will not return until the write has completed.
This flag is a stricter requirement than
.Fl \-blockCacheWriteDelay=0 ,
which merely causes the writes to be initiated as soon as possible (but still after the write request returns).
.Pp
This flag requires
.Fl \-blockCacheWriteDelay
to be zero.
Using this flag is likely to drastically reduce write performance.
.It Fl \-blockCacheRecoverDirtyBlocks
An unclean dismount may leave dirty blocks (blocks written to the local cache file, but not yet flushed to S3) in the cache file.
.Pp
If this option is set,
.Nm
will recover any such dirty blocks and eventually write them back to S3.
If this option is not specified, all dirty data in the cache file are discarded on startup.
.Pp
If the filesystem has been mounted since the cache file was last used,
.Nm
will refuse to mount.
This is verified by checking a unique 32-bit mount token in the cache file against the 'already mounted' flag in the data store.
.Pp
This flag requires
.Fl \-blockCacheFile
to be set.
.It Fl \-blockCacheNumProtected=NUM
Preferentially retain the first
.Ar NUM
blocks in the block cache.
.Pp
Some upper filesystems store highly active data (e.g., write journal) at the beginning of the filesystem.
This option can be used to improve performance by reducing network reads for these regions of the file.
With this option enabled, blocks after the first
.Ar NUM
blocks will be evicted before any protected blocks.
.It Fl \-blockCacheFileAdvise
Immediately after being read or written by the kernel, data in the block cache file can end up being cached twice by the kernel:
once in the cache for the block cache file, and again in the cache for the
.Nm
file.
.Pp
When this flag is given,
.Xr posix_fadvise 2
is used whenever data is read or written to/from the block cache file to tell the kernel not to cache that data
on behalf of the block cache file.
.Pp
This flag is ignored if
.Fl \-blockCacheFile
is not specified.
.It Fl \-blockHashPrefix
Prepend random prefixes (generated deterministically from the block number) to block object names.
This spreads requests more evenly across the namespace, and prevents heavy access to a narrow range of blocks from all being directed to the same backend server.
.Pp
As with
.Fl \-prefix
or
.Ar subdir ,
this flag must be used consistently once a disk image is established.
.It Fl \-blockSize=SIZE
Specify the block size.
This must be a power of two and should be a multiple of the kernel's native page size.
The size may have an optional suffix 'K' for kilobytes, 'M' for megabytes, etc.
The total number of blocks must fit into 32 bits.
.Pp
.Nm
supports partial block operations, though this forces a read before each write;
use of the block cache and proper alignment of the
.Nm
block size with the intended use (e.g., the block size of the `upper' filesystem) will help minimize the extra reads.
Note that even when filesystems are configured for large block sizes, the kernel will often still write page-sized blocks.
.Pp
.Nm
will attempt to auto-detect the block size by reading block number zero at startup.
If this option is not specified, the auto-detected value will be used.
If this option is specified but disagrees with the auto-detected value,
.Nm
will exit with an error unless
.Fl \-force
is also given.
If auto-detection fails because block number zero does not exist, and this option is not specified,
then the default value of 4K (4096) is used.
.It Fl \-cacert=FILE
Specify SSL certificate file to be used when verifying the remote server's identity when operating over SSL connections.
Equivalent to the
.Fl \-cacert
flag documented in
.Xr curl 1 .
.It Fl \-compress[=ALGORITHM]
Compress blocks using the specified compression algorithm before sending them over the network.
This should result in less network traffic (in both directions) and lower storage costs.
.Pp
The
.Ar ALGORITHM
is optional; currently, the default value is
.Ar deflate .
Also supported is
.Ar zstd .
.Pp
This flag only enables compression of newly written blocks; decompression is always enabled and applied when appropriate.
Therefore, it is safe to switch this flag on or off between different invocations of
.Nm
on the same filesystem.
.Pp
This flag is implied when
.Fl \-encrypt
is used.
.Pp
When using an encrypted upper layer filesystem, this flag adds no value because the data will not be compressible.
.It Fl \-compress-level=LEVEL
Configure the compression level for the selected compression algorithm, which must be specified via
.Fl \-compress .
.Pp
The interpretation of
.Ar LEVEL
depends on the algorithm;
for
.Ar deflate ,
it must be an integer between 1 (maximum speed) and 9 (maximum compression), inclusive.
.Pp
If this flag is omitted, the default compression level for the selected algorithm is used.
.It Fl \-configFile=FILE
Insert command line flags and arguments read from the specified file in place of this flag.
.Pp
Leading and trailing whitespace is trimmed from each line, and blank lines and lines starting with
.Ar #
are ignored.
Each line contains exactly one command line argument (including internal whitespace, if any).
.Pp
Nested
.Fl \-configFile
flags are allowed.
.Pp
This option may also be specified as
.Fl F Ar FILE .
.It Fl \-directIO
Disable kernel caching of the backed file.
This will force the kernel to always pass reads and writes directly to
.Nm .
This may reduce performance but also eliminates one source of inconsistency.
.It Fl \-debug
Enable logging of debug messages.
This includes logging HTTP error response payloads.
.Pp
Note that this flag is different from
.Fl d ,
which is a flag to FUSE;
however, the
.Fl d
FUSE flag implies this flag.
.It Fl \-debug-http
Configure
.Xr libcurl 3
to print debugging information (e.g., HTTP request and response headers) to standard error.
.Pp
With this flag you would normally also specify
.Fl f
so
.Nm
runs in the foreground and standard error is visible.
.It Fl \-defaultContentEncoding=VALUE
Use this to workaround S3 backends that fail to send back the
.Pa "Content-Encoding"
header that was sent to them by
.Nm .
If a block read response contains no
.Pa "Content-Encoding"
header, this value will be substituted.
.Pp
If you get errors complaining that the content was expected to be encrypted, try setting this to
.Pa deflate,encrypt-AES-128-CBC .
.It Fl \-encrypt[=CIPHER]
Enable encryption and authentication of block data.
See your OpenSSL documentation for a list of supported ciphers;
the default if no cipher is specified is AES-128 CBC.
.Pp
Currently, only block ciphers are supported.
.Pp
The encryption password may be supplied via one of
.Fl \-password
or
.Fl \-passwordFile .
If neither flag is given,
.Nm
will ask for the password at startup.
.Pp
Note: the actual key used is derived by hashing the password, the bucket name, the prefix name (if any), and the block number.
Therefore, encrypted data cannot be ported to different buckets or prefixes.
.Pp
This flag implies
.Fl \-compress .
.It Fl \-erase
Completely erase the file system by deleting all non-zero blocks, clear the 'already mounted' flag, and then exit.
User confirmation is required unless the
.Fl \-force
flag is also given.
Note, no simultaneous mount detection is performed in this case.
.Pp
This operation bypasses the caching layers, so any leftover cache file must be manually deleted.
.It Fl \-filename=NAME
Specify the name of the backed file that appears in the
.Nm
filesystem.
Default is `file'.
.It Fl \-fileMode=MODE
Specify the UNIX permission bits for the backed file that appears in the
.Nm
filesystem.
Default is 0600, unless
.Fl \-readOnly
is specified, in which case the default is 0400.
.It Fl \-force
Proceed even if the value specified by
.Fl \-blockSize
or
.Fl \-size
disagrees with the auto-detected value, or
.Nm
detects that another
.Nm
instance is still mounted on top of the same S3 bucket (and prefix).
In any of these cases, proceeding will lead to corrupted data, so the
.Fl \-force
flag should be avoided for normal use.
.Pp
The simultaneous mount detection can produce a false positive when a previous
.Nm
instance was not shut down cleanly.
In this case, don't use
.Fl \-force
but rather run
.Nm
once with the
.Fl \-reset-mounted-flag
flag.
.Pp
If
.Fl \-erase
is given,
.Fl \-force
causes
.Nm
to proceed without user confirmation.
.It Fl h Fl \-help
Print a help message and exit.
.It Fl \-http11
Restrict to HTTP version 1.1.
There have been reports of errors and/or reduced performance with some S3-compatible backends when HTTP/2 is used.
This flag can be used to prevent HTTP/2 negotiation.
.It Fl \-initialRetryPause=MILLIS
Specify the initial pause time in milliseconds before the first retry attempt after failed HTTP operations.
Failures include network failures and timeouts, HTTP errors, and reads of stale data
(i.e., MD5 mismatch);
.Nm
will make multiple retry attempts using an exponential backoff algorithm, starting with this initial retry pause time.
Default value is 200ms.
See also
.Fl \-maxRetryPause .
.It Fl \-insecure
Do not verify the remote server's identity when operating over SSL connections.
Equivalent to the
.Fl \-insecure
flag documented in
.Xr curl 1 .
.It Fl \-keyLength
Override the length of the generated block encryption key.
.Pp
Versions of
.Nm
prior to 1.3.6 contained a bug where the length of the generated encryption key was fixed but system-dependent,
causing it to be possibly incompatible on different systems for some ciphers.
In version 1.3.6, this bug was corrected; however, in some cases this changed the generated key length,
making the encryption no longer compatible with previously written data.
This flag can be used to force the older, fixed key length.
The value you want to use is whatever is defined for
.Pa EVP_MAX_KEY_LENGTH
on your system, typically 64.
.Pp
It is an error to specify a value smaller than the cipher's natural key length;
however, a value of zero is allowed and is equivalent to not specifying anything.
.It Fl \-listBlocks
After starting, initiate a background query to determine which blocks already exist.
This enables optimizations whereby, for each block that does not yet exist, reads return zeroes and zeroed writes are omitted,
thereby eliminating any network access.
.Pp
This flag is useful when creating a new backed file, or any time it is expected that a large number of zeroed
blocks will be read or written, such as when initializing a new filesystem.
.Pp
In general, use of this flag is recommended, but it does create additional network traffic during startup in proportion to the number of blocks that already exist.
.It Fl \-listBlocksThreads=NUM
To minimize startup delay, the initial block enumeration of
.Fl \-listBlocks
is performed in parallel using multiple threads.
This flag configures the number of threads used.
.Pp
Default value is 16.
.It Fl \-maxUploadSpeed=BITSPERSEC
.It Fl \-maxDownloadSpeed=BITSPERSEC
These flags set a limit on the bandwidth utilized for individual block uploads and downloads (i.e.,
the setting applies on a per-thread basis).
The limits only apply to HTTP payload data and do not include any additional overhead from HTTP or TCP headers, etc.
.Pp
The value is measured in bits per second, and abbreviations like `256k', `1m', etc. may be used.
By default, there is no fixed limit.
.Pp
Use of these flags may also require setting the
.Fl \-timeout
flag to a higher value.
.It Fl \-maxRetryPause=MILLIS
Specify the total amount of time in milliseconds
.Nm
should pause when retrying failed HTTP operations before giving up.
Failures include network failures and timeouts, HTTP errors, and reads of stale data
(i.e., MD5 mismatch);
.Nm
will make multiple retry attempts using an exponential backoff algorithm, up to this maximum total retry pause time.
This value does not include the time it takes to perform the HTTP operations themselves (use
.Fl \-timeout
for that).
Default value is 30000 (30 seconds).
See also
.Fl \-initialRetryPause .
.It Fl \-minWriteDelay=MILLIS
Specify a minimum time in milliseconds between the successful completion of a write and the initiation
of another write to the same block. This delay ensures that S3 doesn't receive the writes out of order.
This value must be set to zero when
.Fl \-md5CacheSize
is set to zero (MD5 cache disabled).
Default value is 500ms.
.It Fl \-md5CacheSize=SIZE
Specify the size of the MD5 checksum cache (in number of blocks).
If the cache is full when a new block is written, the write will block until there is room.
Therefore, it is important to configure
.Fl \-md5CacheTime
and
.Fl \-md5CacheSize
according to the frequency of writes to the filesystem overall and to the same block repeatedly.
Alternately, a value equal to the number of blocks in the filesystem eliminates this problem but consumes
the most memory when full (each entry in the cache is approximately 40 bytes).
A value of zero disables the MD5 cache.
Default value is zero (disabled).
.It Fl \-md5CacheTime=MILLIS
Specify in milliseconds the time after a block has been successfully written for which the MD5 checksum
of the block's contents should be cached, for the purpose of detecting stale data during subsequent reads.
A value of zero means `infinite' and provides a guarantee against reading stale data; however,
you should only do this when
.Fl \-md5CacheSize
is configured to be equal to the number of blocks; otherwise deadlock will (eventually) occur.
This value must be at least as big as
.Fl \-minWriteDelay.
This value must be set to zero when
.Fl \-md5CacheSize
is set to zero (MD5 cache disabled).
Default value is 10 seconds.
.Pp
The MD5 checksum cache is not persisted across restarts.
Therefore, to ensure the same eventual consistency protection while
.Nm
is not running, you must delay at least
.Fl \-md5CacheTime
milliseconds between stopping and restarting
.Nm .
.It Fl \-nbd
Create a Network Block Device (NBD) instead of a FUSE filesystem.
This is probably a more logical and direct way to do things on systems that support NBD.
.Pp
.Nm
will start up appropriate instances of
.Xr nbdkit 1
and
.Xr nbd-client 1
in order to attach the S3 volume to the specified device.
.Pp
When invoked in this mode, only
.Fl \-doubleDash
style flags are supported, with the exception of
.Fl f
and
.Fl d ;
any other flags (including any FUSE-specific flags) are disallowed.
The
.Fl \-nbd
flag may not appear in a
.Fl \-configFile
file.
.Pp
To detach the S3 volume, just kill the
.Nm
process and it will automatically disconnect the NBD client and shut down the NBD server.
.It Fl \-nbd-flag=string
.It Fl \-nbd-param=string
Add
.Ar string
as an
.Xr nbdkit 1
command line parameter, either before or after the plugin name (respectively).
.It Fl \-noAutoDetect
Disable block and file size auto-detection at startup.
If this flag is given, then the block size defaults to 4096 and the
.Fl \-size
flag is required.
.It Fl \-noCurlCache
Disable caching of cURL handles.
This flag will reduce performance; it exists to help in certain debugging situations.
.It Fl \-password=PASSWORD
Supply the password for encryption and authentication as a command-line parameter.
.It Fl \-passwordFile=FILE
Read the password for encryption and authentication from (the first line of) the specified file.
.It Fl \-prefix=STRING
Specify a prefix to prepend to the resource names within
.Ar bucket
that identify each block.
By using different prefixes, multiple independent
.Nm
disks can live in the same S3 bucket.
.Pp
Instead of using this flag, it is recommended to specify a prefix implicitly by appending a slash and a subdirectory name to the bucket name (the S3 bucket namespace is flat, so the notion of a subdirectory is not real).
The configured prefix then becomes the given subdirectory name, followed by slash.
The AWS web console will then display the virtual disk's blocks grouped into a subfolder.
.Pp
For example, specifying bucket
.Ar mybucket/foo/bar
is equivalent to specifying bucket
.Ar mybucket
along with the flag
.Fl \-prefix=foo/bar/
(note the additional trailing slash).
.Pp
Extraneous slash characters in a subdirectory name are disallowed.
.Pp
This flag (or
.Ar subdir )
must be used consistently once a disk image is established.
.Pp
The default prefix is the empty string.
.It Fl \-quiet
Suppress progress output during initial startup.
.It Fl \-readAhead=NUM
Configure the number of blocks of read ahead.
This determines how many blocks will be read into the block cache ahead of the last block read by the kernel when read ahead is active.
This option has no effect if the block cache is disabled.
Default value is 4 in FUSE mode, zero in NBD mode.
.It Fl \-readAheadTrigger=NUM
Configure the number of blocks that must be read consecutively before the read ahead algorithm is triggered.
Once triggered, read ahead will continue as long as the kernel continues reading blocks sequentially.
This option has no effect if the block cache is disabled.
Default value is 2 in FUSE mode, zero in NBD mode.
.It Fl \-readOnly
Assume the filesystem is going to be mounted read-only, and return
.Er EROFS
in response to any attempt to write.
This flag also changes the default mode of the backed file from 0600 to 0400
and disables the MD5 checksum cache.
.It Fl \-region=REGION
Specify an AWS region.
This flag changes the default base URL to include the region name and automatically sets the
.Fl \-vhost
flag, unless the
.Fl \-no-vhost
flag is used.
.Pp
The default region is
.Pa us-east-1 .
.It Fl \-reset-mounted-flag
Reset the 'already mounted' flag on the underlying S3 data store.
.Pp
.Nm
detects simultaneous mounts by checking a special flag.
If a previous invocation of
.Nm
was not shut down cleanly, the flag may not have been cleared.
Running
.Nm
.Fl \-erase
will clear it manually.
But see also BUGS below.
.It Fl \-size=SIZE
Specify the size (in bytes) of the backed file to be exported by the filesystem.
The size may have an optional suffix 'K' for kilobytes, 'M' for megabytes, 'G' for gigabytes, 'T' for terabytes, 'E' for exabytes, 'Z' for zettabytes (if supported), or 'Y' for yottabytes (if supported).
The total number of blocks must fit into 32 bits.
Suffixes represent powers of two.