You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Chunkservers can greatly optimise performance with many small files by aggregating small chunks into chunk-volumes. The idea is to implement a special storage backend in chunkserver code that stores small chunks in 64 MiB volume-files to reduce overall number of chunk-files. Implementation can leverage LevelDB, SQLite3 or a similar technology for a backend.
Chunkserver's performance suffer dramatically when chunkservers handle millions of files, which is a case when MooseFS is used to store Maildir structures, git repositories and CCTV data. SeaweedFS implementation stores all chunks in volume files (not a chunk-per-file like MooseFS) -- that achieves superior performance on great number of small files.
My original feature request #370 was moved to discussion for some reason so I'm filing feature request again, this time described more precisely. Thanks.
The text was updated successfully, but these errors were encountered:
Chunkservers can greatly optimise performance with many small files by aggregating small chunks into chunk-volumes. The idea is to implement a special storage backend in chunkserver code that stores small chunks in 64 MiB volume-files to reduce overall number of chunk-files. Implementation can leverage LevelDB, SQLite3 or a similar technology for a backend.
Chunkserver's performance suffer dramatically when chunkservers handle millions of files, which is a case when MooseFS is used to store Maildir structures, git repositories and CCTV data. SeaweedFS implementation stores all chunks in volume files (not a chunk-per-file like MooseFS) -- that achieves superior performance on great number of small files.
My original feature request #370 was moved to discussion for some reason so I'm filing feature request again, this time described more precisely. Thanks.
The text was updated successfully, but these errors were encountered: