Moosefs as a general (slower) in-box storage.

[eleaner]

I have been using distributed moose at home for a while and I love it to bits. But I like less my electricity bill from all those servers.

I am thinking about building moose inside of bigger PC case that can hold multiple HDD and use it instead. (New Linux PC is planned anyway)
What will be the draw backs for such home setup?

1. If my PC goes down my wife cannot access the data
2. My wife gets the data from a single server instead of many chunkservers so she might get it slower
3. It probably will be slower than a raid or zfs, but it will tolerate various HDD sizes easily

Upsides?
Will in-box transfers from Moosefs to my PC increase while not really using the network? I am still mounting it.

[chogata]

Cons:

• one machine, so one power supply failure means you are offline until you buy a new power supply unit; with 2-3-4 machines if chunk server's power supply fails, you aren't really impacted, if master's power supply fails, you can quickly move master process to another machine (for example the one that doubles as metalogger); substitute cpu/ram/mainboard failure for power supply failure
• one machine, so parallel access to data will be slower; here all depends on what you do as home user, if you mainly use your storage for music and movies, you probably won't feel any difference, but if you use it to store big software (for example Steam games library of a devoted gamer ;) ), update times might suddenly become less friendly

Pros:

• lower electric bill - you said that yourself ;)
• less space needed - also important in home environment :)
• faster replications - everything between chunk servers and also master goes via loopback and not your home network
• as you said - easier to utilise various sizes and types of drives - I didn't try it, but I feel like having a SSD and a HDD in a RAID 1 is a bad idea, but MooseFS will handle this, no problem
• in case of failure of one drive your system will not wait for you to replace this drive to come back to full security level, it will immediately start copying data to other drives, so if you can go shopping for a new drive only 2 days later, your data will not be endangered for 2 days; with RAID you are just left with one (less) copy for those 2 days

One thing to remember about is to still have your metadata backup done somehow on another machine, but you could have a raspberry or similar for that purpose only, tucked somewhere in a corner of a shelf ;)

[eleaner]

thank you,
I wonder how moosefs in such scenario compares to alternatives - e.g. ceph
I am not sure if there are too many comparisons for this corner-case

[eleaner]

you also write
"one machine, so parallel access to data will be slower; here all depends on what you do as home user, if you mainly use your storage for music and movies, you probably won't feel any difference, but if you use it to store big software (for example Steam games library of a devoted gamer ;) ), update times might suddenly become less friendly"

but the hdd will be connected via SATA instead of 1gb ethernet
would it not be faster then?

[chogata]

If client is also on the same machine, then yes, it might even be faster - or not, depends on you bus speed, remember, there can be 10 very fast SATA drives in your computer, but they share one data bus.
If client is on the network, then all traffic goes through one network card to one machine instead of many cards to many machines.

But I still think in terms of home use with one client even if there are differences, they are not huge. I would still pay more attention to data safety and to potential problems a hardware issue may cause you vs gain on space and electricity, than to speed. Noteworthy speed differences would occur if more than one network client used the the setup.

[Blackpaw]

I've been running a single server setup for years, works the best with the eclectic range of disks I have - 13 disks, a mix of sata drives and usb external disks. Copies=2 for everything. 53TiB total storage.

Using 4 virtual chunkservers and a virtual master, with a metadata backup on a seperate server.

Chunkservers and master run as lxc containers on a Proxmox server.

[eleaner]

do I need multiple chunksevers or would moosefs be able to balance on HDD level?

@Blackpaw
why did you decide for proxmox and not docker (seems to be officially suported?)
or just native install on proxmox box and running mutiple chunkservers on different ports there?

[chogata]

@eleaner - you need multiple chunk servers (processes), a chunk server will not let you record two copies of a chunk, it would be deemed a duplicate even on another drive. One master + one chunk server = each chunk is kept only in one copy, no matter what storage classes say. One physical machine with 5 hdds for data + 5 chunk server processes each serving its own drive = up to 5 copies of each chunk, each copy on a different drive.

[Lindsay-Mathieson]

Yah, to get multiple copies you need at least <n> chunckservers for <n> copies.

I do use docker on my proxmox server (via a VM), but I'm not sure how to set up a chunkserver via docker. It was easy enough via lxc and quite lightweight, only consuming 200-300mb of RAM.

I prefer to keep my Proxmox server clean with no thirdparty installs, so run everything via lxc, docker or a VM.

[eleaner]

I eventually decided to bastardise proxmox setting moosefs directly
I am still not sure if it was the right decision, but my reasoning was that if I want to use moosefs mount as a datasource for (slower) proxmox vm disks - I need it rock solid and would rather avoid situation that vm did not start etc.
I eventually ended up with 4 chunkservers with rotational hdd and I am really impressed. I will likely add some SSD tiering later.

[eleaner]

very unscientific comparison
proxmox 8.1.4, few old HDD in a single machine

storage on ceph on almost default settings (with exception of the switch that allows balancing on single IP, 3copies is default)
vs
storage using mounted moosefs 3.0.117 directory, default settings on moosefs 3.0.117 (4 chunkservers, 2 copies is default)

moose performed about 10x faster while backing up 1TB VM

I am impressed
I am sure ceph can be optimised with some nvme but moose can be also tiered
and since I am not an expert in either - moosefs is a clean winner
(unless I discover something later)

[Lindsay-Mathieson]

I run a ceph cluster at work as well, would never use it for a lightweight home setup. Ceph needs 5+ nodes and SSD disks to get decent performance and is way more complex to setup and maintain than MooseFS, plus being a real memory/resource hog. Its pretty good for really big setups though.