Replies: 2 comments
-
I've noticed that one of CS have value populated in last error: Can this interrupt re-balancing ? |
Beta Was this translation helpful? Give feedback.
-
MooseFS disables replications (including rebalancing) when you have a disconnected chunk server in maintenance mode. So if you shut down your "B" chunk server properly, nothing will happen with the chunks as long as "B" stays in maintenance mode (automatic for half an hour, but can be extended indefinitely). Even "*,*" chunks will not start rebalancing with maintenance mode on. If your "B" chunk server has a hardware failure and is disconnected NOT in maintenance mode, then some replications will happen. "*,*" chunks that are not undergoal could be rebalancing, but only as long as they are not blocked by (higher priority) undergoal replications. "A,B" chunks behaviour will depend on your storage class definition: if it is in normal or loose mode, then a second copy will be created on the other "A" chunk server (so the chunk has 2 copies on 2 different chunk servers). It it is in strict mode, nothing will happen with those chunks, as there is no "B" chunk server available and strict mode is, well ... strict ;) So no, rebalancing "between" As in "A,B" class will not happen in this scenario, because either there will be 2 copies on A, which is not rebalancing, but undergoal mechanics or all the "A,B" chunks will be undergoal forced by strict mode and undergoal chunks cannot be rebalanced (only a chunk that is in "full" goal can be rebalanced). This is the correct behaviour of the system based on storage class definition. As for the error: this means one of your chunk servers has some issues with its hard drive - best shut down the chunk server process on it and run fsck and check your smart parameters, to make sure your disks are not failing. If your disks are OK, use mfschunktool to verify your chunks. P.S. Nodes 3.0.177 vs master 3.0.116 - this is not recommended. If it's not possible to have all the components in one instance in the same version, master should always have the highest version of all. |
Beta Was this translation helpful? Give feedback.
-
Hi,
What are conditions for chunkserver re-balancing ? (Internal within one chunkserver and between chunkservers)
I've checked following scenario
Three chunkservers:
1 - Label A - 1TB - Used: 100%
2 - Label A - 1TB - Used: 0%
3 - Label B - 2TB - Used: 50%
I have storage class: "A,B" so 2 copies, one reside on server A one on B.
Chunkservers A are very unbalanced (one server A is empty and second is full (almost no free space)).
Chunkserver B are only 50% full as it has full copy of A (1TB of 2TB that server has).
When three nodes are online everything works fine, chunkservers are re-balancing correctly.
Then I turn off chunkserver 'B' whole re-balancing stops - nothing is moved between 'A' servers although they should re-balance between each other. (Also internal re-balancing within one chunkserver is also not working).
This situation lasts until I enable server 'B', after B gets back online everything goes back to normal (whole re-balancing starts again).
Is that correct ?
Nodes are running on: 3.0.117
Master is running on: 3.0.116
Beta Was this translation helpful? Give feedback.
All reactions