I found that some of these data sets are not that easy to find, so I hope this list will help somebody in the future. Feel free to contribute.
4800 Images, multiple hands in gaming scenarios, such as playing cards, jenga, puzzle. Ego-centric perspective.
Source: http://vision.soic.indiana.edu/projects/egohands/
Multiple data sets, EGTEA Gaze+ consists of 15,176 images from cooking scenarios. Ego-centric perspective.
Source: http://www.cbi.gatech.edu/fpv/
1290 images from 3 different YouTube Videos, containing various amounts of hands and different scenarios. Ego-centric perspective.
Source: https://aurooj.github.io/Hand-Segmentation-in-the-Wild/
300 images from the internet.
Source: https://aurooj.github.io/Hand-Segmentation-in-the-Wild/
2000 hand pose images from and HCI-like setting.
Source: http://www.ouhands.oulu.fi/
41258 training and 2728 testing images, computer generated.
Source: https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html
I can not find the data set, although the paper states that they want to make the data set publicly available. Therefore, this serves as a placeholder. It contains segmentation masks generated from depth images.
Source: https://www.cs.ubc.ca/~kmyi/pubs/2019/bojja2019handseg/page/