Missing documentation about mtu setting

[jwerak]

Its unclear to me how mtu value saved in /run/flannel/subnet.env is derived and how it can be changed or why it shouldn't be changed.

[eyakubovich]

MTU is calculated and set automatically by flannel. It then reports that value in subnet.env. It's not something the user can change.

MTU calculation is backend specific. I think we should have more backend specific documentation and we can include MTU calculation there.

[zbwright]

adding
MTU is calculated and set automatically by flannel. It then reports that value in subnet.env. This value cannot be changed.
to configuration.md

[kfox1111]

Not sure this really is the fix?

I have a cluster with some vm's in a cloud, and some bare metal nodes in the same flannel. the mtu on the physical hosts is different from the mtu on the vm's. So the overall flannel mtu should be the min(mtu) of all the nodes.

I'm also looking closely at using the kube-api setting. we need a way to manually specify the mtu in that mode I think.

[eyakubovich]

@kfox1111 flannel network is IP based (not L2) so it can have different MTU sizes. It's just like it is on the Internet. IP fragmentation, or more realistically, Path MTU discovery should take care of making it work.

[kfox1111]

flannel is l3 on top of l2 inside l3 on top of l2 via vxlan... there isn't a problem with overlay l2 packets being received if the receiving end has its overlay mtu smaller then the sender? I would expect this to cause packet drops.

container -> l3 -> l2 -> vxlan ------------ to different host ----------------- vxlan -> l2 (here) -> l3 -> container

It might work for tcp. but I don't think MSS will save you from that, nor is ip fragmentation always an option. (udp?)

I still think it would be safer/more performant to allow overriding the overlay mtu?

[eyakubovich]

@kfox1111 I think you're right. PMTUD over tunnels (and specifically VXLAN) is a hairy subject that might not be working. I think vxlan device should forward any "Fragmentation Needed" messages it receives, reducing Next-Hop MTU by its own overhead. But I'm not sure if it does that. So yeah, I agree that for now, adding a network-wide MTU override is a good idea.

However, I'm no longer involved with the flannel project anymore. Therefore I can't influence any addition of features to it.