Possible Approaches Towards P2P Resource Distribution

[dynos01]

Issue Description

Type: feature request

Describe what feature you want

Currently, Sealer heavily relies on SFTP to distribute resources. This leads to several potential issues:

• High resource usage
• Slow network speed
• Complex protocol (SSH, compared with HTTP or other common protocols)
• Difficult to scale up
• Hard to manage (SSH authentication)

Therefore, giving Sealer a fundamentally new method of distributing resources should be considered.

Additional context

Here are several common methods with their pros and cons:

P2P (BitTorrent or IPFS-like)

Pros:

• Highly mature, there are already large-scale deployment examples
• Extremely fast (from the perspective of the entire network)
• Easy to configure

Cons:

• It's necessary to re-design the entire distribution system
• High resource consumption (when the number of peers is large)

Distributed Storage

Pros:

• Possible to save a decent amount of disk space

Cons:

• Relatively difficult to implement

IP Multicast

Pros:

• Possible to save bandwidth

Cons:

• Reliant on IP networks that fully support multicast
• Need to completely get rid of SSH (it is only possible to use UDP when multicasting)

After all the considerations, I believe P2P is the most feasible approach. This is due to the fact that there are many fully tested P2P network implementations, among which some have already served the public for two decades. Distributed storage is a possible way, but implementing it with fault-tolerant replicas is harder, and there will be fundamental changes to the end-user experience. As for IP multicast, it shouldn't be taken for granted that the network fully supports multicasting.

To implement P2P resource distribution, there are two possible ways:

• Completely replace the current SSH-based system
• Build it as a plugin or add-on based on SSH

I personally prefer the second way since it requires minimal modification to the current codebase and is possible to maintain backward compatibility.

[VinceCui]

Iis there any mature IP Multicast file distributer?

[VinceCui]

以下是一些基于IP组播的开源文件传输系统：

P2PSP：它是一个基于IP组播的点对点流媒体传输协议，支持实时视频和音频流的传输，也可以用于文件传输。

UFTP：它是一个安全、可靠、高速的文件传输协议，使用IP组播和UDP协议传输数据，支持多个接收端。

CCNx：它是一个开源的内容中心化网络（Content-Centric Networking）架构，支持基于IP组播的数据传输，实现高效的内容分发和缓存。

StreamControl Transmission Protocol（SCTP）：它是一种可靠的传输协议，支持IP组播传输和多个接收端，适用于高性能数据传输和流媒体传输。

GridFTP：它是一个基于FTP协议的高性能文件传输系统，支持IP组播和多个接收端，适用于超大规模数据传输和分布式计算。

以上是一些常见的基于IP组播的开源文件传输系统，它们都有不同的特点和适用场景，可以根据具体需求选择。

[dynos01]

Thanks, I’ll take this information as reference. It might be possible to add multicasting when the network (esp. the router) has proper support for multicasting, and fall back to P2P otherwise.