[RFC] 009 - 基于 YJS 的跨端同步 #368
Replies: 11 comments 11 replies
-
YJS 集成 Demo将 YJS 的实例化过程独立于 store ，构建相应的双向绑定关系。再通过 provider 将状态同步到远程 代码如下： const ydoc = new Doc();
const persistence = new IndexeddbPersistence('yjs-demo', ydoc);
const provider = new WebrtcProvider('yjs-demo', ydoc);
const ymap = ydoc.getMap();
useSessionStore.subscribe(
(s) => s.activeId,
(t) => {
ymap.set('activeId', t);
},
);
ymap.observe(() => {
const id = ymap.get('activeId') as string;
useSessionStore.getState().activeSession(id);
});
persistence.on('synced', () => {
console.log('inited');
});效果如下： Area.mp4上述为演示 demo ，在工程化的过程中还需要考虑到和组件的集成、SoT 选择、怎么处理水合逻辑等。 |
Beta Was this translation helpful? Give feedback.
-
|
One disadvantage of yjs is that it is difficult to authenticate clients and impose access restrictions since it provides a peer-to-peer service, which may limit the implementation of some advanced features. Some alternative solutions to yjs: https://localfirstweb.dev/ |
Beta Was this translation helpful? Give feedback.
-
I think it seems not a problem, because yjs is not design for authentication, application can make the authentication and then use yjs to sync data. For example, if we use WebRTC Provider of YJS, we need to create a room channel for the webrtc connection. And we can add a password for the room. By using room name and the password ,we can hash them to a room channel id and use this id to create the channel. and I have also discussed this with chatgpt, fyi: https://sharegpt.com/c/5aM8XJs btw, this blog Local-first software is really amazing, I have read it latest year and then knew local-first and yjs. I even think that local-first software is the next-stage web app. |
Beta Was this translation helpful? Give feedback.
-
|
@arvinxx Because of the peer-to-peer design of yjs, any authenticated clients can access and modify any parts of the entire document. I am concerned that this may make it difficult to implement features such as collaborative editing and plugins that restrict read / write to specified data (if lobe chat wants to do that). Since Yjs doesn't care about the provider, we can control permissions in the provider. However, it is still difficult to restrict permissions for specific nodes in the document. We can also take it a step further and isolate content with different permissions in different documents and bind each to a provider. But before that, maybe we can first learn whether there are other out-of-the-box solutions available if the problem really matters. |
Beta Was this translation helpful? Give feedback.
-
|
btw, what is SoT? |
Beta Was this translation helpful? Give feedback.
-
|
SoT: Source of Truth. We need to choose the SoT of data persist in local. Because the data store we want to refactor to is not same with yjs persist. The yjs IndexedDB Provider is storage data to unit8Array like below. We can't use this data structure to implement the database search or similar features. We want to save data to IndexedDB more like relational dabase: So we need to decide how to solve the difference between two ways of saving data. This is also a challenge of engineering. |
Beta Was this translation helpful? Give feedback.
-
|
Didn't know that before. Can I refer to the description in this article? |
Beta Was this translation helpful? Give feedback.
-
|
yeah, you can |
Beta Was this translation helpful? Give feedback.
-
|
在 #378 实现基础上，会发现之前需要考虑的一些问题自然消除了:
如何结合YJS实现跨端同步，一个可能的思路如下： 实现Dexie.js和Yjs之间的映射涉及以下步骤：
下面是一个简化的示例，展示了如何实现Dexie.js和Yjs之间的基本映射： // 引入所需库
import Dexie from 'dexie';
import * as Y from 'yjs';
import { IndexeddbPersistence } from 'y-indexeddb';
// 定义Dexie数据库
const db = new Dexie('myDatabase');
db.version(1).stores({
items: '++id,name'
});
// 创建Yjs文档
const ydoc = new Y.Doc();
// 创建Yjs类型用于同步
const yitems = ydoc.getMap('items');
// Yjs IndexedDB持久化
const persistence = new IndexeddbPersistence('myDatabase', ydoc);
// 将Dexie数据库的数据加载到Yjs
async function loadDataFromDexieToYjs() {
const items = await db.items.toArray();
items.forEach(item => {
yitems.set(item.id.toString(), item);
});
}
// 监听Dexie数据库的变化
db.items.hook('creating', (primaryKey, obj, transaction) => {
yitems.set(primaryKey.toString(), obj);
});
db.items.hook('updating', (mods, primaryKey, obj, transaction) => {
const newYitem = { ...obj, ...mods };
yitems.set(primaryKey.toString(), newYitem);
});
db.items.hook('deleting', (primaryKey, obj, transaction) => {
yitems.delete(primaryKey.toString());
});
// 监听Yjs的变化并更新Dexie数据库
yitems.observe(event => {
event.changes.keys.forEach((change, key) => {
const id = parseInt(key);
if (change.action === 'add' || change.action === 'update') {
const item = yitems.get(key);
db.items.put({ ...item, id }); // Dexie需要id字段
} else if (change.action === 'delete') {
db.items.delete(id);
}
});
});
// 初始化加载数据
loadDataFromDexieToYjs();
// 示例：添加新项到Dexie.js（这也会更新Yjs）
db.items.put({ name: 'New Item' });
// 示例：更新Yjs（这也会更新Dexie.js）
yitems.set('1', { id: 1, name: 'Updated Item' });在这个示例代码中，我们创建了一个Dexie数据库和一个Yjs文档，它们共享一个名为 请注意，这只是一个基础示例，实际应用中可能需要更复杂的映射逻辑和错误处理。您需要根据自己的应用需求调整代码。 |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
Hi,這個存儲架構會考慮放在s3上嗎 |
Beta Was this translation helpful? Give feedback.
-
|
请问这个有计划的发布时间吗？谢谢 |
Beta Was this translation helpful? Give feedback.
-
|
Q1 |
Beta Was this translation helpful? Give feedback.
-
|
配置存储、冲突解决应该是后端比较擅长的，是否考虑在后端实现这些逻辑？ @arvinxx |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
作者大大开发的肿么样啦 |
Beta Was this translation helpful? Give feedback.
-
|
最近在做 #737 。同步需求估计会在2月下旬开始，预计3月份应该能做完 |
Beta Was this translation helpful? Give feedback.
-
|
是否考虑使用一个json file 来管理记录，并且利用Github的Gist 管理这个文件。把Github Gist 看成数据库 |
Beta Was this translation helpful? Give feedback.
-
|
JsonFile 不太现实，原因是后面数据量一大，同步会越来越慢。 举个例子，我现在本地数据近3000条消息，几百个topics，导出 json 已经要几M了。这个量级的同步，用json效率会很低 |
Beta Was this translation helpful? Give feedback.
-
如果能变成标准的「关系型」数据结构，那么是否可以仅仅依靠数据库的ACID特性（MySQL，PostgreSQL等均ACID）来保持同步。那这样的话用户是否可以只需填入一个URL（比如说 |
Beta Was this translation helpful? Give feedback.
-
|
是的，你说的没错 |
Beta Was this translation helpful? Give feedback.
-
Roadmap
|
Beta Was this translation helpful? Give feedback.
-
背景
用户存在跨端同步的诉求。譬如使用 WebDAV 实现跨端同步
调研与讨论
目前来看，单纯使用 WebDAV 技术实现跨端同步，会出现很多关于同步一致性的问题:
原因在于同步时必然存在的数据冲突。而从根本上解决该问题，需要引入 yjs 这样的 CRDT 解决方案。
市面上暂时没有同时结合 Yjs 与 WebDAV 的技术方案，需要自行组装实现。 refs: https://sharegpt.com/c/AlFzX7s
大致思路上，是将现有的 persist 层基础上，添加一个 yjs 作为操作同步层。同时利用 YJS 实现数据在 indexedDB 层的持久化，与在 WebDAV 层的持久化。在这种实现架构下，后续甚至还可以通过接入 WebRTC 层，实现含有 AI 的多人群聊会话模式。
现在存在另外一个问题，是 LobeChat 现有数据结构的可扩展性问题。当前的存储结构并没有考虑到消息无限膨胀后的扩展情况。因此当一个 session 中存在大量消息时，有很大概率导致页面卡死。
为考虑代码的可扩展性，我们需要将现有的存储模式，从「本地化」单一大对象的形态，变成标准的「关系型」数据结构。同时实现按需加载以提升性能。
因此跨端同步的实施前提是在整个数据存储架构上做一次全面的重构，将现在单纯的 JSON 大对象模式转换为标准的关系型数据库结构。这对于实现本 RFC ，以及后续的全局搜索（ #334 (comment) ）、服务端存储 #331 等需求都是重要的基石。
关于此部分重构，在 #378 跟进
实现思路
YJS 的工作原理核心实现原理如下： （refs）
关于自定义 Provider 的实现可以参考：
Beta Was this translation helpful? Give feedback.
All reactions