[ISSUE #4787]integration connector #4790
Conversation
There was a problem hiding this comment.
Since the demo code is just a general idea, I've only looked at the general idea, so I'll leave some minor review comments for now. I think it should be possible to realize "when start runtime, if there are local Connectors, they will be started together with SPI mechanism, and the communication method will be intra-process communication". I agree with this idea, and look forward to hearing from more people in the community.
But there is a question, should we consider this case: the local Connector has already started by itself?
因为该demo代码只是大致的思路，所以只看了个大概，一些细小的review意见就暂时不提了。我觉得该思路应该可以实现“启动runtime时，如果本地有Connectors会利用SPI机制一并启动，并且通信方式将是进程内通信”。我赞同该思路，同时期待社区更多人的看法。
但是有个疑问，要不要考虑这种情况：本地的Connector已经自己启动过了？
|
Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed). |
I see in the issue that you have raised a lot of queries about the community's planning for this part of the demand, so let's leave this minor detail aside for now. I'll keep an eye on these queries as well #4787 (comment) . 我看到issue中您提了很多关于社区对这部分需求的规划的疑问，所以这个次要的细节问题暂不考虑吧。对于这些疑问（#4787 (comment) ），我也会继续关注。 |
Fixes #4787
########
UnMergeable！！！ its demo ，不能合并这只是用于讨论
########
Motivation
Integrate connector with minimal modification
Very simple integration using connecter
Modifications
Integrating 'connecter' means that 'connecter' has to have another running mode, and integrating into 'runtime' has some problems to solve
集成
connecter，意味着connecter要有另一种运行模式，集成到runtime中有一些问题需要解决The first problem that needs to be solved is that the main method is not called, but
connecteris a plug-in module, and it is troublesome to hard-code main once for each plug-in. So the method I think of is that all theconnectermodule's startup class inherits aEmbeddableConnectorinterface, and uses theEventmesh-SPIto load, so that the changes are broughtEmbeddableConnectorAll startup classes in
Connector(in this case,FileConnectServer)All files referred to by
SPIinConnector首先需要解决的问题就是
main方法没有被调用，但是connecter是插件化的模块，为每一个插件都硬编码一次main这样很麻烦，所以我想到的方法是所有connecter模块的启动类都继承一个EmbeddableConnector接口,使用Eventmesh的SPI进行加载，这样带来的修改就有EmbeddableConnector所有
Connector中的启动类（这个例子中的启动类是FileConnectServer）所有
Connector中SPI涉及的文件Second, the original interaction between
connectorandruntimeusestcp protocol,and the purpose of integration is to reduce serialization and communication consumption. Therefore, interaction needs to be realized in other ways. The method I think is to use the form of service. connector itself only has two roles,sinkandsource, which are publish and subscribe in terms of function, so this service needs to use publish and subscribe functions. Meanwhile, in order to preserve the original structure and design to the greatest extent, the connector uses atcp adapter(the adapter does not implement network communication in the middle). Instead, the service is called.) So the design changes areIInnerPubSubService Indicates the interface of the service
InnerPubSubService implementation
IntegrationCloudEventTCPClientAdapter TCP adapter
Application must have a static variable holding the service
The SinkWorker passes the service to the tcp adapter
SourceWorker transfers the service tcp adapter
其次需要解决的就是原有
connector和runtime之间的交互使用的是tcp协议而集成的目的就是减少序列化和通信消耗，所以需要通过其他方式对交互进行实现，我想到的方法是使用服务的形式，connector本身其实只有两种角色，sink和source，从功能来说就是发布和订阅，那么这个服务就需要用发布和订阅的功能即可，同时为了最大程度保留原有结构和设计，使用tcp适配器（适配器中间不在实现网络通信，而是对服务进行调用）所以设计的修改有IInnerPubSubService 服务的接口
InnerPubSubService 服务的实现
IntegrationCloudEventTCPClientAdapter tcp适配器
Application 要有静态变量持有服务
SinkWorker 传递服务给tcp适配器
SourceWorker 传递服务tcp适配器
Unfinished changes:
The implementation of InnerPubSubService
未完成的修改：
InnerPubSubService的具体实现达成的效果：
只要
gradle引用了对应的connector插件就会自动处理 内嵌connector相关逻辑Results achieved:
The built-in
connectorlogic is automatically handled whenevergradlereferences the correspondingconnectorplug-in