Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

支持批量任务分布式执行 #157

Closed
Tracked by #154 ...
baisui1981 opened this issue Nov 7, 2022 · 1 comment
Closed
Tracked by #154 ...

支持批量任务分布式执行 #157

baisui1981 opened this issue Nov 7, 2022 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@baisui1981
Copy link
Member

baisui1981 commented Nov 7, 2022

  • 支持批量任务定时执行
  • 利用PowerJob整合现有TIS分布式调度任务

#97 has been referd

相关执行流程图

@baisui1981 baisui1981 mentioned this issue Nov 7, 2022
6 tasks
@baisui1981 baisui1981 added 3.7.1 enhancement New feature or request labels Nov 7, 2022
@baisui1981
Copy link
Member Author

baisui1981 commented Dec 15, 2022

powerJob相关

https://github.com/PowerJob/PowerJob

利用powerJob 实现DAG任务定义:https://github.com/PowerJob/PowerJob/blob/805046dccbb9dbeb6a59b0fe12a3fe9f1beb9ae7/powerjob-client/src/test/java/tech/powerjob/client/test/TestWorkflow.java

https://github.com/PowerJob/PowerJob/blob/805046dccbb9dbeb6a59b0fe12a3fe9f1beb9ae7/powerjob-client/src/test/java/tech/powerjob/client/test/TestWorkflow.java#L93-L103

        // DAG 图
        List<PEWorkflowDAG.Node> nodes = Lists.newLinkedList();
        List<PEWorkflowDAG.Edge> edges = Lists.newLinkedList();

        nodes.add(new PEWorkflowDAG.Node(nodeList.get(0).getId()));
        nodes.add(new PEWorkflowDAG.Node(nodeList.get(1).getId()));
        nodes.add(new PEWorkflowDAG.Node(nodeList.get(2).getId()));

        edges.add(new PEWorkflowDAG.Edge(nodeList.get(0).getId(), nodeList.get(1).getId()));
        edges.add(new PEWorkflowDAG.Edge(nodeList.get(1).getId(), nodeList.get(2).getId()));
        PEWorkflowDAG peWorkflowDAG = new PEWorkflowDAG(nodes, edges);

需要修改点

  1. 现 DataXAction.launchDataXWorker() 启动 dataX 分布式执行需要的环境,入口:https://github.com/qlangtech/plugins/blob/2a36da841abb279ee0eec3054c1b3662e6bdccbf/tis-k8s-plugin/src/main/java/com/qlangtech/tis/plugin/datax/K8SDataXJobWorker.java
  2. dataX任务在创建的时加一段脚本,需要直接将powerjob的workflow创建出来(更新时也需要去更新一下)
  3. DataXAction.triggerFullbuildTask() 触发全量构建
    需要加一段代码,判断workflow 是否存在?如果不存在需要让用户先填一个workflow的表单,再触发worflow任务执行

@baisui1981 baisui1981 added 3.8.0 and removed 3.7.1 labels Mar 29, 2023
@baisui1981 baisui1981 mentioned this issue Mar 29, 2023
3 tasks
@baisui1981 baisui1981 added this to the v3.8.0 milestone Apr 13, 2023
@baisui1981 baisui1981 modified the milestones: v3.8.0, v4.0.0 Aug 1, 2023
@baisui1981 baisui1981 removed the 3.8.0 label Aug 1, 2023
@baisui1981 baisui1981 pinned this issue Oct 28, 2023
baisui1981 added a commit to qlangtech/plugins that referenced this issue Nov 6, 2023
baisui1981 added a commit that referenced this issue Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant