【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 #3944

wubolive · 2022-12-02T08:53:19Z

wubolive
Dec 2, 2022

DSS+Linkis Ansible一键安装脚本

DSS1.1.1 & Linkis 1.3.0 Ansible 一键部署脚本

作者：wubolive
Q Q：1049635685
邮箱：wubolive@foxmai.com
Github：https://github.com/wubolive/dss-linkis-ansible

一、简介

为解决繁琐的部署流程，简化安装步骤，本脚本提供一键安装最新版本的DSS+Linkis环境；部署包中的软件采用我自己编译的安装包，并且为最新版本：DSS1.1.1 + Linkis1.3.0。

1.1 版本介绍

以下版本及配置信息可参考安装程序hosts文件中的[all:vars]字段。

软件名称	软件版本	应用路径	测试/连接命令
MySQL	mysql-5.6	/usr/local/mysql	mysql -h 127.0.0.1 -uroot -p123456
JDK	jdk1.8.0_171	/usr/local/java	java -version
Python	python 2.7.5	/usr/lib64/python2.7	python -V
Nginx	nginx/1.20.1	/etc/nginx	nginx -t
Hadoop	hadoop-2.7.2	/opt/hadoop	hdfs dfs -ls /
Hive	hive-2.3.3	/opt/hive	hive -e "show databases"
Spark	spark-2.4.3	/opt/spark	spark-sql -e "show databases"
dss	dss-1.1.1	/home/hadoop/dss	http://<服务器IP>:8085
links	linkis-1.3.0	/home/hadoop/linkis	http://<服务器IP>:8188
zookeeper	3.4.6	/usr/local/zookeeper	无
DolphinScheduler	1.3.9	/opt/dolphinscheduler	http://<服务器IP>:12345/dolphinscheduler
Visualis	1.0.0	/opt/visualis-server	http://<服务器IP>:9088
Qualitis	0.9.2	/opt/qualitis	http://<服务器IP>:8090
Streamis	0.2.0	/opt/streamis	http://<服务器IP>:9188
Sqoop	1.4.6	/opt/sqoop	sqoop
Exchangis	1.0.0	/opt/exchangis	http://<服务器IP>:8028

二、部署前注意事项

要求：

本脚本仅在CentOS 7系统上测试过，请确保安装的服务器为CentOS 7。
仅安装DSS+Linkis服务器内存至少16G，安装全部服务内存至少32G。
安装前请关闭服务器防火墙及SElinux，并使用root用户进行操作。
安装服务器必须通畅的访问互联网，脚本需要yum下载一些基础软件。
保证服务器未安装任何软件，包括不限于java、mysql、nginx等，最好是全新系统。
必须保证服务器除lo:127.0.0.1回环地址外，仅只有一个IP地址，可使用echo $(hostname -I)命令测试。

三、部署方法

本案例部署主机IP为192.168.1.52，以下步骤请按照自己实际情况更改。

3.1 安装前设置

### 安装ansible
$ yum -y install epel-release
$ yum -y install ansible

### 配置免密
$ ssh-keygen -t rsa
$ ssh-copy-id root@192.168.1.52

### 关闭防火墙及SELinux
$ systemctl stop firewalld.service && systemctl disable firewalld.service
$ sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0

3.2 部署linkis+dss

### 获取安装包
$ git clone https://github.com/wubolive/dss-linkis-ansible.git
$ cd dss-linkis-ansible

### 目录说明
dss-linkis-ansible
├── ansible.cfg    # ansible 配置文件
├── hosts          # hosts主机及变量配置
├── playbooks      # playbooks剧本
├── README.md      # 说明文档
└── roles          # 角色配置

### 配置部署主机（注：ansible_ssh_host的值不能设置127.0.0.1）
$ vim hosts
[deploy]
dss-service ansible_ssh_host=192.168.1.52 ansible_ssh_port=22

### 下载安装包到download目录(如果下载失败，可以手动下载放到该目录)
$ ansible-playbook playbooks/download.yml

### 一键安装Linkis+DSS
$ ansible-playbook playbooks/all.yml
......
TASK [dss : 打印访问信息] *****************************************************************************************
ok: [dss-service] => {
    "msg": [
        "*****************************************************************", 
        "              访问 http://192.168.1.52 查看访问信息                 ", 
        "*****************************************************************"
    ]
}

执行结束后，即可访问：http://192.168.1.52 查看信息页面，上面记录了所有服务的访问地址及账号密码。

3.3 部署其它服务

# 安装dolphinscheduler
$ ansible-playbook playbooks/dolphinscheduler.yml
### 注: 安装以下服务必须优先安装dolphinscheduler调度系统
# 安装visualis
$ ansible-playbook playbooks/visualis.yml 
# 安装qualitis
$ ansible-playbook playbooks/qualitis.yml
# 安装streamis
$ ansible-playbook playbooks/streamis.yml
# 安装exchangis
$ ansible-playbook playbooks/exchangis.yml

3.4 维护指南

### 查看实时日志
$ su - hadoop
$ tail -f ~/linkis/logs/*.log ~/dss/logs/*.log

### 启动DSS+Linkis服务（如服务器重启可使用此命令一建启动）
$ ansible-playbook playbooks/all.yml -t restart
# 启动zookeeper
$ sh /usr/local/zookeeper/bin/zkServer.sh start
# 启动其它服务
$ su - hadoop
$ cd /opt/dolphinscheduler/bin &&  sh start-all.sh 
$ cd /opt/visualis-server/bin && sh start-visualis-server.sh
$ cd /opt/qualitis/bin/ && sh start.sh
$ cd /opt/streamis/streamis-server/bin/ && sh start-streamis-server.sh
$ cd /opt/exchangis/sbin/ && ./daemon.sh start server

使用问题请访问官方QA文档：https://docs.qq.com/doc/DSGZhdnpMV3lTUUxq

brucedchen1991 · 2022-12-13T02:54:46Z

brucedchen1991
Dec 13, 2022
Collaborator

TASK [安装linkis] ***********************************************************************************************************************************************************************************
fatal: [dss-service]: FAILED! => {"changed": true, "cmd": "source /etc/profile ~/.bashrc ; sh bin/install.sh", "delta": "0:00:00.081167", "end": "2022-12-13 10:46:34.735480", "msg": "non-zero return code", "rc": 1, "start": "2022-12-13 10:46:34.654313", "stderr": "check command fail \n need 'lsof' (your linux command not found)", "stderr_lines": ["check command fail ", " need 'lsof' (your linux command not found)"], "stdout": "======= Step 1: Load deploy-config/* ==========\n\u001b[0;32mSucceed\u001b[0m to load config\n\n======= Step 2: Check env ==========\n<-----start to check used cmd---->\ncheck yum\ncheck java\ncheck mysql\ncheck telnet\ncheck tar\ncheck sed\ncheck lsof\n\u001b[0;31mFailed\u001b[0m to check env", "stdout_lines": ["======= Step 1: Load deploy-config/* ==========", "\u001b[0;32mSucceed\u001b[0m to load config", "", "======= Step 2: Check env ==========", "<-----start to check used cmd---->", "check yum", "check java", "check mysql", "check telnet", "check tar", "check sed", "check lsof", "\u001b[0;31mFailed\u001b[0m to check env"]}

1 reply

linjie830914 Dec 22, 2022

执行一下 yum install lsof -y即可。

brucedchen1991 · 2022-12-13T03:13:25Z

brucedchen1991
Dec 13, 2022
Collaborator

TASK [配置mysql程序目录] ********************************************************************************************************************************************************************************
fatal: [dss-service]: FAILED! => {"changed": true, "cmd": ["/usr/bin/mv", "/tmp/mysql-5.6.49-linux-glibc2.12-x86_64", "/usr/local/mysql"], "delta": "0:00:00.012687", "end": "2022-12-13 11:11:17.307584", "msg": "non-zero return code", "rc": 1, "start": "2022-12-13 11:11:17.294897", "stderr": "/usr/bin/mv: cannot move ‘/tmp/mysql-5.6.49-linux-glibc2.12-x86_64’ to ‘/usr/local/mysql/mysql-5.6.49-linux-glibc2.12-x86_64’: Directory not empty", "stderr_lines": ["/usr/bin/mv: cannot move ‘/tmp/mysql-5.6.49-linux-glibc2.12-x86_64’ to ‘/usr/local/mysql/mysql-5.6.49-linux-glibc2.12-x86_64’: Directory not empty"], "stdout": "", "stdout_lines": []}

2 replies

linjie830914 Dec 22, 2022

你安装之前是不是已经有mysql数据库了？先卸载，再重新部署试试。

wubolive Dec 22, 2022
Author

是不是已经存在了 /usr/local/mysql 目录

brucedchen1991 · 2022-12-13T10:07:31Z

brucedchen1991
Dec 13, 2022
Collaborator

TASK [安装linkis] ***********************************************************************************************************************************************************************************
fatal: [dss-service]: FAILED! => {"changed": true, "cmd": "source /etc/profile ~/.bashrc ; sh bin/install.sh", "delta": "0:00:05.705482", "end": "2022-12-13 18:06:31.920861", "msg": "non-zero return code", "rc": 1, "start": "2022-12-13 18:06:26.215379", "stderr": "Warning: Using a password on the command line interface can be insecure.\nchmod: changing permissions of '/tmp/linkis': Permission denied. user=hadoop is not the owner of inode=linkis", "stderr_lines": ["Warning: Using a password on the command line interface can be insecure.", "chmod: changing permissions of '/tmp/linkis': Permission denied. user=hadoop is not the owner of inode=linkis"], "stdout": "======= Step 1: Load deploy-config/* ==========\n\u001b[0;32mSucceed\u001b[0m to load config\n\n======= Step 2: Check env ==========\n<-----start to check used cmd---->\ncheck yum\ncheck java\ncheck mysql\ncheck telnet\ncheck tar\ncheck sed\ncheck lsof\n<-----end to check used cmd---->\n\u001b[0;32mSucceed\u001b[0m to execute cmd: python --version\n\n\u001b[0;32mSucceed\u001b[0m to execute cmd: java --version\n\n\u001b[0;32mSucceed\u001b[0m to execute cmd: spark-submit --version \n\nYour [hadoop] version may match default support version: 2.7\n\nYour [hive] version may match default support version: 2.3\n\n\u001b[0;32mSucceed\u001b[0m to check env\n\nLINKIS_HOME: /home/hadoop/linkis is alread exists and will be backed up\nmv /home/hadoop/linkis /home/hadoop/linkis-back-1670925987\n\u001b[0;32mSucceed\u001b[0m to back up old LINKIS_HOME:/home/hadoop/linkis to /home/hadoop/linkis-back-1670925987\n\ntry to create dir LINKIS_HOME: /home/hadoop/linkis\n\u001b[0;32mSucceed\u001b[0m to create the dir of LINKIS_HOME:/home/hadoop/linkis\n\nStart to cp /opt/apache-linkis-1.3.0/linkis-package to /home/hadoop/linkis.\n\u001b[0;32mSucceed\u001b[0m to cp /opt/apache-linkis-1.3.0/linkis-package to /home/hadoop/linkis\n\n======= Step 3: Create necessary directory ==========\n[WORKSPACE_USER_ROOT_PATH] try to create directory\n[WORKSPACE_USER_ROOT_PATH] try to create local dir,cmd is: mkdir -p /home/hadoop/linkis/tmp/linkis//hadoop\n\u001b[0;32mSucceed\u001b[0m to create WORKSPACE_USER_ROOT_PATH: file:///home/hadoop/linkis/tmp/linkis/ directory\n\n[HDFS_USER_ROOT_PATH] try to create directory\n[HDFS_USER_ROOT_PATH] try to create hdfs dir,cmd is: hdfs dfs -mkdir -p /tmp/linkis/hadoop\n\u001b[0;32mSucceed\u001b[0m to create HDFS_USER_ROOT_PATH: hdfs:///tmp/linkis directory\n\n[RESULT_SET_ROOT_PATH] try to create directory\n[RESULT_SET_ROOT_PATH] try to create hdfs dir,cmd is: hdfs dfs -mkdir -p /tmp/linkis\n\u001b[0;31mFailed\u001b[0m to create RESULT_SET_ROOT_PATH: hdfs:///tmp/linkis directory", "stdout_lines": ["======= Step 1: Load deploy-config/* ==========", "\u001b[0;32mSucceed\u001b[0m to load config", "", "======= Step 2: Check env ==========", "<-----start to check used cmd---->", "check yum", "check java", "check mysql", "check telnet", "check tar", "check sed", "check lsof", "<-----end to check used cmd---->", "\u001b[0;32mSucceed\u001b[0m to execute cmd: python --version", "", "\u001b[0;32mSucceed\u001b[0m to execute cmd: java --version", "", "\u001b[0;32mSucceed\u001b[0m to execute cmd: spark-submit --version ", "", "Your [hadoop] version may match default support version: 2.7", "", "Your [hive] version may match default support version: 2.3", "", "\u001b[0;32mSucceed\u001b[0m to check env", "", "LINKIS_HOME: /home/hadoop/linkis is alread exists and will be backed up", "mv /home/hadoop/linkis /home/hadoop/linkis-back-1670925987", "\u001b[0;32mSucceed\u001b[0m to back up old LINKIS_HOME:/home/hadoop/linkis to /home/hadoop/linkis-back-1670925987", "", "try to create dir LINKIS_HOME: /home/hadoop/linkis", "\u001b[0;32mSucceed\u001b[0m to create the dir of LINKIS_HOME:/home/hadoop/linkis", "", "Start to cp /opt/apache-linkis-1.3.0/linkis-package to /home/hadoop/linkis.", "\u001b[0;32mSucceed\u001b[0m to cp /opt/apache-linkis-1.3.0/linkis-package to /home/hadoop/linkis", "", "======= Step 3: Create necessary directory ==========", "[WORKSPACE_USER_ROOT_PATH] try to create directory", "[WORKSPACE_USER_ROOT_PATH] try to create local dir,cmd is: mkdir -p /home/hadoop/linkis/tmp/linkis//hadoop", "\u001b[0;32mSucceed\u001b[0m to create WORKSPACE_USER_ROOT_PATH: file:///home/hadoop/linkis/tmp/linkis/ directory", "", "[HDFS_USER_ROOT_PATH] try to create directory", "[HDFS_USER_ROOT_PATH] try to create hdfs dir,cmd is: hdfs dfs -mkdir -p /tmp/linkis/hadoop", "\u001b[0;32mSucceed\u001b[0m to create HDFS_USER_ROOT_PATH: hdfs:///tmp/linkis directory", "", "[RESULT_SET_ROOT_PATH] try to create directory", "[RESULT_SET_ROOT_PATH] try to create hdfs dir,cmd is: hdfs dfs -mkdir -p /tmp/linkis", "\u001b[0;31mFailed\u001b[0m to create RESULT_SET_ROOT_PATH: hdfs:///tmp/linkis directory"]}

0 replies

linjie830914 · 2022-12-22T07:13:48Z

linjie830914
Dec 22, 2022

部署后有如下错误提示，帮忙看看是什么问题吧，是否是缺少linkis.properties文件？：

{
        "changed": true, 
        "cmd": "source /etc/profile ~/.bashrc ; sh bin/linkis-cli -submitUser  hadoop  -engineType shell-1 -codeType shell  -code \"whoami\"", 
        "delta": "0:00:04.151327", 
        "end": "2022-12-22 13:33:13.225659", 
        "msg": "non-zero return code", 
        "rc": 255, 
        "start": "2022-12-22 13:33:09.074332", 
        "stderr": "", 
        "stderr_lines": [ ], 
        "stdout": "=====Java Start Command=====
exec /usr/local/java/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/hadoop/linkis/logs/linkis-cli -XX:ErrorFile=/home/hadoop/linkis/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /home/hadoop/linkis/conf/linkis-cli:/home/hadoop/linkis/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/home/hadoop/linkis/lib/linkis-commons/public-module/*:.:/usr/local/java/lib/dt.jar:/usr/local/java/lib/tools.jar:/usr/local/java/jre/lib:.:/usr/local/java/lib/dt.jar:/usr/local/java/lib/tools.jar:/usr/local/java/jre/lib: -Dconf.root=/home/hadoop/linkis/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/home/hadoop/linkis/logs/linkis-cli -Dlog.file=linkis-client.hadoop.log.20221222133309116867181  org.apache.linkis.cli.application.LinkisClientApplication '-submitUser hadoop -engineType shell-1 -codeType shell -code whoami'
[INFO] LogFile path: /home/hadoop/linkis/logs/linkis-cli/linkis-client.hadoop.log.20221222133309116867181
[INFO] User does not provide usr-configuration file. Will use default config
[INFO] user does not specify proxy-user, will use current submit-user \"hadoop\" by default.
[INFO] connecting to linkis gateway:http://127.0.0.1:9001
[ERROR] EXE0011,Error occured during execution: Failed to submit job
Caused by: org.apache.linkis.httpclient.exception.HttpClientResultException: errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {\"method\":\"/api/rest_j/v1/entrance/submit\",\"status\":1,\"message\":\"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway\",\"data\":{\"data\":\"{\\\"executionContent\\\":{\\\"code\\\":\\\"whoami\\\"},\\\"source\\\":{\\\"scriptPath\\\":\\\"LinkisCli\\\"},\\\"params\\\":{\\\"configuration\\\":{\\\"startup\\\":{},\\\"runtime\\\":{\\\"hive.resultset.use.unique.column.names\\\":true}},\\\"variable\\\":{}},\\\"executeUser\\\":\\\"hadoop\\\",\\\"labels\\\":{\\\"userCreator\\\":\\\"hadoop-LINKISCLI\\\",\\\"codeType\\\":\\\"shell\\\",\\\"engineType\\\":\\\"shell-1\\\",\\\"executeOnce\\\":\\\"\\\"}}\"}}. errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {\"method\":\"/api/rest_j/v1/entrance/submit\",\"status\":1,\"message\":\"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway\",\"data\":{\"data\":\"{\\\"executionContent\\\":{\\\"code\\\":\\\"whoami\\\"},\\\"source\\\":{\\\"scriptPath\\\":\\\"LinkisCli\\\"},\\\"params\\\":{\\\"configuration\\\":{\\\"startup\\\":{},\\\"runtime\\\":{\\\"hive.resultset.use.unique.column.names\\\":true}},\\\"variable\\\":{}},\\\"executeUser\\\":\\\"hadoop\\\",\\\"labels\\\":{\\\"userCreator\\\":\\\"hadoop-LINKISCLI\\\",\\\"codeType\\\":\\\"shell\\\",\\\"engineType\\\":\\\"shell-1\\\",\\\"executeOnce\\\":\\\"\\\"}}\"}}. ,ip: null ,port: 0 ,serviceKind: null ,ip: null ,port: 0 ,serviceKind: null
############Execute Error!!!########", 
        "stdout_lines": [
                "=====Java Start Command=====", 
                "exec /usr/local/java/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/hadoop/linkis/logs/linkis-cli -XX:ErrorFile=/home/hadoop/linkis/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /home/hadoop/linkis/conf/linkis-cli:/home/hadoop/linkis/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/home/hadoop/linkis/lib/linkis-commons/public-module/*:.:/usr/local/java/lib/dt.jar:/usr/local/java/lib/tools.jar:/usr/local/java/jre/lib:.:/usr/local/java/lib/dt.jar:/usr/local/java/lib/tools.jar:/usr/local/java/jre/lib: -Dconf.root=/home/hadoop/linkis/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/home/hadoop/linkis/logs/linkis-cli -Dlog.file=linkis-client.hadoop.log.20221222133309116867181  org.apache.linkis.cli.application.LinkisClientApplication '-submitUser hadoop -engineType shell-1 -codeType shell -code whoami'", 
                "[INFO] LogFile path: /home/hadoop/linkis/logs/linkis-cli/linkis-client.hadoop.log.20221222133309116867181", 
                "[INFO] User does not provide usr-configuration file. Will use default config", 
                "[INFO] user does not specify proxy-user, will use current submit-user \"hadoop\" by default.", 
                "[INFO] connecting to linkis gateway:http://127.0.0.1:9001", 
                "[ERROR] EXE0011,Error occured during execution: Failed to submit job", 
                "Caused by: org.apache.linkis.httpclient.exception.HttpClientResultException: errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {\"method\":\"/api/rest_j/v1/entrance/submit\",\"status\":1,\"message\":\"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway\",\"data\":{\"data\":\"{\\\"executionContent\\\":{\\\"code\\\":\\\"whoami\\\"},\\\"source\\\":{\\\"scriptPath\\\":\\\"LinkisCli\\\"},\\\"params\\\":{\\\"configuration\\\":{\\\"startup\\\":{},\\\"runtime\\\":{\\\"hive.resultset.use.unique.column.names\\\":true}},\\\"variable\\\":{}},\\\"executeUser\\\":\\\"hadoop\\\",\\\"labels\\\":{\\\"userCreator\\\":\\\"hadoop-LINKISCLI\\\",\\\"codeType\\\":\\\"shell\\\",\\\"engineType\\\":\\\"shell-1\\\",\\\"executeOnce\\\":\\\"\\\"}}\"}}. errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {\"method\":\"/api/rest_j/v1/entrance/submit\",\"status\":1,\"message\":\"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway\",\"data\":{\"data\":\"{\\\"executionContent\\\":{\\\"code\\\":\\\"whoami\\\"},\\\"source\\\":{\\\"scriptPath\\\":\\\"LinkisCli\\\"},\\\"params\\\":{\\\"configuration\\\":{\\\"startup\\\":{},\\\"runtime\\\":{\\\"hive.resultset.use.unique.column.names\\\":true}},\\\"variable\\\":{}},\\\"executeUser\\\":\\\"hadoop\\\",\\\"labels\\\":{\\\"userCreator\\\":\\\"hadoop-LINKISCLI\\\",\\\"codeType\\\":\\\"shell\\\",\\\"engineType\\\":\\\"shell-1\\\",\\\"executeOnce\\\":\\\"\\\"}}\"}}. ,ip: null ,port: 0 ,serviceKind: null ,ip: null ,port: 0 ,serviceKind: null", 
                "############Execute Error!!!########"
        ]
}

其中"[INFO] LogFile path: /home/hadoop/linkis/logs/linkis-cli/linkis-client.hadoop.log.20221222133309116867181"日志内容如下：

2022-12-22 13:33:10,792 INFO AbstractFitter(58) - Parsing command: "-submitUser hadoop -engineType shell-1 -codeType shell -code whoami" into template: "UNIVERSAL"
2022-12-22 13:33:10,811 INFO ParsedTplValidator(54) - start validating command "linkis-cli", template "UNIVERSAL"
2022-12-22 13:33:10,812 INFO ParsedTplValidator(60) - Start params-check
2022-12-22 13:33:10,812 INFO ParsedTplValidator(62) - params-check ok.
2022-12-22 13:33:10,908 INFO PropsFilesScanner(41) - Start scanning for properties files. Root path = "/home/hadoop/linkis/conf/linkis-cli"
2022-12-22 13:33:10,944 INFO PropsFilesScanner(71) - Scanned properties files="[/home/hadoop/linkis/conf/linkis-cli/linkis-cli.properties]"
*********************************************************
2022-12-22 13:33:12,569 INFO DWSHttpClient(156) - invoke http://127.0.0.1:9001/api/rest_j/v1/entrance/submit taken: 175
2022-12-22 13:33:13,182 ERROR DefaultExceptionHandler(47) - EXE0011,Error occured during execution: Failed to submit job
org.apache.linkis.cli.core.exception.LinkisClientExecutionException: EXE0011,Error occured during execution: Failed to submit job
	at org.apache.linkis.cli.application.operator.ujes.LinkisJobOperator.submit(LinkisJobOperator.java:134) ~[linkis-cli-application-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.application.interactor.job.LinkisSubmitJob.submit(LinkisSubmitJob.java:118) ~[linkis-cli-application-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.core.interactor.execution.SyncSubmission.ExecWithAsyncBackend(SyncSubmission.java:161) ~[linkis-cli-core-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.core.interactor.execution.SyncSubmission.execute(SyncSubmission.java:80) ~[linkis-cli-core-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.application.LinkisClientApplication.exec(LinkisClientApplication.java:278) ~[linkis-cli-application-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.application.LinkisClientApplication.main(LinkisClientApplication.java:311) ~[linkis-cli-application-1.3.0.jar:1.3.0]
Caused by: org.apache.linkis.httpclient.exception.HttpClientResultException: errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {"method":"/api/rest_j/v1/entrance/submit","status":1,"message":"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway","data":{"data":"{\"executionContent\":{\"code\":\"whoami\"},\"source\":{\"scriptPath\":\"LinkisCli\"},\"params\":{\"configuration\":{\"startup\":{},\"runtime\":{\"hive.resultset.use.unique.column.names\":true}},\"variable\":{}},\"executeUser\":\"hadoop\",\"labels\":{\"userCreator\":\"hadoop-LINKISCLI\",\"codeType\":\"shell\",\"engineType\":\"shell-1\",\"executeOnce\":\"\"}}"}}. errCode: 10905 ,desc: URL /api/rest_j/v1/entrance/submit request failed! ResponseBody is {"method":"/api/rest_j/v1/entrance/submit","status":1,"message":"NoApplicationExistsException: errCode: 10050 ,desc: Application linkis-cg-entrance is not exists any instances. ,ip: dss-service ,port: 9001 ,serviceKind: linkis-mg-gateway","data":{"data":"{\"executionContent\":{\"code\":\"whoami\"},\"source\":{\"scriptPath\":\"LinkisCli\"},\"params\":{\"configuration\":{\"startup\":{},\"runtime\":{\"hive.resultset.use.unique.column.names\":true}},\"variable\":{}},\"executeUser\":\"hadoop\",\"labels\":{\"userCreator\":\"hadoop-LINKISCLI\",\"codeType\":\"shell\",\"engineType\":\"shell-1\",\"executeOnce\":\"\"}}"}}. ,ip: null ,port: 0 ,serviceKind: null ,ip: null ,port: 0 ,serviceKind: null
	at org.apache.linkis.httpclient.dws.response.DWSResult$$anonfun$set$2.apply(DWSResult.scala:85) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.dws.response.DWSResult$$anonfun$set$2.apply(DWSResult.scala:84) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:61) ~[linkis-common-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.dws.response.DWSResult$class.set(DWSResult.scala:84) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at org.apache.linkis.ujes.client.response.JobExecuteResult.set(JobExecuteResult.scala:25) ~[linkis-computation-client-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.dws.DWSHttpClient$$anonfun$httpResponseToResult$1.apply(DWSHttpClient.scala:83) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.dws.DWSHttpClient$$anonfun$httpResponseToResult$1.apply(DWSHttpClient.scala:79) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at scala.Option.map(Option.scala:146) ~[scala-library-2.11.12.jar:?]
	at org.apache.linkis.httpclient.dws.DWSHttpClient.httpResponseToResult(DWSHttpClient.scala:79) ~[linkis-gateway-httpclient-support-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.AbstractHttpClient.responseToResult(AbstractHttpClient.scala:522) ~[linkis-httpclient-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.AbstractHttpClient.execute(AbstractHttpClient.scala:168) ~[linkis-httpclient-1.3.0.jar:1.3.0]
	at org.apache.linkis.httpclient.AbstractHttpClient.execute(AbstractHttpClient.scala:125) ~[linkis-httpclient-1.3.0.jar:1.3.0]
	at org.apache.linkis.ujes.client.UJESClientImpl.executeUJESJob(UJESClientImpl.scala:31) ~[linkis-computation-client-1.3.0.jar:1.3.0]
	at org.apache.linkis.ujes.client.UJESClient.submit(UJESClient.scala:38) ~[linkis-computation-client-1.3.0.jar:1.3.0]
	at org.apache.linkis.cli.application.operator.ujes.LinkisJobOperator.submit(LinkisJobOperator.java:129) ~[linkis-cli-application-1.3.0.jar:1.3.0]
	... 5 more

6 replies

stdnt-xiao Dec 23, 2022

#2642
hostname映射问题，可以设置主机名为dss-service

linjie830914 Dec 23, 2022

访问http://<服务器IP>:20303，看看微服务都注册上了吗

都注册上了，已经部署成功。

linjie830914 Dec 23, 2022

#2642 hostname映射问题，可以设置主机名为dss-service

请问从哪里看出来的跟hostname有关系啊？谢谢了

wubolive Dec 26, 2022
Author

#2642 hostname映射问题，可以设置主机名为dss-service

请问从哪里看出来的跟hostname有关系啊？谢谢了

我roles/basis/tasks/main.yml剧本中有一步是设置主机名，名字默认是dss-service，并且需要设置在/etc/hosts文件第一行

stdnt-xiao Dec 28, 2022

SELINUX是否关闭？

临时关闭命令
setenforce 0

永久生效
vi /etc/selinux/config
SELINUX=permissive

linjie830914 · 2022-12-23T00:48:43Z

linjie830914
Dec 23, 2022

如下图所示都成功了，

3 replies

stdnt-xiao Dec 23, 2022

这个没有部署成功，linkis或者dss部署失败。部署成功后，ip直接访问应该进入dss管理界面

wubolive Dec 23, 2022
Author

成没成功可以访问dss管理界面是8085端口就知道了

linjie830914 Dec 23, 2022

这个没有部署成功，linkis或者dss部署失败。部署成功后，ip直接访问应该进入dss管理界面

我已经进入dss界面了，可以顺利登录。
https://live.csdn.net/v/265036

linjie830914 · 2022-12-23T01:00:00Z

linjie830914
Dec 23, 2022

但是再安装dolphinscheduler，执行如下操作报错，
$ ansible-playbook playbooks/dolphinscheduler.yml
错误信息如下截图，12345后端dolphinscheduler服务的地址不通

eureka截图如下：

但是能打开这个地址：http://192.168.32.251:12345/dolphinscheduler，却不能登录成功，始终提示密码错误。

执行安装脚本时输出信息如下：

DSS+Linkis.Ansible.-dolphinscheduler.mp4


TASK [dolphinscheduler : 检查端口是否运行] *********************************************************************************************************************************************
fatal: [dss-service]: FAILED! => {"changed": false, "elapsed": 30, "msg": "Timeout when waiting for 127.0.0.1:12345"}

PLAY RECAP *********************************************************************************************************************************************************************
dss-service                : ok=21   changed=17   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[root@dss-service dss-linkis-ansible]#

1 reply

linjie830914 Dec 23, 2022

我执行如下脚本后，能登陆了

UPDATE `dolphinscheduler`.`t_ds_user` SET `user_name` = 'hadoop', `user_password` = '3437a3834a6d403b462648428f8642b2', `user_type` = 0, `email` = 'xxx@qq.com', `phone` = '', `tenant_id` = 1, `create_time` = '2018-03-27 15:48:50', `update_time` = '2022-11-29 15:07:53', `queue` = 'default' WHERE `id` = 1;
    # 创建租户
INSERT INTO `dolphinscheduler`.`t_ds_tenant` (`id`, `tenant_code`, `tenant_name`, `description`, `queue_id`, `create_time`, `update_time`) VALUES (1, 'hadoop', 'hadoop', '', 1, '2022-11-29 15:06:39', '2022-11-29 15:06:39');
    # 创建token
INSERT INTO `dolphinscheduler`.`t_ds_access_token` (`id`, `user_id`, `token`, `expire_time`, `create_time`, `update_time`) VALUES (1, 1, 'c42f5ed1fc16c3e8b505739ab8a55c74', '2032-12-31 00:00:00', '2022-11-29 15:10:12', '2022-11-29 15:10:12');

截图如下，不知道是否存在其他未发现的问题，sql脚本为啥没执行需要查一下：

linjie830914 · 2022-12-23T01:26:36Z

linjie830914
Dec 23, 2022

执行这个脚本后ansible-playbook playbooks/qualitis.yml，也提示如下错误：

********
fatal: [dss-service]: FAILED! => {"changed": false, "elapsed": 30, "msg": "Timeout when waiting for 127.0.0.1:8090"}

PLAY RECAP *********************************************************************************************************************************************************************
dss-service                : ok=8    changed=7    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[root@dss-service dss-linkis-ansible]#

但是能登录成功，

DSS+Linkis.Ansible.-qualitis.mp4

1 reply

wubolive Dec 23, 2022
Author

可能只是端口探测超时了

linjie830914 · 2022-12-23T01:30:49Z

linjie830914
Dec 23, 2022

执行如下脚本后ansible-playbook playbooks/streamis.yml，有如下错误提示


fatal: [dss-service]: FAILED! => {"changed": true, "cmd": "source /etc/profile ~/.bashrc ; sh bin/start-streamis-server.sh", "delta": "0:00:02.054659", "end": "2022-12-23 09:16:44.236265", "msg": "non-zero return code", "rc": 1, "start": "2022-12-23 09:16:42.181606", "stderr": "", "stderr_lines": [], "stdout": "Streamis Server start succeeded!\nJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=500m; support was removed in 8.0", "stdout_lines": ["Streamis Server start succeeded!", "Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=500m; support was removed in 8.0"]}
...ignoring

打开系统后如报错，如截图所示：

DSS+Linkis.Ansible.-streamis.mp4

0 replies

linjie830914 · 2022-12-23T01:35:09Z

linjie830914
Dec 23, 2022

安装ansible-playbook playbooks/exchangis.yml时，报如下错误：


TASK [exchangis : 打印启动信息] ******************************************************************************************************************************************************
ok: [dss-service] => {
    "start_info.stdout_lines": [
        "2022-12-23 09:21:02.228 [INFO] (95377) load environment properties", 
        "2022-12-23 09:21:02.232 [INFO] (95377) Launcher: launch to start server [ exchangis-server ]", 
        "2022-12-23 09:21:02.342 [INFO] (95377) /usr/local/java/bin/java", 
        "2022-12-23 09:21:02.347 [INFO] (95377) Launcher: waiting [ exchangis-server ] to start complete ...", 
        "2022-12-23 09:21:24.445 [ERROR] (95377) Launcher: [ exchangis-server ] start fail over 20 seconds, please retry it"
    ]
}

TASK [exchangis : 检查端口是否运行] ****************************************************************************************************************************************************
fatal: [dss-service]: FAILED! => {"changed": false, "elapsed": 30, "msg": "Timeout when waiting for 127.0.0.1:9321"}

PLAY RECAP *********************************************************************************************************************************************************************
dss-service                : ok=21   changed=17   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[root@dss-service dss-linkis-ansible]# 
http://192.168.32.251:9321和Exchangis: http://192.168.32.251:8028/这俩地址都打不开。

DSS+Linkis.Ansible.-exchangis.mp4

0 replies

linkagt · 2022-12-25T15:03:23Z

linkagt
Dec 25, 2022

请教个问题，部署成功后，使用都没问题，问个小白问题，重启后，所有服务都起不来怎么处理

4 replies

wubolive Dec 26, 2022
Author

请教个问题，部署成功后，使用都没问题，问个小白问题，重启后，所有服务都起不来怎么处理

需要依次启动mysql、nginx、hadoop、hive、spark、linkis、dss、dolphinscheduler、及其它，启动方式可参考roles目录中的tasks文件。

linkagt Dec 26, 2022

1\启动mysql
service mysqld start

2\启动hadoop
sh ./opt/hadoop/sbin/start-dfs.sh
sh ./opt/hadoop/sbin/start-yarn.sh

3\启动hive
nohup /opt/hive/bin/hive --service metastore >> /opt/hive/logs/metastore.log 2>&1 &
nohup /opt/hive/bin/hive --service hiveserver2 >> /opt/hive/logs/hiveserver2.log 2>&1 &

4\启动spark
sh ./opt/spark/sbin/start-all.sh

5\启动links
sh ./home/hadoop/linkis/sbin/linkis-start-all.sh

6\启动dss
sh ./home/hadoop/dss/sbin/dss-start-all.sh

7\启动dolphinscheduler
sh ./opt/dolphinscheduler/bin/start-all.sh

8、启动Visualis
sh ./opt/visualis-server/bin/start-visualis-server.sh

9、启动Qualitis
sh ./opt/qualitis/bin/start.sh、

10、启动Streamis
sh ./opt/streamis/streamis-server/bin/start-streamis-server.sh

11、启动sqoop
这个只是测试直接输入指令sqoop

12\启动Exchangis
没有操作

操作后，DSS登录报后端服务错误
因为对DSS不熟悉，请大神帮看下又没哪里有问题

所有操作都在根目录/下操作

安装后是可以正常使用，重启后，很多服务不会自动启动
按你的建议，以上是操作记录

wubolive Dec 27, 2022
Author

1\启动mysql service mysqld start

2\启动hadoop sh ./opt/hadoop/sbin/start-dfs.sh sh ./opt/hadoop/sbin/start-yarn.sh

3\启动hive nohup /opt/hive/bin/hive --service metastore >> /opt/hive/logs/metastore.log 2>&1 & nohup /opt/hive/bin/hive --service hiveserver2 >> /opt/hive/logs/hiveserver2.log 2>&1 &

4\启动spark sh ./opt/spark/sbin/start-all.sh

5\启动links sh ./home/hadoop/linkis/sbin/linkis-start-all.sh

6\启动dss sh ./home/hadoop/dss/sbin/dss-start-all.sh

7\启动dolphinscheduler sh ./opt/dolphinscheduler/bin/start-all.sh

8、启动Visualis sh ./opt/visualis-server/bin/start-visualis-server.sh

9、启动Qualitis sh ./opt/qualitis/bin/start.sh、

10、启动Streamis sh ./opt/streamis/streamis-server/bin/start-streamis-server.sh

11、启动sqoop 这个只是测试直接输入指令sqoop

12\启动Exchangis 没有操作

操作后，DSS登录报后端服务错误因为对DSS不熟悉，请大神帮看下又没哪里有问题

所有操作都在根目录/下操作

安装后是可以正常使用，重启后，很多服务不会自动启动按你的建议，以上是操作记录

DSS默认就不带自动重启的脚本，等我待会更新一版加一个重启服务的脚本吧。

wubolive Dec 27, 2022
Author

请教一个问题，部门成功后，使用都没有问题，问一个小白问题，重新启动后，所有服务都起不来怎么处理

刚刚更新了一下，下载v1.0.1版本部署文件，参考3.4步骤启动服务。

linkagt · 2022-12-28T09:32:14Z

linkagt
Dec 28, 2022

已经关闭了，部署前就关闭防火墙和 selinex

…

---原始邮件--- 发件人: ***@***.***> 发送时间: 2022年12月28日(周三) 下午4:13 收件人: ***@***.***>; 抄送: ***@***.******@***.***>; 主题: Re: [apache/incubator-linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) SELINUX是否关闭？临时关闭命令 setenforce 0 永久生效 vi /etc/selinux/config SELINUX=permissive — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

0 replies

15259236303 · 2022-12-30T04:47:37Z

15259236303
Dec 30, 2022

小白提个问题，再部署完成后，创建项目-创建工作流时，这个是哪里有问题，提示：
错误提示
Warning code( 警告码): 50032，Warningmessage(警告信息): Try to ask Linkis forcreating a new contextld failed(向Linkis请求创建一个 ContextID 失败)! Linkis error msg:HttpClientResultException:errCode: 10905desc.URL/api/rest_j/v1/contextservice/createContextlDrequestfailed!1ResponseBodyfmethod":"/api/rest_j/1/contextservice/createConSupplied VIP Address cannot be null","data":[data":null. errCode: 10905 ,desc: URL/api/rest_j/v1/contextservice/createContextlDrequestfailed!ResponseBodyIsmethod":"/api/rest j/1/contextservice/createConSupplied VIP Address cannot be null","data".data"null.,ip: dss-service ,port: 9003serviceKinddss-framework-orchestrator-dss-service ,port: 9003server-devip:serviceKind.dss-framework-orchestrator-server-dev.

11 replies

15259236303 Dec 30, 2022

服务都正常：

DSS-APISERVICE-SERVER | n/a (1) | (1) | UP (1) - dss-service:dss-apiservice-server:9004
DSS-FLOW-ENTRANCE | n/a (1) | (1) | UP (1) - dss-service:dss-flow-entrance:9006
DSS-FRAMEWORK-ORCHESTRATOR-SERVER-DEV | n/a (1) | (1) | UP (1) - dss-service:dss-framework-orchestrator-server-dev:9003
DSS-FRAMEWORK-PROJECT-SERVER | n/a (1) | (1) | UP (1) - dss-service:dss-framework-project-server:9002
DSS-GUIDE-SERVER | n/a (1) | (1) | UP (1) - dss-service:dss-guide-server:9210
DSS-SCRIPTIS-SERVER | n/a (1) | (1) | UP (1) - dss-service:dss-scriptis-server:9008
DSS-WORKFLOW-SERVER-DEV | n/a (1) | (1) | UP (1) - dss-service:dss-workflow-server-dev:9005
LINKIS-CG-ENGINECONNMANAGER | n/a (1) | (1) | UP (1) - dss-service:linkis-cg-engineconnmanager:9102
LINKIS-CG-ENGINEPLUGIN | n/a (1) | (1) | UP (1) - dss-service:linkis-cg-engineplugin:9103
LINKIS-CG-ENTRANCE | n/a (1) | (1) | UP (1) - dss-service:linkis-cg-entrance:9104
LINKIS-CG-LINKISMANAGER | n/a (1) | (1) | UP (1) - dss-service:linkis-cg-linkismanager:9101
LINKIS-MG-EUREKA | n/a (1) | (1) | UP (1) - dss-service:linkis-mg-eureka:20303
LINKIS-MG-GATEWAY | n/a (1) | (1) | UP (1) - dss-service:linkis-mg-gateway:9001
LINKIS-PS-PUBLICSERVICE | n/a (1) | (1) | UP (1) - dss-service:linkis-ps-publicservice:9105

15259236303 Dec 30, 2022

这个该查看linkis那个日志？

wubolive Dec 30, 2022
Author

这个该查看linkis那个日志？
先tail日志，然后再到控制台操作，查看日志输出。

$ su - hadoop
$ tail -f ~/linkis/logs/*.log ~/dss/logs/*.log

auserwn Feb 15, 2023

亲，你这个问题解决了吗，怎么解决的

SpiderPalace Mar 25, 2023

这个还有后续吗

linkagt · 2023-02-06T06:11:04Z

linkagt
Feb 6, 2023

我的也是有这个问题，在别的群里问，有人说是load问题

…

---原始邮件--- 发件人: ***@***.***> 发送时间: 2023年2月6日(周一) 下午2:09 收件人: ***@***.***>; 抄送: ***@***.******@***.***>; 主题: Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 请问一键部署后，想创建一个新用户来登陆，按照dss官网创建了该用户的工作空间以及HDFS空间，但是还是报错了，请问大家创建新用户都正常创建吗？，谢谢。 dss官方创建用户文档用户创建及相关空间创建还是报错o(╥﹏╥)o — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

0 replies

15259236303 · 2023-02-06T08:04:12Z

15259236303
Feb 6, 2023

群能共分享下吗，或者能告诉下群号吗？也想加一下问下。︶ㄣ︸︺ ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: "apache/linkis" ***@***.***>; 发送时间: 2023年2月6日(星期一) 下午2:11 ***@***.***>; ***@***.******@***.***>; 主题: Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 我的也是有这个问题，在别的群里问，有人说是load问题

---原始邮件--- 发件人: ***@***.***&gt; 发送时间: 2023年2月6日(周一) 下午2:09 收件人: ***@***.***&gt;; 抄送: ***@***.******@***.***&gt;; 主题: Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 请问一键部署后，想创建一个新用户来登陆，按照dss官网创建了该用户的工作空间以及HDFS空间，但是还是报错了，请问大家创建新用户都正常创建吗？，谢谢。 dss官方创建用户文档用户创建及相关空间创建还是报错o(╥﹏╥)o — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt; — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

1 reply

wubolive Feb 8, 2023
Author

群能共分享下吗，或者能告诉下群号吗？也想加一下问下。︶ㄣ︸︺ @.***
…
------------------ 原始邮件 ------------------ 发件人: "apache/linkis" @.>; 发送时间: 2023年2月6日(星期一) 下午2:11 @.>; @.@.>; 主题: Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 我的也是有这个问题，在别的群里问，有人说是load问题
---原始邮件--- 发件人: @.> 发送时间: 2023年2月6日(周一) 下午2:09 收件人: @.>; 抄送: @.@.>; 主题: Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 请问一键部署后，想创建一个新用户来登陆，按照dss官网创建了该用户的工作空间以及HDFS空间，但是还是报错了，请问大家创建新用户都正常创建吗？，谢谢。 dss官方创建用户文档用户创建及相关空间创建还是报错o(╥﹏╥)o — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

你加 wedatasphere 这个人微信，让他拉你进群

auserwn · 2023-02-07T07:36:50Z

auserwn
Feb 7, 2023

这是啥问题，有老哥遇到过吗

2 replies

wubolive Feb 8, 2023
Author

这是啥问题，有老哥遇到过吗

linkis没起来，执行ansible-playbook playbooks/all.yml --start-at="启动linkis服务"重试看看

wangbinhk Feb 21, 2023

我也遇到这个问题，重启playbook同样错误，运行 linkis-stop-all.sh也不行，检查进程发现已经启动但pid不对，需要kill后重新运行剧本。
ps -ef | grep linkis 查出进程kill
然后重启剧本

lkm0522 · 2023-02-08T07:16:42Z

lkm0522
Feb 8, 2023

fatal: [dss-service]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "refusing to convert from file to symlink for /usr/bin/pip3", "owner": "root", "path": "/usr/bin/pip3", "size": 407, "state": "file", "uid": 0}
这个问题是需要提前删除一下/usr/bin/pip软链吗

1 reply

wubolive Feb 9, 2023
Author

fatal: [dss-service]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "refusing to convert from file to symlink for /usr/bin/pip3", "owner": "root", "path": "/usr/bin/pip3", "size": 407, "state": "file", "uid": 0} 这个问题是需要提前删除一下/usr/bin/pip软链吗

不是全新系统吗？可以删除后加上--start-at="当前步骤"重试

lkm0522 · 2023-02-08T07:57:17Z

lkm0522
Feb 8, 2023

大佬，安装启动linkis服务失败，这个是什么问题

3 replies

auserwn Feb 8, 2023

我也遇到这个了，方便加个联系方式吗

wubolive Feb 9, 2023
Author

大佬，安装启动linkis服务失败，这个是什么问题

linkis没起来，执行ansible-playbook playbooks/all.yml --start-at="启动linkis服务"重试看看，如果还不行就看下/home/hadoop/linkis下的日志

wangbinhk Feb 21, 2023

可以试下 ps -ef | grep linkis 查出进程kill
然后重启剧本

auserwn · 2023-02-08T09:21:31Z

auserwn
Feb 8, 2023

/opt/dss/bin/start-all.sh启动完服务访问8085报这个错误是为啥子，请教大佬

4 replies

wubolive Feb 9, 2023
Author

/opt/dss/bin/start-all.sh启动完服务访问8085报这个错误是为啥子，请教大佬

服务都起来了吗？Eureka上注册了几个服务

auserwn Feb 10, 2023

这么些服务，有没起来的吗，重启好几次都是这样，是不是服务器不稳定导致的，我用的虚拟机 8G内存

auserwn Feb 10, 2023

DSS-FRAMEWORK-PROJECT-SERVER这个服务没启动成功，大佬这个服务的启动在哪里

auserwn Feb 21, 2023

内存太小了，服务都掉了，建议大家安装之前保证内存足够用

QuantumXiecao · 2023-02-11T05:31:41Z

QuantumXiecao
Feb 11, 2023

If someone encounters the following problem:
Let me ask you a question：After deployment, the workflow was created incorrectly.
Error Message:
Warning code: 50032，Warning message: Try to ask Linkis forcreating a new contextld failed Linkis error msg:HttpClientResultException:errCode: 10905desc.URL/api/rest_j/v1/contextservice/createContextlDrequestfailed!1ResponseBodyfmethod":"/api/rest_j/1/contextservice/createConSupplied VIP Address cannot be null","data":[data":null. errCode: 10905 ,desc: URL/api/rest_j/v1/contextservice/createContextlDrequestfailed!ResponseBodyIsmethod":"/api/rest j/1/contextservice/createConSupplied VIP Address cannot be null","data".data"null.,ip: dss-service ,port: 9003serviceKinddss-framework-orchestrator-dss-service ,port: 9003server-devip:serviceKind.dss-framework-orchestrator-server-dev.

You can find solutions according to: #4167

0 replies

wangbinhk · 2023-02-21T03:59:12Z

wangbinhk
Feb 21, 2023

分享个问题处理，DSS-FRAMEWORK-PROJECT-SERVER这个服务没启动成功，
检查是 /home/hadoop/dss/conf/application-dss.yml 文件中的 defaultZone 的Host配置错误：两个ip叠加，
检查ifconfig 发现有个 virbr0虚拟网卡，网上查到是图形化centos7带来的，需要删除
参考：https://blog.csdn.net/qq_41934760/article/details/113529846

3 replies

wangbinhk Feb 21, 2023

dss.yml文件也需要修改

wubolive Feb 21, 2023
Author

仔细看过注意事项就不会出现这个问题了

wangbinhk Feb 21, 2023

是的，最后一条没仔细看...大佬的剧本给力

lkm0522 · 2023-03-13T07:57:52Z

lkm0522
Mar 13, 2023

你好，请问一下，部署所有组件ansible-playbook playbooks/all.yml，每次基本上都是这个启动links报错，都是卡在这一步了，每次重试一下就又好了。请问这个怎么解决呢，我有个工作流步骤，一键部署需求是一次性部署成功。
根据ansible报错日志去找的日志文件报错如下：

配置和服务：

1 reply

wubolive Mar 20, 2023
Author

你好，请问一下，部署所有组件ansible-playbook playbooks/all.yml，每次基本上都是这个启动links报错，都是卡在这一步了，每次重试一下就又好了。请问这个怎么解决呢，我有个工作流步骤，一键部署需求是一次性部署成功。根据ansible报错日志去找的日志文件报错如下：

配置和服务：

查看您的linkis日志，看看启动失败的原因

$ su - hadoop
$ tail -f ~/linkis/logs/*.log ~/dss/logs/*.log

jason7805 · 2023-03-15T05:32:00Z

jason7805
Mar 15, 2023

Caused by: com.webank.wedatasphere.dss.common.exception.DSSRuntimeException: errCode: 100000 ,desc: UnknownHostException: 192.168.7.176172.18.0.1172.19.0.1172.20.0.1172.17.0.1172.22.0.1: Name or service not known ,ip: dss-service ,port: 9002 ,serviceKind: dss-framework-project-server
at com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils.lambda$handling$0(DSSExceptionUtils.java:43) ~[dss-common-1.1.1.jar:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_171]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_171]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_171]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_171]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_171]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_171]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_171]
at com.webank.wedatasphere.dss.framework.appconn.restful.AppConnManagerRestfulApi.init(AppConnManagerRestfulApi.java:66) ~[dss-appconn-framework-1.1.1.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
这是hosts文件配置问题？还是EK配置问题呢?

1 reply

wubolive Mar 20, 2023
Author

Caused by: com.webank.wedatasphere.dss.common.exception.DSSRuntimeException: errCode: 100000 ,desc: UnknownHostException: 192.168.7.176172.18.0.1172.19.0.1172.20.0.1172.17.0.1172.22.0.1: Name or service not known ,ip: dss-service ,port: 9002 ,serviceKind: dss-framework-project-server at com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils.lambda$handling$0(DSSExceptionUtils.java:43) ~[dss-common-1.1.1.jar:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_171] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_171] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_171] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_171] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_171] at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_171] at com.webank.wedatasphere.dss.framework.appconn.restful.AppConnManagerRestfulApi.init(AppConnManagerRestfulApi.java:66) ~[dss-appconn-framework-1.1.1.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] 这是hosts文件配置问题？还是EK配置问题呢?

应该是你服务器存在多个IP，请仔细查看部署前注意事项

zmj-upupup · 2023-04-18T16:44:03Z

zmj-upupup
Apr 18, 2023

大佬，请问个问题，部署完成后执行sql报错

2023-04-19 00:41:12.041 INFO Background is starting a new engine for you,execId TaskID_5_otJobId_astJob_4_codeExec_4 mark id is mark_4, it may take several seconds, please wait
2023-04-19 00:41:13.041 ERROR Job with execId-IDE_hadoop_spark_4 + subJobId : 5 execute failed,21304, Task is Failed,errorMsg: errCode: 12003 ,desc: dss-service:9101_4 Failed to async get EngineNode AMErrorException: errCode: 30001 ,desc: Failed to createEngine ask em ServiceInstance(linkis-cg-engineconnmanager, dss-service:9102) ,ip: dss-service ,port: 9101 ,serviceKind: linkis-cg-linkismanager ,ip: dss-service ,port: 9104 ,serviceKind: linkis-cg-entrance

0 replies

cutejun · 2023-05-09T07:50:36Z

cutejun
May 9, 2023

大佬麻烦问一下，这个hadoop初始化卡了，半小时这是为啥

0 replies

Sunny5566 · 2023-07-31T10:08:05Z

Sunny5566
Jul 31, 2023

一键部署成功后，在eureka上发现linkis的服务少了三个。LINKIS-PS-CS、LINKIS-PS-DATA-SOURCE-MANAGER和LINKIS-PS-METADATAMANAGER

0 replies

casionone · 2023-08-01T02:20:24Z

casionone
Aug 1, 2023
Collaborator

These three services have been merged in a certain version and should be merged in version 1.3.0. See website's release note docs https://linkis.apache.org for more information.---- 回复的原邮件 ***@***.***>发送日期2023年07月31日 18:08 ***@***.***> ***@***.***>主题Re: [apache/linkis] 【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 (Discussion #3944) 一键部署成功后，在eureka上发现linkis的服务少了三个。LINKIS-PS-CS、LINKIS-PS-DATA-SOURCE-MANAGER和LINKIS-PS-METADATAMANAGER —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#3944 (comment)", "url": "#3944 (comment)", "name": "View Discussion" }, "description": "View this Discussion on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

0 replies

cg0813 · 2023-08-30T08:08:58Z

cg0813
Aug 30, 2023

e schedulis. com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException: errCode: 50032 ,desc: cannot find the matched SchedulerAppConn with name schedulis ,ip: dss-service ,port: 9003 ,serviceKind: dss-framework-orchestrator-server-dev
at com.webank.wedatasphere.dss.orchestrator.loader.DefaultOrchestratorLoader.lambda$loadOrchestrator$2(DefaultOrchestratorLoader.java:73) ~[dss-orchestrator-loader-1.1.1.jar:?]
at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_171]
at com.webank.wedatasphere.dss.orchestrator.loader.DefaultOrchestratorLoader.loadOrchestrator(DefaultOrchestratorLoader.java:73) ~[dss-orchestrator-loader-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.loader.OrchestratorManager.getOrCreateOrchestrator(OrchestratorManager.java:47) ~[dss-orchestrator-loader-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.server.service.impl.OrchestratorFrameworkServiceImpl.getOrchestrationService(OrchestratorFrameworkServiceImpl.java:246) ~[dss-framework-orchestrator-server-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.server.service.impl.OrchestratorFrameworkServiceImpl.tryOrchestrationOperation(OrchestratorFrameworkServiceImpl.java:148) ~[dss-framework-orchestrator-server-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.server.service.impl.OrchestratorFrameworkServiceImpl.createOrchestrator(OrchestratorFrameworkServiceImpl.java:124) ~[dss-framework-orchestrator-server-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.server.restful.DSSFrameworkOrchestratorRestful.createOrchestrator(DSSFrameworkOrchestratorRestful.java:77) ~[dss-framework-orchestrator-server-1.1.1.jar:?]
at com.webank.wedatasphere.dss.orchestrator.server.restful.DSSFrameworkOrchestratorRestful$$FastClassBySpringCGLIB$$949bc73d.invoke() ~[dss-framework-orchestrator-server-1.1.1.jar:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.apache.linkis.server.restful.SpringRestfulCatchAOP$$anonfun$1.apply(SpringRestfulCatchAOP.scala:39) ~[linkis-module-1.1.1.jar:1.1.1]
at org.apache.linkis.server.restful.SpringRestfulCatchAOP$$anonfun$1.apply(SpringRestfulCatchAOP.scala:39) ~[linkis-module-1.1.1.jar:1.1.1]
at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
at org.apache.linkis.server.package$.catchMsg(package.scala:56) ~[linkis-module-1.1.1.jar:1.1.1]
at org.apache.linkis.server.package$.catchIt(package.scala:88) ~[linkis-module-1.1.1.jar:1.1.1]

大佬们，这个问题有知道的吗

1 reply

Suphx Jan 5, 2024

我也遇到这个问题了

【安装部署】Linkis1.3.0+DSS1.1.1 Ansible 单机一键安装脚本 #3944

DSS+Linkis Ansible一键安装脚本

一、简介

1.1 版本介绍

二、部署前注意事项

三、部署方法

3.1 安装前设置

3.2 部署linkis+dss

3.3 部署其它服务

3.4 维护指南

Replies: 27 comments · 46 replies

brucedchen1991 Dec 13, 2022 Collaborator

brucedchen1991 Dec 13, 2022 Collaborator

wubolive Dec 22, 2022 Author

brucedchen1991 Dec 13, 2022 Collaborator

wubolive Dec 26, 2022 Author

wubolive Dec 23, 2022 Author

wubolive Dec 23, 2022 Author

wubolive Dec 26, 2022 Author

wubolive Dec 27, 2022 Author

wubolive Dec 27, 2022 Author

wubolive Dec 30, 2022 Author

wubolive Feb 8, 2023 Author

wubolive Feb 8, 2023 Author

wubolive Feb 9, 2023 Author

wubolive Feb 9, 2023 Author

wubolive Feb 9, 2023 Author

wubolive Feb 21, 2023 Author

wubolive Mar 20, 2023 Author

Replies: 27 comments 46 replies

brucedchen1991
Dec 13, 2022
Collaborator

brucedchen1991
Dec 13, 2022
Collaborator

wubolive Dec 22, 2022
Author

brucedchen1991
Dec 13, 2022
Collaborator

wubolive Dec 26, 2022
Author

wubolive Dec 23, 2022
Author

wubolive Dec 23, 2022
Author

wubolive Dec 26, 2022
Author

wubolive Dec 27, 2022
Author

wubolive Dec 27, 2022
Author

wubolive Dec 30, 2022
Author

wubolive Feb 8, 2023
Author

wubolive Feb 8, 2023
Author

wubolive Feb 9, 2023
Author

wubolive Feb 9, 2023
Author

wubolive Feb 9, 2023
Author

wubolive Feb 21, 2023
Author

wubolive Mar 20, 2023
Author