【安装部署】基于Kind部署linkis 1.3.0到Kubernetes

[jacktao007]

作者：志强兄
Github：jacktao007

背景介绍

社区的小伙伴们都在等待容器化版本，作为容器化的第一个版本1.3.0对linkis来说具有重大的意义，在此感谢克路兄、武康师兄、华金兄、和平兄、夏晨兄及社区贡献者们的幸苦付出，该版本基于kind实现了将linkis部署到kubernetes中，此外还单独制作了LDH的hadoop容器化发行版，在参与linkis1.3.0版本的测试过程中形成该文档，方便小伙伴们尽早体验Linkis容器化版本和快速开发调试Linkis。 

如何参与贡献？ https://linkis.apache.org/zh-CN/community/how-to-contribute

1. 依赖版本

kind github：https://github.com/kubernetes-sigs/kind

kind官网：kind.sigs.k8s.io/

版本:

kind 0.14.0
docker 20.10.17
centos 7.6
helm 3.x

注意：

1.确保组件依赖版本
2.kind是用docker容器模拟节点的 机器重启回来容器都变了 调度器就不工作了 这个是kind的limitation,官方文档有详细说明。

2.安装docker

2.1 安装教程

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
sudo yum makecache fast
sudo yum -y install docker-ce
systemctl start docker
systemctl enable docker

2.2 设置国内镜像

vi /etc/docker/daemon.json
{
"registry-mirrors": ["http://hub-mirror.c.163.com"],
"insecure-registries": ["https://registry.mydomain.com","http://hub-mirror.c.163.com"]
}

3. 安装kind

（1）手工下载kind二进制

https://github.com/kubernetes-sigs/kind/releases

（2）安装kind二进制

chmod +x ./kind
mv kind-linux-amd64 /usr/bin/kind

4. 安装linkis

0.下载或自行编译linkis1.3.0部署包

使用版本：dev1.3.0分支编译版本

apache-linkis-1.3.0-incubating-bin.tar.gz

1.建立目录

mkdir -p /opt/data/common/extendlib

2.拷贝驱动到 /opt/data/common/extendlib

curl https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar -o /opt/data/common/extendlib/[mysql-connector-java-8.0.28.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar)

3.重置kind（非必要）

./bin/install-linkis-to-kubernetes.sh reset

4.拉取镜像

./bin/install-linkis-to-kubernetes.sh pull -mghcr.dockerproxy.com

5.安装linkis到kind中

./bin/install-linkis-to-kubernetes.sh install -l -mghcr.dockerproxy.com

6.使用命令查看服务，等待全部启动成功

kubectl get pods -A

7.拷贝ldh环境到linkis

./helm/scripts/prepare-for-spark.sh

8.开启端口映射

./helm/scripts/remote-proxy.sh start

9.访问系统

linkis-web: http://10.0.2.101:8088/
eureka: http://10.0.2.101:20303/

10.进入容器

./helm/scripts/login-pod.sh cg-engineconnmanager
执行shell测试
sh ./bin/linkis-cli -engineType shell-1 -codeType shell -code "echo "hello" "  -submitUser hadoop -proxyUser hadoop
执行hive测试
sh ./bin/linkis-cli -engineType hive-2.3.3 -codeType hql -code "show tables"  -submitUser hadoop -proxyUser hadoop
执行spark测试
sh ./bin/linkis-cli -engineType spark-2.4.3 -codeType sql -code "show tables"  -submitUser hadoop -proxyUser hadoop
执行python测试
sh ./bin/linkis-cli -engineType python-python2 -codeType python -code "print(\"hello\")"  -submitUser hadoop -proxyUser hadoop  -confMap  python.version=python

附件：

1.最新脚本install-linkis-to-kubernetes.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#variable
WORK_DIR=`cd $(dirname $0); pwd -P`
ROOT_DIR=${WORK_DIR}/..
MIRRORS="ghcr.io"
TAG="latest"
COMMAND="pull-install"
DEBUG=false
WITH_LDH=false


debug(){
    if [ $DEBUG = true ]; then
        echo $(date "+%Y-%m-%d %H:%M:%S") "debug: "$1
    fi
}

info(){
    echo $(date "+%Y-%m-%d %H:%M:%S") "info: "$1
}


#help info
help() {
    echo "Command        Describe"
    echo "pull-install   pull and install linkis images"
    echo "install        pull linkis images"
    echo "pull           pull linkis images"
    echo "reset          delete the test-helm cluster of kind"
    echo "help           print help info"
    echo ""
    echo "Params         Describe"
    echo "-m             url (default:ghcr.io , eg: ghcr.dockerproxy.com)"
    echo "-d             print debug info"
    echo "-l             install linkis with ldh"
    echo "-t             tag name "
    echo ""
    echo "example:"
    echo "./install-linkis-to-kubernetes.sh pull                                  pull image with ghcr.io"
    echo "./install-linkis-to-kubernetes.sh -tlatest                              pull image with tag"
    echo "./install-linkis-to-kubernetes.sh pull -mghcr.dockerproxy.com           pull image with ghcr.dockerproxy.com or ghcr.nju.edu.cn"
    echo "./install-linkis-to-kubernetes.sh install -l -mghcr.dockerproxy.com     install linkis to kind and kubernetes with ldh"
    echo "./install-linkis-to-kubernetes.sh pull-install -mghcr.dockerproxy.com   pull image and install linkis to kind and kubernetes"
}

#pull the container image of the linkis
pull(){
    debug ${MIRRORS}/apache/incubator-linkis/linkis-ldh:${TAG}
    docker pull ${MIRRORS}/apache/incubator-linkis/linkis-ldh:${TAG}
    debug ${MIRRORS}/apache/incubator-linkis/linkis:${TAG}
    docker pull ${MIRRORS}/apache/incubator-linkis/linkis:${TAG}
    debug ${MIRRORS}/apache/incubator-linkis/linkis-web:${TAG}
    docker pull ${MIRRORS}/apache/incubator-linkis/linkis-web:${TAG}
}
#change the label
tag(){
    docker tag  ${MIRRORS}/apache/incubator-linkis/linkis:${TAG} linkis:dev
    docker tag  ${MIRRORS}/apache/incubator-linkis/linkis-web:${TAG} linkis-web:dev
    docker tag  ${MIRRORS}/apache/incubator-linkis/linkis-ldh:${TAG} linkis-ldh:dev
}
#create an image to carry mysql
make_linikis_image_with_mysql_jdbc(){
    ${ROOT_DIR}/docker/scripts/make-linikis-image-with-mysql-jdbc.sh
    docker tag linkis:with-jdbc linkis:dev
}
#creating a kind cluster
create_kind_cluster(){
    ${ROOT_DIR}/helm/scripts/create-kind-cluster.sh
}
#mysql installation
install_mysql(){
    ${ROOT_DIR}/helm/scripts/install-mysql.sh
}
#ldh installation
install_ldh(){
    ${ROOT_DIR}/helm/scripts/install-ldh.sh
}
#linkis installation
install_linkis(){
    if [ $WITH_LDH = true ];then
      debug with-ldh
      ${ROOT_DIR}/helm/scripts/install-charts-with-ldh.sh
    else
      debug linkis
      ${ROOT_DIR}/helm/scripts/install-linkis.sh
    fi
}
#display pods
display_pods(){
    kubectl get pods -A
}

install(){
    tag
    make_linikis_image_with_mysql_jdbc
    create_kind_cluster
    install_mysql
    install_ldh
    install_linkis
    display_pods
}

reset(){
    kind delete clusters test-helm
}


check_docker(){
    docker -v >> /dev/null 2>&1
    if [ $? -ne  0 ]; then
        echo "Docker is not installed！"
        exit 1
    fi
}

check_kind(){
    kind --version >> /dev/null 2>&1
    if [ $? -ne  0 ]; then
        echo "kind is not installed！"
        exit 1
    fi
}

check_kubectl(){
    kubectl >> /dev/null 2>&1
    if [ $? -ne  0 ]; then
        echo "kubectl is not installed！"
        exit 1
    fi
}

check_helm(){
    helm version >> /dev/null 2>&1
    if [ $? -ne  0 ]; then
        echo "helm is not installed！"
        exit 1
    fi
}


debug $WORK_DIR

#entrance to the program
main(){

    #environmental testing
    check_docker
    check_kind
    check_kubectl
    check_helm

    #argument parsing
    long_opts="debug,mirrors:"
    getopt_cmd=$(getopt -o dm:lt: -n $(basename $0) -- "$@") || \
                { echo -e "\nERROR: Getopt failed. Extra args\n"; exit 1;}

    eval set -- "$getopt_cmd"
    while true; do
        case "$1" in
            -d) DEBUG=true;;
            -m) MIRRORS=$2;;
            -l) WITH_LDH=true;;
            -t) TAG=$2;;
            --) shift; break;;
        esac
        shift
    done

    debug "params num:"$#

    #command parsing
    if [ $# -eq 0 ]; then
        COMMAND="pull-install"
    else
        COMMAND=$1
    fi

    debug "command is:"$COMMAND

    if [ $COMMAND = "pull-install" ]; then
        pull
        install
    fi

    if [ $COMMAND = "install" ]; then
        install
    fi

    if [ $COMMAND = "pull" ]; then
        pull
    fi

    if [ $COMMAND = "reset" ]; then
        reset
    fi

    if [ $COMMAND = "help" ]; then
        help
    fi
}

main $@

[dlimeng]

+1  I try to deploy 在 2022年10月15日 +0800 AM11:43，apache/incubator-linkis ***@***.***>，写道：

…

YourHeroBaby