feat: support nacos storage #796

sjcsjc123 · 2024-01-22T14:21:28Z

Ⅰ. Describe what this PR did

support nacos storage

Ⅱ. Does this pull request fix one issue?

fixes: #642

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Makefile.core.mk

docker/Dockerfile.higress

helm/core/templates/controller-service.yaml

CH3CHO · 2024-01-22T14:30:32Z

helm/core/values.yaml

+## Higress ApiServer Storage Settings
+## Todo simplify this
+apiserver:
+  addr: 127.0.0.1


Would we allow user to use another address here?

If run api server in other pod, we can modify this. If not, we can remove it. It can be fixed.

按照目前的安装方式来看，是否需要移除这个配置，把apiserver固定为higress-controller的其中一个container

helm/core/values.yaml

CH3CHO · 2024-01-22T14:32:24Z

helm/core/values.yaml

+  imagePullPolicy: IfNotPresent
+  securePort: 8443
+  storage: nacos
+  serverAddr: http://127.0.0.1:8848


Is it possible that we have a Nacos running in http://127.0.0.1:8848?

This parameter is used to enable users to specify a nacos address.

这个默认值如果用户不修改的话，安装肯定会失败吧？具体表现是什么？

新增helm参数来控制是否替用户安装nacos，这个地址就修改为替用户安装nacos的时候的nacos地址，这样可以吗

CH3CHO · 2024-01-22T14:34:36Z

tools/hack/gen-keys.sh

+  fi
+}
+
+initializeApiServer


When user installs Higress with API Server enabled in his own K8s cluster, how could API server get these keys and certificates?

I have tried to incorporate this script into the helm installation process. Not successful.

how about making a new keys folder under helm/core dir and copying the certificates and keys generated by gen-keys.sh to the dir, and using .Files.Get from keys dir? and so there is no need to generate certificates and keys during installation

Signed-off-by: sjcsjc123 <1401189096@qq.com>

codecov-commenter · 2024-01-23T04:59:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.11%. Comparing base (b4f72d3) to head (b306f61).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #796      +/-   ##
==========================================
- Coverage   38.14%   38.11%   -0.03%     
==========================================
  Files          61       61              
  Lines       10442    10442              
==========================================
- Hits         3983     3980       -3     
- Misses       6160     6162       +2     
- Partials      299      300       +1

see 1 file with indirect coverage changes

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO · 2024-01-23T08:26:16Z

helm/core/templates/configmap.yaml

@@ -110,7 +110,7 @@ data:
    apiVersion: v1
    kind: Config
    clusters:
-      - name: {{ .Release.Namespace }}
+      - name: {{ .Values.clusterName | default "higress" }}


这句的目的是啥，为什么不能用 Namespace 呢？

这里指的是k8s cluster的name，在make create-cluster中会创建名为higress的k8s集群，这里用higress-system会无法访问apiserver

CH3CHO · 2024-01-23T08:26:44Z

test/e2e/conformance/base/nacos.yaml

@@ -31,9 +31,10 @@ metadata:
 spec:
  containers:
  - name: nacos-standlone-rc3
-    image: registry.cn-hangzhou.aliyuncs.com/hinsteny/nacos-standlone-rc3:1.0.0-RC3 
+    image: swsk33/nacos-standalone:2.2.3


这里可以考虑用官方镜像吗？

官方镜像依赖于数据源，registry.cn-hangzhou.aliyuncs.com是否有2.x版本的nacos

我的意思是，是否可以可以用 nacos 发布的官方镜像

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Makefile.core.mk

tools/hack/gen-secret-configmap.sh

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO · 2024-01-24T02:40:30Z

helm/core/templates/controller-deployment.yaml

+          - configMap:
+              name: higress-config
+          - configMap:
+              name: higress-apiserver


鉴于 gen-secret-configmap.sh 里是允许自定义这个 ConfigMap 的名字的，那这里挂载用的名字需要支持自定义吗？

gen-secret-configmap.sh那边会改成固定的，同higress-config一样

Signed-off-by: sjcsjc123 <1401189096@qq.com>

docker/run-higress.sh

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 · 2024-01-24T09:37:20Z

2024-01-24T09:33:53.978099Z	error	ingress	Fallback service higress-conformance-infra/infra-backend-v1 within ingress higress-conformance-infra/higress-conformance-infra-default-backend is not found
2024-01-24T09:33:53.978130Z	info	ingress	resource type networking.istio.io/v1alpha3/DestinationRule, configs number 1
2024-01-24T09:33:53.978147Z	info	ingress	controller tracing ConstructEnvoyFilters
2024-01-24T09:33:53.978155Z	info	ingress	controller gzip ConstructEnvoyFilters
2024-01-24T09:33:53.978163Z	info	ingress	controller global-option ConstructEnvoyFilters
2024-01-24T09:33:53.978537Z	info	ingress	Append 1 configmap EnvoyFilters
2024-01-24T09:33:53.978553Z	info	ingress	resource type networking.istio.io/v1alpha3/EnvoyFilter, configs number 1
2024-01-24T09:33:53.978637Z	error	ingress	Fallback service higress-conformance-infra/infra-backend-v1 within ingress higress-conformance-infra/higress-conformance-infra-default-backend is not found

看起来是base目录下的service部分也需要通过nacos发布一下

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Makefile.core.mk

CH3CHO · 2024-02-21T02:42:49Z

helm/core/templates/controller-deployment.yaml

+          - --{{ .Values.apiserver.storage }}-ns-id
+          - {{ .Values.apiserver.namespaceID }}
+          - --{{ .Values.apiserver.storage }}-encryption-key-file
+          - /etc/api/nacos.key


storage 如果不是 nacos 的话，是没有以上这些参数的。这里是否要考虑开闭原则？

不太清楚开闭原则在这里的用法，有什么改进的建议吗

CH3CHO · 2024-02-21T02:43:41Z

helm/core/templates/controller-deployment.yaml

+              name: cert-config
+              readOnly: true
+            - mountPath: /tmp/nacos
+              name: nacos-data


下面叫{{ .Values.apiserver.storage }}-data，这里是nacos-data。同样建议考虑一下开闭原则。

CH3CHO · 2024-02-21T02:45:55Z

test/e2e/conformance/base/nacos.yaml

@@ -31,9 +31,10 @@ metadata:
 spec:
  containers:
  - name: nacos-standlone-rc3
-    image: registry.cn-hangzhou.aliyuncs.com/hinsteny/nacos-standlone-rc3:1.0.0-RC3 
+    image: swsk33/nacos-standalone:2.2.3


我的意思是，是否可以可以用 nacos 发布的官方镜像

CH3CHO · 2024-02-21T02:46:41Z

helm/core/values.yaml

+  imagePullPolicy: IfNotPresent
+  securePort: 8443
+  storage: nacos
+  serverAddr: http://127.0.0.1:8848


这个默认值如果用户不修改的话，安装肯定会失败吧？具体表现是什么？

CH3CHO · 2024-02-21T02:47:23Z

test/e2e/conformance/base/nacos.yaml

@@ -47,3 +48,5 @@ spec:
  ports:
  - name: foo # name is not required for single-port Services


现在已经不是单端口服务了，name 还是好好取一个吧

test/e2e/conformance/utils/kubernetes/apply.go

test/e2e/conformance/utils/suite/suite.go

CH3CHO · 2024-02-21T02:51:02Z

tools/hack/conf/nacos.yaml

+    name: nacos
+  clusterIP: None
+  ports:
+    - name: foo # name is not required for single-port Services


johnlanni · 2024-02-21T03:03:47Z

@sjcsjc123 docker.io/swsk33/nacos-standalone:2.2.3 这个镜像变成私有的了，看是否换成nacos官方镜像，参考下：https://github.com/nacos-group/nacos-k8s/tree/master/helm

这个默认也是standalone模式

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 · 2024-02-29T09:00:17Z

@sjcsjc123 docker.io/swsk33/nacos-standalone:2.2.3 这个镜像变成私有的了，看是否换成nacos官方镜像，参考下：https://github.com/nacos-group/nacos-k8s/tree/master/helm

这个默认也是standalone模式

目前github ci上的磁盘空间好像不够用了，我把e2e测试用的base/nacos.yaml也给换成了官方镜像，以节省磁盘开销

johnlanni · 2024-02-29T09:34:08Z

higress/.github/workflows/build-and-test-plugin.yaml

Lines 36 to 44 in 9c112a0

    
           - name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧 
        
             uses: jlumbroso/free-disk-space@main 
        
             with: 
        
               tool-cache: false 
        
               android: true 
        
               dotnet: true 
        
               haskell: true 
        
               large-packages: true 
        
               swap-storage: true

@sjcsjc123 可以配置这个action释放一些磁盘空间

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO · 2024-03-13T06:29:43Z

helm/core/templates/nacos-service.yaml

+      protocol: TCP
+      {{- if eq .Values.nacos.service.type "NodePort" }}
+      nodePort: {{ .Values.nacos.service.nodePort }}
+  {{- end }}


这里格式处理一下吧。另外，什么情况下需要用到 NodePort 呢，还只有 7848？

CH3CHO · 2024-03-13T06:32:13Z

helm/core/values.yaml

+  service:
+    port: 8848
+    type: NodePort
+    nodePort: 30000


默认需要开NodePort吗？

不需要的，也可以改成clusterip，我给改一下

test/e2e/conformance/base/nacos.yaml

johnlanni · 2024-03-26T09:59:46Z

helm/core/values.yaml

+  imagePullPolicy: IfNotPresent
+  securePort: 8443
+  storage: nacos
+  serverAddr: http://higress-nacos-service.higress-system.svc.cluster.local:8848


serverAddr 建议改名为 storageAddr，否则容易误解。
此外能否不要让用户配置 http:// 这个协议头呢，在helpers里根据storage和storageAddr来生成最终的配置参数
@CH3CHO 一起看看

我觉得可以的。之前也考虑过能不能让 nacos 直接支持这种配置方式。短时间可以在生成配置的时候处理一下。

只让用户配置higress-nacos-service.higress-system.svc.cluster.local:8848这个storageAddr参数吗，通过helper.tpl给处理一下http://

或者直接让用户配置 nacos://higress-nacos-service.higress-system.svc.cluster.local:8848？这样 storage 也不需要单独配置了。

johnlanni · 2024-04-23T07:44:04Z

@sjcsjc123 这个PR还有空继续完善下吗

feat: support nacos storage

bd58388

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 requested review from johnlanni, Xunzhuo and 2456868764 as code owners January 22, 2024 14:21

recover something

625c61d

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO reviewed Jan 22, 2024

View reviewed changes

sjcsjc123 added 4 commits January 22, 2024 22:45

recover something

d848bb8

Signed-off-by: sjcsjc123 <1401189096@qq.com>

recover something

e8888cb

Signed-off-by: sjcsjc123 <1401189096@qq.com>

simple

d01a220

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix bug

d8bb8d3

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 added 2 commits January 23, 2024 14:16

modify nacos image

5aed7f4

Signed-off-by: sjcsjc123 <1401189096@qq.com>

update nacos image in Makefile

0ffcc30

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO reviewed Jan 23, 2024

View reviewed changes

sjcsjc123 added 3 commits January 23, 2024 16:39

add nacos client

c0c4e38

Signed-off-by: sjcsjc123 <1401189096@qq.com>

add ci for nacos storage

8eb8f5b

Signed-off-by: sjcsjc123 <1401189096@qq.com>

delete something

9bb4920

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO reviewed Jan 23, 2024

View reviewed changes

Makefile.core.mk Outdated Show resolved Hide resolved

tools/hack/gen-secret-configmap.sh Outdated Show resolved Hide resolved

tools/hack/gen-secret-configmap.sh Show resolved Hide resolved

sjcsjc123 added 3 commits January 24, 2024 00:25

fix bug with start fail

31ca385

Signed-off-by: sjcsjc123 <1401189096@qq.com>

rename make

b5fd8a7

Signed-off-by: sjcsjc123 <1401189096@qq.com>

opt import

85b8274

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO reviewed Jan 24, 2024

View reviewed changes

sjcsjc123 added 5 commits January 24, 2024 12:49

update e2e

370d386

Signed-off-by: sjcsjc123 <1401189096@qq.com>

opt import

01c08fe

Signed-off-by: sjcsjc123 <1401189096@qq.com>

recover something

06651de

Signed-off-by: sjcsjc123 <1401189096@qq.com>

modify default falg value

35c5db1

Signed-off-by: sjcsjc123 <1401189096@qq.com>

make ci print simple

9449862

Signed-off-by: sjcsjc123 <1401189096@qq.com>

2456868764 reviewed Jan 24, 2024

View reviewed changes

docker/run-higress.sh Outdated Show resolved Hide resolved

sjcsjc123 added 2 commits January 24, 2024 15:10

update apply

ecb2672

Signed-off-by: sjcsjc123 <1401189096@qq.com>

update apply

62c421a

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 and others added 9 commits January 24, 2024 17:48

fix ci

0715626

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix bug with wasm plugin

62cb1dd

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Merge branch 'main' into helm/nacos-storage

8ce8df9

Merge branch 'main' into helm/nacos-storage

cd3527e

fix ci

4b5cd88

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Merge branch 'main' into helm/nacos-storage

a3ecec7

fix ci

9349772

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix ci

a3b114b

Signed-off-by: sjcsjc123 <1401189096@qq.com>

Merge branch 'main' into helm/nacos-storage

41bd16a

johnlanni requested a review from CH3CHO February 20, 2024 03:40

Merge branch 'main' into helm/nacos-storage

598aaf0

CH3CHO reviewed Feb 21, 2024

View reviewed changes

sjcsjc123 and others added 4 commits February 28, 2024 13:09

Merge branch 'main' into helm/nacos-storage

a8af197

update nacos image

6cd64b4

Signed-off-by: sjcsjc123 <1401189096@qq.com>

update nacos image

5575512

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix ci

81810c0

Signed-off-by: sjcsjc123 <1401189096@qq.com>

sjcsjc123 added 3 commits February 29, 2024 17:47

fix ci

daba6c7

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix ci

32a6786

Signed-off-by: sjcsjc123 <1401189096@qq.com>

fix ci

0a425dd

Signed-off-by: sjcsjc123 <1401189096@qq.com>

CH3CHO reviewed Mar 13, 2024

View reviewed changes

johnlanni requested changes Mar 26, 2024

View reviewed changes

sjcsjc123 and others added 3 commits March 27, 2024 22:38

Merge branch 'main' into helm/nacos-storage

0f270ce

Merge branch 'main' into helm/nacos-storage

b306f61

Merge branch 'main' into helm/nacos-storage

688442b

johnlanni marked this pull request as draft April 23, 2024 08:16

		@@ -47,3 +48,5 @@ spec:
		ports:
		- name: foo # name is not required for single-port Services

feat: support nacos storage #796

Are you sure you want to change the base?

feat: support nacos storage #796

Conversation

sjcsjc123 commented Jan 22, 2024

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Choose a reason for hiding this comment

sjcsjc123 Jan 22, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjcsjc123 Jan 22, 2024 • edited

Choose a reason for hiding this comment

2456868764 Jan 23, 2024 • edited

Choose a reason for hiding this comment

codecov-commenter commented Jan 23, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjcsjc123 commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnlanni commented Feb 21, 2024

sjcsjc123 commented Feb 29, 2024

johnlanni commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnlanni Mar 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnlanni commented Apr 23, 2024

sjcsjc123 Jan 22, 2024 •

edited

sjcsjc123 Jan 22, 2024 •

edited

2456868764 Jan 23, 2024 •

edited

codecov-commenter commented Jan 23, 2024 •

edited

johnlanni Mar 26, 2024 •

edited