help request: Not able to use custom plugin with composite architecture approach

[shantanu10]

Description

Hi team,

I am facing one issue where I am trying to use a custom plugin with apisix when deployed in composite architecture mode. I followed the approach mentioned here:
https://apisix.apache.org/blog/2023/10/18/ingress-apisix/#install-apisix-ingress-controller
https://apisix.apache.org/docs/ingress-controller/composite/

and to create a custom plugin I used the approach mentioned here:
https://apisix.apache.org/docs/ingress-controller/tutorials/using-custom-plugins/

After this when I tried to deployed the following apisix route resource, its not able to get synced. I checked the ingress controller logs and found these errors:

2024-02-24T21:53:05+08:00	error	apisix/apisix_route.go:379	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unknown plugin [custom-oauth]\n\n"}
2024-02-24T21:53:05+08:00	warn	apisix/apisix_route.go:481	sync ApisixRoute failed, will retry	{"object": {"Type":2,"Object":{"Key":"shared/bytebase-shared-test","OldObject":{"Object":{"metadata":{"name":"bytebase-shared-test","namespace":"shared","uid":"5653149f-27cc-4b12-aedc-59d81fd01d9f","resourceVersion":"9445672","generation":1,"creationTimestamp":"2024-02-23T09:52:43Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2\",\"kind\":\"ApisixRoute\",\"metadata\":{\"annotations\":{},\"name\":\"bytebase-shared-test\",\"namespace\":\"shared\"},\"spec\":{\"http\":[{\"backends\":[{\"serviceName\":\"bytebase-shared\",\"servicePort\":80,\"weight\":10}],\"match\":{\"hosts\":[\"bytebase-test.internal.saas.syfe.com\"],\"paths\":[\"/*\"]},\"name\":\"rule1\"}]}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2024-02-23T09:52:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2024-02-23T09:52:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:http":{}}}}]},"spec":{"http":[{"name":"rule1","match":{"paths":["/*"],"hosts":["bytebase-test.internal.saas.syfe.com"]},"backends":[{"serviceName":"bytebase-shared","servicePort":80,"weight":10}],"websocket":false,"authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{},"ldapAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}},"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unknown plugin [custom-oauth]\n\n"}

To verify whether custom plugin is actually loaded by apisix, I exec into the ingress-controller pod and ran the admin api for plugins (https://apisix.apache.org/docs/apisix/admin-api/#plugin-api). I can see that the plugin was there in the response. I also got the response for the plugins/{{plugin_id}} api for my custom plugin (custom-oauth).

So I can see that the error is only coming when ingress-controller is trying to sync the apisixRoute with apisix. Somehow it's not able to find the custom plugin. The issue is not happening when I am using any built-in plugin. I really need some help out here. I will be very grateful if you can help here. Thanks in advance.

Changes I made in the composite.yaml

apisix:
     enable_control: true
     enable_reuseport: true
     extra_lua_path: /usr/local/apisix/custom_plugins/?.lua   # change 1
... 
...

plugins:   
    ...
    ...
    - ext-plugin-post-resp           # priority: -4000
    - custom-oauth                   # change 2

...
...

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-apisix-composite-deployment
  namespace: ingress-apisix
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-apisix-composite-deployment
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-apisix-composite-deployment
    spec:
      volumes:
        - name: apisix-config-yaml-configmap
          configMap:
            name: apisix-gw-config.yaml
            defaultMode: 420
        - name: plugin-custom-oauth-config     # change 3
          configMap:
            name: custom-oauth-config
            defaultMode: 0744
    ...
    ...

    volumeMounts:
             - name: apisix-config-yaml-configmap
               mountPath: /usr/local/apisix/conf/config.yaml
               subPath: config.yaml
             - mountPath: /usr/local/apisix/custom_plugins/apisix/plugins/custom-oauth.lua  # change 4
               name: plugin-custom-oauth-config
               subPath: custom-oauth.lua


...
...

--- # Change 5
apiVersion: v1
data:
  custom-oauth.lua: |
    -- some required functionalities are provided by apisix.core
    local core = require("apisix.core")

    -- define the schema for the Plugin
    local schema = {
        type = "object",
        properties = {
            body = {
                description = "custom response to replace the Upstream response with.",
                type = "string"
            },
        },
        required = {"body"},
    }

    local plugin_name = "custom-oauth"

    local metadata_schema = {
        name = "custom-oauth",
        priority = "30000"
    }

    -- custom Plugins usually have priority between 1 and 99
    -- higher number = higher priority
    local _M = {
        version = 0.1,
        priority = 30000,
        name = plugin_name,
        schema = schema,
        metadata_schema = metadata_schema
    }

    -- verify the specification
    function _M.check_schema(conf)
        return core.schema.check(schema, conf)
    end

    -- run the Plugin in the access phase of the OpenResty lifecycle
    function _M.access(conf, ctx)
        return 200, conf.body
    end

    return _M
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: custom-oauth-config
  namespace: ingress-apisix

--- # Change 6
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: bytebase-shared-test
  namespace: shared
spec:
  http:
    - name: rule1
      match:
        hosts:
          - bytebase-test.internal.saas.syfe.com
        paths:
          - /*
      backends:
        - serviceName: bytebase-shared
          servicePort: 80
          weight: 10
      plugins:
        - name: "custom-oauth"
          enable: true
          config:
              body: "Hello from your custom Plugin!"

Environment

• APISIX version (run apisix version): 3.2.2
• Operating system (run uname -a):

       18:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  

• OpenResty / Nginx version (run openresty -V or nginx -V):

   nginx version: openresty/1.21.4.1
   built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
   built with OpenSSL 1.1.1s  1 Nov 2022
  

• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):

  {"etcdserver":"3.5.0","etcdcluster":"3.5.0"}
  

  (Its the version I got by hitting curl http://127.0.0.1:12379/version as its a mock etcd service)
• APISIX Dashboard version, if relevant: not using
• Plugin runner version, for issues related to plugin runners: not using
• LuaRocks version, for installation issues (run luarocks --version): not using

[Revolyssup]

@shantanu10 Thanks for reporting. I am checking this one out.

[Revolyssup]

@shantanu10 So your plugin is successfully loaded by APISIX and also it's logic is being executed properly right? So functionally you're all good but you're just getting this error in the ingress controller? Am I right?

[Revolyssup]

@shantanu10 Can you also provide some preceeding and trailing info/debug.. logs from this sync error in ingress controller?

[lucmichea]

@shantanu10 Hi! I was having a similar issue when working with the 'standalone' (composite deployment) of the ingress-controller.

We found out that this is due to the way the schemas (configuration validation) are loaded by the ingress-controller. Here is an approximate snapshot of the trail to follow (if interested):

1. Standalone Config Condition
2. Loading the controller (standalone)
3. Loading of schemas
4. In Memory Validation of schema (standalone)

Long story short, as a temporary fix what I did was create another mounted file for the apisix-schema.json from a configmap that contains the schemas (of all plugins). To do so, you need to:

1. Create a ConfigMap with all schemas

You can obtain all the schemas by calling an endpoint of the apisix container (even the schema of the custom plugins you created) from the composite deployment. To do so, you can port-forward on the deployment:

kubectl port-forward deployment/ingress-apisix-composte-deployment -n ingress-apisix 9090:9090

Then call the endpoint and save the result in a file:

curl http://localhost:9090/v1/schema > /tmp/schema.json

(for a bit of clarity on the result I would recommend | jq before saving it to the file)

Finally, apply the ConfigMap:

kubectl create configmap plugin-schema-json -n ingress-apisix --from-file=apisix-schema.json=/tmp/schema.json

2. Modify the composite deployment

For the sake of brevity I added # [...] to represent skipped sections.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-apisix-composite-deployment
  namespace: ingress-apisix
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-apisix-composite-deployment
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-apisix-composite-deployment
    spec:
      volumes:
        - name: apisix-config-yaml-configmap
          configMap:
            name: apisix-gw-config.yaml
       - name: plugin-schema-json
         configMap:
           name: plugin-schema-json
      containers:
        - livenessProbe:
            tcpSocket:
              port: 8080
            # [...]
          # [...]
          name: ingress-apisix
          image: apache/apisix-ingress-controller:1.7.0
          volumeMounts:
            - mountPath: /ingress-apisix/conf/apisix-schema.json
              name: plugin-schema-json
              subPath: apisix-schema.json
# [...]

---

Hope it helps! Looking forward to having a better solution coming from contributors 😸. I would propose using the ApiSix api in the standalone mode and not a static file to load all the schemas, the solution I provided which creates a ConfigMap containing the result of the call to ApiSix is a placeholder as in composite deployment the apisix-ingress-controller could call the api on localhost:9090 without any issue. (can be reproduced by shell into the ingress-controller container and running the following apk add curl && curl http://localhost:9090/v1/schema)

[shantanu10]

Thanks @lucmichea . This approach worked out!
@Revolyssup We can close this issue. Thanks a lot.

[lucmichea]

Glad I could help!

@Revolyssup if you could provide some feedback on the solution I provided long term, I would love to discuss further! I can open another thread / a draft MR to cover this.

[holdeneoneal]

Thank you for sharing these workarounds. We are adding custom-plugins as well that require us to extend the schema. Sadly this extension is making the configmap too large and so we have had to trim the list of plugins in the schema. It would be very helpful if plugin support were improved for standalone/composite architecture given that it is the recommend architecture for k8s deployments.

[lucmichea]

Hi @holdeneoneal

Yes, the schema.json file that you will obtain will be really large and so it is more difficult to apply it to the cluster.
From my experience, in general, the limiting factor to apply the configmap is the annotations containing the versioning of what you want to apply (kubectl.kubernetes.io/last-applied-configuration). This happens when you use kubectl apply -f <myfile>.yaml. That is why I decided in my previous answer to use kubectl create configmap ... as an alternative.

I usually use Helm for versioning and creating a chart containing the schema.json in a ConfigMap is doable.

If you are experiencing other issues concerning the size of the file I would be happy to take a look as I would be learning another limitation. Although, it would be quite out of subject for this thread.

On another note, I plan on trying to do a PR containing the full integration of custom-plugins in the standalone mode once I have a bit of time. As I didn't receive any feedback here.