Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iter8 controller doesn't work #1644

Open
paloma-rebuelta opened this issue Sep 22, 2023 · 1 comment
Open

Iter8 controller doesn't work #1644

paloma-rebuelta opened this issue Sep 22, 2023 · 1 comment
Labels
kind/bug Something isn't working

Comments

@paloma-rebuelta
Copy link

Describe the bug

I am working on the mirroring trafic use case for modelmesh. I have deployed the traffic controller v0.17.1 as there is no mirroring usecase in the iter8 docs for later versions.
The controller resources get created, the pod spins up, but I get error logs relating to the metrics and A/B?n service and BadgerDB:

time=2023-09-22 12:03:02 level=error msg=cannot read config file: /config/metrics.yaml stack-trace=below ... 
::Trace:: open /config/metrics.yaml: no such file or directory
time=2023-09-22 12:03:02 level=error msg=unable to read metrics configuration: cannot read config file: /config/metrics.yaml
time=2023-09-22 12:03:02 level=error msg=count not start A/B/n metrics service
time=2023-09-22 12:03:02 level=error msg=Unable to configure metrics storage client cannot open BadgerDB
time=2023-09-22 12:03:02 level=error msg=cound not start A/B/n service

This doesn't kill the pod, its status is "Running".
When I deploy a candidate model and use the mirroring traffic template, the virtual service remains intact, it does not get altered based on the weights configmaps.

To Reproduce
the controller was deployed using this kustomization file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- 'https://github.com/iter8-tools/iter8.git/kustomize/controller/clusterScoped?ref=v0.17.1'

namespace: iter8

patches:
- patch: |-
    - op: replace
      path: /data/config.yaml
      value: |
        clusterScoped: true
        defaultResync: 15m
        image: iter8/iter8:0.17
        logLevel: debug
        resourceTypes:
          cm:
            Group: ""
            Resource: configmaps
            Version: v1
          deploy:
            Group: apps
            Resource: deployments
            Version: v1
            conditions:
            - name: Available
              status: "True"
          isvc:
            Group: serving.kserve.io
            Resource: inferenceservices
            Version: v1beta1
            conditions:
            - name: Ready
              status: "True"
          svc:
            Group: ""
            Resource: services
            Version: v1
          vs:
            Group: networking.istio.io
            Resource: virtualservices
            Version: v1alpha3
        storageClassName: ""
  target:
    kind: ConfigMap
    name: iter8

- patch: |-
    - op: replace
      path: /spec/storageClassName
      value: default
  target:
    kind: PersistentVolumeClaim
    name: iter8

Expected behavior
When I deploy a candidate model and the mirroring traffic templates, I would expect traffic to be mirrored to the second model. This does not happen as the virtual service does not get updated with the weights.

Desktop (please complete the following information):

  • OS: MacOS but running in kubernetes
  • Output of the iter8 version command: irrelevant as using controller

Additional context
Add any other context about the problem here.

@paloma-rebuelta paloma-rebuelta added the kind/bug Something isn't working label Sep 22, 2023
@kalantar
Copy link
Member

@paloma-rebuelta I was not able to replicate your problem using your kustomization file (I removed the pvc patch because I could not create a pv with this storageClassName -- I used a kind cluster). The controller started without any errors for me. Inspection of the error suggests the config file could not be found. The pod configuration (kubectl -n iter8 get po -o yaml) should show that comes from the configmap iter8.

That said, mirroring was not available in this version. Inspection of the helm chart shows that it was stubbed out. No tutorials were provided. We would appreciate a contribution of this capacity :). In the latest version of Iter8, the templates of the release chart would need to be updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants