Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky KFServing e2e tests with 404s #604

Open
yuzisun opened this issue Dec 16, 2019 · 8 comments · May be fixed by #3500
Open

Flaky KFServing e2e tests with 404s #604

yuzisun opened this issue Dec 16, 2019 · 8 comments · May be fixed by #3500

Comments

@yuzisun
Copy link
Member

yuzisun commented Dec 16, 2019

/kind bug

What steps did you take and what happened:

predictor/test_xgboost.py:52:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
common/utils.py:78: in predict
   probs = json.loads(response.content.decode('utf-8'))["predictions"]
/usr/local/lib/python3.6/json/__init__.py:354: in loads
   return _default_decoder.decode(s)
/usr/local/lib/python3.6/json/decoder.py:339: in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <json.decoder.JSONDecoder object at 0x7f59ac7cc160>, s = '', idx = 0
   def raw_decode(self, s, idx=0):
       """Decode a JSON document from ``s`` (a ``str`` beginning with
       a JSON document) and return a 2-tuple of the Python
       representation and the index in ``s`` where the document ended.

       This can be used to decode a JSON document from a string that may
       have extraneous data at the end.

       """
       try:
           obj, end = self.scan_once(s, idx)
       except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
/usr/local/lib/python3.6/json/decoder.py:357: JSONDecodeError
----------------------------- Captured stderr call -----------------------------
[I 191216 18:54:12 utils:75] Sending request data: {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
[I 191216 18:54:12 utils:77] Got response code 404, content b''
------------------------------ Captured log call -------------------------------
INFO     root:utils.py:75 Sending request data: {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
INFO     root:utils.py:77 Got response code 404, content b''

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Istio Version:
  • Knative Version:
  • KFServing Version:
  • Kubeflow version:
  • Minikube version:
  • Kubernetes version: (use kubectl version):
  • OS (e.g. from /etc/os-release):
@jinchihe
Copy link
Contributor

@yuzisun Seems the problem cannot be reproduced 100%. Thanks

@yuzisun
Copy link
Member Author

yuzisun commented Dec 24, 2019

@cliveseldon @jinchihe e2e test polls the service ready condition and then sends the request to the service, looks it is a real bug in controller how we report readiness. I suspect here is the issue, we call creating virtual service and immediately mark it "ready" https://github.com/kubeflow/kfserving/blob/master/pkg/controller/inferenceservice/resources/istio/virtualservice.go#L286.

@jinchihe
Copy link
Contributor

@yuzisun Yes, I check the E2E test, I think that's not env problem, may be our code issue.

@jinchihe
Copy link
Contributor

jinchihe commented Jan 2, 2020

/P1
we need to take this as high priority, the CI tests have been broken, and reproduce easily.
@cliveseldon Would you have time to take a look? Thanks!

@yuzisun
Copy link
Member Author

yuzisun commented Jan 7, 2020

/priority p1

@yuzisun
Copy link
Member Author

yuzisun commented Mar 21, 2020

/close

@k8s-ci-robot
Copy link
Contributor

@yuzisun: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sivanantha321
Copy link
Member

This issue still persists. When I removed the time.sleep from the test as part of the Enhance CI PR, I am able to observe this behaviour in CI.

=========================== short test summary info ============================
FAILED predictor/test_sklearn.py::test_sklearn_v2_mixed - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.101.68.127/v2/models/isvc-sklearn-v2-mixed/infer
FAILED predictor/test_pmml.py::test_pmml_v2_kserve - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.101.68.127/v2/models/isvc-pmml-v2-kserve/infer
FAILED batcher/test_batcher.py::test_batcher - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.101.68.127/v1/models/isvc-sklearn-batcher:predict
============ 3 failed, 42 passed, 12 warnings in 1238.08s (0:20:38) ============

@sivanantha321 sivanantha321 linked a pull request Mar 6, 2024 that will close this issue
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants