Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python SDK] REST endpoints mishandle headers #3022

Open
xfu83 opened this issue Jul 6, 2023 · 0 comments · May be fixed by #3669
Open

[Python SDK] REST endpoints mishandle headers #3022

xfu83 opened this issue Jul 6, 2023 · 0 comments · May be fixed by #3669
Labels
kind/bug kserve/sdk

Comments

@xfu83
Copy link
Contributor

xfu83 commented Jul 6, 2023 •
edited

/kind bug

What steps did you take and what happened:

Headers are not handled and returned properly in v1/v2 endpoints in the recent master branch:

  1. Headers are not returned in v1 endpoints

kserve/python/kserve/kserve/protocol/rest/v1_endpoints.py

Lines 74 to 80 in f9e9a07

response, response_headers = self.dataplane.encode(model_name=model_name,
response=response,
headers=headers, req_attributes=req_attributes)
if not isinstance(response, dict):
return Response(content=response, headers=response_headers)
return response

if the response is a dict, then the headers will be completely discarded.

in v2 endpoints, the headers are handled properly:

kserve/python/kserve/kserve/protocol/rest/v2_endpoints.py

Lines 146 to 147 in 97bada5

if response_headers:
raw_response.headers.update(response_headers)

  1. dataplane.encode completely discarded the response headers from the model

kserve/python/kserve/kserve/protocol/dataplane.py

Lines 259 to 260 in 97bada5

def encode(self, model_name, response, headers, req_attributes: Dict) -> Tuple[Dict, Dict[str, str]]:
response_headers = {}

So any custom headers returned back from the model will be completely lost for both v1/v2 endpoints.

kserve/python/kserve/kserve/protocol/rest/v1_endpoints.py

Lines 76 to 85 in 97bada5

response, response_headers = await self.dataplane.infer(model_name=model_name,
request=infer_request,
headers=headers)
response, response_headers = self.dataplane.encode(model_name=model_name,
response=response,
headers=headers, req_attributes=req_attributes)
if not isinstance(response, dict):
return Response(content=response, headers=response_headers)
return response

What did you expect to happen:

Custom headers should be passed through and returned properly, so things won't break if users upgrade their Python SDK to 0.11.x

What's the InferenceService yaml:
[To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

maybe related issue: #3012

Environment:

  • Istio Version:
  • Knative Version:
  • KServe Version:
  • Kubeflow version:
  • Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
  • Minikube/Kind version:
  • Kubernetes version: (use kubectl version):
  • OS (e.g. from /etc/os-release):
@andyi2it andyi2it mentioned this issue Jul 27, 2023
Closed
9 tasks
@yuzisun yuzisun mentioned this issue Feb 24, 2024
Closed
9 tasks
@andyi2it andyi2it linked a pull request May 7, 2024 that will close this issue
Draft
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug kserve/sdk
Projects
None yet
Milestone
No milestone
3 participants
@yuzisun @kserve-oss-bot @xfu83