[Windows] Support CPU shared memory (Client/Frontend) #7048
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
fpetrini15
force-pushed
the
fpetrini-win-cpu-shm
branch
from
86b0362
to
8b63bab
Compare
fpetrini15
force-pushed
the
fpetrini-win-cpu-shm
branch
from
15f94bb
to
a5b6b7e
Compare
| triton_client = httpclient.InferenceServerClient(_url, verbose=True) | ||
| # Custom setup method to allow passing of parameters | ||
| def _setUp(self, protocol, log_file_path): | ||
| self._tritonserver_ipaddr = os.environ.get("TRITONSERVER_IPADDR", "localhost") |
There was a problem hiding this comment.
Does this need to be configurable in practice? Do we expect to use shared memory for anything other than a co-located server on localhost?
There was a problem hiding this comment.
TBD: Currently on the Windows testing side of things, it's passed in as a variable and is different from "localhost". Still trying to get a CI pipeline up to see the new behavior for this test in particular. Will remove if no issue.
| self._build_model_repo() | ||
| self._build_server_args() | ||
| self._shared_memory_test_server_log = open(log_file_path, "w") | ||
| self._server_process = util.run_server( |
There was a problem hiding this comment.
How does util.run_server interact with test.sh also starting server? Is there conflict or issue there?
There was a problem hiding this comment.
I don't believe they should overlap. For this test my ultimate goal is to remove test.sh entirely.
There was a problem hiding this comment.
Isn't this also getting run in the linux case that runs test.sh? or is there changes on the gitlab-side to not run test.sh at all?
There was a problem hiding this comment.
Ah I see your point. There are changes on the gitlab side such that test.sh will not run at all for Windows. I will attempt to change the Linux test case so that it also will not run test.sh
| backend_dir = "C:\\opt\\tritonserver\\backends" | ||
| model_dir = "C:\\opt\\tritonserver\\qa\\L0_shared_memory\\models" | ||
| self._server_executable = "C:\\opt\\tritonserver\\bin\\tritonserver.exe" |
There was a problem hiding this comment.
Probably more of a random note or follow-up, but I was under the impression something like Pathlib.Path("/opt/tritonserver/backends") would translate to "C:\\opt\\tritonserver\\backends" for free when run on Windows. If so you could probably condense the cases to work for both.
Did you see otherwise?
There was a problem hiding this comment.
No, I believe you are right. ATM they are set to my local path and were hardcoded for convenience. They need to be modified and will once I determine the CI environment.
| # Constant members | ||
| shared_memory_test_client_log = Path(os.getcwd()) / "client.log" | ||
| model_dir_path = Path(os.getcwd()) / "models" | ||
| model_source_path = Path(os.getcwd()).parents[0] / "python_models/add_sub/model.py" |
There was a problem hiding this comment.
Future follow-up as we expand python utilities for CI testing, but might be nice to have some kind of utils.relative_path([path, to, thing]).
ex maybe something like this:
model_dir_path = utils.relative_path("models")
model_source_path = utils.relative_path("..", "python_models", "add_sub", "model.py")
Goal: Support CPU shared memory between the server and client for Windows
Sub-goals: Modify L0_shared_memory to run on bare-metal Windows using only Python.
Client changes: triton-inference-server/client#551
Some things to note:
test.sh