[CORE] [REP] Ray OBS support #50

imhy · 2024-04-23T09:52:03Z

Add ability to read working_dir from OBS

ericl · 2024-04-23T21:30:43Z

Proposal makes sense to me... it amounts to removing the URI whitelist of allowed protocols to pass through to smart_open, which I think is not necessary to restrict anyways (cc @edoakes ).

ericl · 2024-04-23T21:31:48Z

reps/2024-04-24-ray-obs-support.md

+
+At the same time, they can specify the location of the project's working directory in a remote file system, for example s3.
+
+This document suggests expanding the list of supported file systems and adding support for OBS (Object Storage Service).


I assume this is https://www.huaweicloud.com/intl/en-us/product/obs.html ? Would be good to add a reference to clarify

Done, added link to docs.

imhy · 2024-04-24T04:00:56Z

Proposal makes sense to me... it amounts to removing the URI whitelist of allowed protocols to pass through to smart_open, which I think is not necessary to restrict anyways (cc @edoakes ).

I`m ready to implement this functionality.

edoakes

LGTM. I don't think we need a full vote on this REP because it is quite small in scope.

@imhy please go ahead and open a PR to implement this support and tag me as a reviewer. Let me know if you have any questions on how to get started.

jjyao

LG

jjyao · 2024-04-26T16:16:27Z

reps/2024-04-24-ray-obs-support.md

+After the implementation of this improvement, the user will be able to run submit tasks specifying the location of the working directory in OBS.
+
+
+The UserCase of submiting ray job via obs is shown in Figure 1:


Suggested change

The UserCase of submiting ray job via obs is shown in Figure 1:

The use case of submiting ray job via obs is shown in Figure 1:

jjyao · 2024-04-26T16:21:26Z

reps/2024-04-24-ray-obs-support.md

+
+- Third, after submitting the ray job, the ray cluster automatically downloads, uncompresses, and executes the codes from the OBS
+
+![Figure 1](2024-04-24-ray-obs-support/fig1.png)


You need to do <img style="background-color:white" src="2024-04-24-ray-obs-support/fig1.png"> to show the image.

jjyao · 2024-04-26T16:22:08Z

reps/2024-04-24-ray-obs-support.md

+
+The UserCase of submiting ray job via obs is shown in Figure 1:
+
+- First, the user startups the remote ray cluster, and then upload the codes to the  obs;


Suggested change

- First, the user startups the remote ray cluster, and then upload the codes to the obs;

- First, the user starts the remote ray cluster, and then uploads the codes to the obs;

jjyao · 2024-04-26T16:22:44Z

reps/2024-04-24-ray-obs-support.md

+
+- First, the user startups the remote ray cluster, and then upload the codes to the  obs;
+
+- Second, the user submits ray job to OBS using the submit_job APIs, as well as setting the obs path in the submit_job API


Suggested change

- Second, the user submits ray job to OBS using the submit_job APIs, as well as setting the obs path in the submit_job API

- Second, the user submits ray job to the ray cluster using the submit_job APIs, as well as setting the obs path in the submit_job API

jjyao · 2024-04-26T16:23:42Z

reps/2024-04-24-ray-obs-support.md

+
+1. Compress the code into a zip or jar package and put it in OBS
+
+2. You can use environment variables or configuration files to access the AK, SK, and Endpoint of OBS


what's AK, SK?

Access Key, Secret Key for access to OBS, something like in S3.
AK SK docs

jjyao · 2024-04-26T16:24:57Z

reps/2024-04-24-ray-obs-support.md

+)
+```
+
+4. The ray cluster automatically downloads the example_file.zip from the specified OBS bucket, decompresses it, and then runs the entry file script.py in the working path


Suggested change

4. The ray cluster automatically downloads the example_file.zip from the specified OBS bucket, decompresses it, and then runs the entry file script.py in the working path

4. The ray cluster automatically downloads the example_file.zip from the specified OBS bucket, decompresses it, and then runs the entry file script.py in the working dir path

jjyao · 2024-04-26T16:26:13Z

reps/2024-04-24-ray-obs-support.md

+We can extend the ray project to support obs protocol, enabling the ray cluster to parse the obs URI and download the codes from obs:
+
+1. parse obs URI, such as "obs://example_bucket/example_file.zip";
+
+2. config the obs AK, SK, and Endpoint via environment variables and config files;
+
+3. the ray cluster automatically downloads and execute the codes from obs.


This is duplicate of the above?

jjyao · 2024-04-26T16:26:30Z

reps/2024-04-24-ray-obs-support.md

+## Implementation Analysis
+
+To extend the ray project for obs, we first need to figure out the workflow of parsing and accessing remote URIs, which is shown in Figure 2.
+![Figure 2](2024-04-24-ray-obs-support/fig2.png)


jjyao · 2024-04-26T16:30:22Z

reps/2024-04-24-ray-obs-support.md

+
+**Step 1**: Extend the **parse_uri** function to parse the OBS URIs, which is implemented in the file [ray/_private/runtime_env/packaging](https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/packaging.py).
+
+**Step 2**: Extend the third-party library [smart_open](https://github.com/piskvorky/smart_open) to read OBS objects, which is suggested to implement 3 interfaces, i.e., parse uri, open_uri, and open, as shown in Extending smart_open.


So you will create a PR in smart_open to support OBS first?

Yes, firstly I implement required logic in smart_open.

Are you planning to upstream this functionality to smart_open or extend it within the Ray codebase?

firstly, I will add functionality to smart_open, then I will add logic to ray packaging.py

Signed-off-by: Sergei <sv.sokolov@gmail.com>

imhy · 2024-04-27T08:19:14Z

LGTM. I don't think we need a full vote on this REP because it is quite small in scope.

@imhy please go ahead and open a PR to implement this support and tag me as a reviewer. Let me know if you have any questions on how to get started.

ok, thanks

imhy force-pushed the 2024-04-24-ray-obs-support branch 2 times, most recently from fcdb934 to 26f6962 Compare April 23, 2024 10:09

ericl reviewed Apr 23, 2024

View reviewed changes

imhy force-pushed the 2024-04-24-ray-obs-support branch from 26f6962 to 2b78dcc Compare April 24, 2024 04:00

edoakes approved these changes Apr 26, 2024

View reviewed changes

jjyao reviewed Apr 26, 2024

View reviewed changes

imhy force-pushed the 2024-04-24-ray-obs-support branch 5 times, most recently from ef6aa52 to cbcc617 Compare April 27, 2024 07:53

Ray OBS support

920603e

Signed-off-by: Sergei <sv.sokolov@gmail.com>

imhy force-pushed the 2024-04-24-ray-obs-support branch from cbcc617 to 920603e Compare April 27, 2024 08:12

This was referenced May 27, 2024

Add support of Huawei Object Storage Service (OBS) piskvorky/smart_open#824

Open

[WIP] [Core] Support of reading Working dir from HUAWEI Object Storage Service (OBS) ray-project/ray#45577

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE] [REP] Ray OBS support #50

[CORE] [REP] Ray OBS support #50

imhy commented Apr 23, 2024

ericl commented Apr 23, 2024

ericl Apr 23, 2024

imhy Apr 24, 2024

imhy commented Apr 24, 2024

edoakes left a comment

jjyao left a comment

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

jjyao Apr 26, 2024

imhy Apr 27, 2024

edoakes Apr 29, 2024

imhy May 3, 2024

imhy commented Apr 27, 2024


		At the same time, they can specify the location of the project's working directory in a remote file system, for example s3.

		This document suggests expanding the list of supported file systems and adding support for OBS (Object Storage Service).

		After the implementation of this improvement, the user will be able to run submit tasks specifying the location of the working directory in OBS.


		The UserCase of submiting ray job via obs is shown in Figure 1:

	The UserCase of submiting ray job via obs is shown in Figure 1:
	The use case of submiting ray job via obs is shown in Figure 1:


		- Third, after submitting the ray job, the ray cluster automatically downloads, uncompresses, and executes the codes from the OBS

		![Figure 1](2024-04-24-ray-obs-support/fig1.png)


		The UserCase of submiting ray job via obs is shown in Figure 1:

		- First, the user startups the remote ray cluster, and then upload the codes to the obs;

	- First, the user startups the remote ray cluster, and then upload the codes to the obs;
	- First, the user starts the remote ray cluster, and then uploads the codes to the obs;


		- First, the user startups the remote ray cluster, and then upload the codes to the obs;

		- Second, the user submits ray job to OBS using the submit_job APIs, as well as setting the obs path in the submit_job API

	- Second, the user submits ray job to OBS using the submit_job APIs, as well as setting the obs path in the submit_job API
	- Second, the user submits ray job to the ray cluster using the submit_job APIs, as well as setting the obs path in the submit_job API


		1. Compress the code into a zip or jar package and put it in OBS

		2. You can use environment variables or configuration files to access the AK, SK, and Endpoint of OBS

	4. The ray cluster automatically downloads the example_file.zip from the specified OBS bucket, decompresses it, and then runs the entry file script.py in the working path
	4. The ray cluster automatically downloads the example_file.zip from the specified OBS bucket, decompresses it, and then runs the entry file script.py in the working dir path


		Step 1: Extend the parse_uri function to parse the OBS URIs, which is implemented in the file [ray/_private/runtime_env/packaging](https://github.com/ray-project/ray/blob/master/python/ray/_private/runtime_env/packaging.py).

		Step 2: Extend the third-party library [smart_open](https://github.com/piskvorky/smart_open) to read OBS objects, which is suggested to implement 3 interfaces, i.e., parse uri, open_uri, and open, as shown in Extending smart_open.

[CORE] [REP] Ray OBS support #50

Are you sure you want to change the base?

[CORE] [REP] Ray OBS support #50

Conversation

imhy commented Apr 23, 2024

ericl commented Apr 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imhy commented Apr 24, 2024

edoakes left a comment

Choose a reason for hiding this comment

jjyao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imhy commented Apr 27, 2024