Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RPC Communication in Distributed RL Training #303

Open
Sharad24 opened this issue Aug 31, 2020 · 2 comments · May be fixed by #327
Open

RPC Communication in Distributed RL Training #303

Sharad24 opened this issue Aug 31, 2020 · 2 comments · May be fixed by #327
Labels
c++ Core enhancement New feature or request no-issue-activity
Projects
Distributed RL

Comments

@Sharad24
Copy link
Member

Sharad24 commented Aug 31, 2020 •
edited

There's three ways that I can think of having distributed training:

  1. Use of Pytorch's Distributed Training infrastructure. Would require establishing communication protocols specific to the case of Deep RL. This would all be in Python (most likely) unless we find a way around.
  2. Use of Reverb
    • Use TF based Datasets (@threewisemonkeys-as )
    • Pytorch wrapper for the conversion of NumPy arrays, etc (that are received) (Short-term, up for grabs)
@Sharad24 Sharad24 added enhancement New feature or request Core c++ labels Aug 31, 2020
@Sharad24 Sharad24 added this to To do in Distributed RL via automation Aug 31, 2020
@threewisemonkeys-as
Copy link
Member

I agree that we should target 2 to begin with. We will still need python multiprocessing over here to run actors and learners seperately right?

As for the structure and fitting it into the rest of the library I was thinking of having DistributedOnPolicyTrainer and DistributedOffPolicyTrainer which will act as the main process and spawn the multile actors while maintaining and updating the central weights. In this case, the agent would only need to implement update_params (to be called in the main process) and select_action (to be called for each actor). The trajectories and weights would be transported through reverb.

I am holding off on #233 since a reverb buffer wrapper would heavily depend on the structure we go with. Plus it is not really useful in the non-distributed case.

@github-actions
Copy link

github-actions bot commented Nov 3, 2020

Stale issue message

@threewisemonkeys-as threewisemonkeys-as linked a pull request Nov 4, 2020 that will close this issue
Distributed Framework #327
Draft
Distributed RL automation moved this from To do to Done Nov 10, 2020
Distributed RL automation moved this from Done to In progress Nov 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c++ Core enhancement New feature or request no-issue-activity
Projects
Distributed RL
  
In progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Distributed Framework threewisemonkeys-as/genrl
2 participants
@threewisemonkeys-as @Sharad24