Deploying a model

[johnlut]

I want to deploy a model in such a way that I can have it observe and act in the real world through an API. It recieves an observation and then it sends back an action.

My understanding is that the trained model parameters are saved in the .ckpt files, and can be loaded through the config files, however I'm lost after that.

Apologies if this information is already out there, I've looked through all the documentation and wasn't able to figure it out myself. Would really appreciate the help!

For reference, this is a simplified version of what I'm trying to achieve, but in stable-baselines3:

evn = gym.make("my_env")    # load env
model = DQN.load("my_model", env=env)    # load model
env.reset()    # initialize model
obs = get_new_obs()    # get latest observation data from real world with custom function
while True:                                                     
    action, _states = model.predict(obs)    # get latest action
    custom_action(action)    # take an action in the world world with custom function
    obs = get_new_obs()    # get latest observation data from real world with custom function

[PaParaZz1]

Sorry about there is no an obvious entry in the documentation about how to evaluate (deploy) the trained policy.

You can refer to this demo and check whether it is enough for you demand.

[johnlut]

Thanks for the reply! I read over the code you linked to and made some progress but I’m still not able to do what I want.

For reference, I’m working with the gym_anytrading env, which I see has it’s own evaluator class and eval function.

I have my own scripts for pulling market data and executing trades through APIs, but I’m having a hard time connecting these to the trained model.

I’m happy to pay for help with this if you’re interested, I see you’ve worked with this env recently.

[PaParaZz1]

I add a new policy deployment demo like what in SB3. You can refer to this example.

[johnlut]

this is great, thanks!

[yourGuy]

@johnlut were you able to figure it out how to make the demo work for gym_anytrading, I'm a little lost at how to make predictions
Thank you

[Cloud-Pku]

Hello, this is a demo (colab) for gym-anytrading. If you have any comments, welcome to discuss.

[yourGuy]

Thank you that helped a lot. I run it on Forex data which has much smaller fluctuations in price and smaller volumes and the reword seems to converge to 0, any ideas how to tweak the settings to make the model more sensitive to smaller fluctuations. Also is it possible to implement this with the PPO/AS2 models ?
Thank you again

[Cloud-Pku]

Can you provide some screenshots of tensorboard and rendering results so that we can check where the problem is?

[yourGuy]

if i print reward form the prediction loop

while True:
        action = forward_fn(obs)
        obs, rew, done, info = env.step(action)
        print(rew)
        returns += rew
        if done:
            break

its always [.0]
I think because in training rewards is 0 or very close to 0
INFO:evaluator_logger:[EVALUATOR]env 0 finish episode, final reward: -0.013898119330406189, current episode: 1
INFO:evaluator_logger:[EVALUATOR]env 1 finish episode, final reward: 0.0, current episode: 2
INFO:evaluator_logger:[EVALUATOR]env 2 finish episode, final reward: 0.0, current episode: 3
INFO:evaluator_logger:[EVALUATOR]env 3 finish episode, final reward: 0.0, current episode: 4
INFO:evaluator_logger:[EVALUATOR]env 4 finish episode, final reward: 0.0, current episode: 5
INFO:evaluator_logger:[EVALUATOR]env 5 finish episode, final reward: 0.0, current episode: 6
INFO:evaluator_logger:[EVALUATOR]env 6 finish episode, final reward: -0.011954807676374912, current episode: 7
INFO:evaluator_logger:[EVALUATOR]env 7 finish episode, final reward: 0.0, current episode: 8
INFO:evaluator_logger:
let me know if you need any other info, all model settings are same as the demo colab just the data is changed to forex data
Thank you