NoneType issue with step_type at a random epoch

[n-chennakeshava]

Hey All,

I am having an issue where I get the following error at a random episode of the training cycle.

Traceback (most recent call last):
File "train_rnn_dqn.py", line 391, in
train_agent(args)
File "train_rnn_dqn.py", line 319, in train_agent
time_step, policy_state = collect_driver.run(time_step, policy_state)
File "/usr/local/lib/python3.8/dist-packages/tf_agents/drivers/py_driver.py", line 125, in run
traj = trajectory.from_transition(
File "/usr/local/lib/python3.8/dist-packages/tf_agents/trajectories/trajectory.py", line 599, in from_transition
next_step_type=next_time_step.step_type,
AttributeError: 'NoneType' object has no attribute 'step_type'

I am using a custom environment with the q_rnn_network agent class given in tf_agents (the same issue occurs when using a sequential CNN). I am using reverb as my data management method (uniform sampler).

I am not sure why it is failing at random episodes? Am I perhaps missing a step_type assignment for some transitions or terminal states? I am pretty sure I am assigning it to every step.

One important thing to note is that I don't have terminal states. I just reset the env after the assigned number of iterations per episode.

Any insight into why this may be happening?

[n-chennakeshava]

The solution was to also include terminal states by manually assigning one, even though it is strictly not correct.