Using Godot, How would do you make a multi-agent scene

[KShivum]

So, I'm relatively new ML/RL I've kinda messed around with Unity's ML agents but did the basics. I'm trying to do a learning project similar to the Unity's Dragon dungeon example where there are 4 agents, one dragon and 3 'players'. The dragon wants to touch all of the players, the players want to avoid the dragon and reach the exit. The dragon gets a reward if it eats a player, the players get a reward if they reach the exit.

So, the issue I'm having is how would you do a multi-agent environment. So initially (or right now) I had the idea to have two managers, one for dragons and the other for players. I was setting up a player environment in a separate scene, however, after some debugging and error testing. The training manager only finds environments that are direct children of the node, so not only can you not use a separate scene to make the training instance, I can't copy the contents of the scene to the main scene since I need to have two separate environments to keep track of two different sets of rewards which both can't be direct children and it be clean.

The other thing I was thinking about was possibly copy and pasting the training manager to each instance, but from my understanding, doesn't that negate the value of having multiple instances since each is learning by itself?

Thanks

[asieradzk]

Hey.

Thanks for trying out RL Matrix.

It's exactly as you said the manager finds all the the children of T where T is your kind of environment. If you have two different environments you would need two managers. But exactly as you said two scripts cannot sit on the same (direct child) node.

I set it up that way because I assumed users would typically want tens if not hundreds of environments.
Thanks for pointing out that you cant have two scripts on single node in Godot, this limitation escaped my attention - I'm actually also fresh to Godot.

I'll replace with a way to search all children instead.

[KShivum]

So I threw together the rest of the project, and I ran into two non-breaking issues. Not 100 percent sure if it is my code or the project.
First being, after I code for a bit I get the error from godotengine/godot#78513 which restarting would be fine if it didn't cause all the exported variables to be unset.
Second, After running the project for a bit, it seems to just freeze and stops responding (maybe permanently? So far has been not responding for about 10 minutes.)
Edit: About 5 minutes later, the program started running around, most of the environments stopped working and Godot is throwing the error

"E 0:13:13:0975   TorchSharp.torch+Tensor RLMatrix.PPOAgent`1.StateToTensor(T): System.InvalidCastException: State must be either float[] or float[,]
 <C++ Error>    System.InvalidCastException
 <C++ Source>   :0 @ TorchSharp.torch+Tensor RLMatrix.PPOAgent`1.StateToTensor(T)
 <Stack Trace>  :0 @ TorchSharp.torch+Tensor RLMatrix.PPOAgent`1.StateToTensor(T)
                :0 @ System.ValueTuple`2[System.Int32[],System.Single[]] RLMatrix.PPOAgent`1.SelectAction(T, bool)
                :0 @ void RLMatrix.PPOAgent`1+Episode.Step()
                :0 @ void RLMatrix.PPOAgent`1.Step()
                Node.cs:2111 @ bool Godot.Node.InvokeGodotClassMethod(Godot.NativeInterop.godot_string_name&, Godot.NativeInterop.NativeVariantPtrArgs, Godot.NativeInterop.godot_variant&)
                :0 @ bool RLMatrix.Godot.TrainingManagerBaseDiscrete`2.InvokeGodotClassMethod(Godot.NativeInterop.godot_string_name&, Godot.NativeInterop.NativeVariantPtrArgs, Godot.NativeInterop.godot_variant&)
                CSharpInstanceBridge.cs:24 @ Godot.NativeInterop.godot_bool Godot.Bridge.CSharpInstanceBridge.Call(nint, Godot.NativeInterop.godot_string_name*, Godot.NativeInterop.godot_variant**, int, Godot.NativeInterop.godot_variant_call_error*, Godot.NativeInterop.godot_variant*)

It very likely could be my code since I'm still learning how the RL project and Godot works and how I should get stuff setup. But I wanted to report just in case anyways.

I can post the project if you want it.

Otherwise, Thanks for making this project, I can't wait to see how it turns out down the line!

[asieradzk]

The error you're seeing is due to invalid cast of the observation vector
Observations should either be float[] or float[,] I wonder what is causing that?

When you inherit GodotEnvironmentDiscrete<TState> what is your TState?

But its odd it runs for a while and then crashes.

Do you have cuda GPU? What is the size of your observation and action space? How many agents do you have? What batch size are you using? I think its best you share your code so we can have a look.

[KShivum]

The error only shows up after the program has stopped freezing.
I am using GodotEnvironmentDiscrete<float[]> on both of the agents

Both the 'dragon' and the 'player' have 3 actions.
Each player has 10 observations (its position, rotation and where the exit is as well where the dragon is.)
The dragon has (3*n) + 1 observations, n being the number of players in its arena. The 3 being the distance from the player, and two bools converted to floats of whether that player has escaped or died.

I am using a 4090 with CUDA drivers installed. I can turn do
wn the environments, right now I have 17.

I have linked the project in this comment; however, I will rework the project when I get off work today since I really messed up with the movement, so the AI can't move as well as they should.
DungeonDragonAI.zip

[asieradzk]

Here Man, I fixed it for you, you were returning null observation due to some reset mess some key points:

1. Do not reset env by yourself, set a bool flag and the RL Matrix resets when episode ends
2. Make sure you dont reset environment multiple times when you dont need it, can cause issues

These I didnt fix for you:
3. Your agents interact with env for many episodes without any reward, the algo will not know which direction to move in (sparse reward problem)
4. In the unlikely event the dragon catches the player or player moves to exit you only give reward at the end, the algo will not know which actions to attribute reward to
DungeonDragonAI.zip
(credit assignment problem)

You should maybe provide some shaped reward depending how close player is to exit and some large reward for escaping and large penalty for losing. You can keep saving and re-starting training while decreasing the distance-based rewards so policy doesn't learn by proxy.

Good luck!

[KShivum]

Thanks! Yeah, with the reward stuff, I wasn't quite finished, so the reward points help quite a bit.
But yeah, looking at the changes and the points you were saying, I can see why it was causing issues.

[asieradzk]

ok updated the nuget hope it doesnt break :)