General Questions

[huiyulhy]

Hi! I've got a question about the scope of editing code in edit_this.py for the competition:

1. In this section of the code, does use_firmware refer to the position controller onboard the crazyflie? If that is set to false, does that mean that the simulated onboard position controller will have to be a PID controller, or is there room to edit this?
   

[JacopoPan]

Hi @huiyulhy,

if you look at any of the configuration files intended for evaluation (e.g. level0.yaml) you'll see that use_firmware will be set to True

use_firmware: True # Whether to use pycffirmware or not (NOTE: only set to False for debugging purposes)

The else branch and the Python PIDController() is only meant for debugging and tests, not the competition.
The on-board controller that you have available (and need to use) with use_firmware == True is the CF firmware's Mellinger one wrapped by pycffirmware. (There is not room to change that as we won't be able to re-build/flash the real life firmware with custom implementations at the time of the competition.)

I just pushed a few changes (fc6a907) to the comments in edit_this to hopefully correct this misunderstanding.

[huiyulhy]

Hi Jacopo, thanks for the clarification on this!

[ZQ2413262560]

Hi! it's great to participate in this competition, but I have some questions about it :

1. When I call the cmdFirmware() function, I can get the information about the gates in the "info" value, but Where can I find the information about the obstacle when I call the Controller.cmdFirmware() function?
   

[ZQ2413262560]

Thank you for your reply！ Does this mean that I can't get the exact location of the obstacle during the whole running process if the obstacle is noisy？

[JacopoPan]

Yes, you would only have access to the noisy positions and the parameters of the distribution used to randomize it.
While the gate positions are initially noisy buy become exact if the drone is within 0.45m of the gate.

The choice was made to encourage the development of solution with both adaptive and robust components. A first naive approach would be to plan for a worst case scenario. However, you should also note that the value of reseed_on_reset in the YAML determines whether obstacles/gates move (if False) or not (if True) between episodes. Thus, in levels below level3 you could start with an "optimistic" trajectory and use the experience (e.g. a collision) of previous runs to determine the exact position of an obstacle, then correct accordingly in subsequent runs. These are just ideas there are certainly more ways to approach the problem than I can think of.

Nonetheless, if you don't like certain aspects of the competition implementation and you have suggestions for things that you'd like to be done differently in future editions, I really encourage you to open another discussion with those recommendations! We definitely aim to create environments that are educational, entertaining, and can support good research for all.

[ZQ2413262560]

Ok, thank you very much for your reply! I also want to ask, do I need to modify the contents exactly in the format of edit_this.py? That is, will I be allowed to add new functions to the Controller class (in addition to init() and cmdFireware()) for the sake of code brevity?

[JacopoPan]

Hello @ZQ2413262560, you are obviously welcome to add as many methods to the Controller class as you want (and/or use functions in the same or a separate file) as long as the interface as used in getting_started is maintained.

(The only think I would be careful about is to minimize the number of external module imports, to make sure that we can easily run the code in the Python environments used for evaluations.)

[ZQ2413262560]

Ok, I understand, and thanks a lot for your reply！

[ZQ2413262560]

Hi Jacopo ! I want to ask a question about the constraints：
I know that "bounded_constraint" means that the agent's pos must obey the constraint range from [x_low,y_low,z_low] to [x_high,y_high,z_high] which has 6 dim. but what "default_constraint" mean exactly? I can't figure it out.

[ZQ2413262560]

Hi Jacopo ~, and I found that in the rule-based example, the "take-off" command will violate the constraint so that self.cnstr_violation=True but I don't know why .
Look forward to your reply ~

[JacopoPan]

Hello @ZQ2413262560

the default_constraint on the input is just a box constrain mirroring the action space limits. I'm tagging @adamhall who could also help clarify further any constraint-related concern.

W.r.t. the second question, please let me know exactly the command you run and the error/message that is thrown. Please make sure you have git pull the code relatively recently (as there was a patch to make the numerical precision explicit that we added a few weeks ago).

[JacopoPan]

@ZQ2413262560

were you able to overcome this with the most recent code?

[ZQ2413262560]

Yes, we have overcome this difficulty in the recent code. Our team plans to submit a version of the solution today or tomorrow because we are worried that the submission format may be wrong, so we may need to trouble you to check whether the code we submitted meets the requirements. Thank you very much!

[JacopoPan]

@ZQ2413262560

sounds good! Have you also confirmed that you will be submitting by email (I sent 2 reminders last week)?

[ZQ2413262560]

Hi Jacopo~ In the process of participating in the competition these days, I have the following doubts:

1. Can participants only modify the contents of edit_this.py? I want to modify _get_reward() in quadrotor.py and .step() in firmware_wrapper.py to get a dense reward, is that okay?
2. While I'm training, I have to set done_on_violation,done_on_completion, and done_on_collision to be True, right?
3. In cmdFirmware(), the neural network infers a command_type and the corresponding args, whereas in interEpisodeLearn, what is added to the Buffer is the quadrotor 4D controllers being executed. That is, the action that the neural network infers and the data that the neural network is trained on is not a batch of data, which doesn't seem to be very conducive to reinforcement learning. Is there something I missed or made a mistake?
4. for q3 , What I'm doing is going to replace the ctrl.interStepLearn(actions, obs, reward, done, info) with ctrl.interStepLearn(args, obs, reward, done, info)， Is that allowed？
   Looking forward to your reply. Thank you very much！

[JacopoPan]

Hi @ZQ2413262560

that's great! Thanks for the questions!

1. (A) I'm on-board with modifying the reward (although you could consider ways to do on top of the existing signal and within the agent in edit_this.py) but I'd prefer if you did it with a PR (or sharing the implementation of the new dense reward with me so that I can (i) check you are not accessing simulator information that wouldn't be realistic to obtain in real-life and (ii) share it in the repo for other people to use as well, for fairness; (B) what do you want to change in firmware_wrapper?
2. Yes, those are all True in the YAMLs used for evaluation (if you think that pre-training the weights of an agent in a "different" environment to have more freedom to explore is convenient, you can do that, but we will only run the level0-3.yaml to score the solutions)
3. (and 4.) that is a good point and I will consider a patch (logging both) for next week (can you explicitly say what args is in your case, I think the change it's ok (or you could implement a workaround in edit_this/Controller, given that the information you want to log originates there anyway).

[ZQ2413262560]

Hi~Jacopo，I'll get this sorted out soon and get back to you. I have one more question, since I can get the exact location of the door when I'm near the door, it would be even better if I could get the exact location of the obstacle when I'm near the obstacle!

[JacopoPan]

Thanks @ZQ2413262560

I know this is something you brought up before: it is a perfectly reasonable concern but it was something deliberately done to encourage use a combination of adaptive and robust methods.

This a change that I don't think is fair to make in the last few weeks before IROS: I made a note of your request and we will consider changing it (e.g., including alternative/multi-modal sensing) in future editions. I would suggest you just "augment" the obstacle shapes with the uncertainty parameters to make sure you steer clear of them regardless of they exact positioning.

[camillo2008]

Hi Jacopo, I was approaching this challenge now, and I know I'm late. But i want to explore it more.
I was wondering to use RL to resolve this, but I can't figure how to use multiple drone to speed up the training, I have seen that in firmware_wrapper multiple drones are not allowed.

1. There is a way to parallelize, or is the simulation limited to one drone?

2. For RL, can I start for example from PPO controller inside safe_contro_gym, if so, is not clear to me how I can instantiate it inside edit_this, or should I use some other RL library and interface it via edit_this?

[JacopoPan]

HI @camillo2008,

For 1.
gym-pybullet-drones supports multi-agent environments while safe-control-gym is single agent only (for now). If you are using a regular CPython interpreter, you should be able to simply exploit multi-process parallelism (creating multiple single agent environments at once), if you want to.

(I'm not 100% if anything in the backend implementation of firmware_wrapper/pycffirmare would prevent that—@spencerteetaert do you know more about this?)

For 2.
You could do multiple things: import some pre-trained models (on scenarios you created) as weights, implement your RL agent, or re-use a loop like stable_baselines' .learn(). Personally, I'd suggest to go the route of "something simple that works" rather than trying to data/compute brute-force the problem with plug-and-play 3rd party agents, especially with the limited time remaining :D

[spencerteetaert]

Hi @camillo2008

Unfortunately we did not design the pycffirmware package with parallelization as a priority.

The firmware wrapper uses SWIG interface to directly wrap the crazyflie firmware. Because the firmware is stateful, so is the pycffirmware package. This means that creating multiple firmware wrapper objects within the same python instance is not expected to work. The two wrapper objects would have different python states but share the same firmware state, which will almost certainly lead to poor results.

If you want to parallelize I think the easiest way would be to run multiple python instances and find a way to intelligently merge the results of each. There are some pretty clear drawbacks with this approach though (python overhead, effective merging, etc).

A more proper approach would be to make edits to pycffirmware to remove the state variables from the C wrapper and instead have them all stored in the python class. This approach I imagine would require significant changes to the structure of the firmware itself which is why we haven't taken it, we have the goal of eventually wrapping the firmware without any edits so that pycffirmware is updated when cffirmware is.

There may also be a way to do it exploiting SWIG and how python imports wrapped modules, but I am not familiar enough with python package import mechanics to comment on an intelligent way to do this.

If I were to try training an RL agent with this environment I would set up two stage training. The first training in parallel on gym-pybullet-drones to get a decent agent, then finetune it single threaded on safe-control-gym with cffirmware.

Hope this helps!