Can not kill Carla by Python script when i set autopilot mode

[Youngzipper]

I am using CARLA 0.9.15 in Ubuntu 18.04
I write a python script to open carla、spawn vehicle、set autopilot、kill carla... repeatedly.
But when i kill carla for the first round, error meassage was raised: aborted (core dumped).
The weird thing is if i do not set autopilot, then the circulation can run, which is open carla、spawn vehicle、kill carla...
My brief codes is as follows:

def find_and_kill_process(name, user=None):
    for proc in psutil.process_iter(['pid', 'name', 'username']):
        try:
            if user and proc.info['username'] != user:
                continue

            if name in proc.info['name']:
                print(proc.info['pid'], proc.info['name'])
                proc.kill()  
                print("force terminate process")
                proc.wait()      
                print("wait process close")
                print(f"Process with PID {proc.info['pid']} ({proc.info['name']}) terminated.")
        except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
            pass

def main():
    try:
        carla_server_path = '/home/user_name/CARLA/CarlaUE4.sh'
        subprocess.Popen(carla_server_path)
        time.sleep(5.0)
        client = carla.Client('localhost', 2000)
        world = client.load_world("Town04")

        x = random.choice([-337, -331, -325, -319])
        y = random.choice([26, 29.5, 33, 36.5])
        ego_location = carla.Location(x, y, z=3)
        ego_rotation = carla.Rotation(yaw=0)
        ego_spawn_point = carla.Transform(ego_location, ego_rotation)
        ego_bp = world.get_blueprint_library().find('vehicle.audi.etron')
        ego_bp.set_attribute('role_name', 'CAV')
        ego_vehicle = world.spawn_actor(ego_bp, ego_spawn_point)

        world.tick()

        ego_vehicle.set_autopilot(True) **# if delete this line, then the the script will run successfully**


        i = 0
        while True:
            i += 1
            time.sleep(1.0)
            if i == 5:
                break
       
            spectator = world.get_spectator()
            spectator_location = carla.Location(ego_vehicle.get_location().x, ego_vehicle.get_location().y, ego_vehicle.get_location().z + 100)
            spectator_rotation = carla.Rotation(pitch=-90)
            spectator_transform = carla.Transform(spectator_location, spectator_rotation)
            spectator.set_transform(spectator_transform)

    except:
        pass


  if __name__ == '__main__':
        for i in range(20):
            print(f"______________episode {i}__________________")
            main()
            find_and_kill_process("CarlaUE4-Linux-Shipping", "user_name")

[Youngzipper]

I even tried ego_vehicle.set_autopilot(False), but the results are same, that once i use function set_autopilot, not matter set True or False, I can not kill Carla by Python script.

[joel-mb]

@Youngzipper which is your use case to start/exit the CARLA server in that way? I would recommend using the client to reload the world (https://carla.readthedocs.io/en/latest/python_api/#methods_8) if necessary. Also you can load a new map if needed without closing the server.

If you still need to start/exit the CARLA server within your python script I would recommend cleaning the simulation before doing that:

while True
  ...

# Destroy all actors spawned
ego_vehicle.destroy()

[Youngzipper]

@Youngzipper which is your use case to start/exit the CARLA server in that way? I would recommend using the client to reload the world (https://carla.readthedocs.io/en/latest/python_api/#methods_8) if necessary. Also you can load a new map if needed without closing the server.

If you still need to start/exit the CARLA server within your python script I would recommend cleaning the simulation before doing that:

while True
  ...

# Destroy all actors spawned
ego_vehicle.destroy()

Thank you ! I have tried ego_vehicle.destroy(), but my problem still exits.
The reason why i start/exit CARLA repeatedly is reload world too many times during multiple episodes may lead to error like :
Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=131160 LargeMemoryPoolOffset=196744 Malloc Size=131160 LargeMemoryPoolOffset=327928 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped)
Additionally, i am using co-simulation between sumo and CARLA, which is a little complex.

[joel-mb]

Could you upload your whole terminal output? Your code is working fine to me.

[Youngzipper]

Could you upload your whole terminal output? Your code is working fine to me.

My terminal output:

(d2rl) ziyuan@ubuntu-3090:~/project/co_simulation/AV_Controller$ python close_carla.py 
______________episode 0__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
79339 CarlaUE4-Linux-Shipping
force terminate process
Killed
wait process close
Process with PID 79339 (CarlaUE4-Linux-Shipping) terminated.
______________episode 1__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
Aborted (core dumped)

And i'm sorry that my codes has one mistake, there is only one while True after setting autopilot:

ego_vehicle.set_autopilot(True) **# if delete this line, then the the script will run successfully**

# while True:  # DELETE THIS LINE
i = 0
while True:
    i += 1
    time.sleep(1.0)
    if i == 5:
        break

[joel-mb]

Hmm I see. Probably this crash is due to the traffic manager not shutting down properly. My proposal would be getting an instance of the traffic manager and before closing the server shutting down the traffic manager.

Try adding the following lines to your code:

# ...

def main():
    try:
        # ...        
        client = carla.Client('localhost', 2000)
        tm = client.get_trafficmanager()  # get an instance of the trafficmanager
        world = client.load_world("Town04")

        # YOUR LOOP CODE

        # clean-up the simulation
        ego_vehicle.destroy()
        tm.shut_down()  # properly shutdown the trafficmanager
    except:
        pass

[Youngzipper]

I add the following lines after my while True loop:

print("after run 5 seconds of autopilot")
ego_vehicle.destroy()
print("after destroy the vehicle")
tm.shut_down()  # properly shutdown the trafficmanager
print("after shut down the tm")

And i got the results like :

(d2rl) ziyuan@ubuntu-3090:~/project/co_simulation/AV_Controller$ python close_carla.py 
______________episode 0__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
after run 5 seconds of autopilot
after destroy the vehicle
Aborted (core dumped)

Seems something wrong with tm.shut_down()

[Youngzipper]

Things changed when i tried again tm.shut_down(), it raised aborted core dumped in episode 2:

(d2rl) ziyuan@ubuntu-3090:~/project/co_simulation/AV_Controller$ python close_carla.py 
______________episode 0__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
<carla.libcarla.TrafficManager object at 0x7d31b6e5e500>
load world town04
after run 5 seconds of autopilot
after destroy the vehicle
after shut down the tm
90540 CarlaUE4-Linux-Shipping
force terminate process
Killed
wait process close
Process with PID 90540 (CarlaUE4-Linux-Shipping) terminated.
______________episode 1__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
<carla.libcarla.TrafficManager object at 0x7d31b6e5e810>
load world town04
after run 5 seconds of autopilot
after destroy the vehicle
after shut down the tm
91003 CarlaUE4-Linux-Shipping
force terminate process
Killed
wait process close
Process with PID 91003 (CarlaUE4-Linux-Shipping) terminated.
______________episode 2__________________
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
<carla.libcarla.TrafficManager object at 0x7d31b6e5e880>
Aborted (core dumped)

And the codes i get traffic manager be like:

carla_server_path = '/home/ziyuan/CARLA/CARLA_0.9.15/CarlaUE4.sh'
subprocess.Popen(carla_server_path)
time.sleep(10.0)
client = carla.Client('localhost', 2000)
tm = client.get_trafficmanager()  # get an instance of the trafficmanager
print(tm)
world = client.load_world("Town04")
print("load world town04")

Something wrong with client.load_world() in episode 2.

[Youngzipper]

@Youngzipper which is your use case to start/exit the CARLA server in that way? I would recommend using the client to reload the world (https://carla.readthedocs.io/en/latest/python_api/#methods_8) if necessary. Also you can load a new map if needed without closing the server.

If you still need to start/exit the CARLA server within your python script I would recommend cleaning the simulation before doing that:

while True
  ...

# Destroy all actors spawned
ego_vehicle.destroy()

I tried reload_world(), and here is the error message:

(base) ziyuan@ubuntu-3090:~/CARLA/CARLA_0.9.15$ ./CarlaUE4.sh
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Malloc Size=131160 LargeMemoryPoolOffset=196744 
Malloc Size=131160 LargeMemoryPoolOffset=327928 
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

it happens when i load_world("Town04") once and later reload_world() once, not immediately after reloading, but after running some codes.

[Youngzipper]

if i tried reload_world() in my brief codes without SUMO cosim, error happens in episode 2:

def main(episode=0):
    try:
        # carla_server_path = '/home/ziyuan/CARLA/CARLA_0.9.15/CarlaUE4.sh'
        # subprocess.Popen(carla_server_path)
        # time.sleep(10.0)
        client = carla.Client('localhost', 2000)
        # tm = client.get_trafficmanager()  # get an instance of the trafficmanager
        # print(tm)

        if episode == 0:
            world = client.load_world("Town04")
            print("load world town04")
            time.sleep(2.0)
        else:
            world = client.reload_world()
            print("reload world town04")

        x = random.choice([-337, -331, -325, -319])
        y = random.choice([26, 29.5, 33, 36.5])
        ego_location = carla.Location(x, y, z=3)
        ego_rotation = carla.Rotation(yaw=0)
        ego_spawn_point = carla.Transform(ego_location, ego_rotation)
        ego_bp = world.get_blueprint_library().find('vehicle.audi.etron')
        ego_bp.set_attribute('role_name', 'CAV')
        ego_vehicle = world.spawn_actor(ego_bp, ego_spawn_point)

        world.tick()

        ego_vehicle.set_autopilot(True) # if delete this line, then the the script will run successfully

        i = 0
        while True:
            i += 1
            time.sleep(1.0)
            if i == 5:
                break
       
            spectator = world.get_spectator()
            spectator_location = carla.Location(ego_vehicle.get_location().x, ego_vehicle.get_location().y, ego_vehicle.get_location().z + 100)
            spectator_rotation = carla.Rotation(pitch=-90)
            spectator_transform = carla.Transform(spectator_location, spectator_rotation)
            spectator.set_transform(spectator_transform)
                # clean-up the simulation
        print("after run 5 seconds of autopilot")
        # ego_vehicle.destroy()
        # print("after destroy the vehicle")
        # tm.shut_down()  # properly shutdown the trafficmanager
        # print("after shut down the tm")

    except:
        print("errror")


if __name__ == '__main__':
    carla_server_path = '/home/ziyuan/CARLA/CARLA_0.9.15/CarlaUE4.sh'
    subprocess.Popen(carla_server_path)
    time.sleep(10.0)
    for i in range(20):
        print(f"______________episode {i}__________________")
        main(i)

(d2rl) ziyuan@ubuntu-3090:~/project/co_simulation/AV_Controller$ python close_carla.py 
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
______________episode 0__________________
load world town04
after run 5 seconds of autopilot
______________episode 1__________________
reload world town04
after run 5 seconds of autopilot
______________episode 2__________________
reload world town04
errror

while CARLA remains alive

[joel-mb]

Still not able to reproduce your issue. Are you running more than one CARLA client at the same time? This can cause issues with the traffic manager and the way you are closing the server as it also works with a primary-secondary architecture.

Alternatively, you cloud try using a bash script to initialize CARLA and your co-simulation scripts instead of doing all this inside the same python process. I believe this could be a safer way to proceed.

[lightlefter]

I meet a same issue.
I am using CARLA 0.9.12 in Windows10
When I running the python Script, just like spawn vehicle. The Script didn't finished and I closed carla server.The Editor of UE4 is not responding. The problem might caused by the running script.
I also want to slove the problem. When I close the carla server, I don't want the unresponding editor, nomatter what python script is running.

[joel-mb]

@lightlefter What do you mean by the script didn't finished? What script are you running? Are you setting the simulator in sync or async mode?

Please provide more information so we can give you a proper answer.

[lightlefter]

1、It means the script is running.
2、The script like this:
import carla
import time
def run_function():
client = carla.Client('localhost',2000)
client.set_timeout(10.0)
world = client.get_world()
for vehicle in world.get_actors().filter('vehicle'):
vehicle.set_autopilot(True)
time.sleep(120)
run_function()
3、I checked the sync mode by the code:
settings = world.get_settings()
print(settings.synchronous_mode)
the result is False
print(settings.fixed_delta_seconds)
the result is None
4、I run carla server in UE4 Editor. When the script and the TM was running, I pressed ESC button in UE4 to close the carla server. The result is unresponding UE4 editor .
Now, I have to wait for the script to finish. That really bother me. Do you have any suggestion? Thanks for yor answer. @joel-mb

[Youngzipper]

@lightlefter Maybe you can end up CARLA in task manager

[lightlefter]

I did end up CARLA in task manager. @Youngzipper
I also trying to avoid this issue. Pretty unconvinient.
Now, I have to code a exit logic in UE4 to avoid this issue.
I want to know a easy way to solve it.