Missing Attributes Runtime Errors

[SeifZizo]

Hi,

For a couple of weeks now, we have been trying to install and run Pylot platform to run on Carla Simulator so we can simulate our algorithms for our graduation project, but we keep getting a lot of errors, whenever we resolve one of the errors many others come up.
We have tried the docker version and the debian local version on several machines with different ubuntu 18.04 OS configuration settings using both Python2 and Python3, we even tried installing a fresh version of the OS and only install any dependencies required for Pylot, nothing more, unfortunately none of the previous attempts worked.

After installing all the dependencies and checking no compilation error would occur due to missing dependencies, and while ensuring Carla simulator is running, the following errors keep showing when trying to run Pylot, after checking the source code, these missing attributes are actually existing, but still compilation throws these errors.

Traceback (most recent call last):
  File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 298, in run
    self.__send_world_data()
  File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 402, in __send_world_data
    self._ego_vehicle.id))
AttributeError: 'CarlaOperator' object has no attribute '_ego_vehicle'
Traceback (most recent call last):
  File "/home/tarek/.local/lib/python3.7/site-packages/erdos/_init_.py", line 253, in internal_watermark_callback
    callback(timestamp, *write_streams)
  File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 225, in on_sensor_ready
    if not self._simulator_in_sync:
AttributeError: 'CarlaOperator' object has no attribute '_simulator_in_sync'

We would appreciate any help that could be provided, as we have exhausted every option to resolve the errors, the internet has no clue about the errors, the Pylot official documentation doesn’t mention any resolution and the errors are unreasonable when trying to debug the source code ourselves. This issue is driving us crazy; as we’re not able to test any of our developed work.

Thanks in advance!

[ICGog]

Hi,

We recommend using the Docker image as that already has installed all the dependencies. Can you please paste the entire error log for your run? From what I can tell it looks like _ego_vehicle_id attribute hasn't been set, which in turn indicates that the CarlaOperator has not initialized correctly as the ego vehicle is set on both paths of the if statement in the source code. Did you run the simulator before you executed Pylot?

[SeifZizo]

Hi @ICGog,

Thanks for for the quick response and your advise!

We have tried the Docker image twice, both gave the same attribute errors.

Below is the entire error log:
2022-03-14 16:52:56.754654: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 I0314 16:53:16.239696 140682789955392 component_creator.py:123] Using ground obstacles from the simulator... I0314 16:53:16.239836 140682789955392 component_creator.py:162] Adding traffic light camera... I0314 16:53:16.240186 140682789955392 component_creator.py:175] Using traffic light detection... I0314 16:53:16.253001 140682789955392 component_creator.py:507] Not using prediction... I0314 16:53:16.253099 140682789955392 component_creator.py:537] Using behavior planning... I0314 16:53:16.256677 140682789955392 component_creator.py:541] Using planning... I0314 16:53:16.260486 140682789955392 component_creator.py:578] Using the manual control/autopilot... I0314 16:53:16.260601 140682789955392 component_creator.py:588] Synchronizing ticking using the waypoints stream pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html 2022-03-14 16:53:18.409153: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2022-03-14 16:53:20.569495: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:20.569788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1050 computeCapability: 6.1 coreClock: 1.442GHz coreCount: 6 deviceMemorySize: 2.95GiB deviceMemoryBandwidth: 78.32GiB/s 2022-03-14 16:53:20.569815: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2022-03-14 16:53:22.096871: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2022-03-14 16:53:22.096981: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2022-03-14 16:53:26.036027: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2022-03-14 16:53:27.013824: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 Traceback (most recent call last): File "<string>", line 31, in <module> File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 80, in _init_ self._initialize_world() File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 325, in _initialize_world self._flags.simulator_town)) RuntimeError: failed to connect to newly created map 2022-03-14 16:53:30.323414: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11 2022-03-14 16:53:30.520127: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11 2022-03-14 16:53:30.771425: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2022-03-14 16:53:30.771575: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:30.771840: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:30.786427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2022-03-14 16:53:57.198010: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-03-14 16:53:57.198436: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:57.198675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1050 computeCapability: 6.1 coreClock: 1.442GHz coreCount: 6 deviceMemorySize: 2.95GiB deviceMemoryBandwidth: 78.32GiB/s 2022-03-14 16:53:57.198781: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:57.199017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:53:57.199202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2022-03-14 16:53:57.243025: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2022-03-14 16:55:03.438912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-03-14 16:55:03.438946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 2022-03-14 16:55:03.438959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N 2022-03-14 16:55:03.439150: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:55:03.439434: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:55:03.439655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-03-14 16:55:03.439855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1412 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1) 2022-03-14 16:55:04.162249: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) 2022-03-14 16:55:04.578059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2400000000 Hz 2022-03-14 16:55:10.271569: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 18874368 exceeds 10% of free system memory. 2022-03-14 16:55:12.864314: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 18874368 exceeds 10% of free system memory. 2022-03-14 16:55:19.148187: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2022-03-14 16:55:42.485890: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8100 2022-03-14 16:56:16.834290: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2022-03-14 16:56:30.861022: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2022-03-14 16:56:31.322443: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:31.491834: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 894.12M (937558016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2022-03-14 16:56:31.713549: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:32.131835: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.08GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:32.650577: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:33.067235: W tensorflow/core/common_runtime/bfc_allocator.cc:337] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2022-03-14 16:56:34.828158: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.36GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:35.611264: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.49GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:35.701694: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.38GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2022-03-14 16:56:35.860288: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.71GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. Traceback (most recent call last): File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 298, in run self.__send_world_data() File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 402, in __send_world_data self._ego_vehicle.id)) AttributeError: 'CarlaOperator' object has no attribute '_ego_vehicle' Traceback (most recent call last): File "/home/tarek/.local/lib/python3.7/site-packages/erdos/_init_.py", line 253, in internal_watermark_callback callback(timestamp, *write_streams) File "/home/tarek/pylot/pylot/simulation/carla_operator.py", line 225, in on_sensor_ready if not self._simulator_in_sync: AttributeError: 'CarlaOperator' object has no attribute '_simulator_in_sync' I0314 16:56:44.186808 140679959271168 geos.py:32] Trying CDLL(libgeos_c.so.1)I0314 16:56:44.714619 140679959271168 geos.py:49] Library path: 'libgeos_c.so.1' I0314 16:56:44.714772 140679959271168 geos.py:50] DLL: <CDLL 'libgeos_c.so.1', handle 7ff28c22aeb0 at 0x7ff263003550> I0314 16:56:44.735809 140679959271168 geos.py:32] TryingCDLL(libc.so.6) I0314 16:56:44.736334 140679959271168 geos.py:49] Library path: 'libc.so.6' I0314 16:56:44.736477 140679959271168 geos.py:50] DLL: <CDLL 'libc.so.6', handle 7ff343b8f000 at 0x7ff2630031d0>

We have ensured that the simulator was running and that it was assigned to port 2000 and the port was open during the execution of Pylot as it was changing the map (town) along with the script. However, sometimes (not usually) it throws a timeout error.

[ICGog]

I've noticed in the logs that the GPU allocator is running out of memory while trying to load our models in the 3GB (possibly just 2GB because CARLA might use 1GB) available on your GPU. We recommend having available at least 12GB of GPU memory to run Pylot. Nonetheless, you can try to run a configuration of Pylot that doesn't deploy any models by executing: python3 pylot.py --flagfile=configs/perfect_detection.conf --visualize_detected_obstacles=False. This configuration deploys a perfect detector that gets bounding boxes from the simulator, and drives using the CARLA autopilot.

[SeifZizo]

Mmmm.. I see, will try the command that doesn't deploy the models and will try running Pylot on a better pc, hopefully that'll get things fixed.

Thanks @ICGog

P.S. Sorry for the late reply