2024-05-06 ODD working group weekly meeting (187)

[Igata-ctrl]

Attendee

• Daniel Shih (Reference Design WG/TIER IV/National Taiwan University)
• Mert Çolak (Leo Drive)
• Berkay Karaman (Leo Drive)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke MITSUDOME (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On April 21, 2024, 428/559 regular scenarios succeeded. 131 scenarios failed.

  • 129/141 public road bus scenarios were successful. 12 scenarios failed.
    • 4 Condition Failures (Npc collides with Ego from aside though "isBLIND: false" is set, etc)
    • 8 Condition Failures (Ego didn't restart after cross traffic passed, etc.)
  • 115/220 pull over scenarios were successful. 105 scenario cases failed.
    • 105 failed with Simulation Time Failure (One of the success conditions was not met, etc.)
  • 184/198 pull out scenarios were successful. 14 scenarios failed.
    • 10 Condition Failures (success condition "act_ego_nostop_check" was not met)
    • 3 Condition Failures (Ego didn't start)
    • 1 Condition Failure (Obstacle stop -> Time out)

• On April 28, 2024, 428/559 regular scenarios succeeded. 131 scenarios failed.

  • 129/141 public road bus scenarios were successful.
  • 112/220 pull over scenarios were successful.
  • 187/198 pull out scenarios were successful.

• On May 5, 2024, 424/559 regular scenarios succeeded.

  • 127/141 public road bus scenarios were successful.
    • Comparison with recent results is here.
  • 113/220 pull over scenarios were successful.
    • Comparison with recent results is here.
  • 184/198 pull out scenarios were successful.
    • Comparison with recent results is here.
  • Each comparison show very similar trend.

• On April 30, @Igata-ctrl and @mitsudome-r joined the meeting with Safety Pool (Deepen.ai + WMG) and discussed that the following issues need to be addressed.

  • [Map] We can now convert OpenDRIVE maps of Safety Pool to Lanelet2 for Autoware using CommRoad Scenario Designer, but the following issues need to be ironed out. One option is to make Autoware accept OpenDRIVE maps instead of Lanelet2.
    • Following information is lost through the conversion.
      • Speed limit, lane change tag, LHT/RHT, etc.
    • The converted Lanelet2 map contains too many unnecessary points.
  • [File format] Safety Pool uses .xosc for OpenSCENARIO files, while Autoware Scenario Simulator v2 on Evaluator (cloud integrated CI/CD pipeline) uses .yaml file.
  • [Autoware] Autoware requires the EGO destination to operate, while no EGO destination is specified in Safety Pool scenarios. May need to consider a sort of roaming mode.
  • [Simulation Tool] Safety Pool uses esmini simulator (OpenSCENARIO player). Safety Pool is kindly sending us the link of the recording of a recent Webinar covering this.
  • [Access] After registering to Safety Pool, @Igata-ctrl can access the limited number (651 out of one million) of scenarios in Safety Pool database. Safety Pool is kindly looking into this issue.

• Daniel joined ODD WG meeting and discussed Low Speed Vehicle ODD over the video of National Taiwan University campus.

• [recap of the discussions until last week]

• @Igata-ctrl prototyped three dense urban scenarios based on the use cases. May need to clarify a few small issues with TIER IV CI/CD team.

• Daniel Shih, the lead of Reference Design WG joined to discuss the low speed (controlled area) ODD. Once it's finalized, it will be merged to the ODD definitions spread sheet.

• @Igata-ctrl shared a tip how to set arbitrary trajectory for NPCs with @Berkay54 on Discord.

• @Igata-ctrl found an example scenario which uses the FollowTrajectoryAction. It seems there are some caveats around the initial settings of NPCs.

• @Igata-ctrl created a couple of example breadboard use cases for the dense urban ODD.

• @Igata-ctrl is continuing communication with Safety Pool. Currently, Map conversion and compatibility issues are discussed. A meeting has been set on April 30 to discuss how many scenarios will be available for Autoware and how.

• As for the "too many points" issue with the converted Lanelet2 map, I learned from TIER IV FMS & map team that this tool maybe helpful to reduce the number of the unnecessary points.

• Due to a recent bug fix of the scenario simulator v2 regarding the maximum speed setting, it turned out some of the scenarios need to be modified. Berkay who ported the basic TIER IV scenarios to AWF is kindly working on the fix now.

• As for the dense urban traffic ODD which Autoware Labs is going to work on, here's the link to the video of typical dense urban traffic which Fatih showed in the TSC. This would be a good start point for the new use case/scenarios.

• @Igata-ctrl has created a step-by-step memo "Map conversion using CommonRoad Scenario Designer (OpenDRIVE-_Lanelet2)-110324-055327" with a help from Ata (Leo Drive) who is familiar with the CommonRoad Scenario Designer also the author of the documentation. A example of converted Lanelet2 map is uploaded to the folder. The sample map has been uploaded to to the Evaluator (CI/CD pipeline) and an example scenario was created on it. Known map issues which require modification are as follows.

  • Linestrings contain too many points (16,000 points in the area map) -> Vector Map Builder response is very slow
  • Max speed was set to 10 kph -> Modified to 100 kph
  • Line strings between lanes were set as "not-lane changeable" -> Set to "lane changeable"

• Regarding the scenario compatibility, Safety Pool scenarios are in .xosc file and Autoware/Evaluator (CI/CD pipeline) uses .yaml file for more flexible handling of the parameters. Though stand alone Scenario Simulator v2 can accept .xosc file, Evaluator cannot. We may perhaps need another converter or some modification of Evaluator side. @Igata-ctrl is inquiring Safety Pool approximately how many scenarios will be available for Autoware in order to identify what level of automation is desired here.

• [Stephen] Is there any way to obtain the ODD information from the suspended EV project of Apple's (Titan)?

• As a preparation for the Autoware Lab discussion, @Igata-ctrl would like to add "(RoboTaxi in) dense traffic area" to the ODD definitions table. Would like to further edit this table during the meeting.

• Safety Pool presented the outline of Safety Pool.

  • The presentation will be shared in the ODD WG Google Drive.
  • A very large scale (half to one million) scenario database and the eco system is already established.
  • Users can search for appropriate scenarios by tag searches and add scenarios to the database.
  • The scenarios of Safety Pool use OpenDRIVE map.
  • @Igata-ctrl shared the sample scenario and the Lanelet2 map with Safety Pool for analysis.
  • Asked Mohammad and Siddartha to join again next week to make a full demonstration of Safety Pool.

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time

[brkay54]

Hi @Igata-ctrl -san, I would like to talk about Autoware Labs scenarios. sheet-link