Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Colcon build fails with UnicodeDecodeError for WstringArrays in Foxy, but successful in dashing #610

Open
karthiknit1 opened this issue Aug 6, 2021 · 7 comments · May be fixed by #620
Open

Colcon build fails with UnicodeDecodeError for WstringArrays in Foxy, but successful in dashing #610

karthiknit1 opened this issue Aug 6, 2021 · 7 comments · May be fixed by #620
Assignees
@sloretz
Labels
bug Something isn't working
Projects
Galactic Patch Release 2

Comments

@karthiknit1
Copy link

karthiknit1 commented Aug 6, 2021 •
edited by sloretz

Bug report

Required Info:

  • Operating System:
    • Windows 10, linux debian 10 and macOs BigSur
  • Installation type:
    • from source( ros2 Foxy )
  • DDS implementation:
    • default shipped with ros2 foxy

Steps to reproduce issue

In Foxy, do colcon build for ros2 custom message with Wstrings.msg as shown below:

Wstrings.msg

wstring wstring_value
wstring wstring_value_default1 "Hello world!"
wstring wstring_value_default2 "Hellö wörld!"
wstring WSTRING_CONST="Hello world!"
wstring<=22 bounded_wstring_value
wstring<=22 bounded_wstring_value_default1 "Hello world!"
wstring[3] array_of_wstrings
wstring[<=3] bounded_sequence_of_wstrings
wstring[] unbounded_sequence_of_wstrings

wstring<=20[<=10] wstring_bounded_var_array
wstring<=10[<=10] wstring_bounded_var_array_default ['Hellö1','Hellö2','Hellö3','Hellö4'] 

wstring[<=10] wstring_bounded_array
wstring[<=10] wstring_bounded_array_default ["Hellö wörld!","Hello world!",'Hellö wörld!']

wstring<=20[5] wstring_limit_var_array
wstring<=20[5] wstring_limit_var_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3',"Hellö wörld!4",'Hellö wörld!5'] 

wstring<=20 bounded_variable
wstring<=20 bounded_variable_default 'Hellö wörld!1'

wstring<=20[] wstring_limit_variable_length_array
wstring<=20[] wstring_limit_variable_length_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3']

Build fails with the following error:

Error processing idl file: D:\ROS_2\ros2-foxy-20201211-windows-release.amd64\ros2_msg\build\ros2_msg\rosidl_adapter\ros2_msg\msg\TestWstring.idl
  Traceback (most recent call last):
    File "D:/dds/Foxy_source/ros2_foxy/install/share/rosidl_generator_cpp/cmake/../../../lib/rosidl_generator_cpp/rosidl_generator_cpp", line 40, in <module>
      sys.exit(main())
    File "D:/dds/Foxy_source/ros2_foxy/install/share/rosidl_generator_cpp/cmake/../../../lib/rosidl_generator_cpp/rosidl_generator_cpp", line 35, in main
      args.generator_arguments_file,
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_generator_cpp\__init__.py", line 40, in generate_cpp
      post_process_callback=prefix_with_bom_if_necessary)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_cmake\__init__.py", line 96, in generate_files
      raise(e)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_cmake\__init__.py", line 74, in generate_files
      idl_file = parse_idl_file(locator)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_parser\parser.py", line 62, in parse_idl_file
      string = locator.get_absolute_path().read_text(encoding='utf-8')
    File "C:\python37\lib\pathlib.py", line 1207, in read_text
      return f.read()
    File "C:\python37\lib\codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 985: invalid start byte

Expected behavior

Build should be successful

Actual behavior

Build failed

Additional information

Build is successful in ros2 dashing but broken in foxy.
In the Wstrings.msg file, these are the fields which are actually causing the failure in foxy.

wstring<=10[<=10] wstring_bounded_var_array_default ['Hellö1','Hellö2','Hellö3','Hellö4'] 
wstring[<=10] wstring_bounded_array_default ["Hellö wörld!","Hello world!",'Hellö wörld!']
wstring<=20[5] wstring_limit_var_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3',"Hellö wörld!4",'Hellö wörld!5'] 
wstring<=20[] wstring_limit_variable_length_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3']

stdout_stderr.log
Also, looking at the ros2 foxy codebase, I see that tests are missing for WstringArrays but present for StringArrays:
@sloretz
Copy link
Contributor

sloretz commented Aug 13, 2021

I'm able to reproduce this using the most recent Foxy binaries

Foxy rosidl packages installed 
ros-foxy-rosidl-adapter/focal,now 1.2.1-1focal.20210423.001432 amd64 [installed]
ros-foxy-rosidl-cmake/focal,now 1.2.1-1focal.20210423.001729 amd64 [installed]
ros-foxy-rosidl-default-generators/focal,now 1.0.1-1focal.20210630.223214 amd64 [installed,automatic]
ros-foxy-rosidl-default-runtime/focal,now 1.0.1-1focal.20210630.223215 amd64 [installed,automatic]
ros-foxy-rosidl-generator-c/focal,now 1.2.1-1focal.20210630.220922 amd64 [installed,automatic]
ros-foxy-rosidl-generator-cpp/focal,now 1.2.1-1focal.20210630.221040 amd64 [installed,automatic]
ros-foxy-rosidl-generator-dds-idl/focal,now 0.7.1-1focal.20210423.001903 amd64 [installed]
ros-foxy-rosidl-generator-py/focal,now 0.9.4-1focal.20210630.222457 amd64 [installed,automatic]
ros-foxy-rosidl-parser/focal,now 1.2.1-1focal.20210423.001559 amd64 [installed]
ros-foxy-rosidl-runtime-c-dbgsym/focal,now 1.2.1-1focal.20210630.220736 amd64 [installed]
ros-foxy-rosidl-runtime-c/focal,now 1.2.1-1focal.20210630.220736 amd64 [installed,automatic]
ros-foxy-rosidl-runtime-cpp/focal,now 1.2.1-1focal.20210423.001410 amd64 [installed]
ros-foxy-rosidl-runtime-py/focal,now 0.9.1-1focal.20210630.233034 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-c-dbgsym/focal,now 1.0.2-1focal.20210630.222356 amd64 [installed]
ros-foxy-rosidl-typesupport-c/focal,now 1.0.2-1focal.20210630.222356 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-connext-c-dbgsym/focal,now 1.0.3-1focal.20210630.222011 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-c/focal,now 1.0.3-1focal.20210630.222011 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-cpp-dbgsym/focal,now 1.0.3-1focal.20210630.221629 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-cpp/focal,now 1.0.3-1focal.20210630.221629 amd64 [installed]
ros-foxy-rosidl-typesupport-cpp-dbgsym/focal,now 1.0.2-1focal.20210630.222647 amd64 [installed]
ros-foxy-rosidl-typesupport-cpp/focal,now 1.0.2-1focal.20210630.222647 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-fastrtps-c-dbgsym/focal,now 1.0.3-1focal.20210630.221752 amd64 [installed]
ros-foxy-rosidl-typesupport-fastrtps-c/focal,now 1.0.3-1focal.20210630.221752 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-fastrtps-cpp-dbgsym/focal,now 1.0.3-1focal.20210630.221557 amd64 [installed]
ros-foxy-rosidl-typesupport-fastrtps-cpp/focal,now 1.0.3-1focal.20210630.221557 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-interface/focal,now 1.2.1-1focal.20210423.001439 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-c-dbgsym/focal,now 1.2.1-1focal.20210630.220923 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-c/focal,now 1.2.1-1focal.20210630.220923 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-introspection-cpp-dbgsym/focal,now 1.2.1-1focal.20210630.221153 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-cpp/focal,now 1.2.1-1focal.20210630.221153 amd64 [installed,automatic]

Build succeeds

wstring test_default "ö"

wstring[] test_default ["H"]

string test_default "ö"

Fails to build with UnicodeDecodeError

wstring[] test_default ["ö"]

string[] test_default ["ö"]

@sloretz
Copy link
Contributor

sloretz commented Aug 13, 2021

I'm also able to reproduce on master branches, so this is probably an issue in ROS Galactic and ROS Rolling too.

@sloretz
Copy link
Contributor

sloretz commented Aug 13, 2021

Looks like an encoding issue in the generated idl file, the problem is probably somewhere in rosidl_adapter

// generated from rosidl_adapter/resource/msg.idl.em
// with input from wstr_msg/msg/WstrTest.msg
// generated code does not contain a copyright notice


module wstr_msg {
  module msg {
    struct WstrTest {
      @default (value="('�',)")
      sequence<string> test_default;
    };
  };
};

@karthiknit1
Copy link
Author

Probably, the issue is coming due to latin-1 encoding which is merged as part of this commit:aae1e9a

This is not merged in dashing and also the build succeeds there.

@sloretz
Copy link
Contributor

sloretz commented Aug 13, 2021

Probably, the issue is coming due to latin-1 encoding which is merged as part of this commit

I'm not sure about that. It seems to be the right encoding to output to: https://www.omg.org/spec/IDL/4.2/PDF Section 7.2

IDL uses the ASCII character set, except for string literals and character literals, which use the ISO Latin-1 (8859-1) 
character set.

@jacobperron jacobperron added this to Proposed in Foxy Patch Release 6 via automation Sep 2, 2021
@karthiknit1
Copy link
Author

Is there any fix/workaround available for this issue?

jacobperron added a commit to ros2/test_interface_files that referenced this issue Sep 30, 2021
@jacobperron
Add array of wstring with default value
d2f14c1 
Regression test for ros2/rosidl#610

Signed-off-by: Jacob Perron <jacob@openrobotics.org>
jacobperron added a commit that referenced this issue Sep 30, 2021
@jacobperron
Use latin-1 encoding for reading interface file content
fdd7053 
Fixes #610

Signed-off-by: Jacob Perron <jacob@openrobotics.org>
jacobperron added a commit that referenced this issue Sep 30, 2021
@jacobperron
Use latin-1 encoding for reading interface file content
548af14 
Fixes #610

Signed-off-by: Jacob Perron <jacob@openrobotics.org>
jacobperron added a commit to ros2/test_interface_files that referenced this issue Sep 30, 2021
@jacobperron
Add array of wstring with default value
8241238 
Regression test for ros2/rosidl#610

Signed-off-by: Jacob Perron <jacob@openrobotics.org>
This was referenced Sep 30, 2021
Open
Open
@jacobperron
Copy link
Member

See #620 for a fix.

@jacobperron jacobperron added the bug Something isn't working label Sep 30, 2021
@jacobperron jacobperron removed this from Proposed in Foxy Patch Release 6 Oct 7, 2021
@jacobperron jacobperron added this to Proposed in Foxy Patch Release 7 via automation Oct 7, 2021
jacobperron added a commit that referenced this issue Oct 14, 2021
@jacobperron
Escape default arrays and sequences the same other default values
8253770 
Fix #610

Apply the same encode/decode pattern and escaping as for other default values.

Signed-off-by: Jacob Perron <jacob@openrobotics.org>
@jacobperron jacobperron removed this from Proposed in Foxy Patch Release 7 Feb 7, 2022
@jacobperron jacobperron added this to Proposed in Foxy Patch Release 8 via automation Feb 7, 2022
@jacobperron jacobperron removed this from Proposed in Foxy Patch Release 8 Sep 23, 2022
@jacobperron jacobperron added this to Proposed in Foxy Patch Release 9 via automation Sep 23, 2022
@quarkytale quarkytale removed this from Proposed in Foxy Patch Release 9 Oct 13, 2022
@quarkytale quarkytale added this to Proposed in Foxy Patch Release 10 Oct 13, 2022
@quarkytale quarkytale removed this from Proposed in Foxy Patch Release 10 Mar 16, 2023
@quarkytale quarkytale added this to Proposed in Foxy Patch Release 11 Mar 16, 2023
@quarkytale quarkytale removed this from Proposed in Foxy Patch Release 11 May 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sloretz sloretz

Labels
bug Something isn't working
Projects
No open projects
Galactic Patch Release 2
Proposed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Escape default arrays and sequences the same other default values ros2/rosidl
3 participants
@jacobperron @sloretz @karthiknit1