Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Socket tracer fails to start on latest Container OS #1900

Closed
ddelnano opened this issue May 13, 2024 · 2 comments · Fixed by #1909
Closed

Socket tracer fails to start on latest Container OS #1900

ddelnano opened this issue May 13, 2024 · 2 comments · Fixed by #1909
Assignees
Labels
area/datacollector Issues related to Stirling (datacollector)

Comments

@ddelnano
Copy link
Member

Running Pixie on a recent (v109) COS GKE cluster results in a failed socket tracer.

To Reproduce

  1. Create GKE cluster using COS latest version (v109)
  2. px deploy
  3. Run a pxl script that uses the socket tracer data tables and see the following error and PEM logs

Screen Shot 2024-05-13 at 2 30 44 PM

pem logs
I20240513 21:23:40.984936  8330 bcc_wrapper.cc:97] Successfully resolved task_struct offsets: {real_start_time=1832, group_leader=1584, exit_code=1416}
I20240513 21:23:40.985049  8330 bcc_wrapper.cc:166] Initializing BPF program ...
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1310:25: error: expected ')'
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                        ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1310:18: note: to match this '('
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                 ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1310:51: error: variable has incomplete type 'typeof(void)' (aka 'void')
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                                                  ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1313:43: error: expected ')'
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                          ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1313:36: note: to match this '('
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                   ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1313:69: error: variable has incomplete type 'typeof(void)' (aka 'void')
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                                                    ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1321:46: error: expected ')'
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1321:31: note: to match this '('
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                              ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1321:20: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec')
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                   ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1434:25: error: expected ')'
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                        ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1434:18: note: to match this '('
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                 ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1434:46: error: variable has incomplete type 'typeof(void)' (aka 'void')
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1437:43: error: expected ')'
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                          ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1437:36: note: to match this '('
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                   ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1437:64: error: variable has incomplete type 'typeof(void)' (aka 'void')
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                                               ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1445:45: error: expected ')'
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                                            ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1445:30: note: to match this '('
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1445:19: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec')
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
12 errors generated.

Expected behavior
Pixie's socket tracer should work with latest COS instances.

App information (please complete the following information):

  • Pixie version: 0.14.9
  • K8s cluster version: 1.28.7
  • Node Kernel version: 6.1.58
  • Node OS:
ddelnano@gke-dev-cluster-ddelnano-default-pool-0f4367ef-x20k ~ $ cat /etc/os-release
NAME="Container-Optimized OS"
ID=cos
PRETTY_NAME="Container-Optimized OS from Google"
HOME_URL="https://cloud.google.com/container-optimized-os/docs"
BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
GOOGLE_CRASH_ID=Lakitu
KERNEL_COMMIT_ID=a75611e516d2d29d1e04e26e345c3313a5b04b90
GOOGLE_METRICS_PRODUCT_ID=26
VERSION=109
VERSION_ID=109
BUILD_ID=17800.66.78
@ddelnano
Copy link
Member Author

ddelnano commented May 17, 2024

I've tracked down the bug to a bcc bug (in combination with llvm).

Screen Shot 2024-05-17 at 1 34 57 PM

Upgrading the pixie BCC fork to include the changes through 0.30.0 results in a functional socket tracer. This also requires upgrading libbpf since we specify libbpf's version separately in our bazel configuration.

More details on the fix can be seen in iovisor/bcc@57ca51b

@ddelnano ddelnano self-assigned this May 17, 2024
@ddelnano ddelnano added the area/datacollector Issues related to Stirling (datacollector) label May 17, 2024
ddelnano added a commit that referenced this issue May 20, 2024
Summary: Upgrade bcc and libbpf to fix Google COS socket tracer issues

See GitHub issue comments below for more details. This pulls in the
following BCC fix
(iovisor/bcc@57ca51b).

Relevant Issues: Closes #1900, part of #1905

Type of change: /kind bug

Test Plan: Skaffold'ed this change to a Ubuntu and Google COS cluster
and verified socket tracer works again
- [x] Verified that all kernel build passes
- [x] Verified that all kernel build used a bcc repo reference that
matches latest commit
```
ddelnano@noble-dev:~/code/bcc/build (pixie8) $ git diff pixie/pixie8 ddelnano/pixie8
ddelnano@noble-dev:~/code/bcc/build (pixie8) $
```

Changelog Message: Fix issue with socket tracer on recent Google COS
instances

---------

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano
Copy link
Member Author

This is a duplicate of #1871.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/datacollector Issues related to Stirling (datacollector)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant