[Tracking Issue] Implement Integraion Tests

[YJDoc2]

As per #56 , we are porting the OCI validation tests into rust. This is the tracking issue for the same.
The folder /youki_integration_test contains the currently implemented tests which are lifecyle, creation and huge_tlb

Here is the documentation for developing the integration tests:
https://containers.github.io/youki/developer/e2e/rust_oci_test.html

Default

List of all test, as per https://github.com/opencontainers/runtime-tools/tree/master/validation :

• config_updates_without_affect
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/config_updates_without_affect/config_updates_without_affect.go
• create(@sivchari )
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/create/create.go
• default
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/default/default.go
• delete
• delete_only_create_resources
• delete_resources
• hooks
• hooks_stdin
• hostname(@chermehdi)
• kill
• kill_no_effect
• killsig
• linux_cgroups_blkio(Add Cgroup V1 block IO integration test #537)
• linux_cgroups_cpus(Integration test linux cgroups cpus #462)
• linux_cgroups_devices
• linux_cgroups_hugetlb
• linux_cgroups_memory(cgroups v1 memory integration test #473)
• linux_cgroups_network(Integration test: cgroup v1 network tests, fix to memory tests #516)
• linux_cgroups_pids(Cgroup v1 pid integration tests #391)
• linux_cgroups_relative_blkio(@furon-kuina)
• linux_cgroups_relative_cpus
• linux_cgroups_relative_devices
• linux_cgroups_relative_hugetlb(@kmpzr)
• linux_cgroups_relative_memory(@omprakaash)
• linux_cgroups_relative_network
• linux_cgroups_relative_pids(@kmpzr)
• linux_devices Add linux_devices test #2708 by @omprakaash
• linux_masked_paths
• linux_mount_label @Gekko0114
• linux_ns_itype(Add ns_itype test #389)
• linux_ns_nopath
• linux_ns_path
• linux_ns_path_type
• linux_process_apparmor_profile
• linux_readonly_paths (Readonly paths #582 )
• linux_rootfs_propagation
• linux_seccomp test(integration_test): port 'runtime-tools/validation/linux_seccomp' #2531 by @xiaoyang-sde
• linux_sysctl test(integration_test): port 'runtime-tools/validation/linux_sysctl' #2527 by @xiaoyang-sde
• linux_uid_mappings
• misc_props
• mounts
• pidfile(Add pidfile test #379)
• poststart
• poststart_fail
• poststop
• poststop_fail
• prestart
• prestart_fail
• process
• process_capabilities
• process_capabilities_fail
• process_oom_score_adj
• process_rlimits
• process_rlimits_fail
• process_user
• root_readonly_true
• start
• state

Extensions

This contains tests are not in https://github.com/opencontainers/runtime-tools/tree/master/validation , or need are not satisfactory enough.

• cgroups v1
• cgroups v2

Out of these, currently only huge_tlb is ported completely. lifecycle and create tests are implemented but need much improvements, and lifecycle tests should be split into individuals as per listed above.
Please read the README of youki_integration_test to get idea for how one should go about implementing the tests, and what utils etc are available.

[YJDoc2]

Waiting for #360 , which moves test_framework into youki_integration_test, but other than that , ready for individual test claiming.

[utam0k]

@YJDoc2 That's great! It would be a good idea to add a link to the sample implementation you implemented to the description of this issue.

[tsturzl]

I'll have some free time soon and I'd love to pitch in on this effort! I think this tooling will be a force multiplier and I think it's really impactful. Are there any pain points that I could tackle that anyone can think of that might not be a good first issue? Otherwise I'm more than happy to just jump in on cgroups controllers where I'm pretty familiar.

[YJDoc2]

Hey @tsturzl , sorry for late reply. Currently there are no obvious pain points that I know, and as you are already familiar with cgroups, it'd be great if you can work on those tests. Also if you take any test from the first list (which oci-tools have) please comment here, so we can keep track.

@utam0k I am a bit busy for few days, but after that I'm planning to work on either pid test or ns_itype test as currently both fail for runc and youki using oci-tests. Assuming that the issue is in test implementation, and not in the runtime, if we could support those tests, we can theoretically be feature equivalent with oci-tests.

[WormOn]

@utam0k i want to implement "config_updates_without_affect"

[utam0k]

@utam0k i want to implement "config_updates_without_affect"

@WormOn
Sure. I assigned you.

[tsturzl]

I'll start off with linux_cgroups_cpus, I think it will be an easy one to start off with.

[Furisto]

I do not want to take too many good first issues away, but I would like to implement at least one test to get familiar with the test framework. So I would take linux_cgroups_pids if that is alright.

[YJDoc2]

Hey @Furisto
Sure, that'd be great!

[WormOn]

@utam0k i want to implement "config_updates_without_affect"

@WormOn Sure. I assigned you.

thx

[tsturzl]

I'm nearly done with the linux_cgroups_cpus integrations tests. I'd like to take on the memory cgroups tasks. I'd really like to fix a problem that currently exists in the current oci validations with these specific tests.

[tsturzl]

Also, maybe we should limit the context of this tracking issue to cgroups v1 and just generally getting this on par with oci runtime-tools? Perhaps then we can open another tracking issue to detail out what cgroups v2 tests should look like and what needs to get implemented.

[YJDoc2]

Hey @tsturzl , sure you can take memory next 😄 , I'll assign it to you then.
It is a good idea that we use this only for tracking migration, and open separate one for cgroup v2, but I feel it'd be good to mention cgroups v2 here, until we have opened that issue, then we can link that here.

[YJDoc2]

Hey @WormOn, how is the implementation going? Let know if you need any help in working on the test implementation 👍

[tsturzl]

I got bored and started working on linux_cgroups_network, so I'll take that one also

[whereistejas]

Hi! I would like to contribute to this issue by adding a few tests, too. I have been working with rust for about 6 months, now. However, I have no exp with container runtimes, thus, would like to start with small and low priority tests.

I though I could start by adding the root_readonly_true test? Any thoughts?

[YJDoc2]

Hey @whereistejas , thanks for your interest in contribution. I would suggest that rather than starting with root_readonly_test , you might start with craete. The reason behind this is that root_readonly uses a function test_inside_runtime a util function, which I am currently implementing for another test, but facing some issues. I hope to get it worked out in some days.

The create test uses the basic functions which are already implemented in the current utils, you can check the lifecycle tests to get an idea how to implement the create tests. If you need any help or are facing some issues, you can ask them here or on the youki discord.

Thanks :)

[whereistejas]

Hi @YJDoc2, if my understanding is correct you are referring to the following create tests. I can see these tests are already implemented as part of lifecycle/container_create.rs. Please, correct me if I'm wrong.

[YJDoc2]

Hey, yes that are the tests, but they were created as part of proof-of-concept of integration tests themselves, and should be replaced in sometime for their own dedicated tests as per the runtime tools.

Or if you are fine with it, you can wait till merging of #582 which implemented the test_inside_container , then you can work on root_readonly as well.

[whereistejas]

and should be replaced in sometime for their own dedicated tests as per the runtime tools.

So, we would like to have the create tests extracted/refactored out of lifecycle into their own module?

[YJDoc2]

Hey @whereistejas , yes I meant that create tests should be refactored in there own module, as you are working on in #583 👍
Thank you for contributing!

[YJDoc2]

@YJDoc2 May I ask you to update tasks in this issue?

Hey, I'll take a look and update them, thanks 👍

I'm still trying to understand the source code. I will let you know soon!

@xiaoyang-sde feel free to ping if you need any help

[utam0k]

Thanks @YJDoc2 😍

[MarceloSpessoto]

Hello! Can I tackle the linux_cgroups_devices test?

[tsturzl]

@MarceloSpessoto Devices is really only feature complete for cgroups v1 right now, so for cgroups v2 the test might need to be skipped. It may be hard to validate the test on a cgroups v2 system, and might require testing in a VM. If your comfortable with that I'm happy to assign you. Otherwise any of the other remaining linux_cgroups_* tests would be a great place to start!

Unfortunately cgroups v2 doesn't really have a mechanism for managing devices, and the workaround is fairly complicated. That may make that particular test a bit confusing to tackle, but if you're still interested in that one in particular you are welcome to.

[MarceloSpessoto]

Hi @tsturzl , I didn't know about these cgroups v2 caveats. I was looking for something easier to start with, so I think I prefer changing to linux_cgroups_relative_blkio.

[tsturzl]

@MarceloSpessoto Assigned! Feel free to reach out if you need help.

[utam0k]

@tsturzl Thanks for your handling and guiding 🙏

[kmpzr]

Hello 👋
I'm with some capacity nowadays and would be up to tackle a thing or two. Since I previously touched PID & hugetlb I could pick linux_cgroups_relative_hugetlb and linux_cgroups_relative_pids. But I'm fine with anything you assign to me given your priority 😄

[omprakaash]

Hello ! Can I take up linux_cgroups_relative_memory ?

[tsturzl]

@kmpzr assigned those two to you. Let me know if you have any questions!

[tsturzl]

@omprakaash assigned! Let me know if you have questions or need help. I'm actually the original author of the v1 and v2 cgroups memory controller, so I'm pretty familiar with that code base.

[utam0k]

@kmpzr @omprakaash I'm looking forward to seeing your PR!

[omprakaash]

@omprakaash assigned! Let me know if you have questions or need help. I'm actually the original author of the v1 and v2 cgroups memory controller, so I'm pretty familiar with that code base.

Thank You. Will do !

[Gekko0114]

Hi @utam0k,
Can I work on linux_mount_label ?

[utam0k]

Hi @utam0k, Can I work on linux_mount_label ?

I'm happy to hear that. I assigned you.

[omprakaash]

Hey, can I take on linux_devices?

[utam0k]

@omprakaash Sure! I assigned you.

[utam0k]

@sivchari Hi 👋 I saw your post in X ;) How about trying to implement this one?

create

This is the same test in Go:
https://github.com/opencontainers/runtime-tools/blob/master/validation/create/create.go

And this code will help you:
https://github.com/containers/youki/tree/main/tests/contest/contest/src/tests/lifecycle

[sivchari]

Hi @utam0k
I'd try to implement it. Please let assign me about it.

[furon-kuina]

Hi! This is my first time with youki.
I'm very interested in this project, and I'd like to contribute to it however I can!
Would you please suggest me with a newbie-friendly test? If there is one, I'd like to take it on!

[utam0k]

@furon-kuina Sure, a good challenge! I appreciate your challenge. I'll assign you to linux_cgroups_relative_blkio. I guess there already exists a similar implemented test case. I suggest referring them.

[furon-kuina]

@utam0k
Thank you so much! I'll take a look at it!

[YJDoc2]

Hey @furon-kuina , check #2722 once, which was opened for the linux_cgroups_relative_blkio test, but the author could not continue. You can use that as a starting point or reference to get an idea for how to implement the test. (that said, the PR has several issues, failing CI etc so you might need to fix / rewrite it)

[utam0k]

Hey @furon-kuina , check #2722 once, which was opened for the linux_cgroups_relative_blkio test, but the author could not continue. You can use that as a starting point or reference to get an idea for how to implement the test. (that said, the PR has several issues, failing CI etc so you might need to fix / rewrite it)

Ops, thanks for your follow-up 🙏

[furon-kuina]

@YJDoc2 @utam0k
Thanks for the help! I really appreciate it.
I'll study the youki User and Developer Documentation and get started with the implementation.