SHM subsystem: Rust #823

yellowhatter · 2024-03-13T15:14:45Z

Closes #685
Closes #827

Conflicts: commons/zenoh-codec/src/core/shm.rs

…generation support

- generation works

- use numeric ID instead of string ID for SHM buffer identification - this feature speeds up segment lookups and reduces wire overhead - remove unnecessary fields from SharedMemoryManager - fix clippy warnings - added comments

- support ProtocolID exchange in establish - convert buffer based on supported protocol ids

- WIP to brush-up the API, eliminate some API flaws - SHM provider is now conceptually made thread-safe for better flexibility

commons/zenoh-shm/src/posix_shm/array.rs

commons/zenoh-shm/src/watchdog/periodic_task.rs

commons/zenoh-shm/src/watchdog/storage.rs

commons/zenoh-shm/src/watchdog/validator.rs

commons/zenoh-sync/Cargo.toml

…to pr/yellowhatter/823

io/zenoh-transport/src/unicast/auth_segment.rs

io/zenoh-transport/src/unicast/auth_unicast.rs

Mallets · 2024-04-18T13:55:57Z

io/zenoh-transport/src/unicast/auth_segment.rs

+    array: ArrayInSHM<AuthSegmentID, AuthChallenge, usize>,
+}
+
+impl AuthSegment {


With the current SHM system there is some leaking due to the shm segment never be closed.
How this new system will clean up all AuthSegment?

AuthSegment, as any other zenoh SHM component, uses posix_shm::array::ArrayInSHM for managing shared memory. Previous SHM system used posix segment mapping on the filesystem - this mechanism required to manually delete SHM-mapped files to free shared memory and was unacceptable because our process might crash with no way to delete mapped files. The new SHM system uses system handles that are automatically unlinked when last linked process terminates, avoiding memory leakage.

Great, that's neat! :)

zenoh/src/bytes.rs

…to pr/yellowhatter/823

yellowhatter · 2024-04-18T15:27:07Z

@Mallets I resolved discussions where I made fixes

commons/zenoh-shm/src/api/provider/types.rs

yellowhatter added 30 commits November 14, 2023 12:19

add watchdog codebase

20d5b26

integrated and tested POC

9bee417

[skip ci] Merge commit '1daaaf512c575b55042b88aa76be94205a93611c'

b78336b

Merge commit '780ec606ba4000a17b2da0e2b0b668e11d37c65e'

d521c05

Conflicts: commons/zenoh-codec/src/core/shm.rs

Update shm.rs

f643f93

WIP: implemented buffer headers in separate SHM segment, made buffer …

f7d053c

…generation support

- refactored POSIX shm wrapper

5523201

- generation works

Merge commit 'd8c27a6e017938f4435d5abea4a0142c9d23ff59'

26e0ac5

WIP on SHM

0c1e385

Fix clippy, additional improvements

43b5391

Implement watchdog periodic tasks with optional realtime scheduling

fcc064f

[skip ci] WIP on SHM provider\client API

00a4d1d

[skip ci] WIP on SHM API

3ccab12

Merge commit 'b669489bc814a758741f09c671ecc3a0683697a0'

2fa05c6

[skip ci] Big WIP on SHM API integration

eb608c9

[skip ci] working SHM (tests passing)

5ee84e7

Merge commit '0943243da1a9eb14644897b13934be25610c4a94'

e234699

[skip ci] WIP on API

db4036a

Merge commit 'e8dca1e0ac7036ee04ab024115d418224492343d'

6e07611

[skip ci] WIP on SHM API

6d05776

Merge commit '2bd4518334d0a9e7e2703c7fd57dfa4547a96819'

fc31e7d

WIP on API

2dc3991

Merge commit '7ebdb3cab481609989f1e9d3124e75d2851458e5'

b65a575

- added SharedMemoryReader to Session and Runtime building API

743b081

- support ProtocolID exchange in establish - convert buffer based on supported protocol ids

[skip ci] correct shm establish logic

0067f63

Remove SharedMemoryFactory

1f1fabd

[skip ci]

3f2e8db

- WIP to brush-up the API, eliminate some API flaws - SHM provider is now conceptually made thread-safe for better flexibility

[skip ci] Final updates to SHM provider API

149ba73

Merge commit '90617ff606b6d4c89d1a3f166dcee6a2b7d5461a'

0acd83d

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/posix_shm/array.rs Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/posix_shm/array.rs Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/watchdog/periodic_task.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/watchdog/storage.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/watchdog/storage.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/watchdog/validator.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-sync/Cargo.toml Outdated Show resolved Hide resolved

Mallets added 3 commits April 18, 2024 15:14

Revert wrong change on log

3df753b

Merge branch 'shm_watchdog_poc' of github.com:ZettaScaleLabs/zenoh in…

0ffad29

…to pr/yellowhatter/823

Merge protocol_changes

5ffadfc

Mallets reviewed Apr 18, 2024

View reviewed changes

io/zenoh-transport/src/unicast/auth_segment.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

io/zenoh-transport/src/unicast/auth_unicast.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

io/zenoh-transport/src/unicast/auth_unicast.rs Outdated Show resolved Hide resolved

Mallets reviewed Apr 18, 2024

View reviewed changes

zenoh/src/bytes.rs Outdated Show resolved Hide resolved

Mallets and others added 6 commits April 18, 2024 16:06

Update zenoh/src/bytes.rs

0903424

Fix use

71f1b02

Merge branch 'shm_watchdog_poc' of github.com:ZettaScaleLabs/zenoh in…

48979a2

…to pr/yellowhatter/823

Fix use

a1cd67e

Review fixes

1dd2c0c

fix recursive call

54691f6

Mallets reviewed Apr 18, 2024

View reviewed changes

commons/zenoh-shm/src/api/provider/types.rs Show resolved Hide resolved

Mallets and others added 4 commits April 18, 2024 18:10

Update commons/zenoh-shm/src/api/provider/types.rs

e79bd02

unstable for shm slice traits

94f37d8

Add more #[zenoh_macros::unstable_doc]

39ee7f2

SHM establishment reorg

cb0908e

Mallets approved these changes Apr 19, 2024

View reviewed changes

add missing ztimeout! in tests

aed5e14

Mallets merged commit 3711d45 into eclipse-zenoh:protocol_changes Apr 19, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SHM subsystem: Rust #823

SHM subsystem: Rust #823

yellowhatter commented Mar 13, 2024 •

edited

Mallets Apr 18, 2024

yellowhatter Apr 18, 2024 •

edited

Mallets Apr 18, 2024

yellowhatter commented Apr 18, 2024

SHM subsystem: Rust #823

SHM subsystem: Rust #823

Conversation

yellowhatter commented Mar 13, 2024 • edited

Mallets Apr 18, 2024

Choose a reason for hiding this comment

yellowhatter Apr 18, 2024 • edited

Choose a reason for hiding this comment

Mallets Apr 18, 2024

Choose a reason for hiding this comment

yellowhatter commented Apr 18, 2024

yellowhatter commented Mar 13, 2024 •

edited

yellowhatter Apr 18, 2024 •

edited