bevy_xwt

[simbleau]

Hey,

When doing my exploration into #132 - I found that I would indeed benefit from an API akin to bevy_matchbox

I want to start building bevy_xwt, at a high level:

bevy_xwt: 2 features (client, server)

• client:
  • commands:
    • commands.start_session(url: &str)
    • commands.close_session()
  • system parameters:
    • ClientSession, derefs to Session
  • events:
    • enum ClientEvent { IdAssigned(SessionId), Connected, Disconnected }
• server:
  • commands:
    • commands.start_server(port: u16)
    • commands.stop_server()
  • system parameters:
    • ServerSessions, derefs to HashMap<SessionId, Session>
  • events:
    • enum ServerEvent { ClientJoined(SessionId), ClientLeft(SessionId) }

That's basically it, extremely simple to start.

Since you mentioned you want to reserve the name bevy_xwt, do you mind starting the seed repo, and adding me as a contributor so I can continue to build it under your repo?

[MOZGIII]

I see the bevy_xwt as a more lower-level than that - just exposing the xwt types though a bevy plugin and setting up I/O loops - but without any opinionated things, like managing how the sessions are represented.

On the API, I'd prefer having separate crates for client and server instead of features. I'm very much appreciate the help, but at the same time I'd like to setup things the certain way, but I'm currently working on a more lower level things and am not yet at the stage when I'm ready to build an full featured bevy integration.

I have a few things in mind for the actual bevy networking libraries, but most of them are kind of transport agnostic to a degree, and would use xwt as just one of the integrations. So, maybe building an abstract bevy_net_transport crate is a way to go here?

To elaborate on my networking thoughs, I'm thinking about building a couple different implementations with deep integration:

• a deterministic networking subsystem with a fixed-frame syncronization and world rollbacks - somewhat similar to Overwatch netcode, with deep bevy integration and custom scheduler and etc; this would be suitable for a variety of games, like Overwatch or MOBAs
• a similar deterministic networking subsystem but without a world rollback system - but with other means of correcting small-ish networking conflicts - for things where modelling the world rollback would be to expensive; for games like Factorio or RTSes
• a non-deterministic simple networking layer for things that need some quick and durty way to syncronize roughtly but cheaply - with maybe a simple layer of predicting on top but not much beyond that

All of those could be built on top of xwt - but they don't really have to, as I am confident that we can build an abtraction layer and power it with an open ecosystem of crates, like xwt - but maybe even other solutions that are not related to WebTranspot at all.

So, maybe you'd be interested in starting that work instead? It will still be some time before I get to work on the xwt bevy integration - but even that I'd like to keep barebones simply, with relying on an abstraction layer to provide the functionality.

[simbleau]

Going to do my best to elaborate here, but I think we may have similar needs. I'm making an MMO so fine-tuned control is important.

A few things:

• I like the idea of separate client/server crates, but in a monorepo.
• State synchronization seems out of scope for bevy_xwt, or at the least a feature, on the premise it's done with data structures and algorithms, not transport.
• Rollback networking seems out of scope/like a feature for the same reasons.
• bevy_net_transport is perhaps a good idea, but what would its API look like?

I think the best way to go here is actually the original proposal: re-exporting the xwt types with bevy ecosystem integration (e.g. commands, system parameters, plugins). I also think bevy_xwt should be low-level, but abstract the ease of starting a server and connecting to a server with bevy.

The reason is control. We shouldn't give up control.

By having a very simple bevy_xwt crate, as I above, you can build all the abstractions above on top of that. I am sure of that because that's exactly how bevy_rtc works, too.

Here is the tree of bevy_rtc:

├── bevy_rtc v0.3.1
│   ├── bevy v0.13.2 (*)
│   ├── bevy_matchbox v0.9.0
│   │   ├── bevy v0.13.2 (*)
│   │   └── matchbox_socket v0.9.0
│   │       ├── matchbox_protocol v0.9.0
│   │       └── webrtc v0.10.1
│   ├── bevy_rtc_macros v0.3.1 (proc-macro)

In my case, I use bevy_matchbox under the hood, which offers control, but I add an ergonomic API on top.

If we translated this to the xwt ecosystem, here would be a glimpse:

├── bevy_xwt_moba
│   ├── bevy
│   ├── bevy_xwt
│   │   ├── bevy
│   │   └── xwt
│   │       ├── xwt-....
│   │       └── wtransport
│   ├── bevy_xwt_moba_macros v0.3.1 (proc-macro)

The idea here is bevy_xwt_moba would be your abstraction for "overwatch" rollback netcode and synchronization. It would use the inner types from bevy_xwt responsibly, and not export those.

In an ideal world, here's what I want:

Protocol (shared dependency):

#[derive(XwtProcol)]  // Ensures it's Serializable, Deserializable, etc.
pub struct PingPayload;

Client:

fn main() {
    App::new()
        .add_plugins(XwtClientPlugin {
            encoding: Encoding::Json,
            validate_certs: false,
            ... // more configuration
        })
        .add_protocol::<PingPayload>()
        .add_systems(
            Startup,
            |mut commands: Commands| {
                commands.connect_xwt_server("127.0.0.1:3536");
            },
        )
        .add_systems(
            Update,
            {
                |mut transport: XwtTransport<PingPayload>| {
                    transport.reliable_to_host(PingPayload);
                    for _pong in client.read() {
                        info!("...Received pong!");
                    }
                }
            }
        )
        .run();
}

Server:

fn main() {
    App::new()
        .add_plugins(XwtServerPlugin {
            encoding: Encoding::Json,
            port: 3536,
            cert: Certs::SelfSigned,
            ... // more configuration
        })
        .add_protocol::<PingPayload>()
        .add_systems(
            Startup,
            |mut commands: Commands| {
                commands.start_listening(); // Start server
            },
        )
        .add_systems(
            Update,
            {
                |mut events: EventReader<XwtServerEvent>, mut server: XwtServer| {
                    for ev in events.read() {
                        if let XwtServerEvent::ConnectionRequest(session) = ev {
                            server.accept(session);
                        }
                    }
                }
            }
        )
        .add_systems(
            Update,
            {
                |mut transport: XwtTransport<PingPayload>| {
                    for (session, _ping) in transport.read() {
                        transport.reliable_to_(session, PingPayload);
                    }
                }
            }
        )
        .run();
}

[MOZGIII]

Oh, I may have confused you: I feel like all my networking ideas should explicitly not be xwt specific, and xwt meaning "cross web transport" should be focus only on the it's core purpose.

This means if I'd were to build the more advanced stuff I'd myself pick a different name for that - and probably something with a separate banding from the xwt completely. Furthermore - those projects would need a transport abstraction layer - so they'd probably not be going to be built on xwt directly - but through an abstraction layer for transports.

For those high level things, the protocol encoding layer is in their domain, because there are differences in how we'd want to model the encoding of the data. Meaning that derive(XwtProcol) would likely be a derive(bevy_net_opt1::Protocol) (throwing this bevy_net_opt1 in place of bevy_net_moba, cause the use of moba in here wouldn't be right, and we need some reference names for this dialog), rather than derive(bevy_xwt::Protocol).

Of course, monorepos would be the go - but we might be go with more than one. As in, one for the networking abstraction layer - that one would also contain the "low-level" bevy bindings; good candidates for drivers there are xwt, quinn and libp2p (as it is also an abstraction layer, and actually an whole system of abstractions). Say, we'll call it bevy_net_* family of crates, with bevy_net_core, bevy_net_server, bevy_net_client.
Then, a separate monorepo for each conceptual thing on top of this - with its own name. Like bevy_opt1net_server, bevy_opt1net_client, bevy_opt1net_scheduler.

Btw, there are already a lot of crates that would benefit from the abstracted transport crate:

• https://github.com/cBournhonesque/lightyear (uses xwt)
• https://github.com/aecsocket/aeronet (uses xwt)
• https://github.com/lucaspoffo/renet (refactored to allow transport abstraction and ready for an integration with xwt)
• https://github.com/projectharmonia/bevy_replicon (refactored to allow transport abstraction and ready for an integration with xwt)
• https://github.com/naia-lib/naia - considering using xwt
• more? I can't recall, but I think there were others too

All those have already shown interest in or integrated with xwt - but I'd rather have a pluggable networking transport layer - so other transports can be added.

That said, I feel like transport layer should actually be fully transparent, and not hide any of its dependencies. What I means is it shouldn't even provide concrete type - just a family of traits, similar to what xwt-core offers. The applications have to have full control over the concrete types they want to use in their implementation, as long as said types implement the traits - meaning it is the end application job to apply the composition of types.

An example of that would look something like this:

type Transport = bevy_net_client::Client<bevy_net_xwt::Client<xwt_web_sys::Client>>;

fn main() {
    let mut transport: Transport = bevy_net_client::Client::new();

    transport.spawn_ioloops().unwrap();

    let net_plugin: bevy_opt1_net::Plugin<Transport, mygame::Protocol> = bevy_opt1_net::client::Plugin {
        transport,
        // ... more configuration
    };

    App::new()
        .add_plugins(net_plugin)
        .add_systems(
            Startup,
            |mut commands: Commands| {
                // `bevy_net_client` command that takes the `bevy_net_xwt`-specific (i.e. driver specific) params
                // This would need to be tied together via generics, so maybe use a different query
                // than the raw `Commands` here...
                commands.bevy_net_client_connect(bevy_net_xwt::ConnectionParams { url: "127.0.0.1:3536", });
            },
        )
        .add_systems(
            Update,
            {
                |frame_idx: bevy_opt1_net::ScheduleFrame, mut replication_tree: bevy_opt1_net::ReplicationTree<mygame::Protocol>| {
                    replication_tree.insert(frame_idx, PingPayload);
                    // The application never sees the network layer, as it only has to interact with deterministic state
                    // which is replicated automatically...
                }
            }
        )
        .add_systems(
            Update,
            {
                |frame_idx: bevy_opt1_net::ScheduleFrame, mut replication_tree: bevy_opt1_net::ReplicationTree<mygame::Protocol>| {
                    replication_tree.insert(frame_idx, mygame::PingPayload);
                    // The application never sees the network layer, as it only has to interact with deterministic state
                    // which is replicated automatically...
                }
            }
        )
        .add_systems(
            Update,
            {
                |mut transport: bevy_opt1_net::TransportResource<Transport>| {
                    // Here's an example of a low-level access to the transport if the need be.
                    // At this point, code knows what concrete transport type is being used, and
                    // allows us to directly invoke the underlying APIs if we need to.
                    let _: &bevy_net_client::Client<bevy_net_xwt::Client<xwt_web_sys::Client>> = transport.as_ref();
                    // For instance, we can access a `xwt_web_sys`-specific function right through the types composition:
                    let driver: &xwt_web_sys::Client = transport.driver();
                    // Call something that's available in the Web WebTransport API driver only
                    // https://w3c.github.io/webtransport/#dom-webtransport-anticipatedconcurrentincomingunidirectionalstreams
                    println!("{}", driver.anticipated_concurrent_incoming_unidirectional_streams);
                }
            }
        )
        .run();
}

This should demonstrate my approach to hiding the types and reexports - for this particular system, at a lot of places I'd pick a transparent the composition of generics instead. I understand the tradeoffs between this and the alternative very well, and I think this, with a lot of care and caution in the API design, is the way to go.

Oh, yeah, note how I did't specify how the mygame::Protocol is declared - that's because it is likely generated based on the complex state machine, more like what async fn codegen does than what a manually defined enum would do.

[MOZGIII]

Or, btw, I want to change the xwt crate such that it doesn't directly import the drivers, so it's dependencies would look something like this:

├── xwt
│   └── xwt-core

And the app that uses xwt would be something like this:

├── xwt
│   └── xwt-core
├── xwt-web-sys (wasm only, if the app needs it)
│   ├── xwt-core
│   └── web-sys
├── xwt-wtransport (native only, if the app needs it)
│   ├── xwt-core
│   └── wtransport

and in the code the app will be responsible for selecting the driver.

In the main.rs you'd specify concrete types, depending on the needs of a particular compilation target:

#[cfg(wasm)]
type Client = xwt_web_sys::Client;
#[cfg(native)]
type Client = xwt_wtransport::Client;

You might also want to be able to switch between the drivers if there are more than one available for a given platform:

#[cfg(native)]
enum Client {
  // https://docs.rs/wtransport
  WTransport(xwt_wtransport::Client),
  // https://docs.rs/webtransport-quinn, if a driver for it was implemented
  WebTransportuinn(xwt_webtransport_quinn::Client),
}

While most of the code can be generic around a driver type:

struct MyState<Client: xwt_core::Client> {
  client: Client,
  // ...
}

impl<Client: xwt_core::Client> MyState<Client> {
  // We can do anything we want from what `xwt_core::Client` offers with the client now.
}

So, after my planned refactor one wouldn't be able to just use an xwt crate, one wouldn't also have to be explicit about the specific driver they want to use.

Current approach the xwt crate of re-exporting its driver is hazardous to the versioning and the API stability. Applications, for the most sub-crates they consist of, should not depend on anything but the xwt-core and be quite happy with the abstracted connection, streams and etc. But if there is a need - at concrete places they can depend on an exact driver and do the driver-specific operations - and compose and mix this concrete things with rest of the code that is generic .

[MOZGIII]

Closing as pot planned because it is currently out of scope for this crate - it needs more fundamental work on the interfaces and standards support before it would be a good time to focus on explicit bevy integration.