Starting a new mediasoup SFU adapter

[Vesper0704]

I have implemented the mediasoup adapter and achieved audio/video transmission just as webrtc/easyrtc adapters have done. It is based on SFU architecture and can greatly improved the efficiency and capacity of audio/video chat. Would it be possible that I can submit a new PR?

[vincentfretin]

That's great! I prefer new adapters be in a separate repository, similar to naf-janus-adapter. If you have a standalone repository, and you want it to be in the networked-aframe organization, we can later move it there. This way you are free to continue developing the adapter further and maintain it.

[Vesper0704]

Glad to hear that! I'll refine some of the code parts and post the repository url here soon.

[Vesper0704]

Here is the repository of mediasoupAdapter.
It is great to let it in the organization, because there are more masters who can help to make it better.
Thanks!

[vincentfretin]

Thanks for sharing it. If you are willing to continue this work, here are some notes you should address before moving it to the networked-aframe organization.

You shouldn't have all the networked-aframe code in your repository, I see that the copy of the code is already outdated.
What you should do is just use a cdn url for networked-aframe, see for example
https://github.com/networked-aframe/naf-janus-adapter/blob/3ad1092af551868d8df48b23fa0737f34223f2ab/examples/index.html
To register the adapter, do this at the end of the adapter, see for example
https://github.com/networked-aframe/naf-janus-adapter/blob/3ad1092af551868d8df48b23fa0737f34223f2ab/src/index.js#L1130
The repository should only include the code of the adapter and only install the required dependencies for the adapter and running the server.
You can look at the old archived https://github.com/networked-aframe/naf-webrtc-adapter repository for an example of repository structure, but don't take those old webpack and babel dependencies.

I see the interesting parts are in
https://github.com/Vesper0704/naf-mediasoup-adapter/blob/master/src/adapters/MediaSoupAdapter.js
and
https://github.com/Vesper0704/naf-mediasoup-adapter/tree/master/server/mediasoup-server

I see you got inspiration from server/socketio-server.js to write
https://github.com/Vesper0704/naf-mediasoup-adapter/blob/master/server/mediasoup-server/server/mediasoup-server.js

If you're starting from scratch, you should try to use latest socketio version instead of keeping the 2.5.0 version.

You should avoid writing comments other than in English if you want contributions in this adapter.

I see you configured the server to take vp8 and opus stereo server side, the adapter is creating audio and video tracks, signaling and data are on the websocket.

What about heartbeat to not close the socket connection? Looking at the code quickly I didn't see anything related to that. See #243 about this issue.
You also need to have some logic to handle the ice failures so you can do an ice restart, or reconnect if the websocket closed. Those are done properly in the janus sfu adapter that I use in production, and also in hubs dialog adapter.

If you didn't see it, I wrote about the hubs dialog/mediasoup adapter here:
https://github.com/networked-aframe/networked-aframe/wiki/NAF-adapters-comparison#hubs-dialog-adapter-mediasoup-stack

What I look forward in a mediasoup adapter and that we don't have currently in the janus sfu adapter is the usage of video simulcast so that the producer send different video quality low/medium/high and the consumer consume the quality they want based on their bandwidth, adding several video tracks in the RTC connection so you share both your webcam and a screen share.

[Vesper0704]

Thanks for telling me that! I haven't noticed these problems before. I'll fix them later.

[Vesper0704]

Hi, I have fixed some problems for now. Here are some updates:

1. Keep adapter-related code only and introduce A-Frame and NAF by cdn url (Because I have some network issues with github sometimes, I download them in advance)
2. update socket.io/socket.io-client to v4.5.3
3. comments in English only.
4. use periodic-full-sync.js to keep ws connection alive
5. handle ICE errors and support restartIce
6. add simulcast function.

[vincentfretin]

That's great. I don't have much time to look at it right now but I added a message to networked-aframe channel on slack to see if there are people interested to try it and give you feedback.
I think this could be a good alternative to janus sfu where the server part installation may not be for everyone...
Can the mediasoup server be run on glitch? Probably if it's just a nodejs server. Did you test that?
I should write a list of things to test for a new adapter, in the meanwhile you can look at easyrtc issues
https://github.com/networked-aframe/networked-aframe/issues?q=is%3Aissue+is%3Aopen+label%3Aeasyrtc
and be sure you do not have those issues in your adapter. :)

For the heartbeat you should probably send a simple websocket message yourself like every 30s (configurable) instead of using periodic-full-sync component though. janus adapter is doing it via the minijanus package, sending a keepalive message every 30s but only if there weren't any message sent. Don't send the message if it's not needed. Relevant code here https://github.com/mozilla/minijanus.js/blob/cf259aee5d87b81b9ffd21dcc4b87efb4c2d6647/minijanus.js#L233-L248
The easyrtc adapter with open-easyrtc dependency sent a stillAlive message, see https://github.com/open-easyrtc/open-easyrtc/search?q=stillAlive

Some things to test:

• switch between wired and Wi-Fi, or Wi-Fi and cellular network and see if your adapter reconnects
• restart the nodejs server and see if your adapter reconnects.
• can connect without giving the camera permission
• can connect initially without camera permission and start a camera or screenshare later, look at my janus example https://github.com/networked-aframe/naf-janus-adapter/blob/master/examples/audio-with-camera.html
• be sure you can connect if you rejected the camera permission or the mic permission
• allow to use getUserMedia for the mic and getDisplayMedia yourself in your own app, and give the stream to the adapter similar to janus adapter setLocalStream.

[kylebakerio]

excellent work! very exciting to have this.

[kylebakerio]

If I get some time tomorrow I'll try to take a stab at seeing how it runs on glitch.

[Vesper0704]

Help wanted!
In this naf-adapter, I have tested some cases in multiple tabs, and the video/audio streams can be acquired and stored correctly. However, when I tried to append them to networked-video-source, the streams can not be rendered expectedly. I debugged and found out that problem is with NAF.utils.getNetworkedEntity(this.el) because the ownerId(networkedEl.components.networked.data.owner) is an empty string, so that getMediaStream can not retrieve correct ones from adapter.videoStreams. However, the id has been assigned already when connected to webSocket server.
If I bracket the NAF.utils.getNetworkedEntity(this.el) in a setTimeout callback, the id can be acquired correctly. However, it is wrong to modify the other source code except the adapter itself.
Looking forward to any help. Perhaps it has something to do with event loop, but I failed to figure it out.

[vincentfretin]

What you described seems to be the same issue as #320. We don't have this issue with the janus adapter where you can see your own video with networked-video-source.
For the easyrtc adapter, the issue is mainly because the adapter is getting the camera stream and set the local media before we are connected so before NAF.clientId is set, so we end up registering the streams with empty string.
The easyrtc adapter has another issue with camera stream, you can't connect if you don't have a webcam or refused the permission #374
This is why I strongly believe that getting mic et video stream should be responsibility of the app, not the adapter.
The janus adapter doesn't have this issue because we connect first then getting the camera stream ourselves and using the setLocalMediaStream API. The janus adapter is not multi streams, it only supports one audio track and one video track so this api is only relevant for the janus adapter.
When we worked on multiple video streams for the easyrtc adapter, we implemented the addLocalMediaStream(stream, streamName) and removeLocalMediaStream(stream, streamName) api (see #269). I think it's a good api, but it should be also used for the camera stream, not just for screenshare stream, so don't automatically get the camera stream if video:true on networked-scene, just ignore this video option.

[kylebakerio]

@vincentfretin is there any barrier or reason to not have the easyrtc API updated to just exactly reflect the janus api? Would be a breaking change, but would be nice to unify the api and make the adapter seamless and I also like the idea of making the stream acquisition explicit in the app side in principle.

Sounds like it wouldn't be a very difficult update in theory, no? I have mentioned I'm considering working with these features and was expecting to perhaps need to update the easyrtc adapter for the streams stuff anyways, might make sense to do that work at the same time.

[vincentfretin]

There is no barrier, it's just developer time. We definitely should use the addLocalMediaStream(stream, streamName) api for the camera stream and stop getting the camera stream automatically from the adapter before we connect, this should be the fix for #374.

[kylebakerio]

This repo is just not wanting to run on glitch, glitch is choking on setup. It keeps giving me an empty node_modules folder. I'm still troubleshooting it, but I'm thinking it's a glitch.com glitch. Probably going to give up and ping support here soon.

[kylebakerio]

Ok, I seem to have it running, though I have not tested it at all:

https://mediasoup-adapter-test.glitch.me/examples/

I had to make several small changes to the server file; it's configured to run locally, not configured to run deployed.

Also, the install process seems to take more memory than a free glitch makes available (512mb). As a result, npm install hangs awkwardly. I had to boost the app to get 2gb of memory, then open the terminal and manually cd into the node_modules of express, mediasoup, and socket.io, and run npm install for each of them.

After doing that, and tweaking the index.js a bit to let glitch handle https and to use the port that glitch provides instead of the hardcoded config port, it finally started to run.

[kylebakerio]

example doesn't even seem to try to display/play received audio and video, not sure what I'm missing. Adapter doesn't seem to implement enableCamera or enableMicrophone.

The logs seem to imply that I am indeed getting streams successfully, though.

[kylebakerio]

Tried adding and adapting basic-audio.html for mediasoup,

MediasoupClient Device load successfully
NafLogger.js:19 occupants changed {occupants: {…}}
NafLogger.js:19 Opening datachannel to Xj7jmI4suwNKXZplAABi
NafLogger.js:19 Opened data channel from Xj7jmI4suwNKXZplAABi
NafLogger.js:19 Creating remote entity {networkId: '1tsug1e', owner: 'Xj7jmI4suwNKXZplAABi', creator: 'Xj7jmI4suwNKXZplAABi', lastOwnerTime: 1667870329436.75, template: '#avatar-template', …}
index.js:327 getMediaStream audio
index.js:330 adapter getMediaStream
NafLogger.js:19 Waiting on audio for Xj7jmI4suwNKXZplAABi
index.js:394 []
index.js:395 no producers currently
index.js:495 produce stream audio
index.js:541 consumer transport connect
index.js:408 subscribe sucessfully {kind: 'audio', consumerstream: MediaStream}
NafLogger.js:19 Received pending audio for Xj7jmI4suwNKXZplAABi

But then nothing happens. Not sure if there's an implied step I'm missing.

[kylebakerio]

so, in conclusion for now:

the provided example has "video: true; audio: false" set, so does not show how to get audio working. adding networked-audio-source alone does not simply work as one would expect. and the a-plane that is a networked-video-source is commented out in the example, and uncommenting it does not work.

improved example needed to show how to implement these features with this adapter, I guess. Either something else is needed to get those streams, or theoretically perhaps something is wrong with the glitch install, though I'm not seeing any errors.

[kylebakerio]

(Also, the keepalive functionality doesn't seem to work properly.)

[Vesper0704]

Hi,
The logs indeed indicate that the produce/subscribe process is successful and if you check NAF.connection.adapter.videoStreams in the console, you will see the object storing id->stream. However, there is a bug I mentioned above:

Help wanted! In this naf-adapter, I have tested some cases in multiple tabs, and the video/audio streams can be acquired and stored correctly. However, when I tried to append them to networked-video-source, the streams can not be rendered expectedly. I debugged and found out that problem is with NAF.utils.getNetworkedEntity(this.el) because the ownerId(networkedEl.components.networked.data.owner) is an empty string, so that getMediaStream can not retrieve correct ones from adapter.videoStreams. However, the id has been assigned already when connected to webSocket server. If I bracket the NAF.utils.getNetworkedEntity(this.el) in a setTimeout callback, the id can be acquired correctly. However, it is wrong to modify the other source code except the adapter itself. Looking forward to any help. Perhaps it has something to do with event loop, but I failed to figure it out.

I haven't fixed it yet. I will figure it out when I get some time. I'll appreciate it if you may offer me some advice. And as Vincent said, it is a good idea to add stream in the app instead of the adapter, I will have a try.

Thanks!

[kylebakerio]

I see, was hoping that was the case. Cool, I'm attempting to debug that now. I forgot that context in between when I first read your post and now!

You mentioned having this issue with networked-video-source, did you have the same issue with networked-audio-source? It's weird, it kind of looks like everything should work fine when debugging, just not hearing things.

[Vesper0704]

Yes, it is the same issue.

createSound() {
    NAF.utils.getNetworkedEntity(this.el).then((networkedEl) => {
      const ownerId = networkedEl.components.networked.data.owner;
     //  HERE: ownerId is en empty string
      console.log('audio-src', {ownerId});
      if (ownerId) {

        NAF.connection.adapter.getMediaStream(ownerId, this.data.streamName, 'audio')
          .then(this._setupSound)
          .catch((e) => naf.log.error(`Error getting media stream for ${ownerId}`, e));
      } else {
        // Correctly configured local entity, perhaps do something here for enabling debug audio loopback
      }
    });
  },

I add some log info in the networked-audio-source. ownerId is always an empty string in the mediasoup adapter, so the getMediaStream is not invoked. That's why the streams can be acquired but not displayed.
Same with networked-video-source because they both call NAF.utils.getNetworkedEntity(this.el) while the resolved networkedEl's id is empty.

[Vesper0704]

Hi Vincent,
I check the issue carefully with #372 (comment). However, they are not the same. The mediasoup-adapter can register the stream with correct clientId instead of the empty string. The problem is described in #372 (comment).
Maybe it is another different reason?

[vincentfretin]

For networked-audio-source and ownerId empty, it's normal for your own avatar. You don't actually need the component on your own avatar. It shouldn't be empty for other participants though.

[vincentfretin]

About the adapter API, setLocalMediaStream(stream) where stream has only zero or one audio track and zero or one video track is really specific to the janus adapter that doesn't support multistreams.
For easyrtc and mediasoup that can be multistreams, instead of setLocalMediaStream(stream), they should implement addLocalMediaStream(stream, streamName) and removeLocalMediaStream(stream, streamName)
and we should ignore the video option on networked-scene that is used currently to auto get a camera stream in the easyrtc adapter, and use getDisplayMedia and addLocalMediaStream in the app. We may also go further and remove the audio:true;video:true usage on the networked-scene component completely.
The janus adapter already ignore both audio and video options in networked-scene, the adapter could be modified to implement addLocalMediaStream and removeLocalMediaStream so that we have the same api but with the limitation of a single audio track and a single video track though.

[Vesper0704]

Thanks Vincent, I have been trying to make users see themselves and taking this as a bug while it is not. And I have fixed the bug that was mentioned by Kyle. By now, you can see each other successfully. @kylebakerio
Here are some other updates:

1. configurable heartbeat interval to keep connection alive instead of using periodic-full-sync
2. test disconnect/reconnect
3. users can enter room and sync with each other without camera permission.
   As for the addLocalMediaStream and removeLocalMediaStream , I will implement them when I get some time. It is a good idea!

Thank you, @vincentfretin and @kylebakerio !

[Vesper0704]

P.S. These changes have been updated in naf-mediasoup-adapter

[vincentfretin]

I looked at the repo again, the repo structure is good now, you did the changes as suggested, that's great.
I don't know much about latest socketio version, is it normal you have both socket.io.min.js and socketio_client_4.5.3.min.js included in your example? I think you need to remove socket.io.min.js? You don't have it in the README example.
You can remove the empty sky img.
You should do your document.querySelector('a-scene')) in a DOMContentLoaded listener, otherwise I could sometimes return null.
You need to modify the copyright in LICENSE.

FYI I had to install sudo apt install python3-pip on ubuntu 22.04 for mediasoup to build. The error was:

$ npm install
npm ERR! code 1
npm ERR! path /home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup
npm ERR! command failed
npm ERR! command sh -c -- node npm-scripts.js postinstall
npm ERR! npm-scripts.js [INFO] running task "postinstall"
npm ERR! npm-scripts.js [INFO] executing command: node npm-scripts.js worker:build
npm ERR! npm-scripts.js [INFO] running task "worker:build"
npm ERR! npm-scripts.js [INFO] executing command: make -C worker
npm ERR! make : on entre dans le répertoire « /home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup/worker »
npm ERR! # Updated pip and setuptools are needed for meson
npm ERR! # `--system` is not present everywhere and is only needed as workaround for
npm ERR! # Debian-specific issue (copied from
npm ERR! # https://github.com/gluster/gstatus/pull/33), fallback to command without
npm ERR! # `--system` if the first one fails.
npm ERR! /usr/bin/python3 -m pip install --system --target=/home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup/worker/out/pip pip setuptools || \
npm ERR! 	/usr/bin/python3 -m pip install --target=/home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup/worker/out/pip pip setuptools || \
npm ERR! 	echo "Installation failed, likely because PIP is unavailable, if you are on Debian/Ubuntu or derivative please install the python3-pip package"
npm ERR! Installation failed, likely because PIP is unavailable, if you are on Debian/Ubuntu or derivative please install the python3-pip package
npm ERR! # Install `meson` and `ninja` using `pip` into custom location, so we don't
npm ERR! # depend on system-wide installation.
npm ERR! /usr/bin/python3 -m pip install --upgrade --target=/home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup/worker/out/pip  meson==0.61.5 ninja
npm ERR! make : on quitte le répertoire « /home/vincentfretin/workspace/naf-mediasoup-adapter/node_modules/mediasoup/worker »
npm ERR! /usr/bin/python3: No module named pip
npm ERR! /usr/bin/python3: No module named pip
npm ERR! /usr/bin/python3: No module named pip
npm ERR! make: *** [Makefile:76 : meson-ninja] Erreur 1

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/vincentfretin/.npm/_logs/2022-11-12T10_19_02_739Z-debug-0.log

You may want to add instructions in the README.

It seems you changed the port to 8185, the README is still pointing to https://127.0.0.1:8181/examples

Replace the room name by something other that shooter, modify also the title of the page.
In your example you should hide the local avatar and remove the gun box. (I see you removed the gun component)
I don't see any button to start a camera share or screen share. How do you test that?
Is this testable on localhost? Or do we need to put it on a server for it to work?
I have no audio, no video for now on localhost

Error play video stream DOMException: The play method is not allowed by the user agent or the platform in the current context, possibly because the user denied permission. [networked-aframe.min.js:1:6596](https://127.0.0.1:8185/dist/networked-aframe.min.js)
DOMException: Starting videoinput failed [index.js:157:36](webpack://networked-aframe/src/index.js)
WebRTC: ICE failed, add a TURN server and see about:webrtc for more details
consumerTransport connect fail and close

Does the changes required to run it on glitch addressed in the repo? Is it possible to document it in the README?

[Vesper0704]

Hi Vincent,
Thanks for your feedback.

1. The socket.io-client should be v4.5.3 and I have commented the cdn url import in the . One of them should be removed.
2. As for the sky img, it is currently a way to avoid users failing to sync with each other(color). I found that if assets are not loaded and cache is used, users fail to sync color and all of them are white by default. It is weird, and I add the useless img to force the asset to load every time, so that everything is normal.
3. Installation of mediasoup should be ok if you follow https://mediasoup.org/documentation/v3/mediasoup/installation/.
4. As for the button to start a camera share or screen share, I think I'd better implement them in addLocalStream as you mentioned above. For now, I just add two buttons to toggle(pause or resume) video/audio streams for simple test. I will turn to addLocalStream in the future.
5. I didn't run into the error you posted in the end before. It seems that users deny the webcam permission? But I have tested this kind of case at localhost and no error came out.

I haven't run it on glitch, and I got video/audio streams successfully at localhost.
Thanks for telling me the issues and I will fix some them of them soon.

[Vesper0704]

And for the copyright in LICENSE, is there anything I can refer to?

[kylebakerio]

As for the sky img, it is currently a way to avoid users failing to sync with each other(color). I found that if assets are not loaded and cache is used, users fail to sync color and all of them are white by default. It is weird, and I add the useless img to force the asset to load every time, so that everything is normal.

@vincent I have a hunch this is related to the bug fixed in aframe master and that having assets causes a script to run which changes the order of execution in aframe... would need to test, but I've run into obscure behavior around having assets and it interacting with NAF's initial load before.

Also, yes, python 3 is required to install mediasoup, but aside from needing a boosted app on glitch for memory reasons, python 3 is available and it isn't a problem there, fwiw.

@Vesper0704 this is good to know, I had removed that as it just looked like a mistake along with some other html mistakes, but I did start to run into that color bug and hadn't tried to fix it.

Also, you say you got video/audio streams, but your current index.html doesn't look like it's even trying to show video and audio streams--do you have a different client? I have still not gotten the audio/video to stream on glitch, but I had tried manually merging in the changes I saw in your recent commits with my changes to make it run in a server environment that wasn't localhost, so it's possible I did something wrong and just need to start fresh again. Basically, my question is just to confirm: did you upload the correct client html to get the streams received by users, or are those local changes you haven't shared?

Edit: just noticed you updated it in the last 9 hours, I last checked a few days ago. Will try again.

[Vesper0704]

Ok. Maybe the latest version will work in your test.

[Vesper0704]

Hi
I have implemented addLocalMediaStream roughly in a new branch feat/addLocalMediaStream-dev here, where the adapter ignores the video/audio options and users are required to open/resume/pause webcam/microphone/screen-share via the buttons in the left bottom corner. Since it is just a quick work, it is likely to have many issues.
I will update it according to your advice in the future and merge it into master branch.
I just added 2 networked-video-source in the a-scene for webcam and sharing screen respectively, and the UI is ugly but the basic functions can be tested already.

[vincentfretin]

For copyright, I mean the year and your name here https://github.com/Vesper0704/naf-mediasoup-adapter/blob/master/LICENSE#L3

[Vesper0704]

Thanks @vincentfretin , I have modified the LICENSE.

[kylebakerio]

Not sure what the deal is with this repo, but running npm install for the first time took almost an hour to complete. I've never seen something like that.

Total install time: 3594716ms

(this was of course on an anemic glitch.com server, though it was boosted, and a boosted glitch server gets more resources than the basic paid servers at say render.com)

Looking at the processes in the beginning, it looked related to the SSL / HTTPS stuff, which I disabled anyways as that's included built-in to glitch and can be expected to be handled in the deployed server environment. Looking at package.json, though, it's not immediately clear what dependency is leading to what sub dependency that's leading to this issue? Didn't dig too deep there yet.

Also, are you removing console.warn?

Anyways: while everything seems to be running and the socket part of the server is clearly up, It's frustrating; in the console, I seem to get the stream, but nothing actually plays. Must be close, but just not sure what the issue is yet, I'm thinking it's likely client side and autoplay related, something small, but that's a guess for now while I poke around--maybe the streams are empty?

https://mediasoup-test-3.glitch.me/

Looking around in the console, the video tracks are muted: true for me, even though I do see my webcam light turn on and I did get video permissions requested when opening the page, fwiw. I sometimes do see this error line being hit in networked video source's init():
return i.log.error("Error getting media stream for ".concat(n), e)
(that's the minified version, but it's easy to find the real one with that log message)

This results in the browser error message,

networked-aframe.min.js:formatted:376 Error play video stream DOMException: The play() request was interrupted by a call to pause(). https://goo.gl/LdLk22

The audio tracks are not muted, though, and I don't hear them either.

@Vesper0704 do you see any reason why this wouldn't be working here? Do you have any working examples up?

[kylebakerio]

Sanity checking with webrtc internals, there is indeed some kind of failure happening. On an easyrtc-based connection I can observe the connectiong staying live, while on the mediasoup demo I'm seeing

connectionstatechange
failed

15 seconds after the connection starts.

Full log:

[kylebakerio]

to be clear, you can use this demo to compare a working easyrtc example, and you can find the tool shown here @ chrome://webrtc-internals/

[kylebakerio]

You'll have to forgive me, answer might be obvious with this info, but it's late and I'm rusty, it's been a while since I had to dig deep in webrtc debugging.

[kylebakerio]

digging around in the mediasoup documentation and the config file in the repo, made some changes around ssl stuff, it looks like the connection doesn't immediately error out and I do see what I think is my local media shown live:

Still haven't heard anything or seen anything in the app itself, and it looks like there are no peer connections...

I am seeing

url: stun:stun2.l.google.com:19302
address: 192.168.0.x
port: 53094
host_candidate: 192.168.0.x:53094
error_text: STUN binding request timed out.
error_code: 701

maybe there's a problem with the stun server setting?

not immediately clear what issue is. let me know if either of you know where to go forward from here, otherwise I'll try to pick this up again later.

[kylebakerio]

(if you want to remix my glitch to try: remember, the install takes an hour and I think it has to be boosted or it won't work.)

[Vesper0704]

Hi @kylebakerio ,
Have you tested locally, or the issue just happened when tested on glitch.com?I can get video/audio streams seen and heard locally like this: . And I haven't run into such issues you mentioned for now. I will have a look at the demo on glitch and try to figure it out.

[kylebakerio]

I have not tested locally, my focus is trying to get this to run in a deployable server environment.

Your image link is broken, but I assume it's an image of it working. :)

You have only gotten this running locally, I take it?

[kylebakerio]

I'm suspecting that the problem may be related to ports being open, which could be the config and/or glitch server environment, still researching though.

[kylebakerio]

Looks like glitch does not currently allow using udp: https://support.glitch.com/t/listen-for-udp-messages/1410/3

that is unfortunate, but we should be able to run mediasoup over tcp, right?

in the meantime, I may also try to get this running on a render.com server instead, which I suspect will be a bit more full featured on the server side--both options will be ideal to have anyways.

[kylebakerio]

I think we may need to use WebRtcServer instead of passing an ip.
https://mediasoup.org/documentation/v3/mediasoup/api/#WebRtcServer

A WebRTC server brings the ability to listen on a single UDP/TCP port to WebRtcTransports. Instead of passing listenIps to router.createWebRtcTransport() pass webRtcServer with an instance of a WebRtcServer so the new WebRTC transport will not listen on its own IP:port(s) but will have its network traffic handled by the WebRTC server instead.

[Vesper0704]

Yes, I have only gotten this running locally.
I didn't know much about the deployment issue happening on glitch. I will try deploying the repo on another public server and find out how it works.
I haven't used the WebRtcServer but it sounds like a reasonable solution to the problem. I'll give it a try.
Thanks!

[vincentfretin]

It's probably like any other sfu, you indeed need an outgoing port range that can be opened for rtp media.

[vincentfretin]

and WebRTC is using DTLS so it's udp, not tcp

[kylebakerio]

mediasoup supports both udp and tcp, though. obviously udp is ideal, but it would need to manage over tcp for a glitch deployment. (docs).

I think we'd need to setup a local reverse proxy within the glitch deploy to get it running in that context. It's starting to become pretty sub-optimal at this point as a useful demo. I've started playing with render.com, but I need to migrate some old heroku projects before they shut down their free tier anyways, so I began with that before I try this... but their free tier might be too anemic for the install process, and their lowest paid tier is more expensive than glitch is...

Mediasoup might just not be able to run on any free servers (even glitch I need to boost for install it seems, to be fair), but open to hear about other ideas. Maybe just accepting it would have to be paid and putting together a guide for digital ocean or google cloud or aws would make more sense?

[Vesper0704]

Hi @kylebakerio .
It took me some time to deploy and test the adapter on a public server due to some network issue in China and the low capacity of my Tencent Cloud Server(the installation of mediasoup requires high memory and CPU usage). However, it finally works anyway.
I did not use WebRTC Server and just exposed some necessary ports for tcp/udp. At first, I didn't receive video/audio either, but the problem is that I forgot to set the ip correctly before I ran it.

// in server/config.js
webRtcTransport: {
        listenIps: [
          {
            ip: '0.0.0.0',
            announcedIp: '162.14.105.224', // server ip
          }
        ],
        maxIncomingBitrate: 1500000,
        initialAvailableOutgoingBitrate: 1000000,
        // reasonable because the outgoing bandwidth is usually less than incoming bandwidth
      }

Did you reconfigure the listenIps?
After this, everything works.
Perhaps in your case, the solution is to use tcp if the udp is not allowed.
Thanks!

[Vesper0704]

Hi @vincentfretin ,
I wonder if the api style in the new branch of mediasoup-adapter is right. I would appreciate it if you take a glance at it in readme.md and offer some advice to help me carry on the implementation of addLocalMediaStream.

[kylebakerio]

I asked some guys in the mediasoup forum, and they mentioned that running it in a heroku-like environment (which glitch is) is full of headaches and very difficult, possibly won't be doable.

I got started trying on google cloud, but other stuff has come up and wasn't able to get around to it. Very good to hear you were able to get it working. Is your hosted version publically available to test?

I still plan to test it soon.

[Vesper0704]

I tested it and then shut it down because the resources on my server is limited. I'll reopen it if you want to have a look.

[vincentfretin]

Hi @vincentfretin , I wonder if the api style in the new branch of mediasoup-adapter is right. I would appreciate it if you take a glance at it in readme.md and offer some advice to help me carry on the implementation of addLocalMediaStream.

Hi,
oh last time I looked only in master, I didn't see you had a branch.
Your readme is completely out of sync already with what you have in index.html where you refactored the async await stuff and declared webcamStream, microphoneStream, screenshareStream. Instead of copying it in the README, just add the link to the index.html really, it's better to read a working example than a stale code snippet.
The logic you implemented with getUserMedia/getDisplayMedia on the app side that call the adapter addLocalMediaStream is what I described yes, perfect.

I see you implemented a generic API resumeStream(type) and pauseStream(type), it's indeed necessary because we can have any stream.
For compatibility with other examples (so we can just switch the adapter on existing examples in networked-aframe repo), you may consider adding this api (add the necessary checks of course, test it)

enableMicrophone(enabled) {
  enabled ? this.resumeStream('audio') : this.pauseStream('audio')
}
enableCamera(enabled) {
  enabled ? this.resumeStream('video') : this.pauseStream('video')
}

It seems you implemented only enabling/disabling a track, you should also implement stopping, really removing the track from the RTCPeerConnection (the equivalent through the mediasoup api) with removeLocalMediaStream.

About your implementation adapter.setHeartbeatTimer(30)
I learned recently that socketio has some options to do this, look at open-easyrtc they removed their own implementation to use the mechanism provided by socketio https://github.com/open-easyrtc/open-easyrtc/pull/100/files

Did you take the changes from Vesper0704/naf-mediasoup-adapter#2 into consideration?

[Vesper0704]

Hi @vincentfretin
I just took a look at the PR you mentioned in open-easyrtc. It seems that socket.io has set reconnection: true as the default value: https://socket.io/docs/v4/client-options/#reconnection.
Maybe we don't have to handle the reconnection ourselves? The heartbeat is necessary, and socket.io does that too: https://socket.io/docs/v4/server-options/#pinginterval. The default value is 25s, maybe it is just the right interval .
I will modify the issues you mentioned. Thanks!

[Vesper0704]

And as for the changes from Vesper0704/naf-mediasoup-adapter#2, Kyle said it was a wrong click. I took a look at it and I think it is better to keep https as the protocol. Since I have already got it run on a public server and received streams as expected, maybe I don't have to take those changes.