Agoraverse- Adding Agora WebSDK for Voice channel

[chitimalli]

Agoraverse- Adding Agora WebSDK for Voice channel..
Replacing easyRCT Adapyor with AgoraWebSDK

[vincentfretin]

Hi, did you wanted to contribute some examples with the agora SDK or just wanted to share you were currently prototyping an adapter for agora? You didn't give lot of instructions in the PR description so I don't know what your plans are here.

Anyway, I changed the certificate to be able to test your agora-rtc example.
For those wanting to test that:

git clone -b agoraverse git@github.com:AgoraIO-Solutions/networked-aframe.git
cd networked-aframe
npm i
# change the certificate in server/easyrtc-server.js to my localhost certificate, remove the port 443 change
npm start
# open https://localhost:8080/agora-rtc.html

It's a mix of the shooter, google-blocks and basic-audio examples, I like it.
The audio was working, going through agora servers when I pressed the "Join Channel" button.
It seems you're still using the easyrtc server for the signaling part as well as the naf updates going through WebRTC datachannels, and you handle only the audio with the agora SDK.
This is good prototype, it's working indeed, but obviously I can't merge that, you changed completely examples/index.html and I won't maintain those examples once merged.

If you want people to learn how to use the agora SDK, you should put your example without all the networked-aframe code in a separate repository, and add it on glitch. I can then add a link in the README to your example.

Question, instead of using the easyrtc server for signaling and naf updates, can you use the agora SDK to send arbitrary stringified json to all participants via an agora websocket? You will need to properly write a NAF adapter similar to wseasyrtc so that all messages transit via the agora SDK.
If you want to write a NAF adapter for this, create it in a separate repo, not in an networked-aframe fork, similar to naf-janus-adapter

For the audio, do you have a single audio stream or one stream per participant?
If agora acts as a MCU and not a SFU, it means the user is receiving a single audio stream. In this case I understand that you didn't write a NAF adapter for the audio. In this case you can't have spatial audio.

Do you have some number to give us concerning the number of participants that can be in an Agora channel?
Even if you can have lots of participants in the voice channel, the example is still using a mesh of WebRTC datachannels between participants so it probably won't scale well. :)
You will probably have a better perf using adapter: wseasyrtc instead of the default easyrtc adapter so that naf updates goes through the websocket instead of a mesh of WebRTC datachannels. I'm talking mainly about the user bandwidth and cpu with more that 30 participants here.

[vincentfretin]

Ah actually it seems agora supports "3-D spatial audio" server side by specifying the pan and gain values for each participant, that's nice. I was just reading this interesting article
https://www.agora.io/en/blog/implementing-spatial-audio-chat-in-unity-using-agora/
so you should be able to have spatial audio with the avatars by sending regularly the pan and gain values to the agora server so that the audio is properly mixed according to those values, if I understand that correctly. It would be awesome to show case that in your example.

When I saw the PR 2446 for janus audiobridge plugin (MCU) last year that added spatial audio by specifying just the pan value (spatial_position ranges from 0 (hard left) to 100 (hard right)), but gain (volume) wasn't supported, I was wondering if we could send periodically the values to the MCU like that to use a MCU but still with spatial audio. I guess the answer yes. :)

[vincentfretin]

Question, instead of using the easyrtc server for signaling and naf updates, can you use the agora SDK to send arbitrary stringified json to all participants via an agora websocket?

I read about the different agora products, it doesn't seem agora gives such a service. Tell me if I'm wrong. NAF updates can occur every 10ms for each participant if they are moving.
So you still need a server with websocket for NAF updates (like socketio or wseasyrtc adapters in this repo).

I'm interested to see how you can get the current active speaker via the Agora SDK and show like an icon above the avatar.

[chitimalli]

@vincentfretin .. Thank you for taking a look at this PR .. The plan is to write a network adaptor similar to naf-janus-adapter

Yes Agora supports spatial audio with options on the serverside, also agora has a signaling layer called rtm, which we are looking into using for NAT updates, also the RTC(A/V) has a data channel that can be used to send small packets of data as well..