PhoneRTC with SIP.js

[lylepratt]

Hi! Thanks for the awesome work on this plugin.

I'm looking for a solution to use SIP.js with with iOS devices. Could PhoneRTC help me with that? Is there anything preventing those libraries from working in conjunction with PhoneRTC?

Edit: @joseph-onsip I noticed that you have forked this repo and the SIP.js is an OnSIP project. Can you comment on this?

[josephfrazier]

Hi Lyle,

Thanks for your interest in SIP.js. We're currently exploring mobile support for SIP.js and surveying everything that has been done in the mobile space. What sort of use cases are you interested in, specifically? It might be more appropriate to continue the conversation in this SIP.js ticket.

Joseph

[alongubkin]

@joseph-onsip I think that PhoneRTC can fit perfectly into SIP.js. Your library could use the PhoneRTC API when on mobile. That being said, PhoneRTC still requires a lot of work. Let's cooperate!

[lylepratt]

I would be willing to sponsor some of this development, especially if we could have a POC fairly quickly.

[josephfrazier]

After discussing this with the other SIP.js developers, we've decided that we'd like to take the PhoneRTC concept and extend it into a free and open plugin that exposes the native WebRTC API to Javascript. This would allow SIP.js to "just work" when dropped into a Cordova app using the plugin. @onsip would like to host the repo for this plugin, so if you are interested in helping out, we can fork the repo and go from there.

[alongubkin]

Exposing the native WebRTC API is a concept that I've thought about before, but it's very complex. How are you planning to do video rendering?

[josephfrazier]

We're... currently thinking that over. In general, we may not be able to get by with just exposing (a subset of) the native WebRTC API, but we'd like to get as close as possible.

[josephfrazier]

Alright, after digging around for a bit, we haven't yet found a good way to render video, so we're probably going to hold off on the WebRTC API plugin for now. In #6 (comment), you mentioned that video support was coming. How are you going about it?

[lylepratt]

Wouldn't a plugin still be worth it JUST for audio support for SIP applications?

[alongubkin]

@joseph-onsip My idea was to call a JavaScript callback every time a new frame is coming. Then, you can draw it on a <canvas>.

Another idea I had, which is probably better for performance but way more complex and limited, is to draw the video on top of the browser, similarly to FastCanvas. I'm not sure if it's works in iOS, though.

Anyway, all of those ways don't work (or barely work) with a native WebRTC API plugin.

@lylepratt I believe that implementing the entire WebRTC API just for audio applications is a waste of time because there isn't much configuration. A better approach would be to improve PhoneRTC IMHO.

[alongubkin]

Okay, after a little bit of research I've thought of the perfect way of implementing video rendering.

Once a new stream (local or remote) is initialized, the JavaScript code is notified and then creates a new Blob. The user can then retrieve the URL for the blob (which is generated using createObjectUrl), that he can then use for the src attribute of a <video> tag. When a new frame is arrived, the plugin calls a JavaScript function that pushes the frame into the Blob.

[egreenmachine]

@alongubkin Can you put together a demo of this? @joseph-onsip and I were researching what you proposed quite a bit and we thought that it was not possible.

My current idea for video is to overlay a native video element on top of the webView. This can be seen here (iOS example): https://github.com/songz/cordova-plugin-opentok/blob/master/src/ios/OpenTokPlugin.m

[josephfrazier]

Well, looking into it more, I think I see how (part of) @alongubkin's idea would work.

When a new frame is arrived, ...

This is the part I wasn't sure about previously, because I didn't know you could learn when a new frame arrives. I see now that it could be done by calling RTCVideoTrack.addRenderer with a custom RTCVideoRenderer that uses the RenderFrame method to pass the frame data to JS.

... the plugin calls a JavaScript function that pushes the frame into the Blob

As far as I can tell, Blobs are immutable, so it may be better to just render the frame into a (or maybe even an )

I don't know Objective-C, but I may try to whip up a Java equivalent to see how it performs on Android.

[alongubkin]

The problem with rendering to canvas is that the frames come in YUV color mode and not RGB. HTML5 canvas only supports RGB image data, so we would need to convert the image to RGB. That would highly reduce the frame rate (especially on mobile).

The traditional solution is to buffer the incoming frames, but that would cause major issues with the audio sync.

Anyway, rendering to canvas is definitely not the solution.

About the Blob solution -- I hadn't realized that Blobs are immutable, so that wouldn't work. If we had some kind of a memory stream in JS that supports createObjectUrl, it would definitely be the best solution for video rendering. But I haven't found anything like that.

Another idea I had is to look at the chromium source code and try to implement createObjectUrl in PhoneRTC. It is impossible.

So right now I see two solutions:

• Drawing the GL surface on top of the WebView. We would need to consider scrolling, orientation change, etc.
• Creating a local HTTP server in the PhoneGap plugin (this is possible in both Android and iOS), and streaming the video on that server. Then, just put it in the src attribute of a video tag. This is pretty complicated, I'm not exactly sure how to implement it.

What's your opinion?

[lylepratt]

For reference: Here is another Cordova plugin that renders video from the device's camera to a canvas element: https://github.com/daraosn/Cordova-CanvasCamera

[alongubkin]

@lylepratt Note that this is for iOS, which is generally much faster in rendering than Android. So even if it is fast (and I'm pretty sure it isn't), in Android it would probably be much slower. But I guess the only way to be sure is to try.

Nonetheless, I think that drawing on top of the WebView is the most practical solution right now (in terms of development speed and a good FPS).

[josephfrazier]

HTML5 canvas only supports RGB image data, so we would need to convert the image to RGB

Oh, I overlooked this bit. Good point.

Another idea I had is to look at the chromium source code and try to implement createObjectUrl in PhoneRTC. It is impossible.

Agreed.

Creating a local HTTP server in the PhoneGap plugin

I hadn't considered this, but it does seem rather complex, involving fairly low-level HTTP server hacking.

I think that drawing on top of the WebView is the most practical solution right now.

Yeah, that's what it's looking like. I believe @egreenmachine is actively working on this for iOS.

[alongubkin]

Great! I guess I will start working on this for Android.

@egreenmachine are you planning to push back to the main repo? (please do!)

After Android and iOS video is ready, I think I'll start adding some more advanced configuration options for PhoneRTC so that it can be integrated to SIP.js with all of its functionallity

[egreenmachine]

I think so. I would like to mostly as a proof of concept so that we know it is possible to do in more advanced scenarios.

[egreenmachine]

My Objective-C is mediocre at best, but I started working on this and my progress can be tracked here. https://github.com/egreenmachine/phonertc/tree/ios-video

[alongubkin]

give me commit access I have some improvements

[egreenmachine]

You should be able to submit a pull request.

[alongubkin]

I've added experimental Android video chat support:

06934a6

[andresfmg]

sorry, anybody find someway to integrate phonertc with sip on phonegap build ??
thanks a lot any help