Media Stream Handler: Core Development Plan & Questions

[dsilhavy]

This issue is supposed to be used to discuss and answer the following questions

• Which programming language do we want to use to implement the MSH?
• What is the main platform that the MSH should run on?
• What whould be the MVP?
• Which media player should we support initially?

[rjb1000]

Proposal on Friday, 14th October from @haudiobe and @dsilhavy is to coalesce the Media Stream Handler into the same process as the 5GMS-Aware Application, wrapping Exoplayer. This approach internalises the M7 Media Stream Handling API between the 5GMS-Aware Application and the Media Stream Handler. (The Media Stream Handler wrapper around Exoplayer could still be structured as a started foreground service to provide a clean realisation of the M7 API, however.)

Any further thoughts, @davidjwbbc?

[davidjwbbc]

For an initial implementation I think this would be fine as the 5GMS-Aware Application would be the only one using this, but ultimately it ought to be a separate Service which provides the M7d API interface to multiple applications and triggers Exoplayer when asked to play some media.

This then allows you to do things like continuing to play the media in a small overlay if the controlling application transitions to the background, and allows other applications which come to the foreground to change the playing media or at least coordinate playback (e.g. play after current media finishes).

[rjb1000]

I'm wondering whether a Media Stream Handler service would ever be released independently of the 5GMS-Aware Application in a separate installable Android APK, @davidjwbbc.

The only concrete Use Case I can think of today would be an 5GMS Application Provider with multiple 5GMS-Aware Applications that depend on the same Media Stream Handler. For the majority of cases, though, each 5GMS Application Provider only has a single 5GMS-Aware Application and it's unlikely that different 5GMS Application Providers would agree amongst themselves to develop a common Media Stream Handler service.

The exception to this might be if Google itself decided to bundle an implementation of a 5GMS Media Stream Handler with Android. (This would very approximately be analogous to the Android System Web View that provides a basic HTML rendering capability for use by third party applications that aren't fully-fledged web browsers.) This hypothetical "Android System Media Stream Handler" could then expose a de facto implementation of the 3GPP M7 client APIs to 5GMS-Aware Applications from any 5GMS Application Provider.

(In this case, though, it might make more sense for Google to provide a composite "Android System 5GMS Client" that bundles the Media Stream Handler service and Media Session Handler service together as a single installable APK offering both M6 and M7 client API implementations to 5GMS-Aware Applications.)

Given that adoption of 3GPP 5G Media Streaming by mobile operating system vendors is still some way off, bundling the Media Stream Handler with the 5GMS-Aware Application in the same APK feels like a reasonable approach to software release for now. But developing the Media Stream Handler as a separate Android service in this repository before marrying it in a single installable APK with a test application that depends on the Media Stream Handler service feels like a nice way of factoring the source code that makes it more reusable in the future.

What do you think, @haudiobe and @dsilhavy?

[dsilhavy]

General

I basically agree with what @rjb1000 wrote. Highlighting again I am not an Android developer so parts of my opinion might not reflect best practices for Android development. Looking at 26.512 Section 13, the M7 interface is used for the following:

• Methods to interact with the Mediaplayer, e.g. attach(MPD) or play()
• Configuration of the player like buffer targets latency etc
• Status information dispatched from the player like updates to metrics, buffer level etc
• Notification and errors: Events dispatched by the player to indicate a certain state, for instance an empty buffer

All of this communication needs to be exposed by the mediaplayer. The methods will probably be simple API calls while status information and notifications are dispatched via events that an application can register for. Similar to this:

player.play()
player.pause()

player.on(dashjs.MediaPlayer.events["PLAYBACK_ENDED"], function () {
     console.log('Playback ended')
 });

All of this needs to be setup as part of the application code. For both mediaplayers (dash.js and Exoplayer) we probably need a wrapper to be compliant with the API calls and events defined in 26.512. At least for Exoplayer the methods and events will most likely be named slightly different.

An essential part is the content decryption, decoding and rendering that is managed by the mediaplayer using the native platform functionality (e.g. EME for license acquisition in a web-based scenario).

Use case dash.js

Assuming we want to implement the architecture depicted in 26.501 and 26.512 with dash.js on Android. dash.js needs MSE and EME for playback of encrypted media streams. For that, we need a web-based environment so we would probably implement the 5GMSd-Aware Application as a webview. Within that webview there is a simple html page that includes dash.js as a library. In addition, there is some glue/wrapper code around the APIs offered by dash.js for instance to trigger playback and to register for certain events. Communication with the Media Session Handler is hopefully possible from a webview via IPC.

Use case Exoplayer

Similar to what is described above for dash.js. However, we launch a native view(activity). The Exoplayer is added as a gradle dependency in the build.gradle. Communication with the Media Session Handler is done via IPC. The Media Session Handler runs in the background as a bound service.

General Options

1. There is a single 5GMS-Aware Application that either starts a webview (dash.js) or a native view(Exoplayer). I am not sure how the information which player to start is provided, though. We implement a wrapper for each of these players to be compliant with 26.512.
2. We have a dedicated application for dash.js and for Exoplayer. Both can use the same Media Session Handler. Each application implements a wrapper to be compliant with 26.512.
3. We have an application that starts the Media Stream Handler (dash.js or Exoplayer) as a service? I am not sure how the communication would happen here, though. Both are libraries that expose certain API functions. In addition, how would the video rendering be linked to the application?

As Richard pointed out, we could also have dedicated repositories for Exoplayer and dash.js that can be included as a dependency in a 5GMS Aware application. Both of these repositories would more or less only use dash.js or Exoplayer and provide a wrapper around the player functions and methods.

Conclusion

At least from my understanding approach 2) feels like the most natural one. As a service provider, I decide which library I use for media playback. Then I implement my app on top of that library.

I suggest we start by adding the Media Stream Handler (Exoplayer) directly to the application and implement the required wrapper and interface to the Media Session Handler as part of the application. Also communication with the Media Session Handler (running as a background service for multiple applications) would be handled here (see also question below).

Additional questions

• There is a link from the MediaPlayer to the Media Session Handler (M6d). The question is how the Media Session Handler subscribes to notifications and status updates from the media player. My assumption is that we have one Media Session Handler running as a service for multiple applications. I think adding this communication between MediaPlayer and Media Session Handler directly to the media player is not a good approach. We should leave Exoplayer and dash.js untouched. I suggest we proxy this through the application or the Media Stream Handler (which would in this case be more than a media player). Maybe that is also something that is defined in the spec? Using the Media Stream Handler or the application to relay information between media player and Media Session Handler.

[dsilhavy]

Tried to put this into a diagram for discussion in the calls

One question in this context: Are notifications, errors an status updates by the media player proxied through the MSH to the application? If so, the app could also go the direct way via the M7 Adapter and receive this information from the media player directly.
Edit: Just saw that in 26.512 13.2 this is the case. The 5GMS Aware Application is querying M7d directly.

[jordijoangimenez]

One question, I'm a bit confused about what is part of the:

• 5GMS Aware Application
• Exoplayer
• Media Stream Handler

What I did in Android is to get a generic app, introduce a layout and put the ExoPlayer inside such layout. In this case I assume Exoplayer is part of the 5GMS Aware Application (yet without the MediaStreamHandler).

Would the MediaStreamHandler be another module that would incorporate Exoplayer inside it or (I think better) just a new function that handles the communication between Exoplayer and any 5GMS related activity? Therefore isolating ExoPlayer from 5GMS?

This goes into a similar direction to what Daniel is commenting in his last comment?

It would be good to get how a generic OTT app works and try to minimize any changes to that app.

[dsilhavy]

If I understand your description correctly I think this would fit into the architecture above. You would include the MediaStreamHandler as a dependency in your application. The MediaStreamHandler incorporates Exoplayer and the two adapters. From your application point of view you would talk to the M7 adapter. At this point I am not sure if the app needs to talk to the MSH adapter for setup or anything. In that sense the Exoplayer would be isolated from the app as you only talk to the adapters.

However, that does probably not align completely with what 26.501 and 26.512 say.

[jordijoangimenez]

This is what I would like to clarify: The MediaStreamHandler incorporates Exoplayer and the two adapters.
I can understand that the MediaStreamHandler incorporates the adapters. However, why does the MediaStreamHandler incorporate Exoplayer when Exoplayer is already part of the existing OTT app? Wouldn't be less invasive to keep the OTT app that contains Exoplayer as such and then have a communication between the MediaStreamHandler and Exoplayer?

Sorry if I'm still missing something. BTW, I am assuming that an existing OTT app already includes Exoplayer as a dependency. In this way you just add extra dependencies to MediaSessionHandler and adapters?
This is what comes to my mind:

[dsilhavy]

Good point, we could also do it like this to make it easier to use the Media Stream Handler with existing applications. For that to work the app could pass an instance of the Exoplayer to the Media Stream Handler?

We should clarify that in our next call. Looking at the specs the Media Stream Handler is essentially the media player. This is why the Exoplayer is incorporated in the Media Stream Handler in my architecture.

[rjb1000]

Good discussion, @jordijoangimenez and @dsilhavy.

In designing the client software architecture, we should take a few things into consideration:

• Logically, the Media Player isn't really part of the Media Stream Handler in TS 26.501: that's just a simplification I have used in my architecture diagrams to make them easier for readers to understand. Strictly speaking, the Media Player is the Media Stream Handler (in the case of downlink media streaming). You can think of the Media Stream Handler as an abstract base class and the Media Player as a concrete subclass of if. (The Media Streamer for uplink media streaming is the other concrete subclass.)
  • The implementation architecture doesn't necessarily have to follow the logical architecture slavishly, of course, but it's good to be clear about what's what.
• Pluggable design. You have an objective to make a DASH.js-based Media Stream Handler and an ExoPlayer-based Media Stream Handler. There are some common aspects (e.g. the handling of inbound M7 API method invocations) and some aspects that are particular to the chosen
  • To help refine our thinking, it would be instructive to draw the DASH.js-based Media Player in relation to Application 2. This would help to identify which aspects of the "adapter" are common regardless of the Media Player in use, and which are unique to the integration.
• Potential reuse of the client-side components. Spiltting things up into smaller chunks makes them more resuable.

[rjb1000]

I'm unclear about the distinction between the two purple adapter boxes in the above figure.

• M7 Adapter.
• MSH Adapter.
  • Is this a Media Session Handler adapter or a Media Stream Handler? Best to avoid using ambiguous abbreviations!

The way I envisage things is that there is a more monolithic M7 server that exposes services to the 5GMS-Aware Application and to the Media Session Handler (as you correctly identified in the specifications) and which adapts the 3GPP-defined API to the chosen Media Player, using a different plug-in for each supported type. (This is somewhat analogous to the plug-in architecture @davidjwbbc used for the 5GMS Application Server so that it can be integrated with different web proxy servers.) The service users then invoke M7 methods on the adapter and can register callbacks to receive notifications.

[rjb1000]

I agree with @jordijoangimenez that, in practice, the M7 service aspects of the Media Stream Handler are going to be pretty tightly integrated with the 5GMS-Aware Application. But it might be clearer to illustrate these two as separate entities that happen to run inside the same app process.

It's then a separate discussion about whether the 5GMS-Aware Application part of the app invokes the M7 API on the Media Session Handler part of the app using local method calls or IPC.

[rjb1000]

Per my comment about the Media Session Handler design on 5G-MAG/rt-5gms-media-session-handler#1, I think it would be instructive to draw dotted lines on the software architecture diagram indicating Android process boundaries (and application packaging scope).

[dsilhavy]

@rjb1000 @jordijoangimenez Thank you for the comments I tried to clarify some aspects with a new version below.

[rjb1000]

Getting better, @dsilhavy . I think the main thing still missing is a second blob inside each of the dotted blue boxes marked "5GMS-Aware Application". This is the main code providing the user interface furniture and business logic that also uses/provides the M6 and M7 APIs, perhaps by instantiating the appropriate purple adapter classes inside itself.

• In one kind of deployment (the one you have in mind here), the 5GMS-adapted Media Player sits alongside the application logic inside the same process.
• In another kind of deployment, an almost identical 5GMS-adapted Media Player with just a slightly different life-cycle wrapper class could instead run as a shared foreground service in another process.

By factoring it this way, both kinds of deployment can be supported from a single code repository. This would make the code more reusable by different flavours of application.

[dsilhavy]

Thank you all for the comments in the last call. Based on our discussion, please find an updated diagram below.

• Blue borders indicate Android process boundaries
• A single adapter class that implements the PlayerAdapter interface and extends the MediaSessionHandlerAdapter

[rjb1000]

Thanks for the update, @dsilhavy. This is getting clearer now. Some quick feedback:

My suggestion last week was to show two separate red boxes in the app process:

1. 5GMS-Aware Application. This contains the M7d client. The box could be usefully annotated "(User Interface and business logic)".
2. Media Stream Handler. This provides the M7d server. Your name "(5GMS-adapted Media Player)" is then the annotation to the main name.

With this arrangement, you can then show reference point M7d as the line (currently labelled "methods") between the 5GMS-Aware Application red box and the Media Player red box, making this implementation architecture compliant with the 5GMSd reference architecture.

In this particular deployment, the two red boxes are peers inside the same app process (e.g. 5GMSDemoApp.class sits alongside 5GMSExoPlayerWrapper.class and ExoPlayer.class) and reference point M7d is realised as local method invocations. The compiled class files for both red boxes are packaged into the same installable APK for distribution.

The reason for depicting it this way is that in a different kind of deployment (which it would be useful to draw as a separate diagram), the 5GMS-Aware Application runs in one Android process (dotted blue line box) and the Media Player runs as an Android Foreground Service (in a different dotted blue line box). In this case, invoking the M7d methods requires Inter-Process Communication since it crosses a process boundary. And the Media Player is installed from a separate APK to the 5GMS-Aware Application.

[dsilhavy]

@rjb1000 Thank you for the quick feedback, is this roughly what you had in mind for the first kind of deployment?

What I had in mind with the previous architecture was a 5GMS aware application that contains business logic and includes the 5GMS-adapted Media Player / Media Stream Handler as a dependency. That's why the smaller red box was included in the bigger red box. Then I do local calls from my 5GMS-Aware-Application to an instance of MediaStreamHandler.class. The MediaStreamHandler.class initializes the ExoPlayerAdapter.class.

[rjb1000]

is this roughly what you had in mind for the first kind of deployment?

Yes, @dsilhavy. (I would attach the "methods" invocation arrow directly to the M7d client.)

Of the two depictions, I prefer the one with two red boxes because the 5GMS-adapted Media Player isn't really included inside the 5GMS-Aware Application. To my mind it's not a containment relationship.

Showing two red boxes will make drawing the alternative implementation architecture diagram simpler, and makes it clear that these are two separate pieces of software to be developed.

[dsilhavy]

Updated version including dash.js.

[rjb1000]

In drawing the three different component configuration models, @dsilhavy, it would be useful to name them so that we can talk about them unambiguously. Here is my take:

• Component configuration model A ("Disintegrated") The 5GMS-Aware Application, Media Stream Handler (including the Media Player) and Media Session Handler run as three separate processes.
  • The Media Stream Handler is built as a foreground service and packaged in its own separate APK.
    • The chosen Media Player is built as a subproject of this.
  • The Media Session Handler is built as a background service and packaged in its own separate APK.
  • The 5GMS-Aware Application is built and packaged in its own separate APK bound to the above two services.
• Component configuration B (Partially integrated) The 5GMS-Aware Application and Media Stream Handler (including the Media Player) run in the same process; the Media Session Handler runs in a separate process.
  • The Media Stream Handler is built as a subproject of the 5GMS-Aware Application and included in the same APK as it.
    • The chosen Media Player is built as a subproject of this.
  • The Media Session Handler is built as a background service and packaged in its own separate APK.
• Component configuration C (Fully integrated) The 5GMS-Aware Application, Media Stream Handler (including the Media Player) and Media Session Handler all run in the same application process.
  • The Media Stream Handler is built as a subproject of the 5GMS-Aware Application and included in the same APK as it.
    • The chosen Media Player is built as a subproject of this.
  • The Media Session Handler is built as a subproject of the 5GMS-Aware Application and included in the same APK as it.

[dsilhavy]

Thanks @rjb1000 that sounds good. Based on your description I updated the illustrations, the three different versions below:

[dsilhavy]

I am converting this to a discussion as we have the initial implementation in place now. Our current implementation follows configuration option B