Will it be easy to support web worker?

[javascripet]

At first I want to say thank you for your sweet lib which really helped me a lot.
What I want is to display several 1080p videos in the page at one time.
I works well when there's only one. I checked the system monitor and found the cpu usage of one core was full. I tried to do some components ways to let the instance of jumxer runs in a iframe, but it didn't work.
It that easy to let JMUXER support web worker or give me a way to inspire me to let me do it myself ?
I know I have no rights to ask you to do anything extra for me...Just want to say thank you. What you did has already helped me so much.

[samirkumardas]

Thank you.

1. First, you have to grasp an overall idea of the code flow

2. You can start from feed method https://github.com/samirkumardas/jmuxer/blob/master/src/jmuxer.js#L79

3. You can move several things in webworker from jumxer.js i.e. getVideoFrames, getAudioFrame

4. You can move full functionality of https://github.com/samirkumardas/jmuxer/blob/master/src/controller/remux.js into webworker

5. You have to provide final buffer coming from webworker to the buffer controller https://github.com/samirkumardas/jmuxer/blob/master/src/controller/buffer.js#L73

Good luck.

[JasinYip]

I spent lots of time for trying this...

Just like @samirkumardas said, I did 1~4 steps and everything looks fine for the worker. But the rendering side still needs to do feed again. Which means do extractNALu again, so it cannot only bring us better performance even worst. Please tell me I'm wrong...😂

I also tried calling onBuffer with the worker returned chunks directly with manually calling createBuffer before. But it still cannot get through because there are a lot of state conditions coupling the remuxs(which we don't need it).

So I stuck here...

[JasinYip]

In my case I need to play multiple videos at the same time and also some user interactions triggers some UI rendering, which cost lots of CPU resources. As we know if we do too much caculations in the single-thread-based JavaScript runtime, the UI might get stucked sometime. So I wanted to take better advantage of the multi-core processers with web worker.

[samirkumardas]

@JasinYip
Sorry, I could not help much with this at the moment as passing very busy days here.
It requires some refactoring obviously.
You can move all code of the feed method into the webworker as it does not have DOM dependency.
Whenever data is feeded, the next processing will take place in the worker. The whole payload is going to be ready in the webworker-end and forward to jmuxer what we process onBuffer method.

Well, I admit the scenario should be more complex when you do actual refactoring.

[witaly-iwanow]

It will work out of the box soon: https://github.com/wolenetz/mse-in-workers-demo/

[mattilaiatostek]

I did some POCing and was able to run jMuxer in a web worker with Chrome 108 with few changes only. You need to pass the handle of the MediaSource instance to the main thread's video element just like in that example. Buffer clearing functionality need to access the videoelement's currentTime and wasn't sure what's the best way to do it. I implemented it by listening video element's timeupdate events and passing currentTime to the worker.

[matjazPrijatelj]

I have also tested and posting my code example if someone wants to play:

Tried with 4 paralell webstreams, working great.

` JMuxer.isSupported = () => MediaSource.canConstructInDedicatedWorker;

  JMuxer.prototype.setupMSE = function () {
    console.warn("override setupMSE");
    this.mediaSource = new MediaSource();
    this.isMSESupported = MediaSource.canConstructInDedicatedWorker;
    this.url = URL.createObjectURL(this.mediaSource);
    // this.node.src = this.url;
    this.mseEnded = false;
    this.mediaSource.addEventListener("sourceopen", this.onMSEOpen.bind(this));
    this.mediaSource.addEventListener("sourceclose", this.onMSEClose.bind(this));
    this.mediaSource.addEventListener("webkitsourceopen", this.onMSEOpen.bind(this));
    this.mediaSource.addEventListener("webkitsourceclose", this.onMSEClose.bind(this));
  };

  JMuxer.prototype.clearBuffer = function () {
    if (this.options.clearBuffer && Date.now() - this.lastCleaningTime > 10000) {
      for (var type in this.bufferControllers) {
        var cleanMaxLimit = this.getSafeClearOffsetOfBuffer();
        this.bufferControllers[type].initCleanup(cleanMaxLimit);
      }

      this.lastCleaningTime = Date.now();
    }
  }; // also depends on currentTime

  JMuxer.prototype.startInterval = function () {
    var _this3 = this;

    if (_this3.options.flushingTime) {
      this.interval = setInterval(function () {
        _this3.applyAndClearBuffer();
      }, this.options.flushingTime || 1000);
    }
  };

  JMuxer.prototype.cancelDelay = function () {}; // implemented in main, depends on currentTime

  JMuxer.prototype.onBuffer = function (data) {
    if (this.options.readFpsFromTrack && typeof data.fps !== "undefined" && this.options.fps != data.fps) {
      this.options.fps = data.fps;
      this.frameDuration = Math.ceil(1000 / data.fps);
      debug.log(`JMuxer changed FPS to ${data.fps} from track data`);
    }

    if (this.env == "browser") {
      if (this.bufferControllers && this.bufferControllers[data.type]) {
        this.bufferControllers[data.type].feed(data.payload);
      }
    } else if (this.stream) {
      this.stream.push(data.payload);
    }

    if (this.options.flushingTime === 0) {
      this.applyAndClearBuffer();
    }
  };

  jmuxer = new JMuxer({
    node: {},
    mode: "video",
    clearBuffer: true,
    flushingTime: 0, // was 30, 0 clears buffer immidiately
    fps: msg.data?.fps ?? 30,
    debug: false
  });

  console.warn(jmuxer.env);
  console.warn(jmuxer.mediaSource);

  let handle = jmuxer.mediaSource.handle;
  postMessage({ topic: "handle", arg: handle }, [handle]);

  wsWorker.addEventListener("message", onMessageHandler);
  wsWorker.addEventListener("error", onErrorHandler);
  wsWorker.addEventListener("close", onCloseHandler);`