WIP: AudioWorkletize

[hughrawlinson]

related to #286

To be clear, this is just my sketch at how it might work, I wanted to understand how to use AudioWorklet. Would love feedback if someone has time, no worries otherwise. I'll hope to keep going with this.

I think generally I'm leaning towards separate packages here, to get around loading two copies of Meyda (or, we can rely on our users tree shaking, or maybe web modules are good).

@jakubfiala I know you worked on this back in February, would love to see some code if you have it :)

[hughrawlinson]

No idea what happened here

[jakubfiala]

Hey @hughrawlinson - thanks for starting this! I'd love to have a look at this properly over Christmas. It's definitely possible to load the worklet code via Webpack, and we can figure out a nice model for sending frame-synced feature data back to the main thread. We could just try a naive technique with postMessage called every time, and if we get perf issues, we could experiment with shared memory etc. But in my previous work simply sending the features in an object down the MessagePort has worked perfectly fine.

[Seanitzel]

Hi,

Would be happy to know what is the state of this PR, since i'm attempting to use Meyda with the standardized-audio-context and it is currently not possible...

createScriptProcessor is not a part of it..

[hughrawlinson]

@Seanitzel I think the current state is that we'll spend time on it when we can, but that use case makes a much stronger case for prioritizing it! Thanks for sharing 😄

[Seanitzel]

No problem!

Anything i can do to speed up things?

[hughrawlinson]

@Seanitzel I'll take a look today, see if there's any issues I can break out on their own, and you could pick one of those to tackle if you like!

[hughrawlinson]

@Seanitzel I got stuck here. The issue is that I haven't been able to get the webpack worklet loader to run the JS in the imported file through webpack before inlining it. That meant I couldn't actually run meyda inside the worklet. I have a vague idea that it might be possible to have webpack do some code splitting and import code into strings that get registered as blob-urls and can then be client-side imported in both the main thread and the worklet. But that's super confusing for me right now. I'll try to come back to it when I have time.

There was also a bit of a design issue in here. Since the feature extraction happens in another thread, the user of meyda would need to asynchronously get the latest audio features, which makes rendering realtime animations based on the audio features pretty tricky without an awful lot of expensive message passing. We'll need to think of a good way to solve that too.

In case it helps, I have been working on a library that does offline audio feature extraction using meyda in a worker, so if you have all your audio up front, it might be a good option. It's called feature-extractor-worker, and there's an alpha on npm. Does that suit your use case?

[Seanitzel]

@hughrawlinson
I see...it does seem a bit complicated and unfortunately i'm not sure how to solve these problems as well.

My use case is real-time extraction too.

I guess i will attempt to implement the extractors i need independently, but will keep an eye on this.

Thank you for the support! :)

[hughrawlinson]

@Seanitzel if you want to just use the feature extractors directly you can import them from Meyda still. Something like import zcr from "meyda/featureExtractors/zcr" should do it, although I'm not at a computer right now to check that's the path.

[Seanitzel]

I should have thought about trying that haha, it works!

Thank you :)

[innerop]

Hi,

Any update on this?

I believe browsers will soon be removing the WebAudio script processor API.

I can try to merge this PR on a local branch.

Is it fully working as far as you know (minus docs/package-lock.json confilicts)?

[hughrawlinson]

Hi! I haven't heard of browsers planning to remove the ScriptProcessorNode interface - can you link to an issue tracker issue for that?

If I remember correctly, this PR works, but has a significant bump in bundle size that I'd like to avoid before merging.

[innerop]

Hi Hugh,

The reason I brought it up was because I've heard the audio script processor works on the main (UI) thread, which has a lot going already in my app. The post I saw was from a while back and it said "deprecated" not removed, so I apologize for raising any alarm.

It would be very nice to have this library up to date with e latest audio processing API, but I am not really sure what the issue with the bundler may be. Is it duplicating some stuff?