AudioWorklet & ScriptProcessor deprecation

[jakubfiala]

I have recently done some contract work using AW to extract real-time features and it's been surprisingly pleasant to implement it. I think we could do an internal refactor to use AW instead of ScriptProcessor if available, giving us out-of-the-box compatibility for when browsers start dropping SP in the future.

This would involve:

• building a new webpack target (with target: 'webworker').
• figuring out a mechanism for main-thread Meyda to locate the worklet code. This could just be provided by the developer as a string, but then we'd have to figure out how to make it work in a webpack-compiled app.
• checking to see if AW is supported at runtime, if so, loading and registering the worklet bundle
• sending computed features via the AudioWorklet's MessagePort
• calling callback when a new message comes through the port

Of course this has the downside of having to load all the extractor code on the main thread in case we need to use the SP, as well as on the worklet thread in case we're using AW. I can see a couple of potential solutions:

1. treat any code that runs within the SP as a separate loadable, just like the worklet. This introduces an additional HTTP request, but ensures we only load the extractor code once while still exporting only one version of the library
2. build a separate version of Meyda which only uses AW, and let developers decide if they want to use it or the "classic" SP version
3. build a version of Meyda which doesn't run any code on the main thread. It would then be the dev's responsibility to write a worklet script which imports Meyda, runs the extraction, etc. However, that's basically just using Meyda's offline extraction in an AudioWorklet.

(hypothetical)
Now at this point I could almost imagine just adding a note to the docs saying "If you'd like to leverage AudioWorklet, just use Meyda as a library of pure extractor functions". In the bright future where SP is gone and evergreen browsers support AW, I can imagine removing the online component of Meyda completely, perhaps providing a LIFO-like utility to buffer incoming signal with a given BS/HS. Yes, it would increase the cognitive load for the developer, but it would also be more flexible and predictable.

What are everyone's thoughts on this? It's likely we'll have to say goodbye to SP one day, so I think having a discussion early on is a good idea.

[2xAA]

Wow, nice one Jakub!

I quite like option 1 as it leaves Meyda rather pluggable(/extensible?).
In a bundle situation, importing another file to add to Meyda or to set up as a Worklet shouldn't be too bad.

Do you think maybe moving this to Meyda 5 would be a good idea?

Would be nice to do something like

MeydaAnalyzer.use([ScriptProcessorNode, AudioWorklet])

and Meyda just figures out what it should do.

Can you create Worklets from blobs as you can Workers?
That way we can abstract away the need of managing a Worklet from the user.

[hughrawlinson]

I also like option 1. I did some work on this at one point but I don't know where it is, I think I may have deleted it. We should be able to create a worklet from a blob, I think that worked from me. Tagging @JamesMilnerUK who I've talked to about this before - James what was the name of that library you used for creating workers on the fly?

[jakubfiala]

@2xAA @hughrawlinson thanks, I do like option 1 too - it is just one more file to load after all.

I wrote a naive test for making an AW from a Blob, but it seems to throw a DOMException in Chromium 72 -- see here https://codepen.io/jakubfiala/pen/gqqZeR
There might be a difference between creating Workers and Worklets (worklets are defined by a separate spec as a subset of workers afaik).
Of course I might be doing something stupid in that example :)

[chrisguttandin]

Hi everyone, I came across this issue today.

There is a bug/feature in Chrome which requires the mime type to be set when creating a Worklet from a Blob. If you specify the mime type when constructing the Blob it should work.

const workletBlob = new Blob([ workletCode ], { type: 'application/javascript; charset=utf-8' });

Just an idea: Maybe my standardized-audio-context package could simplify the migration to AudioWorklets as it will automatically fallback to ScriptProcessorNodes if AudioWorklets are not available. Let me know if there is anything I can help with.

[jakubfiala]

@chrisguttandin ooh thanks, this did indeed do the trick!

And yeah, SAC would be really nice to use here, as we could delegate the compatibility fixes to a separate package :) we should definitely consider it.

[hughrawlinson]

v3 sounds good to me too, since we could also have a separate library that packages it somehow (maybe using standardized-audio-context).

[innerop]

Hi,

Any update on this?

I believe browsers will soon be removing the WebAudio script processor API.

Thanks.

[Twinparadox]

Thank you for your great works.
Any update on this issue?
I can't find any update on this issue in this project.
Other projects also seem to have no other comment on this...

Thanks.

[hughrawlinson]

Hey @innerop and @Twinparadox - no update on this right now. As far as I know, despite ScriptProcessorNode having been deprecated in the spec, it's not expected to be removed from browsers in the near term, so there's no need to worry too much about Meyda not working on Web Audio. The difficulty with AudioWorklet is that because the worker has to be serialized, we end up with lots of duplicate code in the distributed package. When I last checked, the build tooling wasn't able to handle worklets easily either, but that might have changed. When I have time, I'll look into it. Thanks for using Meyda!

[Twinparadox]

@hughrawlinson
Thanks for your reply.
I'm also considering AudioWorklet in my project, and I know it's not easy.
Anyway, Thank you for your care about this issue.

[neuxregime]

@hughrawlinson Thank you for the wonderful library. We are currently using Meyda in a project that already has a lot going on on the main thread. And was wondering if there has been any progress with the bundling issues you've been running into? Thank you in advance.

[hughrawlinson]

Hey @neuxregime

There has been a bit of progress. I submitted a PR to the bundling library we would use, adding support for audio worklets (and a few other worklet style things). That PR hasn't been merged yet though, so I guess maybe we need to fork that package. Hopefully not though.

I've also been working on a change to the public API of meyda to solve a couple of issues - we ideally want meyda to be a configurable function, rather than a singleton module that stores state as it is now. That would make the audio worklet make a lot more sense, so I would love to land that too. I've been having trouble expressing the interface that I want in typescript though, so I kind of lost steam there. Hopefully I can pick it back up soon because I would love to publish a beta of it to get feedback.

Your point about having work on the main thread is a great one that I had forgotten - huge advantage to audio worklet is offloading the work, and I hadn't weighted that into my prioritization well enough. Thanks for pointing it out.

Unfortunately the state of packaging modules that have audio worklets requires serializing code to blobs, because the package has no way of serving the worklet code in the package consumer's website. In our case, I think this inherently means there'll be a pretty chunky string containing duplicated copies of our existing code in our bundle. So, heads up, that might be something you have to consider when this lands.

Hopefully I'm wrong about that last point and someone has figured out an amazing way to package and publish audioworklets to npm - last time I looked into this I was expecting to need to bundle commonjs and esm copies to publish on npm, but based on some dependabot PRs I've got on other projects, it looks like the the community is moving towards publishing esm directly to npm without a commonjs bundle, so this might have some implications that I haven't discovered yet.

Hope that clarifies the state of things, but let me know if you have any other questions.