A way to a) detect if MediaElementAudioSourceNode is CORS-restricted & b) revert createMediaElementSource

[WofWca]

Describe the feature

Perhaps for a) a boolean property and an event emission (in case e.g. element.currentSrc changed from same-origin to cross-origin or vice versa) would do.

b) is needed in order to not "break" the element whose src is CORS-restricted so at least it is possible for the user to hear its sound.
It also can come in handy if you decided to switch to a different AudioContext (e.g. you don't like sampleRate or latencyHint of the current one).

Or perhaps createMediaElementSource needs to be replaced with something that doesn't cause this trouble.

Is there a prototype?
No.

Describe the feature in more detail

For context, I'm developing an extension which calls AudioContext.createMediaElementSource for every media element on every page.
As we know, cross-origin media playback is allowed, but MediaElementAudioSourceNode outputs zeroes for such media (https://webaudio.github.io/web-audio-api/#MediaElementAudioSourceOptions-security), which means that when you call createMediaElementSource for such media it simply gets muted with no conservative (meaning without re-creating the element or something like this) way to revert it, which (at least in my case) is worse than not calling the method at all.
Some websites fetch media data from a different origin (and even a different subdomain is a different origin - see, for example, Zoom recordings), but the Access-Control-Allow-Origin header is not always applied to such media responses, which makes createMediaElementSource (therefore my extension) not only useless but even harmful.

In a perfect world it would also be nice if createMediaElementSource always just worked for browser extensions somehow. Web Audio API may not be the only scope this issue lies in in this case (then I would appreciate if someone gave me directions on how to proceed).

And FYI I'm not the greatest web guru, I may just be missing something.

[WofWca]

Turns out the "b)" part is a duplicate of #1202, sorry. I'll write some comments there.
But a) holds anyway.

[WofWca]

What is the purpose of creating a MediaElementSourceNode for every element on the page?

To analyze the volume of the media. Here's the extension itself: https://github.com/WofWca/jumpcutter.

you can fetch resources outside of CORS and CSP restrictions

The method you described is very hard (if possible) to implement so that it works on (almost) every website, and it's a bit invasive (it requires changing the src (or srcObject)). Also I believe it's gonna breach the CORS protection (a malicious website would become able to access the cross-origin media (e.g. by el.captureStream()).

then observe both loadedmetadata (fired when the header is set) and error (fired when the header is not set) events.

This to me also sounds like a headache to get working everywhere. It requires either creating a new element with the same source, or manipulating the original one.
The former may not always work because the fetch may succeed for the original element but fail for the new one for reasons other than CORS restrictions. Currently this doesn't work, for example, on YouTube, where it uses MediaStream with URL.createObjectURL.
The latter is pretty invasive. For example, the website may also attach an error event listener that may redirect the user to an error page.

Anyway, I appreciate your response.

[Korilakkuma]

How about using Object URL ?

fetch('https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20')
  .then((response) => response.blob())
  .then((blob) => {
    const objectURL = window.URL.createObjectURL(blob);
    const mediaElement = document.querySelector('audio');

    mediaElement.src = objectURL;

    const context = new AudioContext();
    const source = new MediaElementAudioSourceNode(context, { mediaElement });

    source.connect(context.destination);

    mediaElement.play();
  })
  .catch(console.error);

[padenot]

Audio WG call:

• This seems useful indeed. It might be best to have something on the media element: a boolean property stating the cross-origin status, and an event firing when that changes (e.g. server returns 302).

This way this works consistently with HTMLMediaElement.captureStream() + AudioContext.createMediaStreamSource(), or any variation on that. The same-origin status really is a property of the source.

[WofWca]

@guest271314 I'll take time to think about the workarounds you suggested, but currently I'm inclined to think that having such API would be nice either way.

[WofWca]

Also regarding b):
#1285 initially had this commit, which would solve the issue (at least for me), but then it got changed to what we have now.

[hoch]

The idea on #2453 (comment) still stands. The CORS property should be queried via MediaElement.

[chrisguttandin]

This Stack Overflow answer states that captureStream() throws if there is a CORS issue. If that's true than it could be used to get the information asked for in this thread.

But it looks like that behavior is only specified for a HTMLCanvasElement.captureStream().

Content from a canvas that is not origin-clean MUST NOT be captured. This method throws a SecurityError exception if the canvas is not origin-clean.

It is not specified for HTMLMediaElement.captureStream(). The specification for that just says:

The contents of the track might become inaccessible to the current origin due to cross-origin protections. For instance, content that is rendered from an HTTP URL can be subject to a redirect on a request for partial content, or the enabled or selected tracks can change to include cross-origin content.