Audio autoplay / add set audio volume level to kvmd api

[markfrancisonly]

Is your feature request related to a problem? Please describe:

pikvm sessions always start with client audio disabled

---

Describe the solution you'd like:

Ability to enable audio by default and change the setting from an kvmd api call

Chrome browser allows audio autoplay after user interaction. Requesting the ability to start session with client audio enabled and/or enable audio and set the volume level from kvmd

e.g.
api_set_url="https://127.0.0.1/api/streamer/set_params?audio="
post_response=$(curl -s -X POST -k -u "$USERNAME:$PASSWORD" "$API_SET_URL$new_value")

---

Describe alternatives you've considered:

Directly calling the embedded Janus websocket

---

Additional context:

I rely on audio notifications from one of my pcs. When using pikvm, I first switch the input channel using a GPIO menu command and then manually make a couple more mouse clicks to enable audio. Since I'm switching to the PC using a mouse click on a GPIO menu item in the browser, I've satisficed the requirement to automatically play audio using WebRTC in Chrome and the additional mouse clicks are technically unnecessary and could be added to my GPIO menu input script

[mdevaev]

Hello. It's not possible because this is a browser limitation. Browsers do not allow us to play video and audio automatically, only muted video. As you noted, it is allowed after user interaction, but this heuristic is very unreliable and has not always worked.

[markfrancisonly]

Are you referring to the Media Engagement Index in chrome's autoplay policy?

I've had a good experience using audio autoplay code for displaying go2rtc webrtc camera streams from home assistant in javascript using Chrome. I believe it is still true that autoplay is still allowed when the user has interacted with the domain (click, tap, etc.).

I've used a one-time mouse click/key up event listener to the document to unmute the video player.

document.addEventListener('click', ev => { WebRTCsession.enableUnmute(); }, { once: true, capture: true });

or

          const handleKeyUp = (ev) => {
                WebRTCsession.enableUnmute();

                const mute = "KeyT";
                for (const session of iterator) {
                    switch (ev.code) {
                        case mute:
                            session.toggleVolume();
                            break;
                    }
                }
            }

to call

static enableUnmute() {
        if (WebRTCsession.unmuteEnabled)
            return;
        else
            WebRTCsession.unmuteEnabled = true;

        const iterator = WebRTCsession.sessions.values();
        for (const session of iterator) {
            const media = session.media;
            if (media) {
                if (media.classList.contains('unmute-pending')) {
                    media.classList.remove('unmute-pending');
                    media.muted = false;
                }
            }
        }
    }

I did use a custom media element play method to prevent Chrome from pausing the media stream when unmute play isn't allowed

    media.play()
            .then(() => {
                if (!media.muted) {
                    media.classList.remove('unmute-pending');
                    WebRTCsession.enableUnmute();
                }
                media.classList.remove('pause-pending');
            })
            .catch(err => {
                if (err.name == "NotAllowedError" && !media.muted && muted != true) {
                    WebRTCsession.unmuteEnabled = false;

                    media.classList.add('unmute-pending');
                    this.trace('unmuted play failed, reloading media muted');

                    media.muted = true;
                    media.load();
                    this.playMedia(true);
                }
                else if (err.name === "AbortError") {
                    this.trace(`media play aborted: ${err.message}`);
                }
                else {
                    this.trace(`media play failed: ${err.message}`);
                }
            });

autoplay always works after the first mouse click or keypress.

Maybe a hotkey?

[markfrancisonly]

Hard refresh then destroys the permission. I personally would still very much like to attempt to unmute with a single click. In my code, I've used a mute icon overlay when unmuted play fails to provide a visual indication that audio is not enabled

[markfrancisonly]

If there is a way to call the javascript ui from the kvmd api, then the entire DOM can be searched for the media element and the above code can be worked in

[mdevaev]

Are you referring to the Media Engagement Index in chrome's autoplay policy?

Yes. And my testing has shown that it does not work well.

Since you have a positive experience, you can offer a quick PR (maybe a dirty proof of concept) to show how it should work. The sound is turned on here: https://github.com/pikvm/kvmd/blob/master/web/share/js/kvm/stream.js

[mdevaev]

Note that KVMD API can't affect UI.

[mdevaev]

Sup?

[markfrancisonly]

Dirty and untested:

let userInteractionOccurred = false;
let firstMediaElementFound = null;
let unmutePerformed = false;

const observer = new MutationObserver(function(mutationsList, observer) {
    mutationsList.forEach(mutation => {
        mutation.addedNodes.forEach(node => {
            if (node.tagName === 'VIDEO') {
                unmuteMedia(node);
                observer.disconnect();
            }
        });
    });
});

// observing mutations in the DOM for media additions
observer.observe(document.body, { subtree: true, childList: true });

function handleUserInteraction() {
    userInteractionOccurred = true;
    document.removeEventListener('keydown', handleUserInteraction);
    document.removeEventListener('touchstart', handleUserInteraction);
    document.removeEventListener('click', handleUserInteraction);

    if (firstMediaElementFound) {
        unmuteMedia(firstMediaElementFound);
        firstMediaElementFound = null;
    }
}

function unmuteMedia(media) {

    if (unmutePerformed) return true;

    if (!firstMediaElementFound) {
        firstMediaElementFound = media;
        
        media.addEventListener("play", () => {
            unmuteMedia(media);
        });
        media.addEventListener('volumechange', ev => {
            if (media.muted === false)
                unmutePerformed  = true;
        });
        media.addEventListener('pause', ev => {
            unpauseMedia(media);
        });
    }

    if (userInteractionOccurred && !unmutePerformed) {
        media.muted = false;
    }
}

function unpauseMedia(media) {
    if (media.srcObject == null) {
        // cannot play media without source stream
        return;
    }
    media.play()
        .then(() => {})
        .catch(err => {
            if (err.name == "NotAllowedError" && !media.muted) {
                
                // unmuted play failed, reloading media muted

                media.muted = true;
                media.load();
                playMedia(media);
            }
        });
}

document.addEventListener('keydown', handleUserInteraction);
document.addEventListener('touchstart', handleUserInteraction);
document.addEventListener('click', handleUserInteraction);

[mdevaev]

Thank you, I'll try.