Multistreams with Green Screen Effect #310
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| <html> | ||
| <head> | ||
| <!-- Dimitrios Ververidis for CERTH/ITI, 25/1/2022 --> | ||
| <meta charset="utf-8"> | ||
| <title>Multi Streams Example — Networked-Aframe</title> | ||
|
|
||
| <meta name="description" content="Dev Example — Networked-Aframe with Green Screen effect"> | ||
| <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> | ||
| <meta http-equiv="Pragma" content="no-cache" /> | ||
| <meta http-equiv="Expires" content="0" /> | ||
|
|
||
|
|
||
| <script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script> | ||
| <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.3.0/socket.io.slim.js"></script> | ||
| <script src="https://cdnjs.cloudflare.com/ajax/libs/dat-gui/0.7.7/dat.gui.min.js"></script> | ||
| <script src="/easyrtc/easyrtc.js"></script> | ||
| <script src="/dist/networked-aframe.js"></script> | ||
|
|
||
| <script src="https://unpkg.com/aframe-randomizer-components@^3.0.1/dist/aframe-randomizer-components.min.js"></script> | ||
| <!--<script src="https://unpkg.com/aframe-particle-system-component@1.0.5/dist/aframe-particle-system-component.min.js"></script>--> | ||
| <script src="https://cdn.jsdelivr.net/gh/IdeaSpaceVR/aframe-particle-system-component@master/dist/aframe-particle-system-component.min.js"></script> | ||
| <script src="/js/spawn-in-circle.component.js"></script> | ||
| <link rel="stylesheet" type="text/css" href="/css/style.css" /> | ||
| <link rel="stylesheet" type="text/css" href="/css/adv-screen.css" /> | ||
|
|
||
| </head> | ||
| <body> | ||
| <a-scene networked-scene=" | ||
| room: basic-multi-stream-green-screen; | ||
| debug: true; | ||
| adapter: easyrtc; | ||
| audio: true; | ||
| video: true; | ||
| "> | ||
|
|
||
| <a-assets> | ||
| <img id="grid" src="https://img.gs/bbdkhfbzkk/stretch/https://i.imgur.com/25P1geh.png" crossorigin="anonymous"> | ||
| <img id="sky" src="https://i.imgur.com/WqlqEkq.jpg" crossorigin="anonymous" /> | ||
|
|
||
| <!-- Avatar --> | ||
| <template id="avatar-template"> | ||
| <a-entity class="avatar" networked-audio-source> | ||
|
|
||
| <a-plane id="screenPlane" color="#FFF" width="2" height="1.5" position="0 0.6 -0.15" material="side: back" | ||
| networked-video-source-green-screen="streamName: screen; GreenThresholdIn:0.02"></a-plane> | ||
| </a-entity> | ||
| </template> | ||
| </a-assets> | ||
|
|
||
| <!-- Pawn --> | ||
| <a-entity id="player" | ||
| networked="template:#avatar-template;attachTemplateToLocal:false;" | ||
| camera | ||
| position="0 1.6 0" | ||
| spawn-in-circle="radius:3" | ||
| wasd-controls look-controls | ||
| > | ||
|
|
||
| <a-sphere class="head" | ||
| visible="false" | ||
| random-color | ||
| ></a-sphere> | ||
| </a-entity> | ||
|
|
||
| <!-- Grid --> | ||
| <a-entity position="0 0 0" | ||
| geometry="primitive: plane; width: 10000; height: 10000;" rotation="-90 0 0" | ||
| material="src: #grid; repeat: 10000 10000; transparent: true; metalness:0.6; roughness: 0.4; sphericalEnvMap: #sky;"></a-entity> | ||
|
|
||
| <!-- Lights --> | ||
| <a-entity light="color: #ccccff; intensity: 1; type: ambient;" visible=""></a-entity> | ||
| <a-entity light="color: #ffaaff; intensity: 1.5" position="5 5 5"></a-entity> | ||
|
|
||
| <!-- Sky --> | ||
| <a-sky src="#sky" rotation="0 -90 0"></a-sky> | ||
|
|
||
| </a-scene> | ||
|
|
||
| <!-- GitHub Corner. --> | ||
| <a href="https://github.com/networked-aframe/networked-aframe" class="github-corner"> | ||
| <svg width="80" height="80" viewBox="0 0 250 250" style="fill:#222; color:#fff; position: absolute; top: 0; border: 0; right: 0;"> | ||
| <path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path> | ||
| </svg> | ||
| </a> | ||
| <style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}} | ||
| </style> | ||
|
|
||
| <div class="actions"> | ||
| <button id="screen-btn" type="button" class="button" style="height:80px">Send Video</button> | ||
| </div> | ||
|
|
||
| <!-- Preview what stream you send --> | ||
| <div class="videoUI" style="text-align:center; color:white"> | ||
| <span style="font-size: small">This is what you send (Before, After processing)</span> | ||
|
|
||
| <div class="video-canvas-container"> | ||
| <!-- Here we first put the data to process them --> | ||
| <canvas id="canvas_secret" class="output_canvas" width="800px" height="600px" style="width:160px; height:120px"></canvas> | ||
| </div> | ||
|
|
||
| <div class="video-canvas-container" style="left:200px"> | ||
| <!-- Here are the processed data --> | ||
| <canvas id="video_output_canvas" class="output_canvas" width="800px" height="600px" style="width:160px; height:120px"></canvas> | ||
| </div> | ||
| </div> | ||
|
|
||
| <script> | ||
| // --------- here I capture my own stream in order to transmit it as I want ---- | ||
| const constraints = { | ||
| video: true, | ||
| }; | ||
|
|
||
| // Get my camera stream (to do some processing before capturing to canvas | ||
| const myvideo = document.createElement('video'); | ||
| navigator.mediaDevices.getUserMedia(constraints).then((stream) => { | ||
| myvideo.srcObject = stream; | ||
| myvideo.play(); | ||
| }); | ||
|
|
||
| // ---------- The gui for configuring other clients inputs ----------------- | ||
| const API = { | ||
| GreenThresholdIn: 0.02, | ||
| GreenThresholdOut: 5, | ||
| }; | ||
|
|
||
| // GUI for changing thresshold for green screen | ||
| const gui = new dat.GUI( { width: 400 }); | ||
|
|
||
| const videoInbound = gui.addFolder("Video Inbound"); | ||
| videoInbound.open(); | ||
|
|
||
| videoInbound.add( API, 'GreenThresholdIn', 0, 0.7 ).onChange( function () { | ||
| document.getElementById("screenPlane").setAttribute("networked-video-source-green-screen","GreenThresholdIn", | ||
| API.GreenThresholdIn); | ||
|
|
||
| } ); | ||
|
|
||
| const videoOutbound = gui.addFolder("Video outbound"); | ||
| videoOutbound.open(); | ||
|
|
||
| videoOutbound.add(API, "GreenThresholdOut", 0, 255, 1).onChange((val) => { | ||
| API.GreenThresholdOut = val; | ||
| }); | ||
|
|
||
| gui.open(); | ||
|
|
||
| // Process my stream | ||
| var canvasElement = document.getElementById('video_output_canvas'); | ||
| var canvasElementSecret = document.getElementById('canvas_secret'); | ||
|
|
||
| const w = canvasElement.width; | ||
|
|
||
| console.log("w -----> ", w); | ||
| const h = canvasElement.height; | ||
| var canvasCtx = canvasElement.getContext('2d'); | ||
| var canvasCtxSecret = canvasElementSecret.getContext('2d'); | ||
|
|
||
| var canvasStream; | ||
| let r,g,b; | ||
|
|
||
| (function loop() { | ||
| canvasCtxSecret.drawImage(myvideo, 0, 0, w, h); | ||
| let imageData = canvasCtxSecret.getImageData(0, 0, w, h); | ||
|
|
||
| for (var i = 0; i < imageData.data.length; i += 4) { | ||
| r = imageData.data[i]; g = imageData.data[i+1]; b = imageData.data[i+2]; | ||
| if (g-r> API.GreenThresholdOut){ | ||
| imageData.data[i] =0; | ||
| imageData.data[i+1]=255; | ||
| imageData.data[i+2]=0; | ||
| } | ||
| } | ||
| canvasCtx.putImageData(imageData, 0, 0); | ||
| setTimeout(loop, 1000 / 30); // drawing at 30fps | ||
| } | ||
| )(); | ||
|
|
||
| // final processed stream to send | ||
| canvasStream = canvasElement.captureStream(30); | ||
|
|
||
| // Screen share status | ||
| let screenEnabled = false; | ||
|
|
||
| // Screen share button element | ||
| const screenBtnEle = document.getElementById('screen-btn'); | ||
|
|
||
| // Define custom schema for syncing avatar color, set by random-color | ||
| NAF.schemas.add({ | ||
| template: '#avatar-template', | ||
| components: [ | ||
| 'position', | ||
| 'rotation' | ||
| ] | ||
| }); | ||
|
|
||
| // Called by Networked-Aframe when connected to server | ||
| function onConnect () { | ||
| console.log("onConnect", new Date()); | ||
| setTimeout(function(){ | ||
| // This is the processed stream that we want to send | ||
| // We do not want the default stream camera (takes bandwidth) | ||
| NAF.connection.adapter.enableCamera(false); | ||
| },5000); | ||
|
There was a problem hiding this comment. Did you try setting There was a problem hiding this comment. I will try that also. I totally forgot it as it has been a long time ago. There was a problem hiding this comment. Yes you are right, it works with video:false |
||
|
|
||
| // Handle screen button click (Off and On) | ||
| screenBtnEle.addEventListener('click', function() { | ||
| if (screenEnabled) { | ||
| NAF.connection.adapter.removeLocalMediaStream("screen"); | ||
| screenEnabled = false; | ||
| screenBtnEle.textContent = 'Send Video'; | ||
| } else { | ||
| NAF.connection.adapter.addLocalMediaStream(canvasStream, "screen"); // stream | ||
| screenEnabled = true; | ||
| screenBtnEle.textContent = 'Stop sending Video'; | ||
| } | ||
| }); | ||
| } | ||
|
|
||
| </script> | ||
| <!-- | ||
| Known issues with this demo, some cases are not handled: | ||
| - If participant A shares their screen, the partipant B sees the other participant's screen. | ||
| When participant A stops their screen share, the other participant will see a frozen screen, the last image received. | ||
| - If participant A starts screen share, stops, and restarts the screen share, the other participant won't see it. | ||
| --> | ||
| </body> | ||
| </html> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On Linux, a process that runs in a terminal in foreground can be stopped with Ctrl+C.
Is this not working for you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The process might be initialized with a cron job or some other way. We usually put an automatic way in case for some reason the server restarts. So , in order to kill it you must search it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. Generally pm2 is used to deploy a nodejs app, to start it as a service. Last time I deployed easyrtc-server.js on a test server I read https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-16-04 and used this:
Feel free to modify your doc to include that.