This is an exploratory project, which demonstrates the drawing and text recognition possibilities in web VR environment. Drawing part represents the reduced and slightly adapted version of Marlon Lückert's "Whiteboard VR". Text recognition part is made available through Microsoft’s TrOCR model, hosted on HuggingFace. Recognized text is turned into image through another AI model hosted on HuggingFace.
To enable drawing component:
- Attach it to the head of your html file, for example:
<script src="texture-painter-component.js"></script>
- Attach attribute of "texture-painter" to a desired primitive, for example to plane:
<a-plane texture-painter position="0 1.5 -4" rotation="0 0 0" width="5" height="4"></a-plane>
If necessary define the attribute parameters (color: color; background: color; size: numeric; eraseAll: boolean), for example:
texture-painter="color: green; background: red; size: 15"
3. Make sure to make primitive, where drawing will take place, is clickable if used with VR or withouth if it is mobile or desktop. In other words apply a class name and use cursor and raycaster:
VR:
<a-plane texture-painter class="clickable" position="0 1.5 -4" rotation="0 0 0" width="5" height="4"></a-plane>
<a-entity class="controller" laser-controls="hand: left" raycaster="objects: .clickable;" line="color: #000000"></a-entity>
<a-entity class="controller" laser-controls="hand: right" raycaster="objects: .clickable;" line="color: #000000"></a-entity>
Mobile/Desktop:
<a-plane texture-painter position="0 1.5 -4" rotation="0 0 0" width="5" height="4"></a-plane>
<a-entity cursor="rayOrigin: mouse" raycaster="objects: .clickable;"></a-entity>
Text recoginition and image generation based on text are enabled through API endpoints. See how everything works in the provided example.
The project is powered by AFrame and HuggingFace hosted API endpoints.
To see the application at work: Demo application