Welcome to Ubiq-Genie

Ubiq-Genie is a framework that enables you to build server-assisted collaborative mixed reality applications with Unity using the Ubiq framework. This is particularly useful for building multi-user applications that require server-side processing such as generative models, conversational agents, and real-time transcription. For more information, please refer to the Ubiq-Genie paper.

Note: Ubiq-Genie currently uses the Ubiq v0.4.2 client code. An updated version of Ubiq-Genie supporting the latest versions of Ubiq will be released later in summer 2024.

Setup

These instructions will get you a copy of the project up and running to run the samples and to start building your own applications. Please note that Ubiq's server dependencies only support installation on Windows and Linux at the moment. An alternative for macOS users is to use GitHub Codespaces to run the server-side components (in this case, you may skip installing Node.js, step 1, 2, and 3). The client-side components can be run on practically any platform supported by Unity and Ubiq.

0. Install Unity and Node.js.

1. Clone this repository somewhere on your local PC.

2. Open a terminal in the Server folder and run npm install to install the dependencies. This includes the Node.js server of Ubiq.

3. Create a virtual environment using venv or conda.

4. Install torch, torchvision, and torchaudio version 1.13 using the installation instructions on the PyTorch website. Note: please ensure to install the correct version of PyTorch matching the CUDA version of your GPU.

5. From the Server folder, install the Python dependencies by running pip install -r samples/requirements.txt or conda install --file samples/requirements.txt, depending on your virtual environment.

6. In Unity, open the Unity folder. To add Ubiq to the Unity Hub, open the Unity Hub, click Add, then navigate to /Ubiq/Unity and click Select Folder.

7. Read the README file in the corresponding folder in the Server/samples/apps folder for further setup instructions. For a list of available samples, see the Samples section below.

Documentation

For more information on how to use Ubiq-Genie, please refer to the README file in the Server folder.

Samples

The Server/samples folder contains a number of samples that demonstrate how to use Ubiq-Genie. For more information on how to use these samples, please refer to the README files in the corresponding folders. Currently, the following collaborative samples are available:

• Texture Generation: generates a texture based on voice-based input and an optional ray to select target objects
• Multi-user Conversational Agent: a conversational agent that can be interacted with by multiple users
• Transcription: transcribes audio streams of users in a room

For a demo video of the samples, please refer to the Ubiq-Genie demo video.