In-Camera VFX for Indie filmmakers using Unreal Engine

An intro using just an iPhone, a Camera, and Unreal Engine

Unreal and ICVFX (In-Camera VFX) with Scott Kirvan : Getting into Virtual Production without an LED Volume in UE5

Presented by VFX/VP Supervisor, Scott Kirvan, May 16, 2023, at the Seattle Unreal Engine Meetup.

00:00 Intro to virtual production
02:24 ICVFX ideas for independent filmmakers
05:25 Required equipment review
06:50 The "real world" scale and position discussion
08:50 Measuring the "real world"
12:50 Getting into Unreal. (UE_QuickLaunch) - https://github.com/ScottKirvan/UE_Qui...
14:20 (ScooterUtils Edit plugin) - https://github.com/ScottKirvan/Scoote...
15:00 Create and save a default level
16:10 Build the screen from a basic plane inside Unreal
17:28 Create UV material
25:15 Cinecam & LiveLink
27:00 Unreal Remote 2 on iPhone/Tracker
29:40 nDisplay Config
30:40 Add Cluster Node
31:25 Add Static Mesh to nDisplay
32:51 Add ICFVX (inner frustum) Camera to nDisplay
33:46 LiveLink Preset
34:40 Switchboard
38:12 Launching nDisplay
40:30 Camera overheats + general tech fail! haha :-)
42:55 Tech Test sample: Magic Box
43:30 Tech Test sample: Meerkat, nDisplay, multiple monitors

Further details on the project: https://www.scottkirvan.com/ICVFX/
Unreal Seattle's Discord: https://discord.gg/aPdE2QqqeZ
Seattle's Unreal Engine Community page: https://communities.unrealengine.com/...
Filmed on-location by Natt Thangvijit. Edited by Natt & Scott.
Equipment/software used: Asus Nitro 5 laptop w/ NVIDIA 3080, Sigma fp camera, Unreal 5.2.

Required Unreal Project Plugins:

• Static Mesh Editor Modeling Mode (enabled by default in UE5+)
• nDisplay (enabling nDisplay automatically enables the LiveLink plugin too)
• Switchboard

Add'l plugins/tools referenced:

• Scooter Utils- Unreal plugin used to change editor font-size for readability and for quick restarts of the editor.
• UE QuickLaunch - right-click context menu for launching unreal without the Epic Games Launcher.

Process Outline:

1. Collect IRL (In Real Life) Data.
   • Display
     • measurements (X, Y, Height to base of screen -- all in cm)
     • resolution (we used Full HD, 1920x1080px)
   • Camera
     • film back measurements (we had a full-frame DSLR video camera)
     • lens length (45mm fixed lens in our case)
     • addn'l lens stats (f-stop, etc. - which we weren't going to be using in the demo)
     • position relative to display (we set out camera up 1 meter away from the screen, aligned to the bottom of the screen)
   • Computer - ip address (use ipconfig in a windows command prompt to find your current IP)
2. Reconstruct our IRL display screen in Unreal.
   • create a plane actor, duplicate it's static mesh into your content folder - delete the actor
   • drag the new SM plane into your scene, rotate it, align it's uv's, then scale it
     • we created a quick test material to visualize our UVs here
3. Add CineCamera Actor and set it up to match your IRL camera.
   • set the lens length, filmback, disable focus so we're not dealing with DOF
4. Set up Livelink & connect the iPhone using Unreal Remote 2, and save the LiveLink Preset.
   • open livelink window
   • open UnrealRemote2 on iPhone - set the IP address, connect.
   • find the iphone in LiveLink, create a preset, save the preset, set the preset as default in Project Settings
   • add a livelink transform component to your cinecam, and
5. Configure nDisplay - add VP (Viewport) node, add static mesh screen, and add camera.
6. Launch listener.
7. Launch, Configure, and Connect Switchboard.
8. From Switchboard, run your Unreal project in nDisplay mode.
9. Make your movie and get famous.

Video copyright (c) 2023 Natt Thangvijit & Scott Kirvan

Required Equipment

• Computer: capable of running Unreal Engine.
• Camera: any camera that can record video will do.
• iPhone: used as a tracking source and attached to the camera.
• Network: you need the iPhone and the Computer connected to the same network so they can communicate.
• Display: This is the display/TV/monitor/projector/LED Wall/etc. that you will be projecting to and filming with your camera. Your computer monitor will work for this and may be all you need to "tell your story" on film. This setup will work with multiple Displays too.

Resources

• Unreal Engine 4.27 In-Camera VFX Tutrials
• This info on GitHub
• www.ScottKirvan.com/ICVFX
• Contact me: icvfx@skvfx.com