compositor

Warning

GPAC's wiki has moved to wiki.gpac.io.

This github wiki will no longer be updated.

You can contribute to GPAC's documentation here.

Compositor

Register name used to load filter: compositor
This filter may be automatically loaded during graph resolution.

The GPAC compositor allows mixing audio, video, text and graphics in a timed fashion.
The compositor operates either in media-client or filter-only mode.

Media-client mode

In this mode, the compositor acts as a pseudo-sink for the video side and creates its own output window.
The video frames are dispatched to the output video PID in the form of frame pointers requiring later GPU read if used.
The audio part acts as a regular filter, potentially mixing and resampling the audio inputs to generate its output.
User events are directly processed by the filter in this mode.

Filter mode

In this mode, the compositor acts as a regular filter generating frames based on the loaded scene.
It will generate its outputs based on the input video frames, and will process user event sent by consuming filter(s).
If no input video frames (e.g. pure BIFS / SVG / VRML), the filter will generate frames based on the fps, at constant or variable frame rate.
It will stop generating frames as soon as all input streams are done, unless extended/reduced by dur.
If audio streams are loaded, an audio output PID is created.

The default output pixel format in filter mode is:

• rgb when the filter is explicitly loaded by the application
• rgba when the filter is loaded during a link resolution
  This can be changed by assigning the opfmt option.
  If either opfmt specifies alpha channel or bc is not 0 but has alpha=0, background creation in default scene will be skipped.

In filter-only mode, the special URL gpid:// is used to locate PIDs in the scene description, in order to design scenes independently from source media.
When such a PID is associated to a Background2D node in BIFS (no SVG mapping yet), the compositor operates in pass-through mode.
In this mode, only new input frames on the pass-through PID will generate new frames, and the scene clock matches the input packet time.
The output size and pixel format will be set to the input size and pixel format, unless specified otherwise in the filter options.

If only 2D graphics are used and display driver is not forced, 2D rasterizer will happen in the output pixel format (including YUV pixel formats).
In this case, in-place processing (rasterizing over the input frame data) will be used whenever allowed by input data.

If 3D graphics are used or display driver is forced, OpenGL will be used on offscreen surface and the output packet will be an OpenGL texture.

Specific URL syntaxes

The compositor accepts any URL type supported by GPAC. It also accepts the following schemes for URLs:

• views:// : creates an auto-stereo scene of N views from views://v1::.::vN
• mosaic:// : creates a mosaic of N views from mosaic://v1::.::vN

For both syntaxes, vN can be any type of URL supported by GPAC.
For views:// syntax, the number of rendered views is set by nbviews:

• If the URL gives less views than rendered, the views will be repeated
• If the URL gives more views than rendered, the extra views will be ignored

The compositor can act as a source filter when the src option is explicitly set, independently from the operating mode:
Example

gpac compositor:src=source.mp4 vout

The compositor can act as a source filter when the source url uses one of the compositor built-in protocol schemes:
Example

gpac -i mosaic://URL1:URL2 vout

Options

aa (enum, default: all, updatable): set anti-aliasing mode for raster graphics; whether the setting is applied or not depends on the graphics module or graphic card

• none: no anti-aliasing
• text: anti-aliasing for text only
• all: complete anti-aliasing

hlfill (uint, default: 0x0, updatable): set highlight fill color (ARGB)
hlline (uint, default: 0xFF000000, updatable): set highlight stroke color (ARGB)
hllinew (flt, default: 1.0, updatable): set highlight stroke width
sz (bool, default: true, updatable): enable scalable zoom. When scalable zoom is enabled, resizing the output window will also recompute all vectorial objects. Otherwise only the final buffer is stretched
bc (uint, default: 0, updatable): default background color to use when displaying transparent images or video with no scene composition instructions
yuvhw (bool, default: true, updatable): enable YUV hardware for 2D blit
blitp (bool, default: true, updatable): partial hardware blit. If not set, will force more redraw
softblt (bool, default: true): enable software blit/stretch in 2D. If disabled, vector graphics rasterizer will always be used
stress (bool, default: false, updatable): enable stress mode of compositor (rebuild all vector graphics and texture states at each frame)
fast (bool, default: false, updatable): enable speed optimization - whether the setting is applied or not depends on the graphics module / graphic card
bvol (enum, default: no, updatable): draw bounding volume of objects

• no: disable bounding box
• box: draws a rectangle (2D) or box (3D)
• aabb: draws axis-aligned bounding-box tree (3D) or rectangle (2D)

textxt (enum, default: default, updatable): specify whether text shall be drawn to a texture and then rendered or directly rendered. Using textured text can improve text rendering in 3D and also improve text-on-video like content

• default: use texturing for OpenGL rendering, no texture for 2D rasterizer
• never: never uses text textures
• always: always render text to texture before drawing

out8b (bool, default: false, updatable): convert 10-bit video to 8 bit texture before GPU upload
drop (bool, default: false, updatable): drop late frame when drawing. If not set, frames are not dropped until a desynchronization of 1 second or more is observed
sclock (bool, default: false, updatable): force synchronizing all streams on a single clock
sgaze (bool, default: false, updatable): simulate gaze events through mouse
ckey (uint, default: 0, updatable): color key to use in windowless mode (0xFFRRGGBB). GPAC currently does not support true alpha blitting to desktop due to limitations in most windowing toolkit, it therefore uses color keying mechanism. The alpha part of the key is used for global transparency of the output, if supported
timeout (uint, default: 10000, updatable): timeout in ms after which a source is considered dead (0 disable timeout)
fps (frac, default: 30/1, updatable): simulation frame rate when animation-only sources are played (ignored when video is present)
timescale (uint, default: 0, updatable): timescale used for output packets when no input video PID. A value of 0 means fps numerator
autofps (bool, default: true): use video input fps for output, ignored in player mode. If no video or not set, uses fps
vfr (bool, default: false): only emit frames when changes are detected. (always true in player mode and when filter is dynamically loaded)
dur (dbl, default: 0, updatable): duration of generation. Mostly used when no video input is present. Negative values mean number of frames, positive values duration in second, 0 stops as soon as all streams are done
fsize (bool, default: false, updatable): force the scene to resize to the biggest bitmap available if no size info is given in the BIFS configuration
mode2d (enum, default: defer, updatable): specify whether immediate drawing should be used or not

• immediate: the screen is completely redrawn at each frame (always on if pass-through mode is detected)
• defer: object positioning is tracked from frame to frame and dirty rectangles info is collected in order to redraw the minimal amount of the screen buffer
• debug: only renders changed areas, resetting other areas
  Whether the setting is applied or not depends on the graphics module and player mode

amc (bool, default: true): audio multichannel support; if disabled always down-mix to stereo. Useful if the multichannel output does not work properly
asr (uint, default: 0): force output sample rate (0 for auto)
ach (uint, default: 0): force output channels (0 for auto)
alayout (uint, default: 0): force output channel layout (0 for auto)
afmt (afmt, default: s16, minmax: none,u8,s16,s16b,s24,s24b,s32,s32b,flt,fltb,dbl,dblb,u8p,s16p,s24p,s32p,fltp,dblp): force output channel format (0 for auto)
asize (uint, default: 1024): audio output packet size in samples
abuf (uint, default: 100): audio output buffer duration in ms - the audio renderer fills the output PID up to this value. A too low value will lower latency but can have real-time playback issues
avol (uint, default: 100, updatable): audio volume in percent
apan (uint, default: 50, updatable): audio pan in percent, 50 is no pan
async (bool, default: true, updatable): audio resynchronization; if disabled, audio data is never dropped but may get out of sync
max_aspeed (dbl, default: 2.0, updatable): silence audio if playback speed is greater than specified value
max_vspeed (dbl, default: 4.0, updatable): move to i-frame only decoding if playback speed is greater than specified value
buffer (uint, default: 3000, updatable): playout buffer in ms (overridden by BufferLength property of input PID)
rbuffer (uint, default: 1000, updatable): rebuffer trigger in ms (overridden by RebufferLength property of input PID)
mbuffer (uint, default: 3000, updatable): max buffer in ms, must be greater than playout buffer (overridden by BufferMaxOccupancy property of input PID)
ntpsync (uint, default: 0, updatable): ntp resync threshold in ms (drops frame if their NTP is more than the given threshold above local ntp), 0 disables ntp drop
nojs (bool, default: false): disable javascript
noback (bool, default: false): ignore background nodes and viewport fill (useful when dumping to PNG)
ogl (enum, default: auto, updatable): specify 2D rendering mode

• auto: automatically decides between on, off and hybrid based on content
• off: disables OpenGL; 3D will not be rendered
• on: uses OpenGL for all graphics; this will involve polygon tesselation and 2D graphics will not look as nice as 2D mode
• hybrid: the compositor performs software drawing of 2D graphics with no textures (better quality) and uses OpenGL for all 2D objects with textures and 3D objects

pbo (bool, default: false, updatable): enable PixelBufferObjects to push YUV textures to GPU in OpenGL Mode. This may slightly increase the performances of the playback
nav (enum, default: none, updatable): override the default navigation mode of MPEG-4/VRML (Walk) and X3D (Examine)

• none: disables navigation
• walk: 3D world walk
• fly: 3D world fly (no ground detection)
• pan: 2D/3D world zoom/pan
• game: 3D world game (mouse gives walk direction)
• slide: 2D/3D world slide
• exam: 2D/3D object examine
• orbit: 3D object orbit
• vr: 3D world VR (yaw/pitch/roll)

linegl (bool, default: false, updatable): indicate that outlining shall be done through OpenGL pen width rather than vectorial outlining
epow2 (bool, default: true, updatable): emulate power-of-2 textures for OpenGL (old hardware). Ignored if OpenGL rectangular texture extension is enabled

• yes: video texture is not resized but emulated with padding. This usually speeds up video mapping on shapes but disables texture transformations
• no: video is resized to a power of 2 texture when mapping to a shape

paa (bool, default: false, updatable): indicate whether polygon antialiasing should be used in full antialiasing mode. If not set, only lines and points antialiasing are used
bcull (enum, default: on, updatable): indicate whether backface culling shall be disable or not

• on: enables backface culling
• off: disables backface culling
• alpha: only enables backface culling for transparent meshes

wire (enum, default: none, updatable): wireframe mode

• none: objects are drawn as solid
• only: objects are drawn as wireframe only
• solid: objects are drawn as solid and wireframe is then drawn

norms (enum, default: none, updatable): normal vector drawing for debug

• none: no normals drawn
• face: one normal per face drawn
• vertex: one normal per vertex drawn

rext (bool, default: true, updatable): use non power of two (rectangular) texture GL extension
cull (bool, default: true, updatable): use aabb culling: large objects are rendered in multiple calls when not fully in viewport
depth_gl_scale (flt, default: 100, updatable): set depth scaler
depth_gl_type (enum, default: none, updatable): set geometry type used to draw depth video

• none: no geometric conversion
• point: compute point cloud from pixel+depth
• strip: same as point but thins point set

nbviews (uint, default: 0, updatable): number of views to use in stereo mode
stereo (enum, default: none, updatable): stereo output type. If your graphic card does not support OpenGL shaders, only top and side modes will be available

• none: no stereo
• side: images are displayed side by side from left to right
• top: images are displayed from top (laft view) to bottom (right view)
• hmd: same as side except that view aspect ratio is not changed
• ana: standard color anaglyph (red for left view, green and blue for right view) is used (forces views=2)
• cols: images are interleaved by columns, left view on even columns and left view on odd columns (forces views=2)
• rows: images are interleaved by columns, left view on even rows and left view on odd rows (forces views=2)
• spv5: images are interleaved by for SpatialView 5 views display, fullscreen mode (forces views=5)
• alio8: images are interleaved by for Alioscopy 8 views displays, fullscreen mode (forces views=8)
• custom: images are interleaved according to the shader file indicated in mvshader. The shader is exposed each view as uniform sampler2D gfViewX, where X is the view number starting from the left

mvshader (str, updatable): file path to the custom multiview interleaving shader
fpack (enum, default: none, updatable): default frame packing of input video

• none: no frame packing
• top: top bottom frame packing
• side: side by side packing

camlay (enum, default: offaxis, updatable): camera layout in multiview modes

• straight: camera is moved along a straight line, no rotation
• offaxis: off-axis projection is used
• linear: camera is moved along a straight line with rotation
• circular: camera is moved along a circle with rotation

iod (flt, default: 6.4, updatable): inter-ocular distance (eye separation) in cm (distance between the cameras).
rview (bool, default: false, updatable): reverse view order
dbgpack (bool, default: false, updatable): view packed stereo video as single image (show all)
tvtn (uint, default: 30, updatable): number of point sampling for tile visibility algorithm
tvtt (uint, default: 8, updatable): number of points above which the tile is considered visible
tvtd (enum, default: off, updatable): debug tiles and full coverage SRD

• off: regular draw
• partial: only displaying partial tiles, not the full sphere video
• full: only display the full sphere video

tvtf (bool, default: false, updatable): force all tiles to be considered visible, regardless of viewpoint
fov (flt, default: 1.570796326794897, updatable): default field of view for VR
vertshader (str): path to vertex shader file
fragshader (str): path to fragment shader file
autocal (bool, default: false, updatable): auto calibration of znear/zfar in depth rendering mode
dispdepth (sint, default: -1, updatable): display depth, negative value uses default screen height
dispdist (flt, default: 50, updatable): distance in cm between the camera and the zero-disparity plane. There is currently no automatic calibration of depth in GPAC
focdist (flt, default: 0, updatable): distance of focus point
osize (v2di, default: 0x0, updatable): force output size. If not set, size is derived from inputs
dpi (v2di, default: 96x96, updatable): default dpi if not indicated by video output
dbgpvr (flt, default: 0, updatable): debug scene used by PVR addon
player (enum, default: no): set compositor in player mode

• no: regular mode
• base: player mode
• gui: player mode with GUI auto-start

noaudio (bool, default: false): disable audio output
opfmt (pfmt, default: none, minmax: none,yuv420,yvu420,yuv420_10,yuv422,yuv422_10,yuv444,yuv444_10,uyvy,vyuy,yuyv,yvyu,uyvl,vyul,yuyl,yvyl,nv12,nv21,nv1l,nv2l,yuva,yuvd,yuv444a,yuv444p,v308,yuv444ap,v408,v410,v210,grey,algr,gral,rgb4,rgb5,rgb6,rgba,argb,bgra,abgr,rgb,bgr,xrgb,rgbx,xbgr,bgrx,rgbd,rgbds,uncv): pixel format to use for output. Ignored in player mode
drv (enum, default: auto): indicate if graphics driver should be used

• no: never loads a graphics driver, software blit is used, no 3D possible (in player mode, disables OpenGL)
• yes: always loads a graphics driver, output pixel format will be RGB (in player mode, same as auto)
• auto: decides based on the loaded content

src (cstr): URL of source content
gaze_x (sint, default: 0, updatable): horizontal gaze coordinate (0=left, width=right)
gaze_y (sint, default: 0, updatable): vertical gaze coordinate (0=top, height=bottom)
gazer_enabled (bool, default: false, updatable): enable gaze event dispatch
subtx (sint, default: 0, updatable): horizontal translation in pixels towards right for subtitles renderers
subty (sint, default: 0, updatable): vertical translation in pixels towards top for subtitles renderers
subfs (uint, default: 0, updatable): font size for subtitles renderers (0 means automatic)
subd (sint, default: 0, updatable): subtitle delay in milliseconds for subtitles renderers
audd (sint, default: 0, updatable): audio delay in milliseconds
clipframe (bool, default: false): visual output is clipped to bounding rectangle