Perception 1.0.0-preview.1

Upgrade Notes

The Perception package has been upgraded to support versions 2021.3+ and 2022.1+ of the Unity Editor. Along side this change, support for Unity Editor version 2020 has been deprecated. Existing Unity 2020 projects that use the perception package will need to be upgraded to ensure compatibility with the Perception package.

CategoricalParameter type specialization classes (e.g. GameObjectParameter) have been deprecated and will need to be replaced with their template equivalent (e.g. CategoricalParameter).

Added

Added the timestamp field back to the JSON output for captures.

Added the Accumulation feature for rendering techniques that require accumulated frames, with the primary use case being path tracing.

The keypoint labeler can now be configured to show an icon for an occluded keypoint during real-time visualization.

Sample scenes for different labelers

Added full SkinnedMeshRenderer support to the 3D Bounding Box labeler, enabling tight-fitting boxes on humans, animals, and other animated characters.

Added Metadata labeler.

Multiple semantic segmentation labelers can be present at once

RenderedObjectInfo and bounding box labelers now reports hierarchical relationships (such as instance id of parent, child, etc.) for all rendered objects.

Added support for IEnumerable in IMessageBuilder.

Super resolution option to the Unity Camera Sensor

Added a new "Sensor Type" field to the PerceptionCamera to enable users to author, select, and capture data from different camera sensor types and projections within the perception framework.

Added label id and label names for each object reported in an Instance Segmentation annotation.

Added each keypoint's cartesian location in camera space

Editor And Runtime Settings added. This can be used to transfer configs from editor to runtime

Dataset generation can be resumed

Perception endpoint will now serialize annotations that implement the IMessageProvider interface.

Added consumer endpoint validation to check if endpoint properties are valid. Consumer endpoint now calls an endpoint's validation and displays errors to the user.

Added alphaThreshold parameter the PerceptionCamera to configure how translucent materials appear in segmentation images.

Added layerMaskOverride parameter to the PerceptionCamera to configure object filtering during labeler data capture.

Added a check to warn users if multiple GroundTruthRenderFeatures are present in URP projects.

Changed

Changed the default serialization format from perception to Solo.

The Perception package has been upgraded to support versions 2021.3.7+ and 2022.1+ of the Unity Editor.

Support for Unity Editor version 2020 has been deprecated.

Renamed "super resolution" Perception Camera feature to "super sampling" to clarify the exact technique used to generate the anti-aliasing effect.

Authoring new forms of pixel based ground truth data has been simplified with the new CameraSensorChannel interface. A single CameraSensorChannel will now do the work of what originally took three coordinated custom pass classes (URP, HDRP, and CrossPipeline) to accomplish.

The type specializations of the generic CategoricalParameter class can now be used directly. Derived classes like GameObjectParameter are no longer required to define new type specializations for the CategoricalParameter class.

Deprecated the CategoricalParameter.GetCategoryCount() method and replaced it with the new CategoricalParameter.Count property

Semantic segmentation annotations report labels present in the image

Removed

Dataset visualizer removed

URP support removed

Removed redundant type specialization classes that derive from the CategoricalParameter class (e.g. GameObjectParameter).

Fixed

Fixed a bug where frames would not be written to disk until the simulation was finished or quit.

Fixed a bug where using the crash restore feature would cause the --output-path flag to be ignored.

Fixed the name of the description field in the sensor definition JSON output. Changed from definition field description.

Moved Frames Per Iteration field into the Scenario Properties pane in the scenario editor.

Depth Labeler now reports what measurement strategy it's using to capture depth values.

Fixed banding artifacts in darker parts of captured images when super resolution is enabled.

Perception version data provider in case of no Resources folder in the project.

Keypoint visualizations are now a consistent size regardless of screen resolution.

Fixed a UI bug with the Perception Camera's "Override Layer Mask" toggle that caused an exception to be thrown when toggling the option.

Fixed the empty spacing between items in the Categorical Paremeter UI.

Metadata labaler delayed frame output generation.

Fixed burst compilation error with EncodeJob()

Fixed job thread count being set to zero in PixelWeightsUtility when the job system is disabled

Fixed sporadic pixel position labeler test failures due to a shader that was not prewarmed

Issue with keypoint labelers and GameObjects without Animator components

Metrics now report empty strings instead of null.

GenericMetrics now report vector of values instead of just the first value.

Renamed RgbCapture imageEncodingFormat to imageFormat to match Solo spec

Fix FloatRangeElement uxml so that it doesn't cause errors in Unity versions over 2021

Exception in KeypointLabeler when deleting and recreating humans each iteration

The PerceptionCamera component with no labelers will no longer throw NullReferenceExceptions when viewed in the inspector.

AnimationRandomizer now chooses random normalized start time for the selected animation instead of always starting from the beginning.

Known Issues

When using the Accumulation feature, the Scene view should not be visible at runtime, otherwise the captured images will not be accumulated.

Perception 0.11.2-preview.2

Fixed

Install and Open Data Visualizer on Mac

Documentation

Perception 0.11.1-preview.1

Added

Added the ability to define the output of a standalone player with an '--output-path' command line argument

Changed

Exceptions thrown in randomizers will now end the run

Duplicates in categorical parameters are checked and reported

User output paths are now written out in the user's settings

Fixed

Fixed captured RGB images being upside down or dark in various cases

Fixed UI when there are many randomizers in the project

Fixed Scenario UI when the last randomizer is removed

Fixed the Visualizer installation bug, Visualizer can be installed and opened properly now.

What's Changed

• Add missing tutorial and images by @csun in #452

New Contributors

• @csun made their first contribution in #452

Full Changelog: Release-0.10.0-preview.1...Release-0.11.1-preview.1

Perception 0.10.0-preview.1

Added

Added support for consumer endpoints allowing a user to define how they want to generated data to be serialized.

Added perception project settings to allow for different perception settings to be modified, including the selection of an active endpoint.

Added progress bar to the Fixed Length Scenario.

Added the ability to drag and drop items directly onto the UI for Categorical Parameters.

Added required labelerId field for all CameraLabelers.

Added additional generic metrics for bool, bool array, and vector3.

Changed

Changed the dataset capture package to use consumer endpoints instead of only serializing to the perception format.

Changed dataset capture to report data on a per frame basis.

Upgraded com.unity.collections package dependency version to a non-preview release.

Removed dependency on com.unity.simulation packages.

Changed FixedFrameScenario to no longer be cloud specific.

Removed

Removed com.unity.simulation.capture package dependency.

Removed com.unity.simulation.core and related dependencies.

Removed Run in Unity Simulation window.

Fixed

Handled exception from analytics when labeler had an empty label config.

Updated documentation to reflect that the 3D Bounding Box Labeler does not support Skinned Mesh Renderers.

PoissonDiskSampling will no longer generate empty lists of points for editor versions 2020.3.27f1 and above.

Fixed perception endpoint to support multiple cameras.

Fixed potential race condition with annotations being registered from within asynchronous lambda jobs. Image related annotations will now be registered on the main thread in callbacks executed after ImageEncoder jobs have completed.

The PerceptionCamera UI now shows a warning when a missing/null labeler is detected.

Perception 0.9.0-preview.2

[0.9.0-preview.2] - 2021-11-02

Upgrade Notes

The minimum required Unity Editor version has been upgraded to 2020.3.16f1.

Scenario JSON configurations that were generated using previous versions will need to be manually updated due to updates to serialization and deserialization of scenarios.

Known Issues

Added

Added new visualizer tool that can visualize capture information such as Bounding Boxes, Segmentation in a browser window.

Added support for multiple cameras capturing simultaneously (visualization not yet supported)

Users can now delay the current iteration for one frame from within randomizers by calling the DelayIteration function of the active scenario.

Users can now choose the base folder location to store their generated datasets.

Added new configuration options to the Scenario JSON configuration. These include a limits block on numerical Scalars and Samplers to denote a valid range, and a state block on the Randomizers for enabling/disabling them from the config.

Added the AssetSource class for loading assets from generic sources inside randomizers.

Added a projection field in the capture.sensor metadata. Values are either "perspective" or "orthographic".

Added usage analytics for the scenario configuration. Information collected is limited to default (i.e. included in the com.unity.perception package) labelers and randomizers along with their configuration. To disable usage analytics, please refer to the following page: docs.unity3d.com/Manual/EditorAnalytics.html

Changed

Upgraded capture package dependency to 0.0.10-preview.24. This fixes an issue where RGB images were blank on OSX when MSAA was enabled & post-processing was disabled.

Changed the JSON serialization key of Normal Sampler's standard deviation property from "standardDeviation" to "stddev".

Changed the JSON serialization of Randomizers in a scenario to an array of groups rather than a dictionary.

Deprecated

Removed

Fixed

Fixed an indexing issue with the IdLabelConfig editor. When a new label was added to an empty Id Label Config with Auto Assign IDs enabled, the starting id (0 or 1) was ignored and the new label would always have an id of 0.

Perception 0.8.0-preview.4

[0.8.0-preview.4] - 2021-07-05

Upgrade Notes

Known Issues

When using URP in OSX, having MSAA enabled on the camera while the post-processing option is disabled may cause the output RGB images to be blank. As a workaround, you can disable MSAA and use FXAA instead, until the issue is fixed.

Added

Added support for 'step' button in editor.

Added random seed field to the Run in Unity Simulation Window.

Changed

Increased color variety in instance segmentation images.

The PoissonDiskSampling utility now samples a larger region of points to then crop to size of the intended region to prevent edge case bias.

Upgraded capture package dependency to 0.0.10-preview.23 to fix two issues: (1) Post processing effects were not included when capturing images in URP (2) RGB images were upside-down when post processing effects were enabled and FXAA disabled.

Deprecated

Removed

Fixed

Fixed keypoint labeling bug when visualizations are disabled.

Fixed an issue where Simulation Delta Time values larger than 100 seconds in Perception Camera would cause incorrect capture scheduling behavior.

Fixed an issue where Categorical Parameters sometimes tried to fetch items at i = categories.Count, which caused an exception.

Perception 0.8.0-preview.3

[0.8.0-preview.3] - 2021-03-24

Changed

Expanded documentation on the Keypoint Labeler
Updated Keypoint Labeler logic to only report keypoints for visible objects by default
Increased color variety in instance segmentation images

Fixed

Fixed compiler warnings in projects with HDRP on 2020.1 and later

Fixed a bug in the Normal Sampler where it would return values less than the passed in minimum value, or greater than the passed in maximum value, for random values very close to 0 or 1 respectively.

Perception 0.8.0-preview.2

[0.8.0-preview.2] - 2021-03-15

Upgrade Notes

All appearances of the term KeyPoint have been renamed to Keypoint. If you have code that relies on any renamed types or names, make sure to alter your code to reflect the new names. Additionally, if you have create custom keypoint templates, open the template assets using a text editor and replace the keyPoints: line with keypoints:.

ScenarioBase's Awake(), Start(), and Update() functions are now private. If you previously used these, replace the usages with OnAwake(), OnStart(), and OnUpdate().

Known Issues

Added

Added error message when missing Randomizer scripts are detected.

Scenario serialization has been updated to include scalar values on Randomizers and Parameters.

Added new ScenarioBase virtual lifecycle hooks: OnAwake(), OnStart(), OnUpdate(), OnComplete(), and OnIdle().

Keypoint occlusion has been added. No keypoint information will be recorded for a labeled asset completely out of the camera's frustum.

New keypoint tests have been added to test keypoint states.

The color of keypoints and connections are now reported in the annotation definition JSON file for keypoint templates.

The PerceptionScenario abstract class has been added to abstract perception data capture specific functionality from the vanilla Scenario lifecycle.

The newly added LabelManager class now enables custom Labelers to access the list of registered Labeling Components present in the Scene.

Improved UI for KeypointTemplate and added useful default colors for keypoint and skeleton definitions.

Added the ability to switch ground-truth labeling on or off for an object at runtime by enabling or disabling its Labeling component.

Changed

Renamed all appearances of the term KeyPoint within types and names to Keypoint.

ScenarioBase's Awake(), Start(), and Update() methods are now private. The newly added virtual lifecycle hooks are to be used as replacements.

Improved Run in Unity Simulation window UI.

The Run in Unity Simulation window now accepts an optional Scenario JSON configuration to override existing Scenario editor UI settings.

The GetRandomizer() and CreateRandomizer() methods of ScenarioBase have been augmented or replaced with more generic list index style accessors.

The Scenario inspector buttons for serialization and deserialization have been refactored to open a file explorer so that the user can choose where to save the generated JSON configuration or which file to import a configuration from.

RandomizerTags now use OnEnable() and OnDisable() to manage their lifecycle. This allows the user to toggle them on and off in the editor.

Upgraded com.unity.simulation.capture package dependency to integrate new changes that prevent the API updater from looping infinitely when opening the project settings window on new URP projects.

CameraLabeler methods OnBeginRendering() and OnEndRendering() now have an added ScriptableRenderContext parameter.

Deprecated

The Randomizer methods OnCreate(), OnStartRunning(), and OnStopRunning() are now deprecated and have been replaced with OnAwake(), OnEnable() and OnDisable() respectively, so as to better reflect the existing MonoBehaviour lifecycle methods.

Removed

Removed the Entities package dependency.

Fixed

Fixed a null reference error that appeared when adding options to Categorical Parameters.

Fixed ground truth not properly being produced when there are other disabled PerceptionCameras present. Note: this does not yet add support for multiple enabled PerceptionCameras.

Fixed an exception when rendering inspector for Randomizers with private serialized fields.

Fixed an issue preventing the user from adding more options to a Categorical Parameter's list of options with the Add Folder button. Add Folder now correctly appends the contents of the new folder to the existing list.

Fixed a bug where uniform probabilities were not properly reset upon adding or removing options from a Categorical Parameter's list of options.

Fixed keypoints being reported in wrong locations on the first frame in which an object is visible.

Fixed an out of range error that occurred when a keypoint template skeleton relied on a joint that was not available.

Fixed wrong labels on 2d bounding boxes when all labeled objects are deleted in a frame.

0.7.0-preview.2

Added

Added Register() and Unregister() methods to the RandomizerTag API so users can implement RandomizerTag compatible GameObject caching

Changed

Switched accessibility of scenario MonoBehaviour lifecycle functions (Awake, Start, Update) from private to protected to enable users to define their own overrides when deriving the Scenario class.

The GameObjectOneWayCache has been made public for users to cache GameObjects within their own custom Randomizers.

Fixed

Fixed the math offsetting the iteration index of each Unity Simulation instance directly after they deserialize their app-params.

The RandomizerTagManager now uses a LinkedHashSet data structure to register tags to preserve insertion order determinism in Unity Simulation.

GameObjectOneWayCache now correctly registers and unregisters RandomizerTags on cached GameObjects.

0.7.0-preview.1

Upgrade Notes

Randomization Namespace Change

The Randomization toolset has been moved out of the Experimental namespace. After upgrading to this version of the Perception package, please follow these steps:

• Replace all references to UnityEngine.Experimental.Perception.Randomization with UnityEngine.Perception.Randomization in your C# code.
• Open your Unity Scene file in a text editor and replace all mentions of UnityEngine.Experimental.Perception.Randomization with UnityEngine.Perception.Randomization, and save the file.

Random Seed Generation

Replace usages of ScenarioBase.GenerateRandomSeed() with SamplerState.NextRandomState() in your custom Randomizer code.

Sampler Ranges

Before upgrading a project to this version of the Perception package, make sure to keep a record of all sampler ranges in your added Randomizers. Due to a change in how sampler ranges are serialized, after upgrading to this version, ranges for all stock Perception samplers (Uniform and Normal Samplers) will be reset, and will need to be manually reverted by the user.

Tag Querying

The RandomizerTagManager.Query<T> function now returns the tag object itself instead of the GameObject it is attached to. You will need to slightly modify your custom Randomizers to accommodate this change. Please refer to the included sample Randomizers as examples.

Known Issues

The bounding box 3D labeler does not work with labeled assets that utilize a skinned mesh renderer. These are commonly used with animated models.

Added

Added keypoint ground truth labeling

Added animation randomization

Added ScenarioConstants base class for all scenario constants objects

Added ScenarioBase.SerializeToConfigFile()

Randomizer tags now support inheritance

Added AnimationCurveSampler, which returns random values according to a range and probability distribution denoted by a user provided AnimationCurve.

Added ParameterUIElementsEditor class to allow custom ScriptableObjects and MonoBehaviours to render Parameter and Sampler typed public fields correctly in their inspector windows.

Added new capture options to Perception Camera:

• Can now render intermediate frames between captures.
• Capture can now be triggered manually using a function call, instead of automatic capturing on a schedule.

Added 3D bounding box visualizer

Categorical Parameters will now validate that their specified options are unique at runtime.

Changed

Randomizers now access their parent scenario through the static activeScenario property.

Unique seeds per Sampler have been replaced with one global random seed configured via the ScenarioConstants of a Scenario

Samplers now derive their random state from the static SamplerState class instead of individual scenarios to allow parameters and samplers to be used outside of the context of a scenario

Replaced ScenarioBase.GenerateRandomSeed() with SamplerState.NextRandomState() and SamplerState.CreateGenerator()

ScenarioBase.Serialize() now directly returns the serialized scenario configuration JSON string instead of writing directly to a file (use SerializeToConfigFile() instead)

ScenarioBase.Serialize() now not only serializes scenario constants, but also all sampler member fields on randomizers attached to the scenario

RandomizerTagManager.Query() now returns RandomizerTags directly instead of the GameObjects attached to said tags

Semantic Segmentation Labeler now places data in folders with randomized filenames.

The uniform toggle on Categorical Parameters will now reset the Parameter's probability weights to be uniform.

Reorganized Perception MonoBehaviour paths within the AddComponentMenu.

Upgraded the Unity Simulation Capture package dependency to 0.0.10-preview.18 and Unity Simulation Core to 0.0.10-preview.22

Removed

Removed ScenarioBase.GenerateRandomSeedFromIndex()

Removed native sampling (through jobs) capability from all samplers and parameters as it introduced additional complexity to the code and was not a common usage pattern

Removed range as a required ISampler interface property.

Removed randomization tooling from the "Experimental" namespace

Fixed

Fixed an issue where the overlay panel would display a full screen semi-transparent image over the entire screen when the overlay panel is disabled in the UI

Fixed a bug in instance segmentation labeler that erroneously logged that object ID 255 was not supported

Fixed the simulation stopping while the editor/player is not focused

Fixed memory leak or crash occurring at the end of long simulations when using BackgroundObjectPlacementRandomizer or ForegroundObjectPlacementRandomizer

Randomizer.OnCreate() is no longer called in edit-mode when adding a randomizer to a scenario

Fixed a bug where removing all randomizers from a scenario caused the randomizer container UI element to overflow over the end of Scenario component UI

Semantic Segmentation Labeler now produces output in the proper form for distributed data generation on Unity Simulation by placing output in randomized directory names

Texture Randomizer is now compatible with HDRP.

Categorical Parameters no longer produce errors when deleting items from long options lists.

Parameter, ISampler, and non-generic Sampler class UIs now render properly in MonoBehaviours and ScriptableObjects.

Fixed an issue in the perception tutorial sample assets where upon the editor being first opened, and a user generates a dataset by clicking the play button, the first generated image has duplicated textures and hue offsets for all background objects. Enabling the "GPU instancing" boolean in the tutorial's sample material's inspector fixed this issue.