diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 000000000..526c8a38d --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +*.sh text eol=lf \ No newline at end of file diff --git a/Build/Microsoft.Psi.ruleset b/Build/Microsoft.Psi.ruleset index 2dcea87fa..2c43d6868 100644 --- a/Build/Microsoft.Psi.ruleset +++ b/Build/Microsoft.Psi.ruleset @@ -1,5 +1,6 @@  - + + @@ -8,6 +9,7 @@ + @@ -71,7 +73,7 @@ - + \ No newline at end of file diff --git a/Build/Sample.Psi.ruleset b/Build/Sample.Psi.ruleset index 266fc5f6d..f77500b85 100644 --- a/Build/Sample.Psi.ruleset +++ b/Build/Sample.Psi.ruleset @@ -1,9 +1,6 @@  - - - - - + + @@ -12,7 +9,7 @@ - + @@ -74,9 +71,9 @@ + - \ No newline at end of file diff --git a/Build/Security.ruleset b/Build/Security.ruleset new file mode 100644 index 000000000..977293d45 --- /dev/null +++ b/Build/Security.ruleset @@ -0,0 +1,207 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/Build/Test.Psi.ruleset b/Build/Test.Psi.ruleset index 53387a63e..e9ea3c6cf 100644 --- a/Build/Test.Psi.ruleset +++ b/Build/Test.Psi.ruleset @@ -1,18 +1,15 @@  - - - - - + + - + - + @@ -70,15 +67,22 @@ + + + + - + + + + \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 728a94cb7..4a306a281 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ # Contributing to Platform for Situated Intelligence -We welcome contributions from the community in a variety of forms: from simply using it and filing issues and bugs, to writing and releasing your own new components, to creating pull requests for bug fixes or new features, etc. This document describes some of the things you need to know if you are going to contribute to the Platform for Situated Intelligence ecosystem. Please read it carefully before making source code changes. +We welcome contributions from the community in a variety of forms: from simply using it and filing issues and bugs, to writing and releasing your own new components, to creating pull requests for bug fixes or new features, etc. This document describes some of the things you need to know if you are going to contribute to the Platform for Situated Intelligence ecosystem. ## Code of conduct @@ -60,12 +60,14 @@ Below is a description of the directory structure for the Platform for Situated | Sources | Audio | Contains class libraries for audio components. | | Sources | Calibration | Contains class libraries for calibrating cameras. | | Sources | Common | Contains class libraries for common test support. | -| Sources | Extensions | Contains class libraries that extend the \psi runtime class libraries. | +| Sources | Data | Contains class libraries for creating and manipulating datasets. | +| Sources | Devices | Contains class libraries that support enumerating devices. | | Sources | Imaging | Contains class libraries for \psi imaging, e.g. images, video capture, etc. | | Sources | Integrations | Contains integrations - libraries that provide shims around 3rd party libraries. | -| Sources | Kinect | Contains class libraries for Kinect sensor components. | +| Sources | Kinect | Contains class libraries for Azure Kinect and Kinect V2 sensor components. | | Sources | Language | Contains class libraries for natural language processing components. | | Sources | Media | Contains class libraries for media components. | +| Sources | RealSense | Contains class libraries for RealSense sensor component. | | Sources | Runtime | Contains class libraries for \psi runtime. | | Sources | Speech | Contains class libraries for speech components. | | Sources | Toolkits | Contains toolkits - e.g. Finite State Machine toolkit, etc. | @@ -74,10 +76,9 @@ Below is a description of the directory structure for the Platform for Situated ### Coding Style -Platform for Situated Intelligence is an organically grown codebase. The consistency of style reflects this. -For the most part, the team follows these [coding conventions](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions) along with these [design guidelines](https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/). Pull requests that reformat the code will not be accepted. +For the most part, the Platform for Situated Intelligence codebase follows these [coding conventions](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions) along with these [design guidelines](https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/). -In case you would like to add a new project to the `Psi.sln` we require that the project is setup in a similar ways to the other projects to ensure a certain coding standard. +In case you would like to add a new project to the `Psi.sln` we require that the project is setup in a similar ways to the other projects to ensure consistency. ### Build and Test @@ -85,9 +86,7 @@ To fully validate your changes, do a complete rebuild and test for both Debug an ### Pull Requests -We accept __bug fix pull requests__. Please make sure there is a corresponding tracking issue for the bug. When you submit a PR for a bug, please link to the issue. - -We also accept __new feature pull requests__. We are available to discuss new features. We recommend you open an issue if you plan to develop new features. +We accept __bug fix pull requests__ as well as __new feature pull requests__. For bug fixes, please open a corresponding issue for the bug and link to it, if one does not already exist. We also recommend you open an issue if you plan to develop new features, which will help facilitate community discussions about the design, implementation, etc. Pull requests should: diff --git a/Directory.Build.props b/Directory.Build.props index a7856f106..6921f1ec8 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -6,7 +6,7 @@ Microsoft Corporation microsoft,psi Microsoft - 0.11.82.2 + 0.12.53.2 $(AssemblyVersion) $(AssemblyVersion)-beta false @@ -21,6 +21,7 @@ FS2003 latest + false diff --git a/Psi.sln b/Psi.sln index 7af114869..9671802bb 100644 --- a/Psi.sln +++ b/Psi.sln @@ -166,6 +166,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Build", "Build", "{4026C2BE Build\MSLogoGreySmall.png = Build\MSLogoGreySmall.png Build\RunDoxygen.ps1 = Build\RunDoxygen.ps1 Build\Sample.Psi.ruleset = Build\Sample.Psi.ruleset + Build\Security.ruleset = Build\Security.ruleset Build\Test.Psi.ruleset = Build\Test.Psi.ruleset EndProjectSection EndProject @@ -197,6 +198,14 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Psi.DeviceManagem EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Psi.CognitiveServices.Face", "Sources\Integrations\CognitiveServices\Microsoft.Psi.CognitiveServices.Face\Microsoft.Psi.CognitiveServices.Face.csproj", "{084FB05C-4022-40FD-B00B-E3229B882F08}" EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Psi.AzureKinect.Visualization.Windows.x64", "Sources\Kinect\Microsoft.Psi.AzureKinect.Visualization\Microsoft.Psi.AzureKinect.Visualization.Windows.x64.csproj", "{8D33307F-0E96-491A-9D31-9025709310F6}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Psi.AzureKinect.x64", "Sources\Kinect\Microsoft.Psi.AzureKinect.x64\Microsoft.Psi.AzureKinect.x64.csproj", "{C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AzureKinectSample", "Samples\AzureKinectSample\AzureKinectSample.csproj", "{66639311-E7BE-4A5B-A35B-9BFF6D3F69F2}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Psi.Kinect.Visualization.Windows", "Sources\Kinect\Microsoft.Psi.Kinect.Visualization.Windows\Microsoft.Psi.Kinect.Visualization.Windows.csproj", "{F31606FF-3737-45DC-8E89-6256AACD841F}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -435,6 +444,22 @@ Global {084FB05C-4022-40FD-B00B-E3229B882F08}.Debug|Any CPU.Build.0 = Debug|Any CPU {084FB05C-4022-40FD-B00B-E3229B882F08}.Release|Any CPU.ActiveCfg = Release|Any CPU {084FB05C-4022-40FD-B00B-E3229B882F08}.Release|Any CPU.Build.0 = Release|Any CPU + {8D33307F-0E96-491A-9D31-9025709310F6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8D33307F-0E96-491A-9D31-9025709310F6}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8D33307F-0E96-491A-9D31-9025709310F6}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8D33307F-0E96-491A-9D31-9025709310F6}.Release|Any CPU.Build.0 = Release|Any CPU + {C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67}.Debug|Any CPU.ActiveCfg = Debug|x64 + {C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67}.Debug|Any CPU.Build.0 = Debug|x64 + {C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67}.Release|Any CPU.ActiveCfg = Release|x64 + {C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67}.Release|Any CPU.Build.0 = Release|x64 + {66639311-E7BE-4A5B-A35B-9BFF6D3F69F2}.Debug|Any CPU.ActiveCfg = Debug|x64 + {66639311-E7BE-4A5B-A35B-9BFF6D3F69F2}.Debug|Any CPU.Build.0 = Debug|x64 + {66639311-E7BE-4A5B-A35B-9BFF6D3F69F2}.Release|Any CPU.ActiveCfg = Release|x64 + {66639311-E7BE-4A5B-A35B-9BFF6D3F69F2}.Release|Any CPU.Build.0 = Release|x64 + {F31606FF-3737-45DC-8E89-6256AACD841F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {F31606FF-3737-45DC-8E89-6256AACD841F}.Debug|Any CPU.Build.0 = Debug|Any CPU + {F31606FF-3737-45DC-8E89-6256AACD841F}.Release|Any CPU.ActiveCfg = Release|Any CPU + {F31606FF-3737-45DC-8E89-6256AACD841F}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -520,6 +545,10 @@ Global {8AEFDD4F-CF2E-4392-AF46-378DE96126A5} = {A0856299-D28A-4513-B964-3FA5290FF160} {6B572F54-0E2F-4223-8283-14B3BAB7534A} = {8AEFDD4F-CF2E-4392-AF46-378DE96126A5} {084FB05C-4022-40FD-B00B-E3229B882F08} = {05481E26-A4CA-4F7D-B6FC-671A8AAC18B1} + {8D33307F-0E96-491A-9D31-9025709310F6} = {CB8286F5-167B-4416-8FE9-9B97FCF146D5} + {C91D0412-1BB2-40D2-8DCA-A48B6C5B7E67} = {CB8286F5-167B-4416-8FE9-9B97FCF146D5} + {66639311-E7BE-4A5B-A35B-9BFF6D3F69F2} = {1AA38339-B349-4AA7-A0A9-F92ADCFDB2DF} + {F31606FF-3737-45DC-8E89-6256AACD841F} = {CB8286F5-167B-4416-8FE9-9B97FCF146D5} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {EAF15EE9-DCC5-411B-A9E5-7C2F3D132331} diff --git a/README.md b/README.md index b3842492a..2a96cea5c 100644 --- a/README.md +++ b/README.md @@ -3,47 +3,44 @@ ![Build status](https://dev.azure.com/msresearch/psi/_apis/build/status/psi-github-ci?branchName=master) [![Join the chat at https://gitter.im/Microsoft/psi](https://badges.gitter.im/Microsoft/psi.svg)](https://gitter.im/Microsoft/psi?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -**Platform for Situated Intelligence** is an open, extensible framework that enables the development, fielding and study of situated, integrative-AI systems. +**Platform for Situated Intelligence** is an open, extensible framework that enables the development, fielding and study of multimodal, integrative-AI systems. In recent years, we have seen significant progress with machine learning techniques on various perceptual and control problems. At the same time, building end-to-end, multimodal, integrative-AI systems that leverage multiple technologies and act autonomously or interact with people in the open world remains a challenging, error-prone and time-consuming engineering task. Numerous challenges stem from the sheer complexity of these systems and are amplified by the lack of appropriate infrastructure and development tools. -The Platform for Situated Intelligence project aims to address these issues and provide a basis for developing, fielding and studying integrative-AI systems. The platform consists of three layers. The **Runtime** layer provides a parallel programming model centered around temporal streams of data, and enables easy development of components and applications using .NET, while retaining the performance properties of natively written, carefully tuned systems. A set of **Tools** enable multimodal data visualization, annotations, analytics, tuning and machine learning scenarios. Finally, an open ecosystem of **Components** encapsulate various AI technologies and allow for quick compositing of integrative-AI applications. For more information about the goals of the project, the types of systems that you can build using it, and the various layers see [Platform for Situated Intelligence Overview](https://github.com/microsoft/psi/wiki/Platform-Overview). +The Platform for Situated Intelligence project aims to address these issues and provide a basis for __developing, fielding and studying multimodal, integrative-AI systems__. The platform consists of three layers. The **Runtime** layer provides a parallel programming model centered around temporal streams of data, and enables easy development of components and applications using .NET, while retaining the performance properties of natively written, carefully tuned systems. A set of **Tools** enable multimodal data visualization, annotations, analytics, tuning and machine learning scenarios. Finally, an open ecosystem of **Components** encapsulate various AI technologies and allow for quick compositing of integrative-AI applications. + +For more information about the goals of the project, the types of systems that you can build using it, and the various layers see [Platform for Situated Intelligence Overview](https://github.com/microsoft/psi/wiki/Platform-Overview). # Using and Building -Platform for Situated Intelligence is built on the .NET Framework. Large parts of it are built on .NET Standard and therefore run both on Windows and Linux, whereas some components are specific and available only to one operating system (for instance the Kinect sensor component is available only for Windows.) +Platform for Situated Intelligence is built on the .NET Framework. Large parts of it are built on .NET Standard and therefore run both on Windows and Linux, whereas some components are specific and available only to one operating system. You can build applications based on Platform for Situated Intelligence either by leveraging nuget packages, or by cloning and building the code. Below are instructions: * [Using \\psi via Nuget packages](https://github.com/microsoft/psi/wiki/Using-via-NuGet-Packages) * [Building the \\psi codebase](https://github.com/microsoft/psi/wiki/Building-the-Codebase) -# Getting Started +# Documentation and Getting Started -__Brief Introduction__. A number of [tutorials](https://github.com/microsoft/psi/wiki/Basic-Tutorials) are available to get you started with using Platform for Situated Intelligence. We recommend starting with the [Brief Introduction](https://github.com/microsoft/psi/wiki/Brief-Introduction), which provides a guided walk-through for some of the main concepts in \psi. It shows how to create a simple program, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist to and replay streams from disk. We recommend that you first work through the examples in this tutorial to familiarize yourself with these core concepts. The [Writing Components](https://github.com/microsoft/psi/wiki/Writing-Components) tutorial explains how to write new \psi components, and the [Delivery Policies](https://github.com/microsoft/psi/wiki/Delivery-Policies) tutorial describes how to control throughput on streams in your application. +The documentation for Platform for Situated Intelligence is available in the [github project wiki](https://github.com/microsoft/psi/wiki). The documentation is still under construction and in various phases of completion. If you need further explanation in any area, please open an issue and label it `documentation`, as this will help us target our documentation development efforts to the highest priority needs. -__Advanced Topics__. A number of documents on more [advanced topics](https://github.com/microsoft/psi/wiki/More-Advanced-Topics) describe in more detail various aspects of the framework, including [stream operators](https://github.com/microsoft/psi/wiki/Stream-Operators), [synchronization](https://github.com/microsoft/psi/wiki/Synchronization), [remoting](https://github.com/microsoft/psi/wiki/Remoting), [interop](https://github.com/microsoft/psi/wiki/Interop), [shared objects and memory management](https://github.com/microsoft/psi/wiki/Shared-Objects), etc. +__Getting Started__. We recommend starting with the [Brief Introduction](https://github.com/microsoft/psi/wiki/Brief-Introduction) tutorial, which provides a guided walk-through for some of the main concepts in \psi. It shows how to create a simple \\psi application, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist to and replay streams from disk. We recommend that you first work through the examples in the [Brief Introduction](https://github.com/microsoft/psi/wiki/Brief-Introduction) to familiarize yourself with these core concepts, before you peruse the other available [tutorials](https://github.com/microsoft/psi/wiki/Basic-Tutorials). Two other helpful tutorials if you are just getting started are the [Writing Components](https://github.com/microsoft/psi/wiki/Writing-Components) tutorial, which explains how to write new \psi components, and the [Delivery Policies](https://github.com/microsoft/psi/wiki/Delivery-Policies) tutorial, which describes how to control throughput on streams in your application. +__Advanced Topics__. A set of documents on more [advanced topics](https://github.com/microsoft/psi/wiki/More-Advanced-Topics) describe in more detail various aspects of the framework, including [stream fusion and merging](https://github.com/microsoft/psi/wiki/Stream-Fusion-and-Merging), [interpolation and sampling](https://github.com/microsoft/psi/wiki/Interpolation-and-Sampling), [windowing operators](https://github.com/microsoft/psi/wiki/Windowing-Operators), [remoting](https://github.com/microsoft/psi/wiki/Remoting), [interop](https://github.com/microsoft/psi/wiki/Interop), [shared objects and memory management](https://github.com/microsoft/psi/wiki/Shared-Objects), etc. -__Samples__. Besides the tutorials and topics, it may be helpful to look through the set of [Samples](https://github.com/microsoft/psi/wiki/Samples) provided. While some of the samples address specialized topics such as how to leverage speech recognition components or how to bridge to ROS, reading them will give you more insight into programming with \psi. +__Samples__. Besides the tutorials and topics, we also recommend looking through the set of [Samples](https://github.com/microsoft/psi/wiki/Samples) provided. While some of the samples address specialized topics such as how to leverage speech recognition components or how to bridge to ROS, reading them will give you more insight into programming with \psi. __Components__. Additional useful information regarding available packages and components can be found in the [NuGet packages list](https://github.com/microsoft/psi/wiki/List-of-NuGet-Packages) and in the [component list](https://github.com/microsoft/psi/wiki/List-of-Components) pages. The latter page also has pointers to other repositories by third parties containing other \psi components. -__Documentation__. Like the rest of the codebase, the documentation available in the [wiki](https://github.com/microsoft/psi/wiki) is still under construction and in various phases of completion. If you need further explanation in any of these areas, please open an issue, label it `documentation`, as this will help us target our documentation development efforts to the highest priority needs. - -# Disclaimer - -The codebase is currently in beta and various aspects of the platform are at different levels of completion and robustness. There are probably still bugs in the code and we will likely be making breaking API changes. We plan to continuously improve the framework and we encourage the community to contribute. - -The [Roadmap](https://github.com/microsoft/psi/wiki/Roadmap) document provides more information about our future plans. +__API Reference__. An additional [API Reference](https://microsoft.github.io/psi/api/classes.html) is also available. # Getting Help -If you find a reproducible bug or if you would like to request a new feature or additional documentation, please file an [issue on the github repo](https://github.com/microsoft/psi/issues). If you do so, please make sure a corresponding issue has not already been filed. Use the [`bug`](https://github.com/microsoft/psi/labels/bug) label when filing issues that represent code defects, and provide enough information to reproduce. Use the [`feature request`](https://github.com/microsoft/psi/labels/feature%20request) label to request new features, and use the [`documentation`](https://github.com/microsoft/psi/labels/documentation) label to request additional documentation. +If you find a reproducible bug or if you would like to request a new feature or additional documentation, please file an [issue on the github repo](https://github.com/microsoft/psi/issues). If you do so, please first check whether a corresponding issue has already been filed. Use the [`bug`](https://github.com/microsoft/psi/labels/bug) label when filing issues that represent code defects, and provide enough information to reproduce. Use the [`feature request`](https://github.com/microsoft/psi/labels/feature%20request) label to request new features, and use the [`documentation`](https://github.com/microsoft/psi/labels/documentation) label to request additional documentation. # Contributing -We hope the community can help improve and evolve Platform for Situated Intelligence. If you plan to contribute to the codebase, please read the [Contributing Guidelines](https://github.com/microsoft/psi/wiki/Contributing) page. It describes how the source code is organized and things you need to know before making any source code changes. +We hope the community can help improve and evolve Platform for Situated Intelligence, and we welcome contributions in a variety of forms: from simply using it and filing issues and bugs, to writing and releasing your own new components, to creating pull requests for bug fixes or new features, etc. The [Contributing Guidelines](https://github.com/microsoft/psi/wiki/Contributing) page in the wiki describes in more detail a variety of ways in which you can get involved, how the source code is organized, and other useful things to know before starting to make source code changes. # Who is Using @@ -55,6 +52,12 @@ Platform for Situated Intelligence is currently being used in a number of indust If you would like to be added to this list, just add a [GitHub issue](https://github.com/Microsoft/psi/issues) and label it with the [`whoisusing`](https://github.com/Microsoft/psi/labels/whoisusing) label. Add a url for your research lab, website or project that you would like us to link to. +# Disclaimer + +The codebase is currently in beta and various aspects of the platform are at different levels of completion and robustness. There are probably still bugs in the code and we will likely be making breaking API changes. We plan to continuously improve the framework and we encourage the community to contribute. + +The [Roadmap](https://github.com/microsoft/psi/wiki/Roadmap) document provides more information about our future plans. + # License Platform for Situated Intelligence is available under an [MIT License](LICENSE.txt). See also [Third Party Notices](ThirdPartyNotices.txt). diff --git a/Samples/AzureKinectSample/AzureKinectSample.csproj b/Samples/AzureKinectSample/AzureKinectSample.csproj new file mode 100644 index 000000000..a2d68bb44 --- /dev/null +++ b/Samples/AzureKinectSample/AzureKinectSample.csproj @@ -0,0 +1,50 @@ + + + + Exe + netcoreapp3.1 + x64 + ..\..\Build\Sample.Psi.ruleset + false + AzureKinectSample.Program + + + + x64 + true + + bin\Debug\netcoreapp3.1\AzureKinectSample.xml + + + + x64 + true + + bin\Release\netcoreapp3.1\AzureKinectSample.xml + + + + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + diff --git a/Samples/AzureKinectSample/Program.cs b/Samples/AzureKinectSample/Program.cs new file mode 100644 index 000000000..823c85e77 --- /dev/null +++ b/Samples/AzureKinectSample/Program.cs @@ -0,0 +1,165 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace AzureKinectSample +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Text; + using MathNet.Spatial.Euclidean; + using Microsoft.Azure.Kinect.BodyTracking; + using Microsoft.Azure.Kinect.Sensor; + using Microsoft.Psi; + using Microsoft.Psi.AzureKinect; + using Microsoft.Psi.Calibration; + using Microsoft.Psi.Imaging; + + /// + /// Azure Kinect sample program. + /// + public class Program + { + /// + /// Main entry point. + /// + public static void Main() + { + // camera resolution settings + const ColorResolution resolution = ColorResolution.R720p; + const int widthSource = 1280; + const int heightSource = 720; + + // down sampled resolution + const int widthOutput = 80; + const int heightOutput = 45; + const double scaleFactorWidth = (double)widthOutput / widthSource; + const double scaleFactorHeight = (double)heightOutput / heightSource; + + // background subtraction beyond this depth + const double maxDepth = 1.0; // meters + + const SensorOrientation initialOrientation = SensorOrientation.Default; + + using (var pipeline = Pipeline.Create("AzureKinectSample", DeliveryPolicy.LatestMessage)) + { + var azureKinect = new AzureKinectSensor( + pipeline, + new AzureKinectSensorConfiguration() + { + OutputImu = true, + ColorResolution = resolution, + DepthMode = DepthMode.WFOV_Unbinned, + CameraFPS = FPS.FPS15, + BodyTrackerConfiguration = new AzureKinectBodyTrackerConfiguration() + { + CpuOnlyMode = true, // false if CUDA supported GPU available + SensorOrientation = initialOrientation, + }, + }); + + StringBuilder sb = new StringBuilder(); + SensorOrientation lastOrientation = (SensorOrientation)(-1); // detect orientation changes + + // consuming color, depth, IMU, body tracking, calibration + azureKinect.ColorImage.Resize(widthOutput, heightOutput) + .Join(azureKinect.DepthImage) + .Join(azureKinect.Imu, TimeSpan.FromMilliseconds(10)) + .Pair(azureKinect.Bodies) + .Pair(azureKinect.DepthDeviceCalibrationInfo) + .Do(message => + { + var (color, depth, imu, bodies, calib) = message; + + // determine camera orientation from IMU + static SensorOrientation ImuOrientation(ImuSample imu) + { + const double halfGravity = 9.8 / 2; + return + (imu.AccelerometerSample.Z > halfGravity) ? SensorOrientation.Flip180 : + (imu.AccelerometerSample.Y > halfGravity) ? SensorOrientation.Clockwise90 : + (imu.AccelerometerSample.Y < -halfGravity) ? SensorOrientation.CounterClockwise90 : + SensorOrientation.Default; // upright + } + + // enumerate image coordinates while correcting for orientation + static (IEnumerable, IEnumerable, bool) EnumerateCoordinates(SensorOrientation orientation) + { + var w = Enumerable.Range(0, widthOutput); + var h = Enumerable.Range(0, heightOutput); + return orientation switch + { + SensorOrientation.Clockwise90 => (h.Reverse(), w, true), + SensorOrientation.Flip180 => (w.Reverse(), h.Reverse(), false), + SensorOrientation.CounterClockwise90 => (h, w.Reverse(), true), + _ => (w, h, false), // normal + }; + } + + // render color frame as "ASCII art" + sb.Clear(); + var bitmap = color.Resource.ToBitmap(); + var orientation = ImuOrientation(imu); + var (horizontal, vertical, swap) = EnumerateCoordinates(orientation); + foreach (var j in vertical.Where(n => n % 2 == 0)) + { + foreach (var i in horizontal) + { + var (x, y) = swap ? (j, i) : (i, j); + + // subtract background beyond max depth + var d = DepthExtensions.ProjectToCameraSpace(calib, new Point2D(x / scaleFactorWidth, y / scaleFactorHeight), depth); + if (!d.HasValue || d.Value.Z < maxDepth) + { + var p = bitmap.GetPixel(x, y); + sb.Append(" .:-=+*#%@"[(int)((p.R + p.G + p.B) / 76.5)]); + } + else + { + sb.Append(' '); // subtract background + } + } + + sb.Append(Environment.NewLine); + } + + // clear console when orientation changes + if (orientation != lastOrientation) + { + Console.Clear(); + lastOrientation = orientation; + } + + Console.SetCursorPosition(0, 0); + Console.WriteLine(sb.ToString()); + + // overlay head tracking + if (orientation == initialOrientation) + { + // body tracking works only in initially configured orientation + Console.BackgroundColor = ConsoleColor.Red; + foreach (var body in bodies) + { + var p = calib.ToColorSpace(body.Joints[JointId.Head].Pose.Origin); + var x = (int)(p.X * scaleFactorWidth); + var y = (int)(p.Y * scaleFactorHeight / 2); + if (x > 0 && x < widthOutput && y > 0 && y < heightOutput) + { + Console.SetCursorPosition(x, y / 2); + Console.Write(' '); + } + } + + Console.BackgroundColor = ConsoleColor.Black; + } + }); + + Console.BackgroundColor = ConsoleColor.Black; + Console.ForegroundColor = ConsoleColor.White; + Console.Clear(); + pipeline.RunAsync(); + Console.ReadLine(); // press Enter to end + } + } + } +} diff --git a/Samples/AzureKinectSample/README.md b/Samples/AzureKinectSample/README.md new file mode 100644 index 000000000..aaab4918d --- /dev/null +++ b/Samples/AzureKinectSample/README.md @@ -0,0 +1,349 @@ +# Azure Kinect Sample + +This sample demonstrates how to use the Azure Kinect sensor with body tracking and how to use the `Join()` and `Pair()` operators to synchronize and fuse streams. + +# Using the Color Image Stream + +First, let's get a minimal application up and running. The `AzureKinectSensor` component gives us access to various image streams from the device (color, depth, infrared) as well as other information such as IMU and temperature readings. We will start with the `ColorImage` stream. + +```csharp +using (var pipeline = Pipeline.Create("AzureKinectSample", DeliveryPolicy.LatestMessage)) +{ + var azureKinectSensor = new AzureKinectSensor( + pipeline, + new AzureKinectSensorConfiguration() + { + ColorResolution = ColorResolution.R720p, + CameraFPS = FPS.FPS15, + }); + + ... +} +``` +Notice that at construction time we can configure the frame rate (`CameraFPS`) and resolution (`ColorResolution`). A number of other configuration options are also available as part of the `AzureKinectSensorConfiguration`: + +- **DeviceIndex:** The index of the device to open (default 0). +- **ColorResolution:** The resolution of the color camera (default 1080p). +- **DepthMode:** The depth camera mode (default NFOV unbinned). +- **CameraFPS:** The desired frame rate (default 30 FPS). +- **SynchronizedImagesOnly:** Whether color and depth captures should be strictly synchronized (default `true`). +- **OutputColor:** Whether the color stream is emitted (default `true`). +- **OutputDepth:** Whether the depth stream is emitted (default `true`). +- **OutputInfrared:** Whether the infrared stream is emitted (default `true`). +- **OutputImu:** Whether to use the Azure Kinect's IMU (default `false`). +- **OutputCalibration:** Whether the Azure Kinect outputs its calibration settings (default `true`). +- **BodyTrackerConfiguration:** The body tracker configuration (default null). If null, no body tracking is performed. +- **DeviceCaptureTimeout:** The timeout used for device capture (default 1 minute). +- **FrameRateReportingFrequency:** The frequency at which frame rate is reported on the `FrameRate` emitter (default 2 seconds). + +For this demonstration we'll be resizing the color images to render as ASCII art at the console for a cross-platform solution. The `ColorImage` stream is of `Shared` on which several operators exist for cropping, transforming, encoding, etc. Below, we use the `Resize()` operator to scale the image down to 80 by 45 pixels, and then we apply a `Do()` operator in which we convert the image to ASCII art: + +```csharp +// down sampled resolution +const int widthOutput = 80; +const int heightOutput = 45; + +StringBuilder sb = new StringBuilder(); + +// consuming color +azureKinectSensor.ColorImage.Resize(widthOutput, heightOutput).Do(color => +{ + var bitmap = color.Resource.ToBitmap(); + + // render color frame as "ASCII art" + sb.Clear(); + for (int y = 0; y < heightOutput; y += 2) + { + for (int x = 0; x < widthOutput; x++) + { + var p = bitmap.GetPixel(x, y); + sb.Append(" .:-=+*#%@"[(int)((p.R + p.G + p.B) / 76.5)]); + } + + sb.Append(Environment.NewLine); + } + + Console.SetCursorPosition(0, 0); + Console.WriteLine(sb.ToString()); +}); + +Console.BackgroundColor = ConsoleColor.Black; +Console.ForegroundColor = ConsoleColor.White; +Console.Clear(); +pipeline.RunAsync(); +Console.ReadLine(); // press Enter to end +``` + +Here's an example output produced by this application: + +![Sample output](./SampleOutput.png) + +# Using the Depth Image Stream + +A core feature of the Azure Kinect sensor is depth perception. Next we'll use the `DepthImage` stream to perform background subtraction, i.e. to remove pixels beyond a distance threshold. + +```csharp +// background subtraction beyond this depth +const double maxDepth = 1.0; // meters +``` + +We will _not_ be resizing the depth image but will need to scale coordinates: + +```csharp +// camera resolution settings +const ColorResolution resolution = ColorResolution.R720p; +const int widthSource = 1280; +const int heightSource = 720; + +// down sampled resolution +const int widthOutput = 80; +const int heightOutput = 45; +const double scaleFactorWidth = (double)widthOutput / widthSource; +const double scaleFactorHeight = (double)heightOutput / heightSource; +``` + +The `ColorImage` and `DepthImage` streams emit independently. When the component is configured with `SynchronizedImagesOnly = true` (the default) then the images on each stream have matching originating times. However, they remain separate streams and may have varying latencies in the system. In a more complex system they may pass through different paths in the graph of components. We want to ensure that we receive pairs of color and depth images that correspond to the same originating time in the real world regardless of when they arrive in wall clock time at our block of code. To do this we use `Join()`. The operator buffers incoming messages and fuses them in pairs based on their originating times. We'll see later how `Join()` can also be used with a tolerance window to relax the requirement of _exactly_ matching originating times while still guaranteeing reproducibility. + +Additionally, we need the `DepthDeviceCalibrationInfo` to correlate the physical poses of the two cameras involved. This comes as a single message on another stream. Here we use `Pair()` to fuse this with the other data we receive; this time _without_ ensuring synchronicity. To learn more about the different types of fusion and synchronization operators available you can [visit this in-depth tutorial.](https://github.com/microsoft/psi/wiki/Synchronization) + +The result of the consecutive `Join()` and `Pair()` operators is a stream of tuple `message` which we can unpack with `var (color, depth, calib) = message`. + +```csharp +// consuming color, depth, and calibration +azureKinectSensor.ColorImage + .Resize(widthOutput, heightOutput) + .Join(azureKinectSensor.DepthImage) + .Pair(azureKinectSensor.DepthDeviceCalibrationInfo) + .Do(message => +{ + var (color, depth, calib) = message; + + ... + + }); +``` + +The `Microsoft.Psi.Calibration` namespace provides a number of useful functions for dealing with depth information via the `DepthExtensions` static class. We'll use `ProjectToCameraSpace()` to get the depth at each color image pixel. Any pixel known to be beyond the `maxDepth` threshold will be rendered blank. + +```csharp +var d = DepthExtensions.ProjectToCameraSpace(calib, new Point2D(x / scaleFactorWidth, y / scaleFactorHeight), depth); +if (!d.HasValue || d.Value.Z < maxDepth) +{ + var p = bitmap.GetPixel(x, y); + sb.Append(" .:-=+*#%@"[(int)((p.R + p.G + p.B) / 76.5)]); +} +else +{ + sb.Append(' '); // subtract background +} +``` + +# Using the Inertial Measurement Unit (IMU) Stream + +The Azure Kinect provides inertial information as well. A gyro gives instantaneous angular speed when the device is physically rotated and an accelerometer gives linear acceleration. + +We will assume that the device is relatively stationary and will used accelerometer values to measure the direction of gravity. With this we can rotate the output image to remain upright even as the device is physically turned on its side or upside down; much like mobile phones commonly do. + +While `OutputColor` and `OutputDepth` are configured to `true` by default, `OutputIMU` is not. We'll first enable this in the configuration passed in when constructing the sensor. + +```csharp +var azureKinectSensor = new AzureKinectSensor( + pipeline, + new AzureKinectSensorConfiguration() + { + ColorResolution = resolution, + CameraFPS = FPS.FPS15, + OutputImu = true, + }); +``` + +As with the other streams, we will `Join()` with the `Imu` stream. Unlike the color and depth streams, the IMU information flows at a higher rate. It does not obey the `CameraFPS` setting. By default `Join()` correlates messages by _exactly_ matching originating times. This will not work with the IMU because it's on a different cadence. We do want to take samples that are _reasonably_ near to each camera frame. The `Join()` operator allows us to specify what be mean by _reasonably near_; for instance, we can match messages within 10 milliseconds by using `.Join(azureKinectSensor.Imu, TimeSpan.FromMilliseconds(10))`. + +```csharp + // consuming color, depth, IMU, and calibration + azureKinectSensor.ColorImage.Resize(widthOutput, heightOutput) + .Join(azureKinectSensor.DepthImage) + .Join(azureKinectSensor.Imu, TimeSpan.FromMilliseconds(10)) + .Pair(azureKinectSensor.DepthDeviceCalibrationInfo) + .Do(message => + { + var (color, depth, imu, calib) = message; + + ... + + }); +``` + +To determine the orientation we observe the pull of gravity along each axis. + +```csharp +// determine camera orientation from IMU +SensorOrientation ImuOrientation(ImuSample imu) +{ + const double halfGravity = 9.8 / 2; // G ≈ 9.8m/s² + return + (imu.AccelerometerSample.Z > halfGravity) ? SensorOrientation.Flip180 : + (imu.AccelerometerSample.Y > halfGravity) ? SensorOrientation.Clockwise90 : + (imu.AccelerometerSample.Y < -halfGravity) ? SensorOrientation.CounterClockwise90 : + SensorOrientation.Default; // upright +} +``` + +We will enumerate pixels in the order required to render upright (from right-to-left, bottom-to-top when the device is upside down for example). + +```csharp +// enumerate image coordinates while correcting for orientation +(IEnumerable, IEnumerable, bool) EnumerateCoordinates(SensorOrientation orientation) +{ + var w = Enumerable.Range(0, widthOutput); + var h = Enumerable.Range(0, heightOutput); + switch (orientation) + { + case SensorOrientation.Clockwise90: return (h.Reverse(), w, true); + case SensorOrientation.Flip180: return (w.Reverse(), h.Reverse(), false); + case SensorOrientation.CounterClockwise90: return (h, w.Reverse(), true); + default: return (w, h, false); // normal + } +} +``` + +Changing our nested `for` loops to this order, while swapping `x` and `y` when sideways. + +```csharp +var orientation = ImuOrientation(imu); +var (horizontal, vertical, swap) = EnumerateCoordinates(orientation); +foreach (var j in vertical.Where(n => n % 2 == 0)) +{ + foreach (var i in horizontal) + { + var (x, y) = swap ? (j, i) : (i, j); + + ... + } + + sb.Append(Environment.NewLine); +} +``` + +To prevent displaying characters from previous frames we'll keep track of the `lastOrientation` and clear the console when changing between landscape and portrait renderings. + +```csharp +SensorOrientation lastOrientation = (SensorOrientation)(-1); // detect orientation changes + +... + +// clear console when orientation changes +if (orientation != lastOrientation) +{ + Console.Clear(); + lastOrientation = orientation; +} +``` + +# Body Tracking + +Using the depth and infrared image streams, the Azure Kinect may be used to [track one or many human bodies, including detailed joint positions](https://docs.microsoft.com/en-us/azure/kinect-dk/body-joints). + +The body tracker can make use of a CUDA supported GPU if one is available (and configured with `CpuOnlyMode = false`). The following other parameters are available as part of the `AzureKinectBodyTrackerConfiguration`: + +- **TemporalSmoothing:** The temporal smoothing to use across frames for the body tracker. Set between 0 for no smoothing and 1 for full smoothing (default 0.5 seconds). +- **CpuOnlyMode:** Whether to perform body tracking computation only on the CPU. If false, the tracker requires CUDA hardware and drivers (default `false`). +- **SensorOrientation:** The sensor orientation used by body tracking (default upright). + +We configure the Azure Kinect to perform body tracking by providing a body tracker configuration, as follows: + +```csharp +var azureKinectSensor = new AzureKinect( + pipeline, + new AzureKinectSensorConfiguration() + { + OutputImu = true, + ColorResolution = resolution, + DepthMode = DepthMode.WFOV_Unbinned, + CameraFPS = FPS.FPS15, + BodyTrackerConfiguration = + new AzureKinectBodyTrackerConfiguration() + { + CpuOnlyMode = true, // false if CUDA supported GPU available + }, + }); +``` + +Now we can fuse in and make use of the `Bodies` stream. + +```csharp +// consuming color, depth, IMU, body tracking, calibration +azureKinectSensor.ColorImage.Resize(widthOutput, heightOutput) + .Join(azureKinectSensor.DepthImage) + .Join(azureKinectSensor.Imu, TimeSpan.FromMilliseconds(10)) + .Pair(azureKinectSensor.Bodies) + .Pair(azureKinectSensor.DepthDeviceCalibrationInfo) + .Do(message => +{ + var (color, depth, imu, bodies, calib) = message; + + ... + + }); +``` + +Notice that we use `Pair()` to fuse in the `Bodies` stream. The `Bodies` stream generally comes at a lower frequency than the camera image streams. If we were to use a `Join()` here we would have perfectly synchronized data. That is, bodies in sync with color and depth image frames. However the frame rate would drop significantly (especially in `CpuOnlyMode`). + +We could use a tolerance `TimeSpan` as we did with the `Imu` stream, but `Join()` has an interesting side effect to consider. Generally before a joined message may be emitted, the _next_ messages outside of the tolerance window must first be seen to ensure that the _best_ match within the window has been chosen. This necessarily introduces some latency. With the high frequency `Imu` stream, this was fine. With the much lower frequency `Bodies` stream this would cause a significant delay. Instead of `Join()` we choose to use `Pair()` which will _immediately_ fuse the last body message (in wall clock time). No latency, but also no synchronicity or reproducibility guarantees. Reproducibility is the primary difference between `Join()` and `Pair()` as explained in more detail in the [synchronization tutorial.](https://github.com/microsoft/psi/wiki/Synchronization) + +Finally, we highlight each person's head with a red block; correlating the `Head` joint with the color image pixel coordinate using `ToColorSpace()`. + +```csharp +// overlay head tracking +if (orientation == SensorOrientation.Default) +{ + // body tracking works only in initially configured orientation + Console.BackgroundColor = ConsoleColor.Red; + foreach (var body in bodies) + { + var p = calib.ToColorSpace(body.Joints[JointId.Head].Pose.Origin); + var x = (int)(p.X * scaleFactorWidth); + var y = (int)(p.Y * scaleFactorHeight / 2); + if (x > 0 && x < widthOutput && y > 0 && y < heightOutput) + { + Console.SetCursorPosition(x, y / 2); + Console.Write(' '); + } + } + + Console.BackgroundColor = ConsoleColor.Black; +} +``` + +## Decoupled Body Tracking Component + +Body tracking can also be performed without a live, running Azure Kinect sensor, if the depth, IR, and calibration information streams are available. This functionality is implemented by the `AzureKinectBodyTracker` component. + +```csharp +var bodyTracker = new AzureKinectBodyTracker( + pipeline, + new AzureKinectBodyTrackerConfiguration() + { + CpuOnlyMode = true, // false if CUDA supported GPU available + }); +``` + +This component consumes `DepthImage` and `InfraredImage` streams as well as a the 'AzureKinectSensorCalibration` stream that contains the sensor calibration information; these streams are produced by the `AzureKinectSensor` component, and can be persisted and leveraged for running the tracker at a later time. + +For instace, assuming these streams were persisted into a store, we can open them up as follows: + +```csharp +var store = Store.Open(pipeline, "MyRecording", @"C:\Data"); +var depth = store.OpenStream>("DepthImage"); // DepthImage +var infrared = store.OpenStream>("InfraredStream"); // ColorImage +var calibration = store.OpenStream("AzureKinectSensorCalibration"); // AzureKinectSensorCalibration +``` + +The depth and infrared streams are joined and piped to the body tracker. The calibration stream is also separately piped to the body tracker. The tracker generates the resulting bodies on it's `Bodies` output stream. + +```csharp +depth.Join(infrared).PipeTo(bodyTracker); +calibration.PipeTo(bodyTracker.AzureKinectSensorCalibration); + +var bodies = bodyTracker.Bodies; +``` \ No newline at end of file diff --git a/Samples/AzureKinectSample/SampleOutput.png b/Samples/AzureKinectSample/SampleOutput.png new file mode 100644 index 000000000..c00f02f2f Binary files /dev/null and b/Samples/AzureKinectSample/SampleOutput.png differ diff --git a/Samples/AzureKinectSample/build.sh b/Samples/AzureKinectSample/build.sh new file mode 100644 index 000000000..99908579e --- /dev/null +++ b/Samples/AzureKinectSample/build.sh @@ -0,0 +1,3 @@ +#!/usr/bin/env bash + +dotnet build ./AzureKinectSample.csproj diff --git a/Samples/AzureKinectSample/stylecop.json b/Samples/AzureKinectSample/stylecop.json new file mode 100644 index 000000000..6f09427eb --- /dev/null +++ b/Samples/AzureKinectSample/stylecop.json @@ -0,0 +1,16 @@ +{ + // ACTION REQUIRED: This file was automatically added to your project, but it + // will not take effect until additional steps are taken to enable it. See the + // following page for additional information: + // + // https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/EnableConfiguration.md + + "$schema": "https://raw.githubusercontent.com/DotNetAnalyzers/StyleCopAnalyzers/master/StyleCop.Analyzers/StyleCop.Analyzers/Settings/stylecop.schema.json", + "settings": { + "documentationRules": { + "companyName": "Microsoft Corporation", + "copyrightText": "Copyright (c) Microsoft Corporation. All rights reserved.\nLicensed under the MIT license.", + "xmlHeader": false + } + } +} \ No newline at end of file diff --git a/Samples/KinectSample/KinectSample.csproj b/Samples/KinectSample/KinectSample.csproj index d81d614b5..1d1465ab9 100644 --- a/Samples/KinectSample/KinectSample.csproj +++ b/Samples/KinectSample/KinectSample.csproj @@ -1,8 +1,7 @@  net472 - false - false + ../../Build/Sample.Psi.ruleset Exe MultiModalSpeechDetection.Program @@ -43,6 +42,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Samples/KinectSample/README.md b/Samples/KinectSample/README.md index 4174ec8ff..8c51f2953 100644 --- a/Samples/KinectSample/README.md +++ b/Samples/KinectSample/README.md @@ -1,6 +1,6 @@ # Kinect Sample -This sample demostrates how to use the Kinect sensor and how to use the `Join()` operator to synchronize streams. The sample uses the +This sample demonstrates how to use the Kinect sensor and how to use the `Join()` operator to synchronize streams. The sample uses the Kinect's face tracking and the audio and video streams to detect when a user is speaking. The sample compiles and runs on Windows. __NOTE__: In order to run this sample, you must have a valid Cognitive Services Speech subscription key. You may enter this key at runtime, or set it in the static `AzureSubscriptionKey` variable on the `OperatorExtensions` class. For more information on how to obtain a subscription key for the Azure Speech Service, see [https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account](https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account) diff --git a/Samples/LinuxSpeechSample/LinuxSpeechSample.csproj b/Samples/LinuxSpeechSample/LinuxSpeechSample.csproj index 00ca4e5d7..c5d63ebb5 100644 --- a/Samples/LinuxSpeechSample/LinuxSpeechSample.csproj +++ b/Samples/LinuxSpeechSample/LinuxSpeechSample.csproj @@ -2,21 +2,21 @@ Exe - netcoreapp2.0 + netcoreapp3.1 ../../Build/Sample.Psi.ruleset true - bin\Release\netstandard2.0\LinuxSpeechSample.xml + bin\Release\netcoreapp3.1\LinuxSpeechSample.xml ../../Build/Sample.Psi.ruleset true - bin\Debug\netstandard2.0\LinuxSpeechSample.xml + bin\Debug\netcoreapp3.1\LinuxSpeechSample.xml @@ -33,6 +33,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Samples/LinuxSpeechSample/README.md b/Samples/LinuxSpeechSample/README.md index 244a5e741..1a14ff6d5 100644 --- a/Samples/LinuxSpeechSample/README.md +++ b/Samples/LinuxSpeechSample/README.md @@ -1,6 +1,6 @@ # Speech Sample -This sample demostrates how to build a simple speech recognition application using the audio and speech components on Linux. +This sample demonstrates how to build a simple speech recognition application using the audio and speech components on Linux. __NOTES:__ diff --git a/Samples/OpenCVSample/OpenCVSample.Interop/AssemblyInfo.cpp b/Samples/OpenCVSample/OpenCVSample.Interop/AssemblyInfo.cpp index 8b355d8d3..b691ac184 100644 --- a/Samples/OpenCVSample/OpenCVSample.Interop/AssemblyInfo.cpp +++ b/Samples/OpenCVSample/OpenCVSample.Interop/AssemblyInfo.cpp @@ -13,8 +13,8 @@ using namespace System::Security::Permissions; [assembly:AssemblyProductAttribute(L"OpenCVSampleInterop")]; [assembly:AssemblyCompanyAttribute(L"Microsoft Corporation")]; [assembly:AssemblyCopyrightAttribute(L"Copyright (c) Microsoft Corporation. All rights reserved.")]; -[assembly:AssemblyVersionAttribute("0.11.82.2")]; -[assembly:AssemblyFileVersionAttribute("0.11.82.2")]; -[assembly:AssemblyInformationalVersionAttribute("0.11.82.2-beta")]; +[assembly:AssemblyVersionAttribute("0.12.53.2")]; +[assembly:AssemblyFileVersionAttribute("0.12.53.2")]; +[assembly:AssemblyInformationalVersionAttribute("0.12.53.2-beta")]; [assembly:ComVisible(false)]; [assembly:CLSCompliantAttribute(true)]; diff --git a/Samples/OpenCVSample/OpenCVSample.Interop/OpenCVSample.Interop.vcxproj b/Samples/OpenCVSample/OpenCVSample.Interop/OpenCVSample.Interop.vcxproj index 530e46844..73ea77bcf 100644 --- a/Samples/OpenCVSample/OpenCVSample.Interop/OpenCVSample.Interop.vcxproj +++ b/Samples/OpenCVSample/OpenCVSample.Interop/OpenCVSample.Interop.vcxproj @@ -38,7 +38,7 @@ true true Unicode - v141 + v142 Spectre @@ -46,7 +46,7 @@ false true Unicode - v141 + v142 Spectre @@ -162,4 +162,4 @@ copy $(OpenCVDlls) $(OutDir)..\..\..\OpenCVSample\bin\Release $(BuildDependsOn) CheckVariable - + \ No newline at end of file diff --git a/Samples/OpenCVSample/OpenCVSample/MainWindow.xaml.cs b/Samples/OpenCVSample/OpenCVSample/MainWindow.xaml.cs index 891dacb8b..dfa7817ab 100644 --- a/Samples/OpenCVSample/OpenCVSample/MainWindow.xaml.cs +++ b/Samples/OpenCVSample/OpenCVSample/MainWindow.xaml.cs @@ -42,7 +42,7 @@ public static IProducer> ToGrayViaOpenCV(this IProducer { // Our lambda here is called with each image sample from our stream and calls OpenCV to convert - // the image into a grayscale image. We then post the resulting gray scale image to our event queu + // the image into a grayscale image. We then post the resulting gray scale image to our event queue // so that the Psi pipeline will send it to the next component. // Have Psi allocate a new image. We will convert the current image ('srcImage') into this new image. diff --git a/Samples/OpenCVSample/OpenCVSample/OpenCVSample.csproj b/Samples/OpenCVSample/OpenCVSample/OpenCVSample.csproj index 214426b6f..94dcb93cf 100644 --- a/Samples/OpenCVSample/OpenCVSample/OpenCVSample.csproj +++ b/Samples/OpenCVSample/OpenCVSample/OpenCVSample.csproj @@ -1,22 +1,40 @@ - - + net472 false OpenCVSample WinExe + ../../../Build/Sample.Psi.ruleset bin\Debug\ x64 + bin\Debug\OpenCVSample.xml + true bin\Release\ x64 + bin\Release\OpenCVSample.xml + true + + + + + + + + all + runtime; build; native; contentfiles; analyzers + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + diff --git a/Samples/RosArmControlSample/RosArmControlSample.csproj b/Samples/RosArmControlSample/RosArmControlSample.csproj index d62d8312b..3c2501b3d 100644 --- a/Samples/RosArmControlSample/RosArmControlSample.csproj +++ b/Samples/RosArmControlSample/RosArmControlSample.csproj @@ -2,21 +2,21 @@ Exe - netcoreapp2.0 + netcoreapp3.1 ../../Build/Sample.Psi.ruleset true - bin\Debug\netstandard2.0\RosArmControlSample.xml + bin\Debug\netcoreapp3.1\RosArmControlSample.xml ../../Build/Sample.Psi.ruleset true - bin\Release\netstandard2.0\RosArmControlSample.xml + bin\Release\netcoreapp3.1\RosArmControlSample.xml @@ -33,6 +33,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Samples/RosArmControlSample/UArm.cs b/Samples/RosArmControlSample/UArm.cs index 4a01df674..b872051e9 100644 --- a/Samples/RosArmControlSample/UArm.cs +++ b/Samples/RosArmControlSample/UArm.cs @@ -113,7 +113,7 @@ public void Beep(float frequency, float duration) } /// - /// Set absolute cartesian position. + /// Set absolute Cartesian position. /// /// Coordinate X. /// Coordinate Y. @@ -124,7 +124,7 @@ public void AbsolutePosition(float x, float y, float z) } /// - /// Set relative cartesian position. + /// Set relative Cartesian position. /// /// Delta X. /// Delta Y. diff --git a/Samples/RosArmControlSample/UArmComponent.cs b/Samples/RosArmControlSample/UArmComponent.cs index 954c5c8f3..7efb16e94 100644 --- a/Samples/RosArmControlSample/UArmComponent.cs +++ b/Samples/RosArmControlSample/UArmComponent.cs @@ -43,12 +43,12 @@ public UArmComponent(Pipeline pipeline, UArm arm) public Receiver Pump { get; private set; } /// - /// Gets receiver of absolute cartesian positions. + /// Gets receiver of absolute Cartesian positions. /// public Receiver<(float, float, float)> AbsolutePosition { get; private set; } /// - /// Gets receiver of relative cartesian positions. + /// Gets receiver of relative Cartesian positions. /// public Receiver<(float, float, float)> RelativePosition { get; private set; } diff --git a/Samples/RosTurtleSample/RosTurtleSample.csproj b/Samples/RosTurtleSample/RosTurtleSample.csproj index 98ae03ced..e16ba02b9 100644 --- a/Samples/RosTurtleSample/RosTurtleSample.csproj +++ b/Samples/RosTurtleSample/RosTurtleSample.csproj @@ -2,21 +2,21 @@ Exe - netcoreapp2.0 + netcoreapp3.1 ../../Build/Sample.Psi.ruleset true - bin\Debug\netstandard2.0\RosTurtleSample.xml + bin\Debug\netcoreapp3.1\RosTurtleSample.xml ../../Build/Sample.Psi.ruleset true - bin\Release\netstandard2.0\RosTurtleSample.xml + bin\Release\netcoreapp3.1\RosTurtleSample.xml @@ -34,6 +34,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Samples/SpeechSample/Program.cs b/Samples/SpeechSample/Program.cs index 841c86187..c8fc93ddb 100644 --- a/Samples/SpeechSample/Program.cs +++ b/Samples/SpeechSample/Program.cs @@ -116,7 +116,7 @@ public static void RunSystemSpeech(string outputLogPath = null, string inputLogP { // Create the AudioCapture component to capture audio from the default device in 16 kHz 1-channel // PCM format as required by both the voice activity detector and speech recognition components. - audioInput = new AudioCapture(pipeline, new AudioCaptureConfiguration() { OutputFormat = WaveFormat.Create16kHz1Channel16BitPcm() }); + audioInput = new AudioCapture(pipeline, WaveFormat.Create16kHz1Channel16BitPcm()); } // Create System.Speech recognizer component @@ -202,7 +202,7 @@ public static void RunAzureSpeech(string outputLogPath = null, string inputLogPa { // Create the AudioCapture component to capture audio from the default device in 16 kHz 1-channel // PCM format as required by both the voice activity detector and speech recognition components. - audioInput = new AudioCapture(pipeline, new AudioCaptureConfiguration() { OutputFormat = WaveFormat.Create16kHz1Channel16BitPcm() }); + audioInput = new AudioCapture(pipeline, WaveFormat.Create16kHz1Channel16BitPcm()); } // Perform voice activity detection using the voice activity detector component diff --git a/Samples/SpeechSample/README.md b/Samples/SpeechSample/README.md index 2b882f6d6..80da6d2a3 100644 --- a/Samples/SpeechSample/README.md +++ b/Samples/SpeechSample/README.md @@ -1,6 +1,6 @@ # Speech Sample -This sample demostrates how to build a simple speech recognition application using a number of different audio and speech components. In addition, it also demonstrates data logging and replay of logged data. The sample builds and runs on Windows. +This sample demonstrates how to build a simple speech recognition application using a number of different audio and speech components. In addition, it also demonstrates data logging and replay of logged data. The sample builds and runs on Windows. __NOTES:__ diff --git a/Samples/SpeechSample/SpeechSample.csproj b/Samples/SpeechSample/SpeechSample.csproj index c4a5db0c5..f9fdcc6b9 100644 --- a/Samples/SpeechSample/SpeechSample.csproj +++ b/Samples/SpeechSample/SpeechSample.csproj @@ -1,8 +1,7 @@  net472 - false - false + ../../Build/Sample.Psi.ruleset Exe Microsoft.Psi.Samples.SpeechSample.Program @@ -38,6 +37,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Samples/WebcamWithAudioSample/Program.cs b/Samples/WebcamWithAudioSample/Program.cs index e5ed815a3..d9a308255 100644 --- a/Samples/WebcamWithAudioSample/Program.cs +++ b/Samples/WebcamWithAudioSample/Program.cs @@ -70,7 +70,7 @@ public static void RecordAudioVideo(string pathToStore) MediaCapture webcam = new MediaCapture(pipeline, 1920, 1080, 30); // Create the AudioCapture component to capture audio from the default device in 16 kHz 1-channel - IProducer audioInput = new AudioCapture(pipeline, new AudioCaptureConfiguration() { OutputFormat = WaveFormat.Create16kHz1Channel16BitPcm() }); + IProducer audioInput = new AudioCapture(pipeline, WaveFormat.Create16kHz1Channel16BitPcm()); var images = webcam.Out.EncodeJpeg(90, DeliveryPolicy.LatestMessage).Out; diff --git a/Samples/WebcamWithAudioSample/README.md b/Samples/WebcamWithAudioSample/README.md index ab35c27c2..3bd6de6b2 100644 --- a/Samples/WebcamWithAudioSample/README.md +++ b/Samples/WebcamWithAudioSample/README.md @@ -1,4 +1,4 @@ # WebCam + Audio Sample -This sample demostrates how to build a simple application that records audio and video from a webcam and displays the playback using +This sample demonstrates how to build a simple application that records audio and video from a webcam and displays the playback using the Platform for Situated Intelligence Studio's visualization client. The sample builds and runs on Windows. \ No newline at end of file diff --git a/Samples/WebcamWithAudioSample/WebcamWithAudioSample.csproj b/Samples/WebcamWithAudioSample/WebcamWithAudioSample.csproj index e0319beb3..fc109cd48 100644 --- a/Samples/WebcamWithAudioSample/WebcamWithAudioSample.csproj +++ b/Samples/WebcamWithAudioSample/WebcamWithAudioSample.csproj @@ -1,8 +1,7 @@  net472 - false - false + ../../Build/Sample.Psi.ruleset Exe Microsoft.Psi.Samples.WebcamWithAudioSample.Program @@ -40,6 +39,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Samples/WpfSample/README.md b/Samples/WpfSample/README.md index 8987da401..ad1fc9644 100644 --- a/Samples/WpfSample/README.md +++ b/Samples/WpfSample/README.md @@ -1,4 +1,4 @@ # WPF Sample -This sample demostrates how to build a simple application based on Windows Presentation Foundation (WPF). The application +This sample demonstrates how to build a simple application based on Windows Presentation Foundation (WPF). The application connects to a web camera and displays the video stream. The sample builds and runs on Windows. diff --git a/Samples/WpfSample/WpfSample.csproj b/Samples/WpfSample/WpfSample.csproj index 3082d4aa5..3d9b0522b 100644 --- a/Samples/WpfSample/WpfSample.csproj +++ b/Samples/WpfSample/WpfSample.csproj @@ -4,8 +4,7 @@ WinExe PsiWpfSample.App - false - false + ../../Build/Sample.Psi.ruleset true @@ -33,6 +32,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Sources/Audio/Microsoft.Psi.Audio.Linux/AudioCapture.cs b/Sources/Audio/Microsoft.Psi.Audio.Linux/AudioCapture.cs index e29b6e047..64710d608 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Linux/AudioCapture.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Linux/AudioCapture.cs @@ -126,11 +126,11 @@ public void Start(Action notifyCompletionTime) { LinuxAudioInterop.Read(this.audioDevice, buf, blockSize); } - catch (Exception ex) + catch { if (this.audioDevice != null) { - throw ex; + throw; } } diff --git a/Sources/Audio/Microsoft.Psi.Audio.Linux/LinuxAudioInterop.cs b/Sources/Audio/Microsoft.Psi.Audio.Linux/LinuxAudioInterop.cs index 814d42ccc..1d8f13849 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Linux/LinuxAudioInterop.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Linux/LinuxAudioInterop.cs @@ -10,7 +10,7 @@ namespace Microsoft.Psi.Audio /// Structs, enums and static methods for interacting with Advanced Linux Sound Architecture (ALSA) drivers. /// /// - /// This implimentation is based on this spec: http://www.alsa-project.org/alsa-doc/alsa-lib + /// This implementation is based on this spec: http://www.alsa-project.org/alsa-doc/alsa-lib /// The only dependency is on `asound`, which comes with the system. /// internal static class LinuxAudioInterop @@ -347,56 +347,24 @@ internal static Format ConvertFormat(WaveFormat configFormat) internal static unsafe AudioDevice Open(string name, Mode mode, int rate = 44100, int channels = 1, Format format = Format.S16LE, Access access = Access.Interleaved) { void* handle; - if (Open(&handle, name, (int)mode, 0) != 0) - { - throw new ArgumentException("Open failed."); - } + CheckResult(NativeMethods.Open(&handle, name, (int)mode, 0), "Open failed"); void* param; - if (HardwareParamsMalloc(¶m) != 0) - { - throw new ArgumentException("Hardware params malloc failed."); - } - - if (HardwareParamsAny(handle, param) != 0) - { - throw new ArgumentException("Hardware params any failed."); - } - - if (HardwareParamsSetAccess(handle, param, (int)access) != 0) - { - throw new ArgumentException("Hardware params set access failed."); - } - - if (HardwareParamsSetFormat(handle, param, (int)format) != 0) - { - throw new ArgumentException("Hardware params set format failed."); - } + CheckResult(NativeMethods.HardwareParamsMalloc(¶m), "Hardware params malloc failed"); + CheckResult(NativeMethods.HardwareParamsAny(handle, param), "Hardware params any failed"); + CheckResult(NativeMethods.HardwareParamsSetAccess(handle, param, (int)access), "Hardware params set access failed"); + CheckResult(NativeMethods.HardwareParamsSetFormat(handle, param, (int)format), "Hardware params set format failed"); int* ratePtr = &rate; int dir = 0; int* dirPtr = &dir; - if (HardwareParamsSetRate(handle, param, ratePtr, dirPtr) != 0) - { - throw new ArgumentException("Hardware params set rate failed."); - } + CheckResult(NativeMethods.HardwareParamsSetRate(handle, param, ratePtr, dirPtr), "Hardware params set rate failed"); + CheckResult(NativeMethods.HardwareParamsSetChannels(handle, param, (uint)channels), "Hardware params set channels failed"); + CheckResult(NativeMethods.HardwareParams(handle, param), "Hardware set params failed"); - if (HardwareParamsSetChannels(handle, param, (uint)channels) != 0) - { - throw new ArgumentException("Hardware params set channels failed."); - } + NativeMethods.HardwareParamsFree(param); - if (HardwareParams(handle, param) != 0) - { - throw new ArgumentException("Hardware set params failed."); - } - - HardwareParamsFree(param); - - if (PrepareHandle(handle) != 0) - { - throw new ArgumentException("Prepare handle failed."); - } + CheckResult(NativeMethods.PrepareHandle(handle), "Prepare handle failed"); return new AudioDevice(handle); } @@ -411,18 +379,23 @@ internal static unsafe void Read(AudioDevice device, byte[] buffer, int blockSiz { fixed (void* bufferPtr = buffer) { - long err = Read(device.Handle, bufferPtr, (ulong)blockSize); + long err; + if (Environment.Is64BitOperatingSystem) + { + err = NativeMethods.Read64(device.Handle, bufferPtr, (ulong)blockSize); + } + else + { + err = NativeMethods.Read32(device.Handle, bufferPtr, (uint)blockSize); + } + if (err < 0) { - err = Recover(device.Handle, (int)err, 1); - if (err < 0) - { - throw new ArgumentException("Read recovery failed."); - } + CheckResult(NativeMethods.Recover(device.Handle, (int)err, 1), "Read recovery failed"); } else if (err != blockSize) { - throw new ArgumentException("Read failed."); + throw new ArgumentException($"Read failed (ALSA error code: {err})."); } } } @@ -441,14 +414,18 @@ internal static unsafe int Write(AudioDevice device, byte[] buffer, int blockSiz fixed (void* bufferPtr = buffer) { byte* pb = (byte*)bufferPtr + offset; - err = Write(device.Handle, pb, (ulong)blockSize); + if (Environment.Is64BitOperatingSystem) + { + err = NativeMethods.Write64(device.Handle, pb, (ulong)blockSize); + } + else + { + err = NativeMethods.Write32(device.Handle, pb, (uint)blockSize); + } + if (err < 0) { - err = Recover(device.Handle, (int)err, 1); - if (err < 0) - { - throw new ArgumentException("Write recovery failed."); - } + CheckResult(NativeMethods.Recover(device.Handle, (int)err, 1), "Write recovery failed"); } else if (err != blockSize) { @@ -464,53 +441,24 @@ internal static unsafe int Write(AudioDevice device, byte[] buffer, int blockSiz /// Device handle. internal static unsafe void Close(AudioDevice device) { - if (CloseHandle(device.Handle) != 0) + if (NativeMethods.CloseHandle(device.Handle) != 0) { throw new ArgumentException("Close failed."); } } - [DllImport("asound", EntryPoint = "snd_pcm_open")] - private static unsafe extern int Open(void** handle, [MarshalAs(UnmanagedType.LPStr)]string name, int capture, int mode); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_malloc")] - private static unsafe extern int HardwareParamsMalloc(void** param); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_any")] - private static unsafe extern int HardwareParamsAny(void* handle, void* param); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_access")] - private static unsafe extern int HardwareParamsSetAccess(void* handle, void* param, int access); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_format")] - private static unsafe extern int HardwareParamsSetFormat(void* handle, void* param, int format); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_rate_near")] - private static unsafe extern int HardwareParamsSetRate(void* handle, void* param, int* rate, int* dir); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_channels")] - private static unsafe extern int HardwareParamsSetChannels(void* handle, void* param, uint channels); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params")] - private static unsafe extern int HardwareParams(void* handle, void* param); - - [DllImport("asound", EntryPoint = "snd_pcm_hw_params_free")] - private static unsafe extern void HardwareParamsFree(void* param); - - [DllImport("asound", EntryPoint = "snd_pcm_prepare")] - private static unsafe extern int PrepareHandle(void* handle); - - [DllImport("asound", EntryPoint = "snd_pcm_recover")] - private static unsafe extern int Recover(void* handle, int error, int silent); - - [DllImport("asound", EntryPoint = "snd_pcm_readi")] - private static unsafe extern long Read(void* handle, void* buffer, ulong blockSize); - - [DllImport("asound", EntryPoint = "snd_pcm_writei")] - private static unsafe extern long Write(void* handle, void* buffer, ulong blockSize); - - [DllImport("asound", EntryPoint = "snd_pcm_close")] - private static unsafe extern int CloseHandle(void* handle); + /// + /// Check result code and throw argument exception upon failure from ALSA APIs. + /// + /// Result code returned by ALSA API. + /// Error message in case of failure. + private static void CheckResult(long result, string message) + { + if (result != 0) + { + throw new ArgumentException($"{message} (ALSA error code: {result})."); + } + } /// /// Audio device handle. @@ -532,5 +480,56 @@ public unsafe AudioDevice(void* handle) /// public unsafe void* Handle { get; private set; } } + + private static class NativeMethods + { + [DllImport("asound", EntryPoint = "snd_pcm_open", BestFitMapping = false, ThrowOnUnmappableChar = true)] + internal static unsafe extern int Open(void** handle, [MarshalAs(UnmanagedType.LPStr)]string name, int capture, int mode); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_malloc")] + internal static unsafe extern int HardwareParamsMalloc(void** param); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_any")] + internal static unsafe extern int HardwareParamsAny(void* handle, void* param); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_access")] + internal static unsafe extern int HardwareParamsSetAccess(void* handle, void* param, int access); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_format")] + internal static unsafe extern int HardwareParamsSetFormat(void* handle, void* param, int format); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_rate_near")] + internal static unsafe extern int HardwareParamsSetRate(void* handle, void* param, int* rate, int* dir); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_set_channels")] + internal static unsafe extern int HardwareParamsSetChannels(void* handle, void* param, uint channels); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params")] + internal static unsafe extern int HardwareParams(void* handle, void* param); + + [DllImport("asound", EntryPoint = "snd_pcm_hw_params_free")] + internal static unsafe extern void HardwareParamsFree(void* param); + + [DllImport("asound", EntryPoint = "snd_pcm_prepare")] + internal static unsafe extern int PrepareHandle(void* handle); + + [DllImport("asound", EntryPoint = "snd_pcm_recover")] + internal static unsafe extern int Recover(void* handle, int error, int silent); + + [DllImport("asound", EntryPoint = "snd_pcm_readi")] + internal static unsafe extern int Read32(void* handle, void* buffer, uint blockSize); + + [DllImport("asound", EntryPoint = "snd_pcm_readi")] + internal static unsafe extern long Read64(void* handle, void* buffer, ulong blockSize); + + [DllImport("asound", EntryPoint = "snd_pcm_writei")] + internal static unsafe extern int Write32(void* handle, void* buffer, uint blockSize); + + [DllImport("asound", EntryPoint = "snd_pcm_writei")] + internal static unsafe extern long Write64(void* handle, void* buffer, ulong blockSize); + + [DllImport("asound", EntryPoint = "snd_pcm_close")] + internal static unsafe extern int CloseHandle(void* handle); + } } } diff --git a/Sources/Audio/Microsoft.Psi.Audio.Linux/Microsoft.Psi.Audio.Linux.csproj b/Sources/Audio/Microsoft.Psi.Audio.Linux/Microsoft.Psi.Audio.Linux.csproj index 79f661eca..80d2764da 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Linux/Microsoft.Psi.Audio.Linux.csproj +++ b/Sources/Audio/Microsoft.Psi.Audio.Linux/Microsoft.Psi.Audio.Linux.csproj @@ -36,6 +36,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCapture.cs b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCapture.cs index f5ec9c48e..55d37f31a 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCapture.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCapture.cs @@ -88,6 +88,16 @@ public AudioCapture(Pipeline pipeline, string configurationFilename = null) { } + /// + /// Initializes a new instance of the class with a specified output format. + /// + /// The pipeline to add the component to. + /// The output format to use. + public AudioCapture(Pipeline pipeline, WaveFormat outputFormat) + : this(pipeline, new AudioCaptureConfiguration() { OutputFormat = outputFormat }) + { + } + /// /// Gets the output stream of audio buffers. /// diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCaptureConfiguration.cs b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCaptureConfiguration.cs index 38a22a912..57cead244 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCaptureConfiguration.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioCaptureConfiguration.cs @@ -17,15 +17,6 @@ public sealed class AudioCaptureConfiguration /// public AudioCaptureConfiguration() { - this.DeviceName = string.Empty; - this.TargetLatencyInMs = 20; - this.AudioEngineBufferInMs = 500; - this.AudioLevel = -1; - this.Gain = 1.0f; - this.OptimizeForSpeech = false; - this.UseEventDrivenCapture = true; - this.DropOutOfOrderPackets = false; - this.OutputFormat = null; } /// @@ -37,7 +28,7 @@ public AudioCaptureConfiguration() /// static method. If not specified, the /// default recording device will be selected. /// - public string DeviceName { get; set; } + public string DeviceName { get; set; } = string.Empty; /// /// Gets or sets the target audio latency (pull capture mode only). @@ -52,7 +43,7 @@ public AudioCaptureConfiguration() /// is set to true. In event-driven capture mode, the latency /// is determined by the rate at which the audio engine signals that it has new data available. /// - public int TargetLatencyInMs { get; set; } + public int TargetLatencyInMs { get; set; } = 20; /// /// Gets or sets the audio engine buffer. @@ -66,7 +57,7 @@ public AudioCaptureConfiguration() /// audio packets fast enough. Setting this to a larger value reduces the likelihood of /// encountering glitches in the captured audio stream. /// - public int AudioEngineBufferInMs { get; set; } + public int AudioEngineBufferInMs { get; set; } = 500; /// /// Gets or sets the audio input level. @@ -76,7 +67,7 @@ public AudioCaptureConfiguration() /// between 0.0 and 1.0 inclusive. If not specified, the current level of the selected /// recording device will be left unchanged. /// - public double AudioLevel { get; set; } + public double AudioLevel { get; set; } = -1; /// /// Gets or sets the additional gain to be applied to the captured audio. @@ -86,7 +77,7 @@ public AudioCaptureConfiguration() /// audio signal. Values greater than 1.0 boost the audio signal, while values in the range /// of 0.0 to 1.0 attenuate it. The default value is 1.0 (no additional gain). /// - public float Gain { get; set; } + public float Gain { get; set; } = 1.0f; /// /// Gets or sets a value indicating whether the captured audio should be pre-processed for @@ -98,7 +89,7 @@ public AudioCaptureConfiguration() /// the audio for speech recognition applications. By default, this option is set to false. /// This feature may not be available for all capture devices. /// - public bool OptimizeForSpeech { get; set; } + public bool OptimizeForSpeech { get; set; } = false; /// /// Gets or sets a value indicating whether to use event-driven or pull capture mode. When using @@ -109,7 +100,7 @@ public AudioCaptureConfiguration() /// by the audio engine (up to an amount equivalent to ) should /// the application be unable to consume the audio data quickly enough. /// - public bool UseEventDrivenCapture { get; set; } + public bool UseEventDrivenCapture { get; set; } = true; /// /// Gets or sets a value indicating whether the component should @@ -118,7 +109,7 @@ public AudioCaptureConfiguration() /// /// This is for internal use only and may be removed in future versions. /// - public bool DropOutOfOrderPackets { get; set; } + public bool DropOutOfOrderPackets { get; set; } = false; /// /// Gets or sets the desired format for the captured audio. @@ -128,6 +119,6 @@ public AudioCaptureConfiguration() /// Use this to specify a different format for the Out stream of /// the component. /// - public WaveFormat OutputFormat { get; set; } + public WaveFormat OutputFormat { get; set; } = null; } } diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioPlayerConfiguration.cs b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioPlayerConfiguration.cs index 4e3063481..42329051d 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioPlayerConfiguration.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioPlayerConfiguration.cs @@ -17,14 +17,6 @@ public sealed class AudioPlayerConfiguration /// public AudioPlayerConfiguration() { - this.DeviceName = string.Empty; - this.TargetLatencyInMs = 20; - this.BufferLengthSeconds = 0.1; - this.AudioLevel = -1; - this.Gain = 1.0f; - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -36,7 +28,7 @@ public AudioPlayerConfiguration() /// static method. If not specified, the /// default playback device will be selected. /// - public string DeviceName { get; set; } + public string DeviceName { get; set; } = string.Empty; /// /// Gets or sets the target audio latency. @@ -46,9 +38,9 @@ public AudioPlayerConfiguration() /// at a time. This in turn determines the latency of the audio output (i.e. the amount of lag /// between when the audio was available and when the corresponding sound is produced). For /// live audio playback, we normally want this to be small. By default, this value is set to - /// 20 milliseconds. Is is safe to leave this unchanged. + /// 20 milliseconds. It is safe to leave this unchanged. /// - public int TargetLatencyInMs { get; set; } + public int TargetLatencyInMs { get; set; } = 20; /// /// Gets or sets the maximum duration of audio that can be buffered for playback. @@ -57,7 +49,7 @@ public AudioPlayerConfiguration() /// This controls the amount of audio that can be buffered while waiting for the playback /// device to be ready to render it. The default value is 0.1 seconds. /// - public double BufferLengthSeconds { get; set; } + public double BufferLengthSeconds { get; set; } = 0.1; /// /// Gets or sets the audio output level. @@ -67,7 +59,7 @@ public AudioPlayerConfiguration() /// between 0.0 and 1.0 inclusive. If not specified, the current level of the selected /// playback device will be left unchanged. /// - public float AudioLevel { get; set; } + public float AudioLevel { get; set; } = -1; /// /// Gets or sets the additional gain to be applied to the audio data. @@ -77,7 +69,7 @@ public AudioPlayerConfiguration() /// signal. Values greater than 1.0 boost the audio signal, while values in the range /// of 0.0 to 1.0 attenuate it. The default value is 1.0 (no additional gain). /// - public float Gain { get; set; } + public float Gain { get; set; } = 1.0f; /// /// Gets or sets the input format of the audio stream. @@ -87,6 +79,6 @@ public AudioPlayerConfiguration() /// set, the component will attempt to infer the audio format /// from the messages arriving on the input stream. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResampler.cs b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResampler.cs index 1f4fabbee..7479fc6a8 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResampler.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResampler.cs @@ -184,7 +184,7 @@ private void AudioDataAvailableCallback(IntPtr data, int length, long timestamp) timestamp + (10000000L * length / this.Configuration.OutputFormat.AvgBytesPerSec), DateTimeKind.Utc); - if (originatingTime < this.lastOutputPostTime) + if (originatingTime <= this.lastOutputPostTime) { // If the input audio packet is larger than the output packet (as determined by the // target latency), then the packet will be split into multiple packets for resampling. @@ -195,10 +195,10 @@ private void AudioDataAvailableCallback(IntPtr data, int length, long timestamp) // This could happen if the two consecutive input packets overlap in time, for example // if an automatic system time adjustment occurred between the capture of the two packets. // These adjustments occur from time to time to account for system clock drift w.r.t. - // UTC time. In order to ensure that this does not lead to resampled output sub-packets - // regressing in time, we manually enforce the output originating time to be no less than - // that of the previous packet. - originatingTime = this.lastOutputPostTime; + // UTC time. As this could in result in output message originating times not advancing + // or even regressing, we check for this and ensure that they are always monotonically + // increasing. + originatingTime = this.lastOutputPostTime + TimeSpan.FromTicks(1); } // post the data to the output stream diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResamplerConfiguration.cs b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResamplerConfiguration.cs index 532889cf6..62afd7af5 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResamplerConfiguration.cs +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/AudioResamplerConfiguration.cs @@ -17,9 +17,6 @@ public sealed class AudioResamplerConfiguration /// public AudioResamplerConfiguration() { - this.TargetLatencyInMs = 20; - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); - this.OutputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -30,9 +27,9 @@ public AudioResamplerConfiguration() /// turn determines the latency of the audio output. The larger this value, the more audio /// data is carried in each and the longer the audio latency. For /// live audio capture, we normally want this value to be small as possible. By default, - /// this value is set to 20 milliseconds. Is is safe to leave this unchanged. + /// this value is set to 20 milliseconds. It is safe to leave this unchanged. /// - public int TargetLatencyInMs { get; set; } + public int TargetLatencyInMs { get; set; } = 20; /// /// Gets or sets the input format of the audio stream to be resampled. @@ -42,11 +39,11 @@ public AudioResamplerConfiguration() /// set, the component will attempt to infer the audio format /// from the messages arriving on the input stream. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); /// /// Gets or sets the output format for the resampled audio. /// - public WaveFormat OutputFormat { get; set; } + public WaveFormat OutputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Audio/Microsoft.Psi.Audio.Windows/Microsoft.Psi.Audio.Windows.csproj b/Sources/Audio/Microsoft.Psi.Audio.Windows/Microsoft.Psi.Audio.Windows.csproj index 15b8e47bf..c0c73b894 100644 --- a/Sources/Audio/Microsoft.Psi.Audio.Windows/Microsoft.Psi.Audio.Windows.csproj +++ b/Sources/Audio/Microsoft.Psi.Audio.Windows/Microsoft.Psi.Audio.Windows.csproj @@ -30,6 +30,10 @@ + + all + runtime; build; native; contentfiles; analyzers + \ No newline at end of file diff --git a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractor.cs b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractor.cs index 31d934cf0..40ad3dc16 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractor.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractor.cs @@ -17,7 +17,7 @@ namespace Microsoft.Psi.Audio /// public sealed class AcousticFeaturesExtractor : IConsumer { - private Connector inAudio; + private readonly Connector inAudio; /// /// Initializes a new instance of the class. @@ -43,11 +43,9 @@ public AcousticFeaturesExtractor(Pipeline pipeline, AcousticFeaturesExtractorCon float frameRate = configuration.FrameRateInHz; int frameSize = (int)((configuration.InputFormat.SamplesPerSec * configuration.FrameDurationInSeconds) + 0.5); int frameShift = (int)((configuration.InputFormat.SamplesPerSec / frameRate) + 0.5); - int frameOverlap = frameSize - frameShift; int bytesPerSample = configuration.InputFormat.BlockAlign; int bytesPerFrame = bytesPerSample * frameSize; int bytesPerFrameShift = bytesPerSample * frameShift; - int bytesPerOverlap = bytesPerFrame - bytesPerFrameShift; int fftSize = 2; while (fftSize < frameSize) { diff --git a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractorConfiguration.cs b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractorConfiguration.cs index 913d05023..f607f28a7 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractorConfiguration.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/AcousticFeaturesExtractorConfiguration.cs @@ -34,23 +34,8 @@ public sealed class AcousticFeaturesExtractorConfiguration public AcousticFeaturesExtractorConfiguration() { // Default parameters for acoustic features computation - this.FrameDurationInSeconds = 0.025f; - this.FrameRateInHz = 100.0f; - this.AddDither = true; - this.DitherScaleFactor = 1.0f; - this.StartFrequency = 250.0f; - this.EndFrequency = 7000.0f; - this.LowEndFrequency = 3000.0f; - this.HighStartFrequency = 2500.0f; - this.EntropyBandwidth = 2500.0f; - this.ComputeLogEnergy = true; - this.ComputeZeroCrossingRate = true; - this.ComputeFrequencyDomainEnergy = true; - this.ComputeLowFrequencyEnergy = true; - this.ComputeHighFrequencyEnergy = true; - this.ComputeSpectralEntropy = true; - this.ComputeFFT = false; - this.ComputeFFTPower = false; + this.computeFFT = false; + this.computeFFTPower = false; // Defaults to 16 kHz, 16-bit, 1-channel PCM samples this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); @@ -59,58 +44,78 @@ public AcousticFeaturesExtractorConfiguration() /// /// Gets or sets the duration of the frame of audio over which the acoustic features will be computed. /// - public float FrameDurationInSeconds { get; set; } + public float FrameDurationInSeconds { get; set; } = 0.025f; /// /// Gets or sets the frame rate at which the acoustic features will be computed. /// - public float FrameRateInHz { get; set; } + public float FrameRateInHz { get; set; } = 100.0f; /// /// Gets or sets a value indicating whether dither is to be applied to the audio data. /// - public bool AddDither { get; set; } + public bool AddDither { get; set; } = true; /// /// Gets or sets the scale factor by which the dither to be applied will be multiplied. /// A scale factor of 1.0 will result in a dither with a range of -1.0 to +1.0. /// - public float DitherScaleFactor { get; set; } + public float DitherScaleFactor { get; set; } = 1.0f; /// /// Gets or sets the start frequency for frequency-domain features. /// - public float StartFrequency { get; set; } + public float StartFrequency { get; set; } = 250.0f; /// /// Gets or sets the end frequency for frequency-domain features. /// - public float EndFrequency { get; set; } + public float EndFrequency { get; set; } = 7000.0f; /// /// Gets or sets the end frequency for low-frequency features. /// - public float LowEndFrequency { get; set; } + public float LowEndFrequency { get; set; } = 3000.0f; /// /// Gets or sets the start frequency for high-frequency features. /// - public float HighStartFrequency { get; set; } + public float HighStartFrequency { get; set; } = 2500.0f; /// /// Gets or sets the bandwidth for entropy features. /// - public float EntropyBandwidth { get; set; } + public float EntropyBandwidth { get; set; } = 2500.0f; /// /// Gets or sets a value indicating whether to compute the log energy stream. /// - public bool ComputeLogEnergy { get; set; } + public bool ComputeLogEnergy { get; set; } = true; /// /// Gets or sets a value indicating whether to compute the zero-crossing rate stream. /// - public bool ComputeZeroCrossingRate { get; set; } + public bool ComputeZeroCrossingRate { get; set; } = true; + + /// + /// Gets or sets a value indicating whether to compute the frequency domain energy stream. + /// + public bool ComputeFrequencyDomainEnergy { get; set; } = true; + + /// + /// Gets or sets a value indicating whether to compute the low frequency energy stream. + /// + public bool ComputeLowFrequencyEnergy { get; set; } = true; + + /// + /// Gets or sets a value indicating whether to compute the high frequency energy stream. + /// + public bool ComputeHighFrequencyEnergy { get; set; } = true; + + /// + /// Gets or sets a value indicating whether to compute the spectral entropy stream. + /// + public bool ComputeSpectralEntropy { get; set; } = true; /// /// Gets or sets a value indicating whether to compute the FFT stream. @@ -148,26 +153,6 @@ public bool ComputeFFTPower } } - /// - /// Gets or sets a value indicating whether to compute the frequency domain energy stream. - /// - public bool ComputeFrequencyDomainEnergy { get; set; } - - /// - /// Gets or sets a value indicating whether to compute the low frequency energy stream. - /// - public bool ComputeLowFrequencyEnergy { get; set; } - - /// - /// Gets or sets a value indicating whether to compute the high frequency energy stream. - /// - public bool ComputeHighFrequencyEnergy { get; set; } - - /// - /// Gets or sets a value indicating whether to compute the spectral entropy stream. - /// - public bool ComputeSpectralEntropy { get; set; } - /// /// Gets or sets the format of the audio stream. /// diff --git a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FastFourierTransform.cs b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FastFourierTransform.cs index a12405211..0f499f422 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FastFourierTransform.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FastFourierTransform.cs @@ -10,14 +10,12 @@ namespace Microsoft.Psi.Audio /// internal sealed class FastFourierTransform { - private int fftSize; // FFT size - private int windowSize; // Size of the data window. FFTSize - WindowSize = ZeroPadSize - private int fftPow2; // FFT size in form of POW of 2 - private int halfFftSize; + private readonly int fftSize; // FFT size + private readonly int windowSize; // Size of the data window. FFTSize - WindowSize = ZeroPadSize + private readonly int fftPow2; // FFT size in form of POW of 2 - private float[] wriFactors; // SinCos(theta) array - private float[] alignedWriFactors; // SinCos(theta) array - 16 byte aligned - private short[] revMap; + private readonly float[] alignedWriFactors; // SinCos(theta) array - 16 byte aligned + private readonly short[] revMap; /// /// Initializes a new instance of the class. @@ -28,17 +26,15 @@ public FastFourierTransform(int fftSize, int windowSize) { this.fftSize = fftSize; this.windowSize = windowSize; - this.halfFftSize = this.fftSize >> 1; this.fftPow2 = 1; int size = 2; while (size < fftSize) { - size = size << 1; + size <<= 1; this.fftPow2++; } this.alignedWriFactors = new float[this.fftSize * 2]; - this.wriFactors = new float[(this.fftSize * 2) + 20]; this.revMap = new short[this.fftSize / 2]; this.alignedWriFactors[0] = 1.0f; this.alignedWriFactors[1] = -1.0f; @@ -68,7 +64,7 @@ public FastFourierTransform(int fftSize, int windowSize) while (j >= k) { j -= k; - k = k >> 1; + k >>= 1; } j += k; @@ -240,7 +236,7 @@ public void ComputeFFT(float[] input, ref float[] output) kk += limit; limit = incr; - incr = incr + incr; + incr += incr; } float xr1, xi1, xr2, xi2; diff --git a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FrequencyDomainEnergy.cs b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FrequencyDomainEnergy.cs index ccbc732b3..f00c0bf0a 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FrequencyDomainEnergy.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/AcousticFeatures/FrequencyDomainEnergy.cs @@ -10,8 +10,8 @@ namespace Microsoft.Psi.Audio /// public sealed class FrequencyDomainEnergy : ConsumerProducer { - private int start; - private int end; + private readonly int start; + private readonly int end; /// /// Initializes a new instance of the class. diff --git a/Sources/Audio/Microsoft.Psi.Audio/AudioBuffer.cs b/Sources/Audio/Microsoft.Psi.Audio/AudioBuffer.cs index f55119309..a199ac1bc 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/AudioBuffer.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/AudioBuffer.cs @@ -18,8 +18,8 @@ namespace Microsoft.Psi.Audio /// public struct AudioBuffer { - private WaveFormat format; - private byte[] data; + private readonly WaveFormat format; + private readonly byte[] data; /// /// Initializes a new instance of the structure. diff --git a/Sources/Audio/Microsoft.Psi.Audio/Microsoft.Psi.Audio.csproj b/Sources/Audio/Microsoft.Psi.Audio/Microsoft.Psi.Audio.csproj index 7ade1b59f..9b4dca4f3 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/Microsoft.Psi.Audio.csproj +++ b/Sources/Audio/Microsoft.Psi.Audio/Microsoft.Psi.Audio.csproj @@ -35,6 +35,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Audio/Microsoft.Psi.Audio/WaveFileHelper.cs b/Sources/Audio/Microsoft.Psi.Audio/WaveFileHelper.cs index 9b207b06a..7665a5d35 100644 --- a/Sources/Audio/Microsoft.Psi.Audio/WaveFileHelper.cs +++ b/Sources/Audio/Microsoft.Psi.Audio/WaveFileHelper.cs @@ -57,7 +57,7 @@ public static WaveFormat ReadWaveFileHeader(string filename) } /// - /// Reads the lenth in bytes of the data section of a Wave file. + /// Reads the length in bytes of the data section of a Wave file. /// /// The binary reader to read from. /// The number of byte of wave data that follow. diff --git a/Sources/Audio/Test.Psi.Audio/Test.Psi.Audio.csproj b/Sources/Audio/Test.Psi.Audio/Test.Psi.Audio.csproj index 5844bc0df..f146dbecf 100644 --- a/Sources/Audio/Test.Psi.Audio/Test.Psi.Audio.csproj +++ b/Sources/Audio/Test.Psi.Audio/Test.Psi.Audio.csproj @@ -2,7 +2,7 @@ Exe - netcoreapp2.0 + netcoreapp3.1 false Audio unit tests Test.Psi.Audio.ConsoleMain @@ -27,10 +27,14 @@ - + + all + runtime; build; native; contentfiles; analyzers + + - - + + diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/CameraIntrinsics.cs b/Sources/Calibration/Microsoft.Psi.Calibration/CameraIntrinsics.cs index 55fe188db..7011eaae0 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/CameraIntrinsics.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/CameraIntrinsics.cs @@ -3,6 +3,7 @@ namespace Microsoft.Psi.Calibration { + using System.Runtime.Serialization; using MathNet.Numerics.LinearAlgebra; using MathNet.Spatial.Euclidean; @@ -13,6 +14,9 @@ public class CameraIntrinsics : ICameraIntrinsics { private Matrix transform; + [OptionalField] + private bool closedFormDistorts; + /// /// Initializes a new instance of the class. /// @@ -21,12 +25,14 @@ public class CameraIntrinsics : ICameraIntrinsics /// The intrinsics transform matrix. /// The radial distortion parameters. /// The tangential distortion parameters. + /// Indicates which direction the closed form equation for Brown-Conrady Distortion model goes. I.e. does it perform distortion or undistortion. Default is to distort (thus making projection simpler and unprojection more complicated). public CameraIntrinsics( int imageWidth, int imageHeight, Matrix transform, Vector radialDistortion = null, - Vector tangentialDistortion = null) + Vector tangentialDistortion = null, + bool closedFormDistorts = true) { this.ImageWidth = imageWidth; this.ImageHeight = imageHeight; @@ -35,6 +41,7 @@ public class CameraIntrinsics : ICameraIntrinsics this.TangentialDistortion = tangentialDistortion ?? Vector.Build.Dense(2, 0); this.FocalLengthXY = new Point2D(this.Transform[0, 0], this.Transform[1, 1]); this.PrincipalPoint = new Point2D(this.Transform[0, 2], this.Transform[1, 2]); + this.ClosedFormDistorts = closedFormDistorts; } /// @@ -70,6 +77,20 @@ private set /// public Point2D PrincipalPoint { get; private set; } + /// + public bool ClosedFormDistorts + { + get + { + return this.closedFormDistorts; + } + + private set + { + this.closedFormDistorts = value; + } + } + /// public int ImageWidth { get; private set; } @@ -88,7 +109,7 @@ public Point2D ToPixelSpace(Point3D pt, bool distort) Point3D tmp = new Point3D(pixelPt.X, pixelPt.Y, 1.0); tmp = tmp.TransformBy(this.transform); - return new Point2D(tmp.X, this.ImageHeight - tmp.Y); + return new Point2D(tmp.X, tmp.Y); } /// @@ -98,33 +119,63 @@ public Point3D ToCameraSpace(Point2D pt, double depth, bool undistort) Point3D tmp = new Point3D(pt.X, pt.Y, 1.0); tmp = tmp.TransformBy(this.InvTransform); - // Undistort the pixel + // Distort the pixel Point2D pixelPt = new Point2D(tmp.X, tmp.Y); if (undistort) { - pixelPt = this.UndistortPoint(pixelPt); + this.UndistortPoint(pixelPt, out pixelPt); } // X points in the depth dimension. Y points to the left, and Z points up. return new Point3D(depth, -pixelPt.X * depth, -pixelPt.Y * depth); } + /// + public bool UndistortPoint(Point2D distortedPt, out Point2D undistortedPt) + { + if (this.ClosedFormDistorts) + { + return this.InverseOfClosedForm(distortedPt, out undistortedPt); + } + + return this.ClosedForm(distortedPt, out undistortedPt); + } + /// public bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt) { - double x = undistortedPt.X; - double y = undistortedPt.Y; + if (this.ClosedFormDistorts) + { + return this.ClosedForm(undistortedPt, out distortedPt); + } + + return this.InverseOfClosedForm(undistortedPt, out distortedPt); + } + + private bool InverseOfClosedForm(Point2D inputPt, out Point2D outputPt) + { + double k1 = this.RadialDistortion[0]; + double k2 = this.RadialDistortion[1]; + double k3 = this.RadialDistortion[2]; + double k4 = this.RadialDistortion[3]; + double k5 = this.RadialDistortion[4]; + double k6 = this.RadialDistortion[5]; + double t0 = this.TangentialDistortion[0]; + double t1 = this.TangentialDistortion[1]; + + double x = inputPt.X; + double y = inputPt.Y; // Our distortion model is defined as: // See https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=convertpointshomogeneous // r^2 = x^2 + y^2 - // (1+k1*r^2+k2*r^3+k3^r^6) + // (1+k1*r^2+k2*r^4+k3^r^6) // Fx = x ------------------------ + t1*(r^2+ 2 * x^2) + 2 * t0 * x*y - // (1+k4*r^2+k5*r^3+k6^r^6) + // (1+k4*r^2+k5*r^4+k6^r^6) // - // (1+k1*r^2+k2*r^3+k3^r^6) + // (1+k1*r^2+k2*r^4+k3^r^6) // Fy = y ------------------------ + t0*(r^2+ 2 * y^2) + 2 * t1 * x*y - // (1+k4*r^2+k5*r^3+k6^r^6) + // (1+k4*r^2+k5*r^4+k6^r^6) // // We want to solve for: // 1 | @Fy/@y -@Fx/@y | @@ -147,15 +198,6 @@ public bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt) // @Fy/@x = y @d/@x + 2*t0*y + 2*t1*x // // In the code below @/@ is named 'dd'. - double k1 = this.RadialDistortion[0]; - double k2 = this.RadialDistortion[1]; - double k3 = this.RadialDistortion[2]; - double k4 = this.RadialDistortion[3]; - double k5 = this.RadialDistortion[4]; - double k6 = this.RadialDistortion[5]; - double t0 = this.TangentialDistortion[0]; - double t1 = this.TangentialDistortion[1]; - #pragma warning disable SA1305 bool converged = false; for (int j = 0; j < 100 && !converged; j++) @@ -169,7 +211,7 @@ public bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt) double dr2dy = 2 * y; double d = g / h; double dgdr2 = k1 + 2 * k2 * radiusSq + 3 * k3 * radiusSqSq; - double dhdr2 = k4 * 2 * k5 * radiusSq + 3 * k6 * radiusSqSq; + double dhdr2 = k4 + 2 * k5 * radiusSq + 3 * k6 * radiusSqSq; double dddr2 = (dgdr2 * h - g * dhdr2) / (h * h); double dddx = dddr2 * 2 * x; double dddy = dddr2 * 2 * y; @@ -183,7 +225,7 @@ public bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt) if (System.Math.Abs(det) < 1E-16) { // Not invertible. Perform no distortion - distortedPt = new Point2D(undistortedPt.X, undistortedPt.Y); + outputPt = new Point2D(inputPt.X, inputPt.Y); return false; } @@ -199,31 +241,40 @@ public bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt) // to be equal to 0: // 0 = F(xp) - Xu // 0 = F(yp) - Yu - xp -= undistortedPt.X; - yp -= undistortedPt.Y; + double errx = xp - inputPt.X; + double erry = yp - inputPt.Y; - if ((xp * xp) + (yp * yp) < 1E-16) + double err = (errx * errx) + (erry * erry); + if (err < 1.0e-16) { converged = true; break; } // Update our new guess (i.e. x = x - J(F(x))^-1 * F(x)) - x = x - ((dFydy * xp) - (dFxdy * yp)) / det; - y = y - ((-dFydx * xp) + (dFxdx * yp)) / det; + x = x - ((dFydy * errx) - (dFxdy * erry)) / det; + y = y - ((-dFydx * errx) + (dFxdx * erry)) / det; + #pragma warning restore SA1305 } - distortedPt = new Point2D(x, y); - return true; + if (converged) + { + outputPt = new Point2D(x, y); + } + else + { + outputPt = new Point2D(inputPt.X, inputPt.Y); + } + + return converged; } - /// - public Point2D UndistortPoint(Point2D distortedPt) + private bool ClosedForm(Point2D inputPt, out Point2D outputPt) { // Undistort pixel double xp, yp; - double radiusSquared = (distortedPt.X * distortedPt.X) + (distortedPt.Y * distortedPt.Y); + double radiusSquared = (inputPt.X * inputPt.X) + (inputPt.Y * inputPt.Y); if (this.RadialDistortion != null) { double k1 = this.RadialDistortion[0]; @@ -236,26 +287,27 @@ public Point2D UndistortPoint(Point2D distortedPt) double h = 1 + k4 * radiusSquared + k5 * radiusSquared * radiusSquared + k6 * radiusSquared * radiusSquared * radiusSquared; double d = g / h; - xp = distortedPt.X * d; - yp = distortedPt.Y * d; + xp = inputPt.X * d; + yp = inputPt.Y * d; } else { - xp = distortedPt.X; - yp = distortedPt.Y; + xp = inputPt.X; + yp = inputPt.Y; } // If we are incorporating tangential distortion, include that here if (this.TangentialDistortion != null && (this.TangentialDistortion[0] != 0.0 || this.TangentialDistortion[1] != 0.0)) { - double xy = 2.0 * distortedPt.X * distortedPt.Y; - double x2 = 2.0 * distortedPt.X * distortedPt.X; - double y2 = 2.0 * distortedPt.Y * distortedPt.Y; + double xy = 2.0 * inputPt.X * inputPt.Y; + double x2 = 2.0 * inputPt.X * inputPt.X; + double y2 = 2.0 * inputPt.Y * inputPt.Y; xp += (this.TangentialDistortion[1] * (radiusSquared + x2)) + (this.TangentialDistortion[0] * xy); yp += (this.TangentialDistortion[0] * (radiusSquared + y2)) + (this.TangentialDistortion[1] * xy); } - return new Point2D(xp, yp); + outputPt = new Point2D(xp, yp); + return true; } } } diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/DepthDeviceCalibrationInfo.cs b/Sources/Calibration/Microsoft.Psi.Calibration/DepthDeviceCalibrationInfo.cs index 66471cce3..d88ab6d24 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/DepthDeviceCalibrationInfo.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/DepthDeviceCalibrationInfo.cs @@ -60,6 +60,7 @@ public DepthDeviceCalibrationInfo() Vector.Build.DenseOfArray(colorTangentialDistortionCoefficients)); this.ColorExtrinsics = new CoordinateSystem(depthToColorTransform); + this.ColorPose = this.ColorExtrinsics.Invert(); this.DepthIntrinsics = new CameraIntrinsics( depthWidth, @@ -69,24 +70,34 @@ public DepthDeviceCalibrationInfo() Vector.Build.DenseOfArray(depthTangentialDistortionCoefficients)); this.DepthExtrinsics = new CoordinateSystem(depthExtrinsics); + this.DepthPose = this.DepthExtrinsics.Invert(); } /// public CoordinateSystem ColorExtrinsics { get; } + /// + public CoordinateSystem ColorPose { get; } + /// public ICameraIntrinsics ColorIntrinsics { get; } /// public CoordinateSystem DepthExtrinsics { get; } + /// + public CoordinateSystem DepthPose { get; } + /// public ICameraIntrinsics DepthIntrinsics { get; } /// public Point2D ToColorSpace(Point3D point3D) { + // First convert the point into camera coordinates. var point3DInColorCamera = this.ColorExtrinsics.Transform(point3D); + + // Then convert to pixel space. return this.ColorIntrinsics.ToPixelSpace(point3DInColorCamera, true); } } diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/DepthExtensions.cs b/Sources/Calibration/Microsoft.Psi.Calibration/DepthExtensions.cs index a36593d23..83d67619c 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/DepthExtensions.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/DepthExtensions.cs @@ -4,7 +4,9 @@ namespace Microsoft.Psi.Calibration { using System; + using System.Collections.Generic; using MathNet.Spatial.Euclidean; + using Microsoft.Psi; using Microsoft.Psi.Imaging; /// @@ -19,9 +21,9 @@ public static class DepthExtensions /// Pixel coordinates in the color camera. /// Depth map. /// Point in camera coordinates. - internal static Point3D? ProjectToCameraSpace(IDepthDeviceCalibrationInfo depthDeviceCalibrationInfo, Point2D point2D, Shared depthImage) + public static Point3D? ProjectToCameraSpace(IDepthDeviceCalibrationInfo depthDeviceCalibrationInfo, Point2D point2D, Shared depthImage) { - var colorExtrinsicsInverse = depthDeviceCalibrationInfo.ColorExtrinsics.Inverse(); + var colorExtrinsicsInverse = depthDeviceCalibrationInfo.ColorPose; var pointInCameraSpace = depthDeviceCalibrationInfo.ColorIntrinsics.ToCameraSpace(point2D, 1.0, true); double x = pointInCameraSpace.X * colorExtrinsicsInverse[0, 0] + pointInCameraSpace.Y * colorExtrinsicsInverse[0, 1] + pointInCameraSpace.Z * colorExtrinsicsInverse[0, 2] + colorExtrinsicsInverse[0, 3]; double y = pointInCameraSpace.X * colorExtrinsicsInverse[1, 0] + pointInCameraSpace.Y * colorExtrinsicsInverse[1, 1] + pointInCameraSpace.Z * colorExtrinsicsInverse[1, 2] + colorExtrinsicsInverse[1, 3]; @@ -32,6 +34,20 @@ public static class DepthExtensions return IntersectLineWithDepthMesh(depthDeviceCalibrationInfo, rgbLine, depthImage.Resource, 0.1); } + /// + /// Projects set of 2D image points into 3D. + /// + /// Tuple of depth image, list of points to project, and calibration information. + /// An optional delivery policy. + /// Returns a producer that generates a list of corresponding 3D points in Kinect camera space. + public static IProducer> ProjectTo3D( + this IProducer<(Shared, List, IDepthDeviceCalibrationInfo)> source, DeliveryPolicy<(Shared, List, IDepthDeviceCalibrationInfo)> deliveryPolicy = null) + { + var projectTo3D = new ProjectTo3D(source.Out.Pipeline); + source.PipeTo(projectTo3D, deliveryPolicy); + return projectTo3D; + } + /// /// Performs a ray/mesh intersection with the depth map. /// @@ -41,7 +57,7 @@ public static class DepthExtensions /// Distance to march on each step along ray. /// Whether undistortion should be applied to the point. /// Returns point of intersection. - internal static Point3D? IntersectLineWithDepthMesh(IDepthDeviceCalibrationInfo calibration, Line3D line, Image depthImage, double skipFactor, bool undistort = true) + internal static Point3D? IntersectLineWithDepthMesh(IDepthDeviceCalibrationInfo calibration, Line3D line, DepthImage depthImage, double skipFactor, bool undistort = true) { // max distance to check for intersection with the scene double totalDistance = 5; @@ -67,7 +83,7 @@ public static class DepthExtensions return null; } - private static float GetMeshDepthAtPoint(IDepthDeviceCalibrationInfo calibration, Image depthImage, Point3D point, bool undistort) + private static float GetMeshDepthAtPoint(IDepthDeviceCalibrationInfo calibration, DepthImage depthImage, Point3D point, bool undistort) { Point2D depthSpacePoint = calibration.DepthIntrinsics.ToPixelSpace(point, undistort); diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/ICameraIntrinsics.cs b/Sources/Calibration/Microsoft.Psi.Calibration/ICameraIntrinsics.cs index 4d901769a..831bef178 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/ICameraIntrinsics.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/ICameraIntrinsics.cs @@ -49,6 +49,15 @@ public interface ICameraIntrinsics /// Point2D PrincipalPoint { get; } + /// + /// Gets a value indicating whether the closed form equation of the Brown-Conrady Distortion model + /// distorts or undistorts. i.e. if true then: + /// Xdistorted = Xundistorted * (1+K1*R2+K2*R3+... + /// otherwise: + /// Xundistorted = Xdistorted * (1+K1*R2+K2*R3+... + /// + bool ClosedFormDistorts { get; } + /// /// Gets the width of the camera's image (in pixels). /// @@ -92,10 +101,10 @@ public interface ICameraIntrinsics /// Newton's method is used to find the inverse of this. That is /// Xd(n+1) = Xd(n) + J^-1 * F(Xd,Yd). /// - /// The undistorted point in camera post-projection coordinates. - /// The distorted point. - /// True if 'distortedPoint' contains the distorted point, or false if the algorithm did not converge. - bool DistortPoint(Point2D undistortedPoint, out Point2D distortedPoint); + /// The undistorted point in camera post-projection coordinates. + /// The distorted point. + /// True if 'distortedPt' contains the distorted point, or false if the algorithm did not converge. + bool DistortPoint(Point2D undistortedPt, out Point2D distortedPt); /// /// Applies the camera's radial and tangential undistortion to the specified (distorted) point. @@ -110,8 +119,9 @@ public interface ICameraIntrinsics /// T0,T1 - tangential distortion coefficients. /// /// - /// Distorted point in camera post-projection coordinates. - /// Undistorted coordinates in camera post-projection coordinates. - Point2D UndistortPoint(Point2D distortedPoint); + /// Distorted point in camera post-projection coordinates. + /// Returns the undistorted point in camera post-projection coordinates. + /// True if 'undistortedPoint' contains the undistorted point, or false if the algorithm did not converge. + bool UndistortPoint(Point2D distortedPt, out Point2D undistortedPt); } } diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/IDepthDeviceCalibrationInfo.cs b/Sources/Calibration/Microsoft.Psi.Calibration/IDepthDeviceCalibrationInfo.cs index 873966a0b..e52d1e3ae 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/IDepthDeviceCalibrationInfo.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/IDepthDeviceCalibrationInfo.cs @@ -11,20 +11,30 @@ namespace Microsoft.Psi.Calibration public interface IDepthDeviceCalibrationInfo { /// - /// Gets the extrinsics defining the color camera's position with respect to the depth camera. + /// Gets the extrinsics associated with the color camera, which describes how to transform points in world coordinates to color camera coordinates (world => camera). /// CoordinateSystem ColorExtrinsics { get; } + /// + /// Gets the pose of the color camera in the world, which is obtained by inverting the extrinsics matrix (camera => world). + /// + CoordinateSystem ColorPose { get; } + /// /// Gets the intrinsics associated with the color camera. /// ICameraIntrinsics ColorIntrinsics { get; } /// - /// Gets the extrinsics defining the depth camera's position in the world. + /// Gets the extrinsics associated with the depth camera, which describes how to transform points in world coordinates to depth camera coordinates (world => camera). /// CoordinateSystem DepthExtrinsics { get; } + /// + /// Gets the pose of the depth camera in the world, which is obtained by inverting the extrinsics matrix (camera => world). + /// + CoordinateSystem DepthPose { get; } + /// /// Gets the intrinsics associated with the depth camera. /// diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/LevenbergMarquardt.cs b/Sources/Calibration/Microsoft.Psi.Calibration/LevenbergMarquardt.cs similarity index 55% rename from Sources/Kinect/Microsoft.Psi.Kinect.Windows/LevenbergMarquardt.cs rename to Sources/Calibration/Microsoft.Psi.Calibration/LevenbergMarquardt.cs index f8ef6a4fd..cda820a21 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/LevenbergMarquardt.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/LevenbergMarquardt.cs @@ -1,15 +1,15 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT license. -namespace Microsoft.Psi.Kinect +namespace Microsoft.Psi.Calibration { using System; - using System.IO; + using MathNet.Numerics.LinearAlgebra; /// /// Defines a class for performing Levenberg-Marquardt optimization. /// - internal class LevenbergMarquardt + public class LevenbergMarquardt { private int maximumIterations = 100; private double minimumReduction = 1.0e-5; @@ -18,9 +18,6 @@ internal class LevenbergMarquardt private double initialLambda = 1.0e-3; private Function function; private Jacobian jacobianFunction; - private States state = States.Running; - private double rmsError; - private System.Diagnostics.Stopwatch stopWatch = new System.Diagnostics.Stopwatch(); /// /// Initializes a new instance of the class. @@ -35,7 +32,7 @@ public LevenbergMarquardt(Function function) /// Initializes a new instance of the class. /// /// Cost function. - /// Jacobian. + /// Jacobian function. public LevenbergMarquardt(Function function, Jacobian jacobianFunction) { this.function = function; @@ -47,198 +44,116 @@ public LevenbergMarquardt(Function function, Jacobian jacobianFunction) /// /// Parameters. /// Matrix. - public delegate Matrix Function(Matrix parameters); + public delegate Vector Function(Vector parameters); /// /// J_ij, ith error from function, jth parameter. /// /// Parameters. /// Matrix. - public delegate Matrix Jacobian(Matrix parameters); + public delegate Matrix Jacobian(Vector parameters); /// /// States for optimization. /// public enum States { -#pragma warning disable SA1602 // Enumeration items must be documented + /// + /// Running. + /// Running, + + /// + /// Maximum iterations. + /// MaximumIterations, + + /// + /// Lambda too large. + /// LambdaTooLarge, + + /// + /// Reduction step too small. + /// ReductionStepTooSmall, -#pragma warning restore SA1602 // Enumeration items must be documented } /// /// Gets the RMS error. /// - public double RMSError - { - get { return this.rmsError; } - } + public double RMSError { get; private set; } /// /// Gets the optimization state. /// - public States State - { - get { return this.state; } - } - - /// - /// Performs unit test. - /// - public static void Test() - { - // generate x_i, y_i observations on test function - var random = new Random(); - - int n = 200; - - var matX = new Matrix(n, 1); - var matY = new Matrix(n, 1); - - double a = 100; - double b = 102; - for (int i = 0; i < n; i++) - { - double x = (random.NextDouble() / (Math.PI / 4.0)) - (Math.PI / 8.0); - double y = (a * Math.Cos(b * x)) + (b * Math.Sin(a * x)) + (random.NextDouble() * 0.1); - matX[i] = x; - matY[i] = y; - } - - Function f = (Matrix parameters) => - { - // return y_i - f(x_i, parameters) as column vector - var error = new Matrix(n, 1); - - double a2 = parameters[0]; - double b2 = parameters[1]; - - for (int i = 0; i < n; i++) - { - double y = (a2 * Math.Cos(b2 * matX[i])) + (b2 * Math.Sin(a2 * matX[i])); - error[i] = matY[i] - y; - } - - return error; - }; - - var levenbergMarquardt = new LevenbergMarquardt(f); - - var parameters0 = new Matrix(2, 1); - parameters0[0] = 90; - parameters0[1] = 96; - - var rmsError = levenbergMarquardt.Minimize(parameters0); - } + public States State { get; private set; } = States.Running; /// /// Minimizes function. /// /// Parameters. /// Returns the RMS. - public double Minimize(Matrix parameters) + public double Minimize(Vector parameters) { - this.state = States.Running; + this.State = States.Running; for (int iteration = 0; iteration < this.maximumIterations; iteration++) { this.MinimizeOneStep(parameters); - if (this.state != States.Running) + if (this.State != States.Running) { return this.RMSError; } } - this.state = States.MaximumIterations; + this.State = States.MaximumIterations; return this.RMSError; } - /// - /// Writes the specified matrix to the specified file. - /// - /// Matrix to write. - /// Name of output file. - public void WriteMatrixToFile(Matrix matA, string filename) - { - var file = new StreamWriter(filename); - for (int i = 0; i < matA.Rows; i++) - { - for (int j = 0; j < matA.Cols; j++) - { - file.Write(matA[i, j] + "\t"); - } - - file.WriteLine(); - } - - file.Close(); - } - /// /// Single step of the optimization. /// /// Parameters. /// Returns the error. - public double MinimizeOneStep(Matrix parameters) + public double MinimizeOneStep(Vector parameters) { // initial value of the function; callee knows the size of the returned vector var errorVector = this.function(parameters); - var error = errorVector.Dot(errorVector); + var error = errorVector.DotProduct(errorVector); // Jacobian; callee knows the size of the returned matrix var matJ = this.jacobianFunction(parameters); // J'*J - var matJtJ = new Matrix(parameters.Size, parameters.Size); - - // stopWatch.Restart(); - // JtJ.MultATA(J, J); // this is the big calculation that could be parallelized - matJtJ.MultATAParallel(matJ, matJ); - - // Console.WriteLine("JtJ: J size {0}x{1} {2}ms", J.Rows, J.Cols, stopWatch.ElapsedMilliseconds); + var matJtJ = matJ.TransposeThisAndMultiply(matJ); // J'*error - var matJtError = new Matrix(parameters.Size, 1); - - // stopWatch.Restart(); - matJtError.MultATA(matJ, errorVector); // error vector must be a column vector - - // Console.WriteLine("JtError: errorVector size {0}x{1} {2}ms", errorVector.Rows, errorVector.Cols, stopWatch.ElapsedMilliseconds); + var matJtError = matJ.TransposeThisAndMultiply(errorVector); // allocate some space - var matJtJaugmented = new Matrix(parameters.Size, parameters.Size); - var matJtJinv = new Matrix(parameters.Size, parameters.Size); - var matDelta = new Matrix(parameters.Size, 1); - var matNewParameters = new Matrix(parameters.Size, 1); + var matJtJaugmented = Matrix.Build.Dense(parameters.Count, parameters.Count); // find a value of lambda that reduces error double lambda = this.initialLambda; while (true) { // augment J'*J: J'*J += lambda*(diag(J)) - matJtJaugmented.Copy(matJtJ); - for (int i = 0; i < parameters.Size; i++) + matJtJ.CopyTo(matJtJaugmented); + for (int i = 0; i < parameters.Count; i++) { matJtJaugmented[i, i] = (1.0 + lambda) * matJtJ[i, i]; } - // WriteMatrixToFile(errorVector, "errorVector"); - // WriteMatrixToFile(J, "J"); - // WriteMatrixToFile(JtJaugmented, "JtJaugmented"); - // WriteMatrixToFile(JtError, "JtError"); - // solve for delta: (J'*J + lambda*(diag(J)))*delta = J'*error - matJtJinv.Inverse(matJtJaugmented); - matDelta.Mult(matJtJinv, matJtError); + var matJtJinv = matJtJaugmented.Inverse(); + var matDelta = matJtJinv * matJtError; // new parameters = parameters - delta [why not add?] - matNewParameters.Sub(parameters, matDelta); + var matNewParameters = parameters - matDelta; // evaluate function, compute error var newErrorVector = this.function(matNewParameters); - double newError = newErrorVector.Dot(newErrorVector); + double newError = newErrorVector.DotProduct(newErrorVector); // if error is reduced, divide lambda by 10 bool improvement; @@ -255,20 +170,19 @@ public double MinimizeOneStep(Matrix parameters) // termination criteria: // reduction in error is too small - var diff = new Matrix(errorVector.Size, 1); - diff.Sub(errorVector, newErrorVector); - double diffSq = diff.Dot(diff); + var diff = errorVector - newErrorVector; + double diffSq = diff.DotProduct(diff); double errorDelta = Math.Sqrt(diffSq / error); if (errorDelta < this.minimumReduction) { - this.state = States.ReductionStepTooSmall; + this.State = States.ReductionStepTooSmall; } // lambda is too big if (lambda > this.maximumLambda) { - this.state = States.LambdaTooLarge; + this.State = States.LambdaTooLarge; } // change in parameters is too small [not implemented] @@ -276,20 +190,20 @@ public double MinimizeOneStep(Matrix parameters) // if we made an improvement, accept the new parameters if (improvement) { - parameters.Copy(matNewParameters); + matNewParameters.CopyTo(parameters); error = newError; break; } // if we meet termination criteria, break - if (this.state != States.Running) + if (this.State != States.Running) { break; } } - this.rmsError = Math.Sqrt(error / errorVector.Size); - return this.rmsError; + this.RMSError = Math.Sqrt(error / errorVector.Count); + return this.RMSError; } /// @@ -314,17 +228,17 @@ public NumericalDifferentiation(Function function) /// /// Parameters. /// Returns Jacobian. - public Matrix Jacobian(Matrix parameters) + public Matrix Jacobian(Vector parameters) { const double deltaFactor = 1.0e-6; const double minDelta = 1.0e-6; // evaluate the function at the current solution var errorVector0 = this.function(parameters); - var matJ = new Matrix(errorVector0.Size, parameters.Size); + var matJ = Matrix.Build.Dense(errorVector0.Count, parameters.Count); // vary each paremeter - for (int j = 0; j < parameters.Size; j++) + for (int j = 0; j < parameters.Count; j++) { double parameterValue = parameters[j]; // save the original value @@ -338,9 +252,9 @@ public Matrix Jacobian(Matrix parameters) // we only get error from function, but error(p + d) - error(p) = f(p + d) - f(p) var errorVector = this.function(parameters); - errorVector.Sub(errorVector0); + errorVector -= errorVector0; - for (int i = 0; i < errorVector0.Rows; i++) + for (int i = 0; i < errorVector0.Count; i++) { matJ[i, j] = errorVector[i] / delta; } diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/Microsoft.Psi.Calibration.csproj b/Sources/Calibration/Microsoft.Psi.Calibration/Microsoft.Psi.Calibration.csproj index 830d89597..1dc01080a 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/Microsoft.Psi.Calibration.csproj +++ b/Sources/Calibration/Microsoft.Psi.Calibration/Microsoft.Psi.Calibration.csproj @@ -25,12 +25,19 @@ + + all + runtime; build; native; contentfiles; analyzers + - - + + + + + \ No newline at end of file diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/MultiCameraCalibration.cs b/Sources/Calibration/Microsoft.Psi.Calibration/MultiCameraCalibration.cs index bc24025dd..65bb053b0 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/MultiCameraCalibration.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/MultiCameraCalibration.cs @@ -132,14 +132,11 @@ public class CameraCalibrationResult /// Values are stored in column-major order and assumes column-vectors /// (i.e. Matrix * Point versus Point * Matrix). /// Units are millimeters. - /// OpenCV basis is asssumed here (Forward=Z, Right=X, Down=Y): - /// Z (forward) - /// / - /// / - /// +----> X (right) - /// | - /// | - /// Y (down). + /// MathNet basis is asssumed here + /// Z ^ X + /// | / + /// |/ + /// Y ----+. /// [XmlArray] public double[] Extrinsics { get; set; } @@ -241,13 +238,7 @@ public MathNet.Spatial.Euclidean.CoordinateSystem Pose mtx.SetColumn(3, mtx.Column(3) / 1000.0); mtx[3, 3] = 1; - // Extrinsics are stored in OpenCV basis, so convert here to MathNet basis. - var openCVBasis = new MathNet.Spatial.Euclidean.CoordinateSystem( - default, - MathNet.Spatial.Euclidean.UnitVector3D.ZAxis, - MathNet.Spatial.Euclidean.UnitVector3D.XAxis.Negate(), - MathNet.Spatial.Euclidean.UnitVector3D.YAxis.Negate()); - return new MathNet.Spatial.Euclidean.CoordinateSystem(openCVBasis.Invert() * mtx.Inverse() * openCVBasis); + return new MathNet.Spatial.Euclidean.CoordinateSystem(mtx.Inverse()); } } } diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/ProjectTo3D.cs b/Sources/Calibration/Microsoft.Psi.Calibration/ProjectTo3D.cs index 6e355e545..4f4569e77 100644 --- a/Sources/Calibration/Microsoft.Psi.Calibration/ProjectTo3D.cs +++ b/Sources/Calibration/Microsoft.Psi.Calibration/ProjectTo3D.cs @@ -15,7 +15,7 @@ namespace Microsoft.Psi.Calibration /// Inputs are the depth image, list of 2D points from the color image, and the camera calibration. /// Outputs the 3D points projected into the depth camera's coordinate system. /// - public sealed class ProjectTo3D : ConsumerProducer<(Shared, List, IDepthDeviceCalibrationInfo), List> + public sealed class ProjectTo3D : ConsumerProducer<(Shared, List, IDepthDeviceCalibrationInfo), List> { /// /// Initializes a new instance of the class. @@ -27,7 +27,7 @@ public ProjectTo3D(Pipeline pipeline) } /// - protected override void Receive((Shared, List, IDepthDeviceCalibrationInfo) data, Envelope e) + protected override void Receive((Shared, List, IDepthDeviceCalibrationInfo) data, Envelope e) { var point2DList = data.Item2; var depthImage = data.Item1; diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/RotationExtensions.cs b/Sources/Calibration/Microsoft.Psi.Calibration/RotationExtensions.cs new file mode 100644 index 000000000..c2f326e93 --- /dev/null +++ b/Sources/Calibration/Microsoft.Psi.Calibration/RotationExtensions.cs @@ -0,0 +1,165 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Calibration +{ + using System; + using MathNet.Numerics.LinearAlgebra; + + /// + /// Define set of extensions for dealing with rotation matrices and vectors. + /// + public static class RotationExtensions + { + /// + /// Use the Rodrigues formula for transforming a given rotation from axis-angle representation to a 3x3 matrix. + /// Where 'r' is a rotation vector: + /// theta = norm(r) + /// M = skew(r/theta) + /// R = I + M * sin(theta) + M*M * (1-cos(theta)). + /// + /// Rotation in axis-angle vector representation, + /// where the angle is represented by the length (L2-norm) of the vector. + /// Rotation in a 3x3 matrix representation. + public static Matrix AxisAngleToMatrix(Vector vectorRotation) + { + if (vectorRotation.Count != 3) + { + throw new InvalidOperationException("The input must be a valid 3-element vector representing an axis-angle rotation."); + } + + double theta = vectorRotation.L2Norm(); + + var matR = Matrix.Build.DenseIdentity(3, 3); + + // if there is no rotation (theta == 0) return identity rotation + if (theta == 0) + { + return matR; + } + + // Create a skew-symmetric matrix from the normalized axis vector + var rn = vectorRotation.Normalize(2); + var matM = Matrix.Build.Dense(3, 3); + matM[0, 0] = 0; + matM[0, 1] = -rn[2]; + matM[0, 2] = rn[1]; + matM[1, 0] = rn[2]; + matM[1, 1] = 0; + matM[1, 2] = -rn[0]; + matM[2, 0] = -rn[1]; + matM[2, 1] = rn[0]; + matM[2, 2] = 0; + + // I + M * sin(theta) + M*M * (1 - cos(theta)) + var sinThetaM = matM * Math.Sin(theta); + matR += sinThetaM; + var matMM = matM * matM; + var cosThetaMM = matMM * (1 - Math.Cos(theta)); + matR += cosThetaMM; + + return matR; + } + + /// + /// Convert a rotation matrix to axis-angle representation (a unit vector scaled by the angular distance to rotate). + /// + /// Input rotation matrix. + /// Same rotation in axis-angle representation (L2-Norm of the vector represents angular distance). + public static Vector MatrixToAxisAngle(Matrix m) + { + if (m.RowCount != 3 || m.ColumnCount != 3) + { + throw new InvalidOperationException("The input must be a valid 3x3 rotation matrix in order to compute its axis-angle representation."); + } + + double epsilon = 0.01; + + // theta = arccos((Trace(m) - 1) / 2) + double angle = Math.Acos((m.Trace() - 1.0) / 2.0); + + // Create the axis vector. + var v = Vector.Build.Dense(3, 0); + + if (angle < epsilon) + { + // If the angular distance to rotate is 0, we just return a vector of all zeroes. + return v; + } + + // Otherwise, the axis of rotation is extracted from the matrix as follows. + v[0] = m[2, 1] - m[1, 2]; + v[1] = m[0, 2] - m[2, 0]; + v[2] = m[1, 0] - m[0, 1]; + + if (v.L2Norm() < epsilon) + { + // if the axis to rotate around has 0 length, we are in a singularity where the angle has to be 180 degrees. + angle = Math.PI; + + // We can extract the axis of rotation, knowing that v*vT = (m + I) / 2; + // First compute r = (m + I) / 2 + var r = Matrix.Build.Dense(3, 3); + m.CopyTo(r); + r[0, 0] += 1; + r[1, 1] += 1; + r[2, 2] += 1; + r /= 2.0; + + // r = v*vT = + // | v_x*v_x, v_x*v_y, v_x*v_z | + // | v_x*v_y, v_y*v_y, v_y*v_z | + // | v_x*v_z, v_y*v_z, v_z*v_z | + // Extract x, y, and z as the square roots of the diagonal elements. + var x = Math.Sqrt(r[0, 0]); + var y = Math.Sqrt(r[1, 1]); + var z = Math.Sqrt(r[2, 2]); + + // Now we need to determine the signs of x, y, and z. + double xsign; + double ysign; + double zsign; + + double xy = r[0, 1]; + double xz = r[0, 2]; + + if (xy > 0) + { + if (xz > 0) + { + xsign = 1; + ysign = 1; + zsign = 1; + } + else + { + xsign = 1; + ysign = 1; + zsign = -1; + } + } + else + { + if (xz > 0) + { + xsign = 1; + ysign = -1; + zsign = 1; + } + else + { + xsign = 1; + ysign = -1; + zsign = -1; + } + } + + v[0] = x * xsign; + v[1] = y * ysign; + v[2] = z * zsign; + } + + return v.Normalize(2) * angle; + } + } +} diff --git a/Sources/Calibration/Microsoft.Psi.Calibration/ThirdPartyNotices.txt b/Sources/Calibration/Microsoft.Psi.Calibration/ThirdPartyNotices.txt new file mode 100644 index 000000000..b19d6eb9e --- /dev/null +++ b/Sources/Calibration/Microsoft.Psi.Calibration/ThirdPartyNotices.txt @@ -0,0 +1,38 @@ +Microsoft Platform for Situated Intelligence + +THIRD-PARTY SOFTWARE NOTICES AND INFORMATION +Do Not Translate or Localize + +This project is based on or incorporates material from the projects listed below (Third Party IP). The original copyright notice and the license under which Microsoft received such Third Party IP, are set forth below. Such licenses and notices are provided for informational purposes only. Microsoft licenses the Third Party IP to you under the licensing terms for the Microsoft Platform for Situated Intelligence product. Microsoft reserves all other rights not expressly granted under this agreement, whether by implication, estoppel or otherwise. + +1. RoomAlive Toolkit (https://github.com/Microsoft/RoomAliveToolkit) + +%% RoomAlive Toolkit NOTICES AND INFORMATION BEGIN HERE +========================================= +RoomAlive Toolkit + +Copyright (c) Microsoft Corporation + +All rights reserved. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +========================================= +END OF RoomAlive Toolkit NOTICES AND INFORMATION diff --git a/Sources/Calibration/Test.Psi.Calibration/DistortionTests.cs b/Sources/Calibration/Test.Psi.Calibration/DistortionTests.cs index 2e3d328c3..4bb3513c1 100644 --- a/Sources/Calibration/Test.Psi.Calibration/DistortionTests.cs +++ b/Sources/Calibration/Test.Psi.Calibration/DistortionTests.cs @@ -5,9 +5,9 @@ namespace Test.Psi.Calibration { using System; using System.IO; - using System.Net.Http.Headers; using System.Runtime.InteropServices; using MathNet.Numerics.LinearAlgebra; + using MathNet.Spatial.Euclidean; using Microsoft.Psi.Calibration; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -58,7 +58,11 @@ public static void SaveColorToBMP(string filePath, Microsoft.Psi.Imaging.Image r public void TestDistortion() { // Create a checkerboard image - var img = Microsoft.Psi.Imaging.Image.Create(1024, 1024, Microsoft.Psi.Imaging.PixelFormat.BGR_24bpp); + bool useColor = false; + bool reverseDirection = true; + int width = useColor ? 1280 : 640; + int height = useColor ? 720 : 576; + var img = new Microsoft.Psi.Imaging.Image(width, height, Microsoft.Psi.Imaging.PixelFormat.BGR_24bpp); unsafe { byte* row = (byte*)img.ImageData.ToPointer(); @@ -69,15 +73,15 @@ public void TestDistortion() { if ((i / 20 + j / 20) % 2 == 0) { - col[0] = 255; - col[1] = 0; + col[0] = (byte)(255.0f * (float)j / (float)img.Width); + col[1] = (byte)(255.0f * (1.0f - (float)j / (float)img.Width)); col[2] = 0; } else { col[0] = 0; - col[1] = 0; - col[2] = 255; + col[1] = (byte)(255.0f * (float)i / (float)img.Height); + col[2] = (byte)(255.0f * (1.0f - (float)i / (float)img.Height)); } col += img.BitsPerPixel / 8; @@ -90,19 +94,67 @@ public void TestDistortion() #endif // DUMP_IMAGES } + double[] colorAzureDistortionCoefficients = new double[6] + { + 0.609246314, + -2.84837151, + 1.63566089, + 0.483219713, + -2.66301942, + 1.55776918, + }; + double[] colorAzureTangentialCoefficients = new double[2] + { + -0.000216085638, + 0.000744335062, + }; + double[] colorIntrinsics = new double[4] + { + 638.904968, // cx + 350.822327, // cy + 607.090698, // fx + 607.030762, // fy + }; + double[] depthIntrinsics = new double[4] + { + 326.131775, // cx + 324.755524, // cy + 504.679749, // fx + 504.865875, // fy + }; + + double[] depthAzureDistortionCoefficients = new double[6] + { + 0.228193134, + -0.0650567561, + -0.000764187891, + 0.568694472, + -0.0599768497, + -0.0119919786, + }; + double[] depthAzureTangentialCoefficients = new double[2] + { + -9.04210319e-05, + -9.16166828e-05, + }; + // Setup our distortion coefficients - double[] distortionCoefficients = new double[6] { 1.10156359448570129, -0.049757665717193485, -0.0018714899575029596, 0.0, 0.0, 0.0 }; - double[] tangentialCoefficients = new double[2] { 0.0083588278483703853, 0.0 }; + var distortionCoefficients = useColor ? colorAzureDistortionCoefficients : depthAzureDistortionCoefficients; + var tangentialCoefficients = useColor ? colorAzureTangentialCoefficients : depthAzureTangentialCoefficients; // Next run distort on the image - var distortedImage = Microsoft.Psi.Imaging.Image.Create(img.Width, img.Height, img.PixelFormat); - var intrinsicMat = CreateMatrix.Dense(3, 3, new double[9] { 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0 }); + var distortedImage = new Microsoft.Psi.Imaging.Image(img.Width, img.Height, img.PixelFormat); + double[] colorArray = new double[9] { colorIntrinsics[2], 0.0, 0.0, 0.0, colorIntrinsics[3], 0.0, colorIntrinsics[0], colorIntrinsics[1], 1.0, }; + double[] depthArray = new double[9] { depthIntrinsics[2], 0.0, 0.0, 0.0, depthIntrinsics[3], 0.0, depthIntrinsics[0], depthIntrinsics[1], 1.0, }; + var intrinsicMat = CreateMatrix.Dense(3, 3, useColor ? colorArray : depthArray); var ci = new CameraIntrinsics( img.Width, img.Height, intrinsicMat, Vector.Build.DenseOfArray(distortionCoefficients), - Vector.Build.DenseOfArray(tangentialCoefficients)); + Vector.Build.DenseOfArray(tangentialCoefficients), + reverseDirection); + unsafe { byte* dstrow = (byte*)distortedImage.ImageData.ToPointer(); @@ -111,15 +163,18 @@ public void TestDistortion() byte* dstcol = dstrow; for (int j = 0; j < distortedImage.Width; j++) { - MathNet.Spatial.Euclidean.Point2D pixelCoord = new MathNet.Spatial.Euclidean.Point2D((i - 512.0) / 1024.0, (j - 512.0) / 1024.0); - MathNet.Spatial.Euclidean.Point2D distortedPixelCoord; - ci.DistortPoint(pixelCoord, out distortedPixelCoord); + MathNet.Spatial.Euclidean.Point2D pixelCoord = new MathNet.Spatial.Euclidean.Point2D( + ((float)j - ci.PrincipalPoint.X) / ci.FocalLengthXY.X, + ((float)i - ci.PrincipalPoint.Y) / ci.FocalLengthXY.Y); + + Point2D undistortedPoint; + bool converged = ci.UndistortPoint(pixelCoord, out undistortedPoint); - int px = (int)(distortedPixelCoord.X * 1024.0 + 512.0); - int py = (int)(distortedPixelCoord.Y * 1024.0 + 512.0); - if (px >= 0 && px < img.Width && py >= 0 && py < img.Height) + int px = (int)(undistortedPoint.X * ci.FocalLengthXY.X + ci.PrincipalPoint.X); + int py = (int)(undistortedPoint.Y * ci.FocalLengthXY.Y + ci.PrincipalPoint.Y); + if (converged && px >= 0 && px < img.Width && py >= 0 && py < img.Height) { - byte* src = (byte*)img.ImageData.ToPointer() + py * distortedImage.Stride + px * distortedImage.BitsPerPixel / 8; + byte* src = (byte*)img.ImageData.ToPointer() + py * img.Stride + px * img.BitsPerPixel / 8; dstcol[0] = src[0]; dstcol[1] = src[1]; dstcol[2] = src[2]; @@ -136,7 +191,7 @@ public void TestDistortion() } // Finally run undistort on the result - var undistortedImage = Microsoft.Psi.Imaging.Image.Create(img.Width, img.Height, img.PixelFormat); + var undistortedImage = new Microsoft.Psi.Imaging.Image(img.Width, img.Height, img.PixelFormat); unsafe { double err = 0.0; @@ -147,14 +202,16 @@ public void TestDistortion() byte* dstcol = dstrow; for (int j = 0; j < undistortedImage.Width; j++) { - MathNet.Spatial.Euclidean.Point2D pixelCoord = new MathNet.Spatial.Euclidean.Point2D((i - 512.0) / 1024.0, (j - 512.0) / 1024.0); - MathNet.Spatial.Euclidean.Point2D distortedPixelCoord; + MathNet.Spatial.Euclidean.Point2D pixelCoord = new MathNet.Spatial.Euclidean.Point2D( + ((float)j - ci.PrincipalPoint.X) / ci.FocalLengthXY.X, + ((float)i - ci.PrincipalPoint.Y) / ci.FocalLengthXY.Y); + MathNet.Spatial.Euclidean.Point2D distortedPixelCoord, undistortedPixelCoord; ci.DistortPoint(pixelCoord, out distortedPixelCoord); - var undistortedPixelCoord = ci.UndistortPoint(distortedPixelCoord); + bool converged = ci.UndistortPoint(distortedPixelCoord, out undistortedPixelCoord); - int px = (int)(undistortedPixelCoord.X * 1024.0 + 512.0); - int py = (int)(undistortedPixelCoord.Y * 1024.0 + 512.0); - if (px >= 0 && px < img.Width && py >= 0 && py < img.Height) + int px = (int)(undistortedPixelCoord.X * ci.FocalLengthXY.X + ci.PrincipalPoint.X); + int py = (int)(undistortedPixelCoord.Y * ci.FocalLengthXY.Y + ci.PrincipalPoint.Y); + if (converged && px >= 0 && px < img.Width && py >= 0 && py < img.Height) { byte* src = (byte*)img.ImageData.ToPointer() + py * img.Stride + px * img.BitsPerPixel / 8; dstcol[0] = src[0]; diff --git a/Sources/Calibration/Test.Psi.Calibration/LevenbergMarquardtTests.cs b/Sources/Calibration/Test.Psi.Calibration/LevenbergMarquardtTests.cs new file mode 100644 index 000000000..7bec416bd --- /dev/null +++ b/Sources/Calibration/Test.Psi.Calibration/LevenbergMarquardtTests.cs @@ -0,0 +1,65 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Test.Psi.Calibration +{ + using System; + using MathNet.Numerics.LinearAlgebra; + using Microsoft.Psi.Calibration; + using Microsoft.VisualStudio.TestTools.UnitTesting; + + /// + /// Distortion tests. + /// + [TestClass] + public class LevenbergMarquardtTests + { + [TestMethod] + [Timeout(60000)] + public void TestOptimization() + { + // generate x_i, y_i observations on test function + var random = new Random(); + + int n = 200; + + var matX = Vector.Build.Dense(n); + var matY = Vector.Build.Dense(n); + + double a = 100; + double b = 102; + for (int i = 0; i < n; i++) + { + double x = (random.NextDouble() / (Math.PI / 4.0)) - (Math.PI / 8.0); + double y = (a * Math.Cos(b * x)) + (b * Math.Sin(a * x)) + (random.NextDouble() * 0.1); + matX[i] = x; + matY[i] = y; + } + + LevenbergMarquardt.Function f = (Vector parameters) => + { + // return y_i - f(x_i, parameters) as column vector + var error = Vector.Build.Dense(n); + + double a2 = parameters[0]; + double b2 = parameters[1]; + + for (int i = 0; i < n; i++) + { + double y = (a2 * Math.Cos(b2 * matX[i])) + (b2 * Math.Sin(a2 * matX[i])); + error[i] = matY[i] - y; + } + + return error; + }; + + var levenbergMarquardt = new LevenbergMarquardt(f); + + var parameters0 = Vector.Build.Dense(2); + parameters0[0] = 90; + parameters0[1] = 96; + + var rmsError = levenbergMarquardt.Minimize(parameters0); + } + } +} diff --git a/Sources/Calibration/Test.Psi.Calibration/Test.Psi.Calibration.csproj b/Sources/Calibration/Test.Psi.Calibration/Test.Psi.Calibration.csproj index c964a9760..7af970a68 100644 --- a/Sources/Calibration/Test.Psi.Calibration/Test.Psi.Calibration.csproj +++ b/Sources/Calibration/Test.Psi.Calibration/Test.Psi.Calibration.csproj @@ -2,7 +2,7 @@ Exe - netcoreapp2.0 + netcoreapp3.1 false Calibration unit tests Test.Psi.Calibration.ConsoleMain @@ -30,10 +30,10 @@ - + - - + + diff --git a/Sources/Common/Test.Psi.Common/Test.Psi.Common.csproj b/Sources/Common/Test.Psi.Common/Test.Psi.Common.csproj index b74493516..e33166bcf 100644 --- a/Sources/Common/Test.Psi.Common/Test.Psi.Common.csproj +++ b/Sources/Common/Test.Psi.Common/Test.Psi.Common.csproj @@ -20,8 +20,6 @@ true true - false - Microsoft Corporation @@ -31,8 +29,12 @@ + + all + runtime; build; native; contentfiles; analyzers + - + diff --git a/Sources/Common/Test.Psi.Common/TestRunner.cs b/Sources/Common/Test.Psi.Common/TestRunner.cs index a5df3d2a8..2d3a1ce53 100644 --- a/Sources/Common/Test.Psi.Common/TestRunner.cs +++ b/Sources/Common/Test.Psi.Common/TestRunner.cs @@ -195,7 +195,7 @@ public static void RunAll(string nameSubstring = "") } /// - /// Due to the runtime's asynchronous behaviour, we may try to + /// Due to the runtime's asynchronous behavior, we may try to /// delete our test directory before the runtime has finished /// messing with it. This method will keep trying to delete /// the directory until the runtime shuts down. diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotatedEvent.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotatedEvent.cs index 099111571..03ad512bf 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotatedEvent.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotatedEvent.cs @@ -21,10 +21,10 @@ public class AnnotatedEvent /// The end time of the annotated event. public AnnotatedEvent(DateTime startTime, DateTime endTime) { + this.CheckArguments(startTime, endTime); this.StartTime = startTime; this.EndTime = endTime; this.InternalAnnotations = new List(); - this.InitNew(); } /// @@ -85,14 +85,11 @@ public void SetAnnotation(int index, string value, AnnotationSchema schema) } /// - /// Overridable method to allow derived object to initialize properties as part of object construction or after deserialization. + /// Overridable method to allow derived object to initialize properties after deserialization. /// protected virtual void InitNew() { - if (this.EndTime < this.StartTime) - { - throw new ArgumentException("startTime must preceed endTime.", "startTime"); - } + this.CheckArguments(this.StartTime, this.EndTime); } [OnDeserialized] @@ -100,5 +97,13 @@ private void OnDeserialized(StreamingContext context) { this.InitNew(); } + + private void CheckArguments(DateTime startTime, DateTime endTime) + { + if (endTime < startTime) + { + throw new ArgumentException("startTime must preceed endTime.", "startTime"); + } + } } } diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationPartition.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationPartition.cs index 41780d36c..9fe5db3a3 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationPartition.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationPartition.cs @@ -16,6 +16,7 @@ public class AnnotationPartition : Partition private AnnotationPartition(Session session, string storeName, string storePath, string name) : base(session, storeName, storePath, name, typeof(AnnotationSimpleReader)) { + this.Reader = new AnnotationSimpleReader(storeName, storePath); } private AnnotationPartition() @@ -36,7 +37,7 @@ public static AnnotationPartition Create(Session session, string storeName, stri using (var writer = new AnnotationSimpleWriter(definition)) { writer.CreateStore(storeName, storePath); - writer.CreateStream(new JsonStreamMetadata(definition.Name, 0, typeof(AnnotatedEvent).AssemblyQualifiedName, storeName, storePath), new List>()); + writer.CreateStream(new JsonStreamMetadata(definition.Name, 0, typeof(AnnotatedEvent).AssemblyQualifiedName, null, storeName, storePath), new List>()); writer.WriteAll(ReplayDescriptor.ReplayAll); } @@ -57,7 +58,7 @@ public static AnnotationPartition CreateFromExistingStore(Session session, strin } /// - /// Overridable method to allow derived object to initialize properties as part of object construction or after deserialization. + /// Overridable method to allow derived object to initialize properties after deserialization. /// protected override void InitNew() { diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchema.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchema.cs index 9fc58d08b..5737d6ada 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchema.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchema.cs @@ -95,7 +95,7 @@ public bool IsValidValue(string value) /// /// Removes an annotation schema value from the current annotation schema. /// - /// The annotation schema value to remove from teh current annotation schema. + /// The annotation schema value to remove from the current annotation schema. public void RemoveSchemaValue(AnnotationSchemaValue schemaValue) { this.InternalValues.Remove(schemaValue); diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchemaRegistry.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchemaRegistry.cs index c83e6eb78..21174d1b6 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchemaRegistry.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSchemaRegistry.cs @@ -8,7 +8,7 @@ namespace Microsoft.Psi.Data.Annotations using System.Runtime.Serialization; /// - /// Provides a singleton resistry for annotation schemas. + /// Provides a singleton registry for annotation schemas. /// [DataContract(Namespace = "http://www.microsoft.com/psi")] public class AnnotationSchemaRegistry @@ -22,7 +22,6 @@ public class AnnotationSchemaRegistry private AnnotationSchemaRegistry() { this.InternalSchemas = new List(); - this.InitNew(); } /// @@ -53,7 +52,7 @@ public void Unregister(AnnotationSchema schema) } /// - /// Overridable method to allow derived object to initialize properties as part of object construction or after deserialization. + /// Overridable method to allow derived object to initialize properties after deserialization. /// protected virtual void InitNew() { diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleReader.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleReader.cs index cacad2419..8b480f2d0 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleReader.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleReader.cs @@ -19,8 +19,9 @@ public class AnnotationSimpleReader : JsonSimpleReader /// The name of the application that generated the persisted files, or the root name of the files. /// The directory in which the main persisted file resides or will reside, or null to create a volatile data store. public AnnotationSimpleReader(string name, string path) - : base(name, path, AnnotationStoreCommon.DefaultExtension) + : this() { + this.Reader = new AnnotationStoreReader(name, path); } /// @@ -36,7 +37,7 @@ public AnnotationSimpleReader() /// /// Existing used to initialize new instance. public AnnotationSimpleReader(AnnotationSimpleReader that) - : base(that) + : this(that.Name, that.Path) { } diff --git a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleWriter.cs b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleWriter.cs index f0b34882f..f9bd0bd26 100644 --- a/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleWriter.cs +++ b/Sources/Data/Microsoft.Psi.Data/Annotations/AnnotationSimpleWriter.cs @@ -22,9 +22,9 @@ public class AnnotationSimpleWriter : JsonSimpleWriter /// The annotated event definition used to create and validate annotated events for this store. /// If true, a numbered subdirectory is created for this store. public AnnotationSimpleWriter(string name, string path, AnnotatedEventDefinition definition, bool createSubdirectory = true) - : base(name, path, createSubdirectory, AnnotationStoreCommon.DefaultExtension) + : this(definition) { - this.definition = definition; + this.Writer = new AnnotationStoreWriter(name, path, definition, createSubdirectory); } /// diff --git a/Sources/Data/Microsoft.Psi.Data/Dataset.cs b/Sources/Data/Microsoft.Psi.Data/Dataset.cs index 9a4279799..0e4920e94 100644 --- a/Sources/Data/Microsoft.Psi.Data/Dataset.cs +++ b/Sources/Data/Microsoft.Psi.Data/Dataset.cs @@ -43,7 +43,7 @@ public Dataset(string name = Dataset.DefaultName) public string Name { get; set; } /// - /// Gets the orginating time interval (earliest to latest) of the messages in this dataset. + /// Gets the originating time interval (earliest to latest) of the messages in this dataset. /// [IgnoreDataMember] public TimeInterval OriginatingTimeInterval => @@ -87,7 +87,7 @@ public static Dataset Load(string filename) } /// - /// Creates a new dataset from an exising data store. + /// Creates a new dataset from an existing data store. /// /// The name of the data store. /// The path of the data store. @@ -227,7 +227,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// The name of the output data store. Default is null. /// The path of the output data store. Default is null. /// The replay descriptor to us. - /// A token for cancelling the asynchronous task. + /// A token for canceling the asynchronous task. /// A task that represents the asynchronous operation. public async Task CreateDerivedPartitionAsync( Action computeDerived, @@ -259,7 +259,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// A function to determine output path from the given Session. /// The replay descriptor to us. /// An object that can be used for reporting progress. - /// A token for cancelling the asynchronous task. + /// A token for canceling the asynchronous task. /// A task that represents the asynchronous operation. public async Task CreateDerivedPartitionAsync( Action computeDerived, @@ -286,7 +286,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// /// Asynchronously computes a derived partition for each session in the dataset. /// - /// The type of paramater passed to the action. + /// The type of parameter passed to the action. /// The action to be invoked to derive partitions. /// The parameter to be passed to the action. /// The output partition name to be created. @@ -295,7 +295,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// A function to determine output path from the given Session. /// The replay descriptor to us. /// An object that can be used for reporting progress. - /// A token for cancelling the asynchronous task. + /// A token for canceling the asynchronous task. /// A task that represents the asynchronous operation. public async Task CreateDerivedPartitionAsync( Action computeDerived, @@ -328,7 +328,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// /// Asynchronously computes a derived partition for each session in the dataset. /// - /// The type of paramater passed to the action. + /// The type of parameter passed to the action. /// The action to be invoked to derive partitions. /// The parameter to be passed to the action. /// The output partition name to be created. @@ -336,7 +336,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// The name of the output data store. Default is null. /// The path of the output data store. Default is null. /// The replay descriptor to us. - /// A token for cancelling the asynchronous task. + /// A token for canceling the asynchronous task. /// A task that represents the asynchronous operation. public async Task CreateDerivedPartitionAsync( Action computeDerived, @@ -372,7 +372,7 @@ public Session AddSessionFromExistingStore(string sessionName, string storeName, /// Adds sessions from data stores located in the specified path. /// /// The path that contains the data stores. - /// The name of the partion to be added when adding a new session. Default is null. + /// The name of the partition to be added when adding a new session. Default is null. public void AddSessionsFromExistingStores(string path, string partitionName = null) { this.AddSessionsFromExistingStores(path, path, partitionName); diff --git a/Sources/Data/Microsoft.Psi.Data/IPartition.cs b/Sources/Data/Microsoft.Psi.Data/IPartition.cs index 1e7953b4e..780e3bfbd 100644 --- a/Sources/Data/Microsoft.Psi.Data/IPartition.cs +++ b/Sources/Data/Microsoft.Psi.Data/IPartition.cs @@ -16,7 +16,7 @@ public interface IPartition string Name { get; set; } /// - /// Gets the orginating time interval (earliest to latest) of the messages in this partition. + /// Gets the originating time interval (earliest to latest) of the messages in this partition. /// TimeInterval OriginatingTimeInterval { get; } diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonExporter.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonExporter.cs index 74c7fbdbb..fc6a43742 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonExporter.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonExporter.cs @@ -69,6 +69,8 @@ public override void Dispose() { this.writer.Dispose(); } + + this.throttle.Dispose(); } /// diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonGenerator.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonGenerator.cs index 32ae5cc6a..7e45ae5b1 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonGenerator.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonGenerator.cs @@ -61,7 +61,7 @@ protected JsonGenerator(Pipeline pipeline, JsonStoreReader reader) public IEnumerable AvailableStreams => this.reader.AvailableStreams; /// - /// Gets the orginating time interval (earliest to latest) of the messages in the underlying data store. + /// Gets the originating time interval (earliest to latest) of the messages in the underlying data store. /// public TimeInterval OriginatingTimeInterval => this.reader.OriginatingTimeInterval; @@ -90,7 +90,7 @@ public void Dispose() /// /// Type of data in underlying stream. /// The name of the stream. - /// The newly created emmitte that generates messages from the stream of type . + /// The newly created emitter that generates messages from the stream of type . public Emitter OpenStream(string streamName) { // if stream already opened, return emitter diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleReader.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleReader.cs index 805d76897..9c6cb0eb3 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleReader.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleReader.cs @@ -27,7 +27,7 @@ public class JsonSimpleReader : ISimpleReader, IDisposable public JsonSimpleReader(string name, string path, string extension = JsonStoreBase.DefaultExtension) : this(extension) { - this.OpenStore(name, path); + this.Reader = new JsonStoreReader(name, path, extension); } /// diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleWriter.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleWriter.cs index b6cf6135a..b39a387a5 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleWriter.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonSimpleWriter.cs @@ -28,7 +28,7 @@ public class JsonSimpleWriter : ISimpleWriter, IDisposable public JsonSimpleWriter(string name, string path, bool createSubdirectory = true, string extension = JsonStoreBase.DefaultExtension) : this(extension) { - this.CreateStore(name, path, createSubdirectory); + this.Writer = new JsonStoreWriter(name, path, createSubdirectory, extension); } /// diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreReader.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreReader.cs index 48214738a..d6c0249a5 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreReader.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreReader.cs @@ -62,7 +62,7 @@ public JsonStoreReader(string name, string path, string extension = DefaultExten public IEnumerable AvailableStreams => this.catalog; /// - /// Gets the orginating time interval (earliest to latest) of the messages in the data store. + /// Gets the originating time interval (earliest to latest) of the messages in the data store. /// public TimeInterval OriginatingTimeInterval => this.originatingTimeInterval; @@ -113,6 +113,7 @@ public override void Dispose() { this.streamReader?.Dispose(); this.streamReader = null; + this.jsonReader?.Close(); this.jsonReader = null; } diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreWriter.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreWriter.cs index ca2f89dd3..26e444699 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreWriter.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonStoreWriter.cs @@ -67,6 +67,7 @@ public override void Dispose() this.jsonWriter.WriteEndArray(); this.streamWriter.Dispose(); this.streamWriter = null; + this.jsonWriter.Close(); this.jsonWriter = null; } diff --git a/Sources/Data/Microsoft.Psi.Data/Json/JsonStreamMetadata.cs b/Sources/Data/Microsoft.Psi.Data/Json/JsonStreamMetadata.cs index d6177b653..ac409eb09 100644 --- a/Sources/Data/Microsoft.Psi.Data/Json/JsonStreamMetadata.cs +++ b/Sources/Data/Microsoft.Psi.Data/Json/JsonStreamMetadata.cs @@ -18,7 +18,6 @@ public class JsonStreamMetadata : IStreamMetadata /// public JsonStreamMetadata() { - this.Reset(); } /// @@ -26,15 +25,17 @@ public JsonStreamMetadata() /// /// The name of the stream the metadata represents. /// The id of the stream the metadata represents. - /// The name of the type of data conatined in the stream the metadata represents. - /// The name of the partation where the stream is stored. - /// The path of the partation where the stream is stored. - public JsonStreamMetadata(string name, int id, string typeName, string partitionName, string partitionPath) + /// The name of the type of data contained in the stream the metadata represents. + /// The name of the type of supplemental metadata for the stream the metadata represents. + /// The name of the partition where the stream is stored. + /// The path of the partition where the stream is stored. + public JsonStreamMetadata(string name, int id, string typeName, string supplementalMetadataTypeName, string partitionName, string partitionPath) : this() { this.Name = name; this.Id = id; this.TypeName = typeName; + this.SupplementalMetadataTypeName = supplementalMetadataTypeName; this.PartitionName = partitionName; this.PartitionPath = partitionPath; } @@ -87,19 +88,9 @@ public JsonStreamMetadata(string name, int id, string typeName, string partition [JsonProperty(Order = 12)] public int MessageCount { get; set; } - /// - /// Reset statistics for this stream metadata instance. - /// - public virtual void Reset() - { - this.FirstMessageTime = default(DateTime); - this.LastMessageTime = default(DateTime); - this.FirstMessageOriginatingTime = default(DateTime); - this.LastMessageOriginatingTime = default(DateTime); - this.AverageMessageSize = 0; - this.AverageLatency = 0; - this.MessageCount = 0; - } + /// + [JsonProperty(Order = 13)] + public string SupplementalMetadataTypeName { get; set; } /// public void Update(Envelope envelope, int size) diff --git a/Sources/Data/Microsoft.Psi.Data/Microsoft.Psi.Data.csproj b/Sources/Data/Microsoft.Psi.Data/Microsoft.Psi.Data.csproj index 7f8d87952..987c0b3c4 100644 --- a/Sources/Data/Microsoft.Psi.Data/Microsoft.Psi.Data.csproj +++ b/Sources/Data/Microsoft.Psi.Data/Microsoft.Psi.Data.csproj @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Data/Microsoft.Psi.Data/Partition.cs b/Sources/Data/Microsoft.Psi.Data/Partition.cs index ea0f9dd0a..a68f5d931 100644 --- a/Sources/Data/Microsoft.Psi.Data/Partition.cs +++ b/Sources/Data/Microsoft.Psi.Data/Partition.cs @@ -33,7 +33,6 @@ protected Partition(Session session, string storeName, string storePath, string this.StoreName = storeName; this.StorePath = storePath; this.Name = name ?? storeName; - this.InitNew(); } /// @@ -100,7 +99,7 @@ protected set public IEnumerable AvailableStreams => this.Reader?.AvailableStreams; /// - /// Overridable method to allow derived object to initialize properties as part of object construction or after deserialization. + /// Overridable method to allow derived object to initialize properties after deserialization. /// protected virtual void InitNew() { diff --git a/Sources/Data/Microsoft.Psi.Data/Session.cs b/Sources/Data/Microsoft.Psi.Data/Session.cs index 07eb942c6..dc712268f 100644 --- a/Sources/Data/Microsoft.Psi.Data/Session.cs +++ b/Sources/Data/Microsoft.Psi.Data/Session.cs @@ -71,7 +71,7 @@ public string Name } /// - /// Gets the orginating time interval (earliest to latest) of the messages in this session. + /// Gets the originating time interval (earliest to latest) of the messages in this session. /// [IgnoreDataMember] public TimeInterval OriginatingTimeInterval => @@ -122,7 +122,7 @@ public StorePartition AddStorePartition(string storeName, string storePath, stri /// /// The name of the annotation store. /// The path of the annotation store. - /// The annotated event definition to use when creating new annoted events in the newly created annotation partition. + /// The annotated event definition to use when creating new annotated events in the newly created annotation partition. /// The partition name. Default is null. /// The newly added annotation partition. public AnnotationPartition CreateAnnotationPartition(string storeName, string storePath, AnnotatedEventDefinition definition, string partitionName = null) @@ -135,7 +135,7 @@ public AnnotationPartition CreateAnnotationPartition(string storeName, string st /// /// Asynchronously computes a derived partition for the session. /// - /// The type of paramater passed to the action. + /// The type of parameter passed to the action. /// The action to be invoked to derive partitions. /// The parameter to be passed to the action. /// The output partition name to be created. @@ -144,7 +144,7 @@ public AnnotationPartition CreateAnnotationPartition(string storeName, string st /// The path of the output partition. Default is null. /// The replay descriptor to us. /// An object that can be used for reporting progress. - /// A token for cancelling the asynchronous task. + /// A token for canceling the asynchronous task. /// A task that represents the asynchronous operation. public async Task CreateDerivedPartitionAsync( Action computeDerived, @@ -274,7 +274,7 @@ private void OnDeserialized(StreamingContext context) } /// - /// Due to the runtime's asynchronous behaviour, we may try to + /// Due to the runtime's asynchronous behavior, we may try to /// delete our test directory before the runtime has finished /// messing with it. This method will keep trying to delete /// the directory until the runtime shuts down. diff --git a/Sources/Data/Microsoft.Psi.Data/SessionImporter.cs b/Sources/Data/Microsoft.Psi.Data/SessionImporter.cs index 03bf4f99b..a272d72ab 100644 --- a/Sources/Data/Microsoft.Psi.Data/SessionImporter.cs +++ b/Sources/Data/Microsoft.Psi.Data/SessionImporter.cs @@ -31,7 +31,7 @@ private SessionImporter(Pipeline pipeline, Session session) public string Name { get; private set; } /// - /// Gets the orginating time interval (earliest to latest) of the messages in the session. + /// Gets the originating time interval (earliest to latest) of the messages in the session. /// public TimeInterval OriginatingTimeInterval { get; private set; } @@ -76,7 +76,7 @@ public bool HasStream(string streamName) } /// - /// Determines if a specicif importer contains the named stream. + /// Determines if a specific importer contains the named stream. /// /// Partition name of the specific importer. /// The stream to search for. diff --git a/Sources/Data/Microsoft.Psi.Data/SimpleReader.cs b/Sources/Data/Microsoft.Psi.Data/SimpleReader.cs index 54447f2c9..0208d7a0a 100644 --- a/Sources/Data/Microsoft.Psi.Data/SimpleReader.cs +++ b/Sources/Data/Microsoft.Psi.Data/SimpleReader.cs @@ -114,6 +114,18 @@ public ISimpleReader OpenNew() /// The metadata describing the specified stream. public PsiStreamMetadata GetMetadata(string streamName) => this.reader.GetMetadata(streamName); + /// + /// Returns the supplemental metadata for a specified storage stream. + /// + /// Type of supplemental metadata. + /// The name of the storage stream. + /// The metadata associated with the storage stream. + public T GetSupplementalMetadata(string streamName) + { + var meta = this.reader.GetMetadata(streamName); + return meta.GetSupplementalMetadata(this.Serializers); + } + /// /// Checks whether the specified storage stream exist in this store. /// diff --git a/Sources/Data/Microsoft.Psi.Data/StorePartition.cs b/Sources/Data/Microsoft.Psi.Data/StorePartition.cs index 0bc584937..af94bd61b 100644 --- a/Sources/Data/Microsoft.Psi.Data/StorePartition.cs +++ b/Sources/Data/Microsoft.Psi.Data/StorePartition.cs @@ -15,6 +15,7 @@ public class StorePartition : Partition, IDisposable private StorePartition(Session session, string storeName, string storePath, string name) : base(session, storeName, storePath, name, typeof(SimpleReader)) { + this.Reader = new SimpleReader(storeName, storePath); } private StorePartition() @@ -40,7 +41,7 @@ public static StorePartition Create(Session session, string storeName, string st } /// - /// Creates a store partition from an exisitng data store. + /// Creates a store partition from an existing data store. /// /// The session that this partition belongs to. /// The store name of this partition. @@ -60,7 +61,7 @@ public void Dispose() } /// - /// Overridable method to allow derived object to initialize properties as part of object construction or after deserialization. + /// Overridable method to allow derived object to initialize properties after deserialization. /// protected override void InitNew() { diff --git a/Sources/Data/Test.Psi.Data/AnnotationTests.cs b/Sources/Data/Test.Psi.Data/AnnotationTests.cs index 30df4fc07..73368a226 100644 --- a/Sources/Data/Test.Psi.Data/AnnotationTests.cs +++ b/Sources/Data/Test.Psi.Data/AnnotationTests.cs @@ -49,7 +49,7 @@ public void Initialize() this.definition = new AnnotatedEventDefinition("Definition"); this.definition.AddSchema(this.booleanSchema); - this.metadata = new JsonStreamMetadata("Range", 1, typeof(AnnotatedEvent).AssemblyQualifiedName, this.name, this.path); + this.metadata = new JsonStreamMetadata("Range", 1, typeof(AnnotatedEvent).AssemblyQualifiedName, null, this.name, this.path); } /// diff --git a/Sources/Data/Test.Psi.Data/DatasetTests.cs b/Sources/Data/Test.Psi.Data/DatasetTests.cs index 0fa6e4e61..bbbc332c0 100644 --- a/Sources/Data/Test.Psi.Data/DatasetTests.cs +++ b/Sources/Data/Test.Psi.Data/DatasetTests.cs @@ -388,7 +388,7 @@ public async Task SessionCreateDerivedPartitionCancellation() var inputStream = importer.OpenStream("Root"); var derivedStream = inputStream.Sample(TimeSpan.FromMinutes(1), RelativeTimeInterval.Infinite).Select(x => x * parameter).Write("DerivedStream", exporter); - // add a dummy source and propose a long time interval so that the operation will block (and eventually be cancelled) + // add a dummy source and propose a long time interval so that the operation will block (and eventually be canceled) var generator = Generators.Repeat(pipeline, 0, int.MaxValue, TimeSpan.FromMilliseconds(1000)); var replayTimeInterval = TimeInterval.LeftBounded(importer.OriginatingTimeInterval.Left); pipeline.ProposeReplayTime(replayTimeInterval); diff --git a/Sources/Data/Test.Psi.Data/JsonTests.cs b/Sources/Data/Test.Psi.Data/JsonTests.cs index fbb4129dd..5a898b9c0 100644 --- a/Sources/Data/Test.Psi.Data/JsonTests.cs +++ b/Sources/Data/Test.Psi.Data/JsonTests.cs @@ -117,8 +117,8 @@ public void JsonSimpleWriterTest() { writer.CreateStore(StoreName, OutputPath, false); - IStreamMetadata metadata1 = new JsonStreamMetadata("Stream1", 1, TypeName, PartitionName, OutputPath); - IStreamMetadata metadata2 = new JsonStreamMetadata("Stream2", 2, TypeName, PartitionName, OutputPath); + IStreamMetadata metadata1 = new JsonStreamMetadata("Stream1", 1, TypeName, null, PartitionName, OutputPath); + IStreamMetadata metadata2 = new JsonStreamMetadata("Stream2", 2, TypeName, null, PartitionName, OutputPath); List> stream1 = new List>(); stream1.Add(new Message(Data, FirstTime, FirstTime, 1, 0)); @@ -134,7 +134,7 @@ public void JsonSimpleWriterTest() } var escapedOutputPath = OutputPath.Replace(@"\", @"\\"); - string expectedCatalog = "[{\"Name\":\"Stream1\",\"Id\":1,\"TypeName\":\"Test.Psi.Data.SimpleObject\",\"PartitionName\":\"JsonStore\",\"PartitionPath\":\"" + escapedOutputPath + "\",\"FirstMessageTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageTime\":\"2017-11-01T09:15:34.12345Z\",\"FirstMessageOriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageOriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"AverageMessageSize\":303,\"AverageLatency\":0,\"MessageCount\":2},{\"Name\":\"Stream2\",\"Id\":2,\"TypeName\":\"Test.Psi.Data.SimpleObject\",\"PartitionName\":\"JsonStore\",\"PartitionPath\":\"" + escapedOutputPath + "\",\"FirstMessageTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageTime\":\"2017-11-01T09:15:34.12345Z\",\"FirstMessageOriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageOriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"AverageMessageSize\":303,\"AverageLatency\":0,\"MessageCount\":2}]"; + string expectedCatalog = "[{\"Name\":\"Stream1\",\"Id\":1,\"TypeName\":\"Test.Psi.Data.SimpleObject\",\"PartitionName\":\"JsonStore\",\"PartitionPath\":\"" + escapedOutputPath + "\",\"FirstMessageTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageTime\":\"2017-11-01T09:15:34.12345Z\",\"FirstMessageOriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageOriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"AverageMessageSize\":303,\"AverageLatency\":0,\"MessageCount\":2,\"SupplementalMetadataTypeName\":null},{\"Name\":\"Stream2\",\"Id\":2,\"TypeName\":\"Test.Psi.Data.SimpleObject\",\"PartitionName\":\"JsonStore\",\"PartitionPath\":\"" + escapedOutputPath + "\",\"FirstMessageTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageTime\":\"2017-11-01T09:15:34.12345Z\",\"FirstMessageOriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"LastMessageOriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"AverageMessageSize\":303,\"AverageLatency\":0,\"MessageCount\":2,\"SupplementalMetadataTypeName\":null}]"; string expectedData = "[{\"Envelope\":{\"SourceId\":1,\"SequenceId\":0,\"OriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"Time\":\"2017-11-01T09:15:30.12345Z\"},\"Data\":{\"ArrayValue\":[0,1,2,3],\"BoolValue\":true,\"DateTimeValue\":\"2017-11-30T12:59:41.896745Z\",\"DoubleValue\":0.123456,\"IntValue\":123456,\"ListValue\":[4,5,6,7],\"StringValue\":\"abc\",\"TimeSpanValue\":\"01:02:03.4567890\"}},{\"Envelope\":{\"SourceId\":2,\"SequenceId\":0,\"OriginatingTime\":\"2017-11-01T09:15:30.12345Z\",\"Time\":\"2017-11-01T09:15:30.12345Z\"},\"Data\":{\"ArrayValue\":[0,1,2,3],\"BoolValue\":true,\"DateTimeValue\":\"2017-11-30T12:59:41.896745Z\",\"DoubleValue\":0.123456,\"IntValue\":123456,\"ListValue\":[4,5,6,7],\"StringValue\":\"abc\",\"TimeSpanValue\":\"01:02:03.4567890\"}},{\"Envelope\":{\"SourceId\":1,\"SequenceId\":1,\"OriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"Time\":\"2017-11-01T09:15:34.12345Z\"},\"Data\":{\"ArrayValue\":[0,1,2,3],\"BoolValue\":true,\"DateTimeValue\":\"2017-11-30T12:59:41.896745Z\",\"DoubleValue\":0.123456,\"IntValue\":123456,\"ListValue\":[4,5,6,7],\"StringValue\":\"abc\",\"TimeSpanValue\":\"01:02:03.4567890\"}},{\"Envelope\":{\"SourceId\":2,\"SequenceId\":1,\"OriginatingTime\":\"2017-11-01T09:15:34.12345Z\",\"Time\":\"2017-11-01T09:15:34.12345Z\"},\"Data\":{\"ArrayValue\":[0,1,2,3],\"BoolValue\":true,\"DateTimeValue\":\"2017-11-30T12:59:41.896745Z\",\"DoubleValue\":0.123456,\"IntValue\":123456,\"ListValue\":[4,5,6,7],\"StringValue\":\"abc\",\"TimeSpanValue\":\"01:02:03.4567890\"}}]"; string actualCatalog = string.Empty; string actualData = string.Empty; diff --git a/Sources/Data/Test.Psi.Data/SimpleReaderTests.cs b/Sources/Data/Test.Psi.Data/SimpleReaderTests.cs index 3bde4602d..4c0733886 100644 --- a/Sources/Data/Test.Psi.Data/SimpleReaderTests.cs +++ b/Sources/Data/Test.Psi.Data/SimpleReaderTests.cs @@ -98,5 +98,30 @@ public void SimpleReaderLargeStream() Assert.AreEqual(result.Sum(x => x), probe * size); } } + + [TestMethod] + [Timeout(60000)] + public void RetrieveStreamSupplementalMetadata() + { + var name = nameof(this.RetrieveStreamSupplementalMetadata); + + // create store with supplemental meta + using (var p = Pipeline.Create("write")) + { + var store = Store.Create(p, name, this.path); + var stream0 = Generators.Range(p, 0, 10, TimeSpan.FromTicks(1)); + var stream1 = Generators.Range(p, 0, 10, TimeSpan.FromTicks(1)); + stream0.Write("NoMeta", store, true); + stream1.Write(("Favorite irrational number", Math.E), "WithMeta", store); + } + + // read it back with a simple reader + var reader = new SimpleReader(name, this.path); + Assert.IsNull(reader.GetMetadata("NoMeta").SupplementalMetadataTypeName); + Assert.AreEqual(typeof(ValueTuple).AssemblyQualifiedName, reader.GetMetadata("WithMeta").SupplementalMetadataTypeName); + var supplemental1 = reader.GetSupplementalMetadata<(string, double)>("WithMeta"); + Assert.AreEqual("Favorite irrational number", supplemental1.Item1); + Assert.AreEqual(Math.E, supplemental1.Item2); + } } } diff --git a/Sources/Data/Test.Psi.Data/Test.Psi.Data.csproj b/Sources/Data/Test.Psi.Data/Test.Psi.Data.csproj index 2a3dd7983..720f44fa4 100644 --- a/Sources/Data/Test.Psi.Data/Test.Psi.Data.csproj +++ b/Sources/Data/Test.Psi.Data/Test.Psi.Data.csproj @@ -1,7 +1,7 @@ - + - netcoreapp2.0 + netcoreapp3.1 false Test.Psi.Data.ConsoleMain @@ -31,10 +31,14 @@ - + + all + runtime; build; native; contentfiles; analyzers + + - - + + diff --git a/Sources/Devices/Microsoft.Psi.DeviceManagement/CameraDeviceInfo.cs b/Sources/Devices/Microsoft.Psi.DeviceManagement/CameraDeviceInfo.cs index adeb75229..61436b869 100644 --- a/Sources/Devices/Microsoft.Psi.DeviceManagement/CameraDeviceInfo.cs +++ b/Sources/Devices/Microsoft.Psi.DeviceManagement/CameraDeviceInfo.cs @@ -48,6 +48,11 @@ public CameraDeviceInfo(string friendlyName, string deviceName, string serialNum /// public string DeviceName { get; set; } + /// + /// Gets or sets the device id (index of device from systems perspective). + /// + public int DeviceId { get; set; } + /// /// Gets or sets the serial number for this device. Maybe empty string. /// diff --git a/Sources/Devices/Microsoft.Psi.DeviceManagement/Microsoft.Psi.DeviceManagement.csproj b/Sources/Devices/Microsoft.Psi.DeviceManagement/Microsoft.Psi.DeviceManagement.csproj index 99e197189..13e937f90 100644 --- a/Sources/Devices/Microsoft.Psi.DeviceManagement/Microsoft.Psi.DeviceManagement.csproj +++ b/Sources/Devices/Microsoft.Psi.DeviceManagement/Microsoft.Psi.DeviceManagement.csproj @@ -31,6 +31,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageCompressor.cs new file mode 100644 index 000000000..166c1748c --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageCompressor.cs @@ -0,0 +1,67 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; + + /// + /// Implements a compressor used by the serialization layer for compressing streams + /// of depth images in a generic fashion. This object should not be called directly, + /// but instead is used by the class. + /// + public class DepthImageCompressor : IDepthImageCompressor + { + private readonly IDepthImageToStreamEncoder encoder = null; + private readonly IDepthImageFromStreamDecoder decoder = null; + + /// + /// Initializes a new instance of the class. + /// + /// The depth compression method to be used. + public DepthImageCompressor(DepthCompressionMethod depthCompressionMethod) + { + this.DepthCompressionMethod = depthCompressionMethod; + switch (this.DepthCompressionMethod) + { + case DepthCompressionMethod.Png: + this.encoder = new DepthImageToPngStreamEncoder(); + break; + case DepthCompressionMethod.None: + break; + } + + this.decoder = new DepthImageFromStreamDecoder(); + } + + /// + public DepthCompressionMethod DepthCompressionMethod { get; set; } = DepthCompressionMethod.Png; + + /// + public void Serialize(BufferWriter writer, DepthImage depthImage, SerializationContext context) + { + if (this.encoder != null) + { + using var sharedEncodedDepthImage = EncodedDepthImagePool.GetOrCreate(depthImage.Width, depthImage.Height); + sharedEncodedDepthImage.Resource.EncodeFrom(depthImage, this.encoder); + Serializer.Serialize(writer, sharedEncodedDepthImage, context); + } + else + { + Serializer.Serialize(writer, depthImage, context); + } + } + + /// + public void Deserialize(BufferReader reader, ref DepthImage depthImage, SerializationContext context) + { + Shared sharedEncodedDepthImage = null; + Serializer.Deserialize(reader, ref sharedEncodedDepthImage, context); + using var sharedDepthImage = DepthImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height); + sharedDepthImage.Resource.DecodeFrom(sharedEncodedDepthImage.Resource, this.decoder); + depthImage = sharedDepthImage.Resource.DeepClone(); + sharedEncodedDepthImage.Dispose(); + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageFromStreamDecoder.cs new file mode 100644 index 000000000..8cc8eb577 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageFromStreamDecoder.cs @@ -0,0 +1,22 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Runtime.InteropServices; + using SkiaSharp; + + /// + /// Implements a depth image decoder. + /// + public class DepthImageFromStreamDecoder : IDepthImageFromStreamDecoder + { + /// + public void DecodeFromStream(Stream stream, DepthImage depthImage) + { + var decoded = SKBitmap.Decode(stream); + Marshal.Copy(decoded.Bytes, 0, depthImage.ImageData, decoded.ByteCount); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageToPngStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageToPngStreamEncoder.cs new file mode 100644 index 000000000..991df2ef7 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/DepthImageToPngStreamEncoder.cs @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.IO; + using SkiaSharp; + + /// + /// Implements a depth image encoder for PNG format. + /// + public class DepthImageToPngStreamEncoder : IDepthImageToStreamEncoder + { + /// + public void EncodeToStream(DepthImage depthImage, Stream stream) + { + depthImage.AsSKImage().Encode(SKEncodedImageFormat.Png, 100).SaveTo(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/IBitmapEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/IBitmapEncoder.cs deleted file mode 100644 index 3453916aa..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/IBitmapEncoder.cs +++ /dev/null @@ -1,20 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System.IO; - - /// - /// Bitmap encoder interface. - /// - public interface IBitmapEncoder - { - /// - /// Encode image to stream. - /// - /// Image to encode. - /// Stream to which to encode. - void Encode(Image image, Stream stream); - } -} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageCompressor.cs index 194c79fcd..990bc49ed 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageCompressor.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageCompressor.cs @@ -7,74 +7,69 @@ namespace Microsoft.Psi.Imaging using Microsoft.Psi.Serialization; /// - /// ImageCompressor defines an object used by the serialization layer - /// for compressing streams of images in a generic fashion. This object - /// should not be called directly but instead if used by Microsoft.Psi.Imaging. + /// Implements a compressor used by the serialization layer for compressing streams + /// of images in a generic fashion. This object should not be called directly, but + /// instead is used by the class. /// public class ImageCompressor : IImageCompressor { - /// - /// Initializes a new instance of the class. - /// - public ImageCompressor() - { - } + private readonly IImageToStreamEncoder encoder = null; + private readonly IImageFromStreamDecoder decoder = null; /// /// Initializes a new instance of the class. /// - /// Compression method to be used by compressor. + /// The image compression method to be used. public ImageCompressor(CompressionMethod compressionMethod) { this.CompressionMethod = compressionMethod; - } - /// - /// Gets or sets the compression method being used by the compressor. - /// - public CompressionMethod CompressionMethod { get; set; } = CompressionMethod.PNG; - - /// - public void Serialize(BufferWriter writer, Image instance, SerializationContext context) - { - // Note: encoder instances are created here because they may have thread affinity. - IBitmapEncoder encoder = null; switch (this.CompressionMethod) { - case CompressionMethod.JPEG: - encoder = new JpegBitmapEncoder { QualityLevel = 90 }; + case CompressionMethod.Jpeg: + this.encoder = new ImageToJpegStreamEncoder { QualityLevel = 90 }; break; - case CompressionMethod.PNG: - encoder = new PngBitmapEncoder(); + case CompressionMethod.Png: + this.encoder = new ImageToPngStreamEncoder(); break; case CompressionMethod.None: break; } - if (encoder != null) + this.decoder = new ImageFromStreamDecoder(); + } + + /// + public CompressionMethod CompressionMethod { get; set; } = CompressionMethod.Png; + + /// + public void Serialize(BufferWriter writer, Image image, SerializationContext context) + { + if (this.encoder != null) { - using (var sharedEncodedImage = EncodedImagePool.GetOrCreate()) - { - sharedEncodedImage.Resource.EncodeFrom(instance, encoder.Encode); - Serializer.Serialize(writer, sharedEncodedImage, context); - } + using var sharedEncodedImage = EncodedImagePool.GetOrCreate(image.Width, image.Height, image.PixelFormat); + sharedEncodedImage.Resource.EncodeFrom(image, this.encoder); + Serializer.Serialize(writer, sharedEncodedImage, context); } else { - Serializer.Serialize(writer, instance, context); + Serializer.Serialize(writer, image, context); } } /// - public void Deserialize(BufferReader reader, ref Image target, SerializationContext context) + public void Deserialize(BufferReader reader, ref Image image, SerializationContext context) { - Shared encodedImage = null; - Serializer.Deserialize(reader, ref encodedImage, context); - using (var image = ImagePool.GetOrCreate(encodedImage.Resource.Width, encodedImage.Resource.Height, PixelFormat.BGR_24bpp)) - { - ImageDecoder.DecodeTo(encodedImage.Resource, image.Resource); - target = image.Resource.DeepClone(); - } + Shared sharedEncodedImage = null; + Serializer.Deserialize(reader, ref sharedEncodedImage, context); + + using var sharedImage = ImagePool.GetOrCreate( + sharedEncodedImage.Resource.Width, + sharedEncodedImage.Resource.Height, + sharedEncodedImage.Resource.PixelFormat); + sharedImage.Resource.DecodeFrom(sharedEncodedImage.Resource, this.decoder); + image = sharedImage.Resource.DeepClone(); + sharedEncodedImage.Dispose(); } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageDecoder.cs deleted file mode 100644 index 893b70fac..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageDecoder.cs +++ /dev/null @@ -1,50 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System.Runtime.InteropServices; - using Microsoft.Psi; - using Microsoft.Psi.Components; - using SkiaSharp; - - /// - /// Pipeline component for decoding an image. - /// - public class ImageDecoder : ConsumerProducer, Shared> - { - /// - /// Initializes a new instance of the class. - /// - /// Pipeline this component is a part of. - public ImageDecoder(Pipeline pipeline) - : base(pipeline) - { - } - - /// - /// Decodes an encoded image into the given image instance. - /// - /// Encoded image to decode. - /// Image into which to decode. - public static void DecodeTo(EncodedImage encodedImage, Image image) - { - var decoded = SKBitmap.Decode(encodedImage.GetStream()); - Marshal.Copy(decoded.Bytes, 0, image.ImageData, decoded.ByteCount); - } - - /// - /// Pipeline callback method for decoding a sample. - /// - /// Encoded image to decode. - /// Pipeline information about the sample. - protected override void Receive(Shared encodedImage, Envelope e) - { - using (var image = ImagePool.GetOrCreate(encodedImage.Resource.Width, encodedImage.Resource.Height, PixelFormat.BGR_24bpp)) - { - DecodeTo(encodedImage.Resource, image.Resource); - this.Out.Post(image, e.OriginatingTime); - } - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageEncoder.cs deleted file mode 100644 index bcccc13a8..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageEncoder.cs +++ /dev/null @@ -1,46 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - using System.IO; - using Microsoft.Psi; - using Microsoft.Psi.Components; - - /// - /// Pipeline component for encoding an image. - /// - public class ImageEncoder : ConsumerProducer, Shared> - { - private readonly Func encoderFn; - - /// - /// Initializes a new instance of the class. - /// - /// Pipeline this component is a part of. - /// Callback method for encoding a single image sample. - public ImageEncoder(Pipeline pipeline, Func encoderFn) - : base(pipeline) - { - this.encoderFn = encoderFn; - } - - /// - /// Pipeline callback function for encoding an image sample. - /// - /// Image to be encoded. - /// Pipeline information about the sample. - protected override void Receive(Shared sharedImage, Envelope e) - { - // the encoder has thread affinity, so we need to re-create it (we can't dispatch the call since we sdon't know if the thread that created us is pumping messages) - var encoder = this.encoderFn(); - - using (var sharedEncodedImage = EncodedImagePool.GetOrCreate()) - { - sharedEncodedImage.Resource.EncodeFrom(sharedImage.Resource, encoder.Encode); - this.Out.Post(sharedEncodedImage, e.OriginatingTime); - } - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageFromStreamDecoder.cs new file mode 100644 index 000000000..f4be85153 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageFromStreamDecoder.cs @@ -0,0 +1,34 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Runtime.InteropServices; + using SkiaSharp; + + /// + /// Implements an image decoder. + /// + public class ImageFromStreamDecoder : IImageFromStreamDecoder + { + /// + public void DecodeFromStream(Stream stream, Image image) + { + var decoded = SKBitmap.Decode(stream); + Marshal.Copy(decoded.Bytes, 0, image.ImageData, decoded.ByteCount); + } + + /// + public PixelFormat GetPixelFormat(Stream stream) + { + var decoded = SKBitmap.Decode(stream); + return decoded.ColorType switch + { + SKColorType.Bgra8888 => PixelFormat.BGRA_32bpp, + SKColorType.Gray8 => PixelFormat.Gray_8bpp, + _ => PixelFormat.Undefined, + }; + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToJpegStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToJpegStreamEncoder.cs new file mode 100644 index 000000000..9fdf5c11f --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToJpegStreamEncoder.cs @@ -0,0 +1,25 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using SkiaSharp; + + /// + /// Implements an image encoder for JPEG format. + /// + public class ImageToJpegStreamEncoder : IImageToStreamEncoder + { + /// + /// Gets or sets JPEG image quality (0-100). + /// + public int QualityLevel { get; set; } = 100; + + /// + public void EncodeToStream(Image image, Stream stream) + { + image.AsSKImage().Encode(SKEncodedImageFormat.Jpeg, this.QualityLevel).SaveTo(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToPngStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToPngStreamEncoder.cs new file mode 100644 index 000000000..3294c4605 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImageToPngStreamEncoder.cs @@ -0,0 +1,20 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using SkiaSharp; + + /// + /// Implements an image encoder for PNG format. + /// + public class ImageToPngStreamEncoder : IImageToStreamEncoder + { + /// + public void EncodeToStream(Image image, Stream stream) + { + image.AsSKImage().Encode(SKEncodedImageFormat.Png, 100).SaveTo(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImagingOperators.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImagingOperators.cs new file mode 100644 index 000000000..ec6e0c60f --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/ImagingOperators.cs @@ -0,0 +1,91 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using SkiaSharp; + + /// + /// Implements stream operator methods for Imaging. + /// + public static partial class ImagingOperators + { + /// + /// Encodes an image to a JPEG format. + /// + /// A producer of images to encode. + /// JPEG quality to use. + /// An optional delivery policy. + /// A producer that generates the JPEG images. + public static IProducer> EncodeJpeg(this IProducer> source, int quality = 90, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new ImageToJpegStreamEncoder { QualityLevel = quality }, deliveryPolicy); + } + + /// + /// Encodes an image to a PNG format. + /// + /// A producer of images to encoder. + /// An optional delivery policy. + /// A producer that generates the PNG images. + public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new ImageToPngStreamEncoder(), deliveryPolicy); + } + + /// + /// Decodes an encoded image. + /// + /// A producer of encoded images to decode. + /// An optional delivery policy. + /// A producer that generates the decoded images. + public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(new ImageFromStreamDecoder(), deliveryPolicy); + } + + /// + /// Encodes a depth image to a PNG format. + /// + /// A producer of depth images to encode. + /// An optional delivery policy. + /// A producer that generates the PNG-encoded depth images. + public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new DepthImageToPngStreamEncoder(), deliveryPolicy); + } + + /// + /// Decodes an encoded depth image. + /// + /// A producer of encoded depth images to decode. + /// An optional delivery policy. + /// A producer that generates the decoded depth images. + public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(new DepthImageFromStreamDecoder(), deliveryPolicy); + } + + /// + /// Converts an image to a SkiaSharp SKImage. + /// + /// Image to convert to SKImage type. + /// SKImage. + internal static SKImage AsSKImage(this ImageBase image) + { + var data = SKData.Create(image.ImageData, image.Size); + var colorType = image.PixelFormat switch + { + // These are unsupported by SkiaSharp: BGRX_32bpp, BGR_24bpp, Gray_16bpp, RGBA_64bpp + PixelFormat.BGRA_32bpp => SKColorType.Bgra8888, + PixelFormat.Gray_8bpp => SKColorType.Gray8, + PixelFormat.Undefined => SKColorType.Unknown, + PixelFormat.Gray_16bpp => throw new ArgumentException($"Unsupported pixel format: {image.PixelFormat} (e.g. DepthImage)"), + _ => throw new ArgumentException($"Unsupported pixel format: {image.PixelFormat}"), + }; + var info = new SKImageInfo(image.Width, image.Height, colorType); + return SKImage.FromPixelData(info, data, image.Stride); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/JpegBitmapEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/JpegBitmapEncoder.cs deleted file mode 100644 index a1bd771fe..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/JpegBitmapEncoder.cs +++ /dev/null @@ -1,32 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System.IO; - using SkiaSharp; - - /// - /// JPEG bitmap encoder. - /// - public class JpegBitmapEncoder : IBitmapEncoder - { - /// - /// Gets or sets JPEG image quality (0-100). - /// - public int QualityLevel { get; set; } - - /// - /// Encode image to stream. - /// - /// Image to be encoded. - /// Stream to which to encode. - public void Encode(Image image, Stream stream) - { - var data = SKData.Create(image.ImageData, image.Size); - var img = SKImage.FromPixelData(SKImageInfo.Empty, data, image.Stride); - var jpeg = img.Encode(SKEncodedImageFormat.Jpeg, this.QualityLevel); - jpeg.SaveTo(stream); - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/Microsoft.Psi.Imaging.Linux.csproj b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/Microsoft.Psi.Imaging.Linux.csproj index 90c3b2d0d..2e472ba83 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/Microsoft.Psi.Imaging.Linux.csproj +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/Microsoft.Psi.Imaging.Linux.csproj @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PngBitmapEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PngBitmapEncoder.cs deleted file mode 100644 index 088323cc3..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PngBitmapEncoder.cs +++ /dev/null @@ -1,27 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System.IO; - using SkiaSharp; - - /// - /// PNG bitmap encoder. - /// - public class PngBitmapEncoder : IBitmapEncoder - { - /// - /// Encode image to stream. - /// - /// Image to be encoded. - /// Stream to which to encode. - public void Encode(Image image, Stream stream) - { - var data = SKData.Create(image.ImageData, image.Size); - var img = SKImage.FromPixelData(SKImageInfo.Empty, data, image.Stride); - var png = img.Encode(SKEncodedImageFormat.Png, 100); - png.SaveTo(stream); - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PsiImaging.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PsiImaging.cs deleted file mode 100644 index b15608735..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Linux/PsiImaging.cs +++ /dev/null @@ -1,59 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - - /// - /// Implements stream operator methods for Imaging. - /// - public static partial class ImagingOperators - { - /// - /// Converts from an Image to a compressed (encoded) image. - /// - /// Source image to encode. - /// Method to perform encoding. - /// An optional delivery policy. - /// Returns a producer that generates the encoded images. - public static IProducer> Encode(this IProducer> source, Func encoderFn, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ImageEncoder(source.Out.Pipeline, encoderFn), deliveryPolicy); - } - - /// - /// Converts from an Image to a compressed JPEG image. - /// - /// Source image to compress. - /// JPEG quality to use. - /// An optional delivery policy. - /// Returns a producer that generates the JPEG images. - public static IProducer> EncodeJpeg(this IProducer> source, int quality = 90, DeliveryPolicy> deliveryPolicy = null) - { - return Encode(source, () => new JpegBitmapEncoder { QualityLevel = quality }, deliveryPolicy); - } - - /// - /// Converts from an Image to a compressed PNG image. - /// - /// Source image to compress. - /// An optional delivery policy. - /// Returns a producer that generates the PNG images. - public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) - { - return Encode(source, () => new PngBitmapEncoder(), deliveryPolicy); - } - - /// - /// Decodes an image that was previously encoded. - /// - /// Source image to compress. - /// An optional delivery policy. - /// Returns a producer that generates the decoded images. - public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ImageDecoder(source.Out.Pipeline), deliveryPolicy); - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageCompressor.cs new file mode 100644 index 000000000..0cc44bd6e --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageCompressor.cs @@ -0,0 +1,68 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; + + /// + /// Implements a compressor used by the serialization layer for compressing streams + /// of depth images in a generic fashion. This object should not be called directly, + /// but instead is used by the class. + /// + public class DepthImageCompressor : IDepthImageCompressor + { + private readonly IDepthImageToStreamEncoder encoder = null; + private readonly IDepthImageFromStreamDecoder decoder = null; + + /// + /// Initializes a new instance of the class. + /// + /// The depth image compression method to be used. + public DepthImageCompressor(DepthCompressionMethod depthCompressionMethod) + { + this.DepthCompressionMethod = depthCompressionMethod; + switch (this.DepthCompressionMethod) + { + case DepthCompressionMethod.Png: + this.encoder = new DepthImageToPngStreamEncoder(); + break; + case DepthCompressionMethod.None: + break; + } + + this.decoder = new DepthImageFromStreamDecoder(); + } + + /// + public DepthCompressionMethod DepthCompressionMethod { get; set; } = DepthCompressionMethod.Png; + + /// + public void Serialize(BufferWriter writer, DepthImage depthImage, SerializationContext context) + { + if (this.encoder != null) + { + using var sharedEncodedDepthImage = EncodedDepthImagePool.GetOrCreate( + depthImage.Width, depthImage.Height); + sharedEncodedDepthImage.Resource.EncodeFrom(depthImage, this.encoder); + Serializer.Serialize(writer, sharedEncodedDepthImage, context); + } + else + { + Serializer.Serialize(writer, depthImage, context); + } + } + + /// + public void Deserialize(BufferReader reader, ref DepthImage depthImage, SerializationContext context) + { + Shared sharedEncodedDepthImage = null; + Serializer.Deserialize(reader, ref sharedEncodedDepthImage, context); + using var sharedDepthImage = DepthImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height); + sharedDepthImage.Resource.DecodeFrom(sharedEncodedDepthImage.Resource, this.decoder); + depthImage = sharedDepthImage.Resource.DeepClone(); + sharedEncodedDepthImage.Dispose(); + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageFromStreamDecoder.cs new file mode 100644 index 000000000..f54004045 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageFromStreamDecoder.cs @@ -0,0 +1,23 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Windows; + using System.Windows.Media.Imaging; + + /// + /// Implements a depth image decoder. + /// + public class DepthImageFromStreamDecoder : IDepthImageFromStreamDecoder + { + /// + public void DecodeFromStream(Stream stream, DepthImage depthImage) + { + var decoder = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default); + BitmapSource bitmapSource = decoder.Frames[0]; + bitmapSource.CopyPixels(Int32Rect.Empty, depthImage.ImageData, depthImage.Stride * depthImage.Height, depthImage.Stride); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageToPngStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageToPngStreamEncoder.cs new file mode 100644 index 000000000..cc45b495e --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/DepthImageToPngStreamEncoder.cs @@ -0,0 +1,32 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Windows.Media.Imaging; + + /// + /// Implements a depth image encoder for PNG format. + /// + public class DepthImageToPngStreamEncoder : IDepthImageToStreamEncoder + { + /// + public void EncodeToStream(DepthImage depthImage, Stream stream) + { + var encoder = new PngBitmapEncoder(); + var bitmapSource = BitmapSource.Create( + depthImage.Width, + depthImage.Height, + 96, + 96, + depthImage.PixelFormat.ToWindowsMediaPixelFormat(), + null, + depthImage.ImageData, + depthImage.Stride * depthImage.Height, + depthImage.Stride); + encoder.Frames.Add(BitmapFrame.Create(bitmapSource)); + encoder.Save(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageCompressor.cs index ca0044fdd..e99072822 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageCompressor.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageCompressor.cs @@ -3,86 +3,70 @@ namespace Microsoft.Psi.Imaging { - using System.IO; - using System.Windows; - using System.Windows.Media.Imaging; using Microsoft.Psi.Common; using Microsoft.Psi.Serialization; /// - /// ImageCompressor defines an object used by the serialization layer - /// for compressing streams of images in a generic fashion. This object - /// should not be called directly but instead if used by Microsoft.Psi.Imaging. + /// Implements a compressor used by the serialization layer for compressing streams + /// of images in a generic fashion. This object should not be called directly, but + /// instead is used by the class. /// public class ImageCompressor : IImageCompressor { - /// - /// Initializes a new instance of the class. - /// - public ImageCompressor() - { - } + private readonly IImageToStreamEncoder encoder = null; + private readonly IImageFromStreamDecoder decoder = null; /// /// Initializes a new instance of the class. /// - /// Compression method to be used by compressor. + /// The image compression method to be used. public ImageCompressor(CompressionMethod compressionMethod) { this.CompressionMethod = compressionMethod; - } - - /// - /// Gets or sets the compression method being used by the compressor. - /// - public CompressionMethod CompressionMethod { get; set; } = CompressionMethod.PNG; - - /// - public void Serialize(BufferWriter writer, Image instance, SerializationContext context) - { - // Note: encoder instances are created here because they (in the case of media foundation) may have thread affinity. - BitmapEncoder encoder = null; switch (this.CompressionMethod) { - case CompressionMethod.JPEG: - encoder = new JpegBitmapEncoder { QualityLevel = 90 }; + case CompressionMethod.Jpeg: + this.encoder = new ImageToJpegStreamEncoder { QualityLevel = 90 }; break; - case CompressionMethod.PNG: - encoder = new PngBitmapEncoder(); + case CompressionMethod.Png: + this.encoder = new ImageToPngStreamEncoder(); break; case CompressionMethod.None: break; } - if (encoder != null) + this.decoder = new ImageFromStreamDecoder(); + } + + /// + public CompressionMethod CompressionMethod { get; set; } = CompressionMethod.Png; + + /// + public void Serialize(BufferWriter writer, Image image, SerializationContext context) + { + if (this.encoder != null) { - using (var sharedEncodedImage = EncodedImagePool.GetOrCreate()) - { - ImageEncoder.EncodeFrom(sharedEncodedImage.Resource, instance, encoder); - Serializer.Serialize(writer, sharedEncodedImage, context); - } + using var sharedEncodedImage = EncodedImagePool.GetOrCreate( + image.Width, image.Height, image.PixelFormat); + sharedEncodedImage.Resource.EncodeFrom(image, this.encoder); + Serializer.Serialize(writer, sharedEncodedImage, context); } else { - Serializer.Serialize(writer, instance, context); + Serializer.Serialize(writer, image, context); } } /// - public void Deserialize(BufferReader reader, ref Image target, SerializationContext context) + public void Deserialize(BufferReader reader, ref Image image, SerializationContext context) { - Shared encodedImage = null; - Serializer.Deserialize(reader, ref encodedImage, context); - using (var image = ImagePool.GetOrCreate(encodedImage.Resource.Width, encodedImage.Resource.Height, Imaging.PixelFormat.BGR_24bpp)) - { - ImageDecoder.DecodeTo(encodedImage.Resource, image.Resource); - target = image.Resource.DeepClone(); - } - - if (encodedImage != null) - { - encodedImage.Dispose(); - } + Shared sharedEncodedImage = null; + Serializer.Deserialize(reader, ref sharedEncodedImage, context); + using var sharedImage = ImagePool.GetOrCreate( + sharedEncodedImage.Resource.Width, sharedEncodedImage.Resource.Height, sharedEncodedImage.Resource.PixelFormat); + sharedImage.Resource.DecodeFrom(sharedEncodedImage.Resource, this.decoder); + image = sharedImage.Resource.DeepClone(); + sharedEncodedImage.Dispose(); } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageDecoder.cs deleted file mode 100644 index 398e5adc0..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageDecoder.cs +++ /dev/null @@ -1,98 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - using System.Windows; - using System.Windows.Media.Imaging; - using Microsoft.Psi; - using Microsoft.Psi.Components; - - /// - /// Component that decodes an image using a specified decoder (e.g. JPEG, PNG). - /// - public class ImageDecoder : ConsumerProducer, Shared> - { - private PixelFormat imagePixelFormat = PixelFormat.Undefined; - - /// - /// Initializes a new instance of the class. - /// - /// Pipeline this component is a part of. - public ImageDecoder(Pipeline pipeline) - : base(pipeline) - { - } - - /// - /// Decodes an encoded image into the given image instance. - /// - /// Encoded image to decode. - /// Image into which to decode. - public static void DecodeTo(EncodedImage encodedImage, Image image) - { - var decoder = BitmapDecoder.Create(encodedImage.GetStream(), BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default); - BitmapSource bitmapSource = decoder.Frames[0]; - bitmapSource.CopyPixels(Int32Rect.Empty, image.ImageData, image.Stride * image.Height, image.Stride); - } - - /// - /// Returns the pixel format of the image. - /// - /// Encoded image from which to get pixel format. - /// Returns the image's pixel format. - public static PixelFormat GetPixelFormat(EncodedImage encodedImage) - { - var decoder = BitmapDecoder.Create(encodedImage.GetStream(), BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default); - BitmapSource bitmapSource = decoder.Frames[0]; - if (bitmapSource.Format == System.Windows.Media.PixelFormats.Bgr24) - { - return PixelFormat.BGR_24bpp; - } - else if (bitmapSource.Format == System.Windows.Media.PixelFormats.Gray16) - { - return PixelFormat.Gray_16bpp; - } - else if (bitmapSource.Format == System.Windows.Media.PixelFormats.Gray8) - { - return PixelFormat.Gray_8bpp; - } - else if (bitmapSource.Format == System.Windows.Media.PixelFormats.Bgr32) - { - return PixelFormat.BGRX_32bpp; - } - else if (bitmapSource.Format == System.Windows.Media.PixelFormats.Bgra32) - { - return PixelFormat.BGRA_32bpp; - } - else if (bitmapSource.Format == System.Windows.Media.PixelFormats.Rgba64) - { - return PixelFormat.RGBA_64bpp; - } - else - { - throw new NotImplementedException("Format not supported."); - } - } - - /// - /// Pipeline callback method for decoding a sample. - /// - /// Encoded image to decode. - /// Pipeline information about the sample. - protected override void Receive(Shared encodedImage, Envelope e) - { - if (this.imagePixelFormat == PixelFormat.Undefined) - { - this.imagePixelFormat = GetPixelFormat(encodedImage.Resource); - } - - using (var image = ImagePool.GetOrCreate(encodedImage.Resource.Width, encodedImage.Resource.Height, this.imagePixelFormat)) - { - DecodeTo(encodedImage.Resource, image.Resource); - this.Out.Post(image, e.OriginatingTime); - } - } - } -} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageEncoder.cs deleted file mode 100644 index 1e6f27ad1..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageEncoder.cs +++ /dev/null @@ -1,81 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - using System.IO; - using System.Windows.Media.Imaging; - using Microsoft.Psi; - using Microsoft.Psi.Components; - - /// - /// Component that encodes an image using a specified encoder (e.g. JPEG, PNG). - /// - public class ImageEncoder : ConsumerProducer, Shared> - { - private readonly Func encoderFn; - - /// - /// Initializes a new instance of the class. - /// - /// Pipeline this component is a part of. - /// Callback method for encoding a single image sample. - public ImageEncoder(Pipeline pipeline, Func encoderFn) - : base(pipeline) - { - this.encoderFn = encoderFn; - } - - /// - /// Encodes an image in-place into the given encoded image instance using the specified encoder. - /// - /// Encoded image into which to encode in-place. - /// Image to be encoded. - /// Encoder to use. - public static void EncodeFrom(EncodedImage encodedImage, Image image, BitmapEncoder encoder) - { - System.Windows.Media.PixelFormat format; - if (image.PixelFormat == PixelFormat.BGR_24bpp) - { - format = System.Windows.Media.PixelFormats.Bgr24; - } - else if (image.PixelFormat == PixelFormat.Gray_16bpp) - { - format = System.Windows.Media.PixelFormats.Gray16; - } - else if (image.PixelFormat == PixelFormat.Gray_8bpp) - { - format = System.Windows.Media.PixelFormats.Gray8; - } - else - { - format = System.Windows.Media.PixelFormats.Bgr32; - } - - encodedImage.EncodeFrom(image, (_, stream) => - { - BitmapSource bitmapSource = BitmapSource.Create(image.Width, image.Height, 96, 96, format, null, image.ImageData, image.Stride * image.Height, image.Stride); - encoder.Frames.Add(BitmapFrame.Create(bitmapSource)); - encoder.Save(stream); - }); - } - - /// - /// Pipeline callback function for encoding an image sample. - /// - /// Image to be encoded. - /// Pipeline information about the sample. - protected override void Receive(Shared sharedImage, Envelope e) - { - // the encoder has thread affinity, so we need to re-create it (we can't dispatch the call since we sdon't know if the thread that created us is pumping messages) - var encoder = this.encoderFn(); - - using (var sharedEncodedImage = EncodedImagePool.GetOrCreate()) - { - EncodeFrom(sharedEncodedImage.Resource, sharedImage.Resource, encoder); - this.Out.Post(sharedEncodedImage, e.OriginatingTime); - } - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageFromStreamDecoder.cs new file mode 100644 index 000000000..38c5549c2 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageFromStreamDecoder.cs @@ -0,0 +1,41 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Windows; + using System.Windows.Media.Imaging; + + /// + /// Implements an image decoder. + /// + public class ImageFromStreamDecoder : IImageFromStreamDecoder + { + /// + public void DecodeFromStream(Stream stream, Image image) + { + var decoder = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default); + BitmapSource bitmapSource = decoder.Frames[0]; + var fmt = bitmapSource.Format.ToPixelFormat(); + if (fmt != image.PixelFormat) + { + using var img = Microsoft.Psi.Imaging.ImagePool.GetOrCreate(image.Width, image.Height, fmt); + bitmapSource.CopyPixels(Int32Rect.Empty, img.Resource.ImageData, img.Resource.Stride * img.Resource.Height, img.Resource.Stride); + img.Resource.CopyTo(image); + } + else + { + bitmapSource.CopyPixels(Int32Rect.Empty, image.ImageData, image.Stride * image.Height, image.Stride); + } + } + + /// + public PixelFormat GetPixelFormat(Stream stream) + { + var decoder = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default); + BitmapSource bitmapSource = decoder.Frames[0]; + return bitmapSource.Format.ToPixelFormat(); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToJpegStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToJpegStreamEncoder.cs new file mode 100644 index 000000000..b918736ae --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToJpegStreamEncoder.cs @@ -0,0 +1,39 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Windows.Media.Imaging; + + /// + /// Implements an image encoder for JPEG format. + /// + public class ImageToJpegStreamEncoder : IImageToStreamEncoder + { + /// + /// Gets or sets JPEG image quality (0-100). + /// + public int QualityLevel { get; set; } = 100; + + /// + public void EncodeToStream(Image image, Stream stream) + { + // The encoder is created on every call (rather than in a constructor) + // b/c the encoder has thread affinity + var encoder = new JpegBitmapEncoder { QualityLevel = this.QualityLevel }; + var bitmapSource = BitmapSource.Create( + image.Width, + image.Height, + 96, + 96, + image.PixelFormat.ToWindowsMediaPixelFormat(), + null, + image.ImageData, + image.Stride * image.Height, + image.Stride); + encoder.Frames.Add(BitmapFrame.Create(bitmapSource)); + encoder.Save(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToPngStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToPngStreamEncoder.cs new file mode 100644 index 000000000..353138f76 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImageToPngStreamEncoder.cs @@ -0,0 +1,34 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + using System.Windows.Media.Imaging; + + /// + /// Implements an image encoder for PNG format. + /// + public class ImageToPngStreamEncoder : IImageToStreamEncoder + { + /// + public void EncodeToStream(Image image, Stream stream) + { + // The encoder is created on every call (rather than in a constructor) + // b/c the encoder has thread affinity + var encoder = new PngBitmapEncoder(); + var bitmapSource = BitmapSource.Create( + image.Width, + image.Height, + 96, + 96, + image.PixelFormat.ToWindowsMediaPixelFormat(), + null, + image.ImageData, + image.Stride * image.Height, + image.Stride); + encoder.Frames.Add(BitmapFrame.Create(bitmapSource)); + encoder.Save(stream); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImagingOperators.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImagingOperators.cs new file mode 100644 index 000000000..bc9db9009 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/ImagingOperators.cs @@ -0,0 +1,67 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + /// + /// Implements stream operator methods for Imaging. + /// + public static partial class ImagingOperators + { + /// + /// Encodes an image to a JPEG format. + /// + /// A producer of images to encode. + /// JPEG quality to use. + /// An optional delivery policy. + /// A producer that generates the JPEG images. + public static IProducer> EncodeJpeg(this IProducer> source, int quality = 90, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new ImageToJpegStreamEncoder { QualityLevel = quality }, deliveryPolicy); + } + + /// + /// Encodes an image to a PNG format. + /// + /// A producer of images to encoder. + /// An optional delivery policy. + /// A producer that generates the PNG images. + public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new ImageToPngStreamEncoder(), deliveryPolicy); + } + + /// + /// Decodes an encoded image. + /// + /// A producer of encoded images to decode. + /// An optional delivery policy. + /// A producer that generates the decoded images. + public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(new ImageFromStreamDecoder(), deliveryPolicy); + } + + /// + /// Encodes a depth image to a PNG format. + /// + /// A producer of depth images to encode. + /// An optional delivery policy. + /// A producer that generates the PNG-encoded depth images. + public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(new DepthImageToPngStreamEncoder(), deliveryPolicy); + } + + /// + /// Decodes an encoded depth image. + /// + /// A producer of encoded depth images to decode. + /// An optional delivery policy. + /// A producer that generates the decoded depth images. + public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(new DepthImageFromStreamDecoder(), deliveryPolicy); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/Microsoft.Psi.Imaging.Windows.csproj b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/Microsoft.Psi.Imaging.Windows.csproj index 730b2bfd8..761a15724 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/Microsoft.Psi.Imaging.Windows.csproj +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/Microsoft.Psi.Imaging.Windows.csproj @@ -24,6 +24,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PixelFormatHelpers.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PixelFormatHelpers.cs new file mode 100644 index 000000000..2b70870a8 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PixelFormatHelpers.cs @@ -0,0 +1,80 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using MediaPixelFormat = System.Windows.Media.PixelFormat; + + /// + /// Implements helper functions for manipulating instances. + /// + public static class PixelFormatHelpers + { + /// + /// Converts from \psi imaging pixel format to Windows.Media pixel format. + /// + /// The \psi imaging pixel format. + /// The corresponding Windows.Media pixel format. + public static MediaPixelFormat ToWindowsMediaPixelFormat(this PixelFormat pixelFormat) + { + if (pixelFormat == PixelFormat.Undefined) + { + throw new InvalidOperationException("Cannot convert the Undefined pixel format to a Windows.Media format."); + } + else if (pixelFormat == PixelFormat.BGR_24bpp) + { + return System.Windows.Media.PixelFormats.Bgr24; + } + else if (pixelFormat == PixelFormat.Gray_16bpp) + { + return System.Windows.Media.PixelFormats.Gray16; + } + else if (pixelFormat == PixelFormat.Gray_8bpp) + { + return System.Windows.Media.PixelFormats.Gray8; + } + else + { + return System.Windows.Media.PixelFormats.Bgr32; + } + } + + /// + /// Converts Windows.Media pixel format to \psi imaging pixel format. + /// + /// The Windows.Media pixel format. + /// The corresponding \psi imaging pixel format. + public static PixelFormat ToPixelFormat(this MediaPixelFormat mediaPixelFormat) + { + if (mediaPixelFormat == System.Windows.Media.PixelFormats.Bgr24) + { + return PixelFormat.BGR_24bpp; + } + else if (mediaPixelFormat == System.Windows.Media.PixelFormats.Gray16) + { + return PixelFormat.Gray_16bpp; + } + else if (mediaPixelFormat == System.Windows.Media.PixelFormats.Gray8) + { + return PixelFormat.Gray_8bpp; + } + else if (mediaPixelFormat == System.Windows.Media.PixelFormats.Bgr32) + { + return PixelFormat.BGRX_32bpp; + } + else if (mediaPixelFormat == System.Windows.Media.PixelFormats.Bgra32) + { + return PixelFormat.BGRA_32bpp; + } + else if (mediaPixelFormat == System.Windows.Media.PixelFormats.Rgba64) + { + return PixelFormat.RGBA_64bpp; + } + else + { + throw new NotSupportedException($"The {mediaPixelFormat} pixel format is not supported for Microsoft.Psi.Imaging"); + } + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PsiImaging.cs b/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PsiImaging.cs deleted file mode 100644 index 059bb400a..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging.Windows/PsiImaging.cs +++ /dev/null @@ -1,60 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - using System.Windows.Media.Imaging; - - /// - /// Implements stream operator methods for Imaging. - /// - public static partial class ImagingOperators - { - /// - /// Converts from an Image to a compressed (encoded) image. - /// - /// Source image to encode. - /// Method to perform encoding. - /// An optional delivery policy. - /// Returns a producer that generates the encoded images. - public static IProducer> Encode(this IProducer> source, Func encoderFn, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ImageEncoder(source.Out.Pipeline, encoderFn), deliveryPolicy); - } - - /// - /// Converts from an Image to a compressed JPEG image. - /// - /// Source image to compress. - /// JPEG quality to use. - /// An optional delivery policy. - /// Returns a producer that generates the JPEG images. - public static IProducer> EncodeJpeg(this IProducer> source, int quality = 90, DeliveryPolicy> deliveryPolicy = null) - { - return Encode(source, () => new JpegBitmapEncoder { QualityLevel = quality }, deliveryPolicy); - } - - /// - /// Converts from an Image to a compressed PNG image. - /// - /// Source image to compress. - /// An optional delivery policy. - /// Returns a producer that generates the PNG images. - public static IProducer> EncodePng(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) - { - return Encode(source, () => new PngBitmapEncoder(), deliveryPolicy); - } - - /// - /// Decodes an image that was previously encoded. - /// - /// Source image to compress. - /// An optional delivery policy. - /// Returns a producer that generates the decoded images. - public static IProducer> Decode(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ImageDecoder(source.Out.Pipeline), deliveryPolicy); - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/DepthImage.cs b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImage.cs new file mode 100644 index 000000000..852171181 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImage.cs @@ -0,0 +1,330 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.Diagnostics; + using System.Drawing; + using System.Drawing.Imaging; + using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; + + /// + /// Represents a depth image, stored in unmanaged memory. + /// + /// Using this class it is possible as to allocate a new depth image in unmanaged memory, + /// as to just wrap provided pointer to unmanaged memory, where an image is stored. + [Serializer(typeof(DepthImage.CustomSerializer))] + public class DepthImage : ImageBase + { + /// + /// Initializes a new instance of the class. + /// + /// The unmanaged array containing the image. + /// Depth image width in pixels. + /// Depth image height in pixels. + /// Depth image stride (line size in bytes). + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public DepthImage(UnmanagedBuffer unmanagedBuffer, int width, int height, int stride) + : base(unmanagedBuffer, width, height, stride, PixelFormat.Gray_16bpp) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Pointer to image data in unmanaged memory. + /// Depth image width in pixels. + /// Depth image height in pixels. + /// Depth image stride (line size in bytes). + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public DepthImage(IntPtr imageData, int width, int height, int stride) + : base(imageData, width, height, stride, PixelFormat.Gray_16bpp) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Depth image width in pixels. + /// Depth image height in pixels. + public DepthImage(int width, int height) + : base(width, height, PixelFormat.Gray_16bpp) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Depth image width in pixels. + /// Depth image height in pixels. + /// Depth image stride (line size in bytes). + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public DepthImage(int width, int height, int stride) + : base(width, height, stride, PixelFormat.Gray_16bpp) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Locked bitmap data. + /// Indicates whether a copy is made (default is false). + /// + /// When the parameter is false (default), the depth image simply wraps + /// the bitmap data. As such, the bitmap data must stay locked for the duration of using the object. + /// + /// If the parameter is set to true, a copy of the bitmap + /// data is made, and the bitmap data can be released right after the has been constructed. + /// + /// + public DepthImage(BitmapData bitmapData, bool makeCopy = false) + : base(bitmapData, makeCopy) + { + CheckPixelFormat(bitmapData.PixelFormat); + } + + /// + /// Create a new from a specified bitmap. + /// + /// A bitmap to create the depth image from. + /// A new depth image, which contains a copy of the specified bitmap. + public static DepthImage CreateFrom(Bitmap bitmap) + { + CheckPixelFormat(bitmap.PixelFormat); + + DepthImage depthImage = null; + BitmapData sourceData = bitmap.LockBits( + new Rectangle(0, 0, bitmap.Width, bitmap.Height), + ImageLockMode.ReadOnly, + bitmap.PixelFormat); + + try + { + depthImage = new DepthImage(sourceData, true); + } + finally + { + bitmap.UnlockBits(sourceData); + } + + return depthImage; + } + + /// + /// Copies the depth image contents from a specified source locked bitmap data. + /// + /// Source locked bitmap data. + /// The method copies data from the specified bitmap into the depth image. + /// The depth image must be allocated and must have the same size as the specified + /// bitmap data. + public void CopyFrom(BitmapData bitmapData) + { + CheckPixelFormat(bitmapData.PixelFormat); + this.UnmanagedBuffer.CopyFrom(bitmapData.Scan0, this.UnmanagedBuffer.Size); + } + + /// + /// Copies the depth image contents from a specified bitmap. + /// + /// A bitmap to copy from. + /// The method copies data from the specified bitmap into the image. + /// The image must be allocated and must have the same size. + public void CopyFrom(Bitmap bitmap) + { + BitmapData bitmapData = bitmap.LockBits( + new Rectangle(0, 0, this.Width, this.Height), + ImageLockMode.ReadWrite, + PixelFormatHelper.ToSystemPixelFormat(this.PixelFormat)); + try + { + this.UnmanagedBuffer.CopyFrom(bitmapData.Scan0, this.UnmanagedBuffer.Size); + } + finally + { + bitmap.UnlockBits(bitmapData); + } + } + + /// + /// Copies the depth image from a specified source depth image of the same size. + /// + /// Source depth image to copy the depth image from. + /// The method copies the current depth image from the specified source depth image. + /// The size of the images must be the same. + public void CopyFrom(DepthImage source) + { + source.CopyTo(this); + } + + /// + /// Copies the depth image from a specified source image of the same size and format. + /// + /// Source image to copy the depth image from. + /// The method copies the current depth image from the specified source image. + /// The size of the images must be the same, and the source image must have format. + public void CopyFrom(Image source) + { + source.CopyTo(this); + } + + /// + /// Decodes a specified encoded depth image with a specified decoder into the current depth image. + /// + /// The encoded depth image to decode. + /// The depth image decoder to use. + /// The depth image width, height and pixel format must match. The method should not be called concurrently. + public void DecodeFrom(EncodedDepthImage encodedDepthImage, IDepthImageFromStreamDecoder depthImageDecoder) + { + if (encodedDepthImage.Width != this.Width || encodedDepthImage.Height != this.Height || encodedDepthImage.PixelFormat != this.PixelFormat) + { + throw new InvalidOperationException("Cannot decode from an encoded depth image that has a different width, height, or pixel format."); + } + + depthImageDecoder.DecodeFromStream(encodedDepthImage.ToStream(), this); + } + + /// + /// Encodes the depth image using a specified encoder. + /// + /// The depth image encoder to use. + /// A new, corresponding encoded depth image. + public EncodedDepthImage Encode(IDepthImageToStreamEncoder depthImageEncoder) + { + var encodedDepthImage = new EncodedDepthImage(this.Width, this.Height); + encodedDepthImage.EncodeFrom(this, depthImageEncoder); + return encodedDepthImage; + } + + /// + /// Copies the depth image into a target depth image of the same size. + /// + /// Target depth image to copy this depth image to. + /// The method copies the current depth image into the specified depth image. + /// The size of the images must be the same. + public void CopyTo(DepthImage target) + { + this.CopyTo(target.ImageData, target.Width, target.Height, target.Stride, target.PixelFormat); + } + + /// + /// Copies the depth image into a target image of the same size. + /// + /// Target image to copy this depth image to. + /// The method copies the current depth image into the specified image. + /// The size of the images must be the same. The method implements a translation of pixel formats. + public void CopyTo(Image target) + { + this.CopyTo(target.ImageData, target.Width, target.Height, target.Stride, target.PixelFormat); + } + + /// + /// Sets a pixel in the depth image. + /// + /// Pixel's X coordinate. + /// Pixel's Y coordinate. + /// Gray value to set pixel to. + public void SetPixel(int x, int y, int gray) + { + if (x < 0 || y < 0 || x >= this.Width || y >= this.Height) + { + return; + } + + unsafe + { + byte* src = (byte*)this.ImageData.ToPointer(); + int pixelOffset = x * this.BitsPerPixel / 8 + y * this.Stride; + switch (this.PixelFormat) + { + case PixelFormat.Gray_16bpp: + src[pixelOffset + 0] = (byte)((gray >> 8) & 0xff); + src[pixelOffset + 1] = (byte)(gray & 0xff); + break; + case PixelFormat.Gray_8bpp: + case PixelFormat.BGRA_32bpp: + case PixelFormat.BGR_24bpp: + case PixelFormat.BGRX_32bpp: + case PixelFormat.RGBA_64bpp: + throw new InvalidOperationException(ExceptionDescriptionUnexpectedPixelFormat); + case PixelFormat.Undefined: + default: + throw new ArgumentException(ExceptionDescriptionUnexpectedPixelFormat); + } + } + } + + /// + public override ImageBase CreateEmptyOfSameSize() + { + return new DepthImage(this.Width, this.Height); + } + + private static void CheckPixelFormat(System.Drawing.Imaging.PixelFormat pixelFormat) + { + if (pixelFormat != System.Drawing.Imaging.PixelFormat.Format16bppGrayScale) + { + throw new InvalidOperationException( + $"Depth images can only be constructed from bitmaps with {nameof(System.Drawing.Imaging.PixelFormat.Format16bppGrayScale)} format."); + } + } + + /// + /// Custom serializer used for reading/writing depth images. + /// + public class CustomSerializer : ImageBase.CustomSerializer + { + private static IDepthImageCompressor depthImageCompressor = null; + + /// + /// Configure the type of compression to use when serializing depth images. Default is no compression. + /// + /// Compressor to be used. + public static void ConfigureCompression(IDepthImageCompressor depthImageCompressor) + { + CustomSerializer.depthImageCompressor = depthImageCompressor; + } + + /// + public override void Serialize(BufferWriter writer, DepthImage instance, SerializationContext context) + { + DepthCompressionMethod depthCompressionMethod = (depthImageCompressor == null) ? DepthCompressionMethod.None : depthImageCompressor.DepthCompressionMethod; + Serializer.Serialize(writer, depthCompressionMethod, context); + if (depthCompressionMethod == DepthCompressionMethod.None) + { + base.Serialize(writer, instance, context); + } + else + { + depthImageCompressor.Serialize(writer, instance, context); + } + } + + /// + public override void Deserialize(BufferReader reader, ref DepthImage target, SerializationContext context) + { + var depthCompressionMethod = DepthCompressionMethod.None; + if (this.Schema.Version >= 4) + { + Serializer.Deserialize(reader, ref depthCompressionMethod, context); + } + + if (depthCompressionMethod == DepthCompressionMethod.None) + { + base.Deserialize(reader, ref target, context); + } + else + { + depthImageCompressor.Deserialize(reader, ref target, context); + } + } + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageDecoder.cs new file mode 100644 index 000000000..c3fe58daa --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageDecoder.cs @@ -0,0 +1,35 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi; + using Microsoft.Psi.Components; + + /// + /// Component that decodes a depth image using a specified . + /// + public class DepthImageDecoder : ConsumerProducer, Shared> + { + private readonly IDepthImageFromStreamDecoder decoder; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// The depth image decoder to use. + public DepthImageDecoder(Pipeline pipeline, IDepthImageFromStreamDecoder decoder) + : base(pipeline) + { + this.decoder = decoder; + } + + /// + protected override void Receive(Shared sharedEncodedDepthImage, Envelope envelope) + { + using var sharedDepthImage = DepthImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height); + sharedDepthImage.Resource.DecodeFrom(sharedEncodedDepthImage.Resource, this.decoder); + this.Out.Post(sharedDepthImage, envelope.OriginatingTime); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageEncoder.cs new file mode 100644 index 000000000..48f6229cc --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImageEncoder.cs @@ -0,0 +1,36 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi; + using Microsoft.Psi.Components; + + /// + /// Component that encodes an image using a specified . + /// + public class DepthImageEncoder : ConsumerProducer, Shared> + { + private readonly IDepthImageToStreamEncoder encoder; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// The depth image encoder to use. + public DepthImageEncoder(Pipeline pipeline, IDepthImageToStreamEncoder encoder) + : base(pipeline) + { + this.encoder = encoder; + } + + /// + protected override void Receive(Shared sharedDepthImage, Envelope e) + { + using var sharedEncodedDepthImage = EncodedDepthImagePool.GetOrCreate( + sharedDepthImage.Resource.Width, sharedDepthImage.Resource.Height); + sharedEncodedDepthImage.Resource.EncodeFrom(sharedDepthImage.Resource, this.encoder); + this.Out.Post(sharedEncodedDepthImage, e.OriginatingTime); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/DepthImagePool.cs b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImagePool.cs new file mode 100644 index 000000000..9a2f7b0d9 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/DepthImagePool.cs @@ -0,0 +1,53 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.Drawing; + using System.Drawing.Imaging; + + /// + /// Provides a pool of shared depth images. + /// + public static class DepthImagePool + { + private static readonly KeyedSharedPool Instance = + new KeyedSharedPool(key => new DepthImage(key.width, key.height)); + + /// + /// Gets or creates a depth image from the pool. + /// + /// The requested image width. + /// The requested image height. + /// A shared depth image from the pool. + public static Shared GetOrCreate(int width, int height) + { + return Instance.GetOrCreate((width, height)); + } + + /// + /// Gets or creates a depth image from the pool and initializes it with a managed object. + /// + /// A bitmap from which to copy the image data. + /// A shared depth image from the pool containing a copy of the image data from . + public static Shared GetOrCreateFrom(Bitmap bitmap) + { + BitmapData sourceData = bitmap.LockBits( + new Rectangle(0, 0, bitmap.Width, bitmap.Height), + ImageLockMode.ReadOnly, + bitmap.PixelFormat); + Shared sharedDepthImage = null; + try + { + sharedDepthImage = GetOrCreate(bitmap.Width, bitmap.Height); + sharedDepthImage.Resource.CopyFrom(sourceData); + } + finally + { + bitmap.UnlockBits(sourceData); + } + + return sharedDepthImage; + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImage.cs b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImage.cs new file mode 100644 index 000000000..2300a078f --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImage.cs @@ -0,0 +1,126 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.IO; + + /// + /// Defines an encoded depth image. + /// + public class EncodedDepthImage : IDisposable + { + /// + /// The memory stream storing the encoded bytes. + /// + private MemoryStream stream; + + /// + /// Initializes a new instance of the class. + /// + /// Width of encoded depth image in pixels. + /// Height of encoded depth image in pixels. + public EncodedDepthImage(int width, int height) + { + this.Width = width; + this.Height = height; + this.PixelFormat = PixelFormat.Gray_16bpp; + this.stream = new MemoryStream(); + } + + /// + /// Initializes a new instance of the class. + /// + /// Width of image in pixels. + /// Height of image in pixels. + /// Byte array used to initialize the image data. + public EncodedDepthImage(int width, int height, byte[] contents) + { + this.Width = width; + this.Height = height; + this.PixelFormat = PixelFormat.Gray_16bpp; + this.stream = new MemoryStream(); + this.stream.Write(contents, 0, contents.Length); + this.stream.Position = 0; + } + + /// + /// Gets the width of the depth image in pixels. + /// + public int Width { get; } + + /// + /// Gets the height of the depth image in pixels. + /// + public int Height { get; } + + /// + /// Gets the pixel format for the depth image. + /// + public PixelFormat PixelFormat { get; } + + /// + /// Releases the depth image. + /// + public void Dispose() + { + this.stream.Dispose(); + this.stream = null; + } + + /// + /// Returns the image data as stream. + /// + /// A new stream containing the image data. + public Stream ToStream() + { + // This method will only fail if the internal buffer is not set to be publicly + // visible, but we create the memory stream ourselves so this should not be an issue + if (!this.stream.TryGetBuffer(out ArraySegment buffer)) + { + throw new InvalidOperationException("The internal buffer is not publicly visible"); + } + + return new MemoryStream(buffer.Array, buffer.Offset, buffer.Count); + } + + /// + /// Returns the depth image data as a byte array. + /// + /// Byte array containing the image data. + public byte[] GetBuffer() + { + return this.stream.GetBuffer(); + } + + /// + /// Encodes a specified depth image with a specified encoder into the current encoded image. + /// + /// The depth image to encode. + /// The depth image encoder to use. + /// The depth image width, height and pixel format must match. The method should not be called concurrently. + public void EncodeFrom(DepthImage depthImage, IDepthImageToStreamEncoder depthImageEncoder) + { + if (depthImage.Width != this.Width || depthImage.Height != this.Height || depthImage.PixelFormat != this.PixelFormat) + { + throw new InvalidOperationException("Cannot encode from an image that has a different width, height, or pixel format."); + } + + this.stream.Position = 0; + depthImageEncoder.EncodeToStream(depthImage, this.stream); + } + + /// + /// Decodes the depth image using a specified decoder. + /// + /// The depth image decoder to use. + /// A new, corresponding decoded depth image. + public DepthImage Decode(IDepthImageFromStreamDecoder depthImageDecoder) + { + var depthImage = new DepthImage(this.Width, this.Height); + depthImage.DecodeFrom(this, depthImageDecoder); + return depthImage; + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImagePool.cs b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImagePool.cs new file mode 100644 index 000000000..be70c6a94 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedDepthImagePool.cs @@ -0,0 +1,25 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + /// + /// Provides a pool of shared encoded depth images. + /// + public static class EncodedDepthImagePool + { + private static readonly KeyedSharedPool Instance = + new KeyedSharedPool(key => new EncodedDepthImage(key.width, key.height)); + + /// + /// Gets or creates an encoded depth image from the pool. + /// + /// A shared encoded depth image from the pool. + /// The requested encoded depth image width. + /// The requested encoded depth image height. + public static Shared GetOrCreate(int width, int height) + { + return Instance.GetOrCreate((width, height)); + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImage.cs b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImage.cs index c3faf9abf..ec4c7f225 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImage.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImage.cs @@ -5,48 +5,54 @@ namespace Microsoft.Psi.Imaging { using System; using System.IO; - using System.Runtime.InteropServices; - using System.Runtime.InteropServices.ComTypes; + using System.Runtime.Serialization; /// /// Defines an encoded image. /// public class EncodedImage : IDisposable { - private MemoryStream stream; + /// + /// The pixel format was added as a private optional field backed property + /// in order to maintain back-compatibility with an earlier version where + /// no pixel format was stored on the image. + /// + [OptionalField] + private readonly PixelFormat pixelFormat; /// - /// Initializes a new instance of the class. + /// The memory stream storing the encoded bytes. /// - public EncodedImage() - { - this.stream = new MemoryStream(); - } + private MemoryStream stream; /// /// Initializes a new instance of the class. /// - /// Width of image in pixels. - /// Height of image in pixels. - /// Byte array used to initialize the image data. - public EncodedImage(int width, int height, byte[] contents) + /// Width of encoded image in pixels. + /// Height of encoded image in pixels. + /// Pixel format of the encoded image. + public EncodedImage(int width, int height, PixelFormat pixelFormat) { this.Width = width; this.Height = height; + this.pixelFormat = pixelFormat; this.stream = new MemoryStream(); - this.stream.Write(contents, 0, contents.Length); - this.stream.Position = 0; } /// /// Gets the width of the image in pixels. /// - public int Width { get; internal set; } + public int Width { get; } /// /// Gets the height of the image in pixels. /// - public int Height { get; internal set; } + public int Height { get; } + + /// + /// Gets the pixel format for the encoded image. + /// + public PixelFormat PixelFormat => this.pixelFormat; /// /// Releases the image. @@ -60,11 +66,17 @@ public void Dispose() /// /// Returns the image data as stream. /// - /// Stream containing the image data. - public Stream GetStream() + /// A new stream containing the image data. + public Stream ToStream() { - this.stream.Position = 0; - return this.stream; + // This method will only fail if the internal buffer is not set to be publicly + // visible, but we create the memory stream ourselves so this should not be an issue + if (!this.stream.TryGetBuffer(out ArraySegment buffer)) + { + throw new InvalidOperationException("The internal buffer is not publicly visible"); + } + + return new MemoryStream(buffer.Array, buffer.Offset, buffer.Count, false); } /// @@ -77,15 +89,32 @@ public byte[] GetBuffer() } /// - /// Compresses an image using the specified encoder. + /// Encodes a specified image with a specified encoder into the current encoded image. + /// + /// The image to encode. + /// The image encoder to use. + /// The image width, height and pixel format must match. The method should not be called concurrently. + public void EncodeFrom(Image image, IImageToStreamEncoder imageEncoder) + { + if (image.Width != this.Width || image.Height != this.Height || image.PixelFormat != this.PixelFormat) + { + throw new InvalidOperationException("Cannot encode from an image that has a different width, height, or pixel format."); + } + + this.stream.Position = 0; + imageEncoder.EncodeToStream(image, this.stream); + } + + /// + /// Decodes the image using a specified decoder. /// - /// Image to compress. - /// Encoder to use to compress. - public void EncodeFrom(Image image, Action encodeFn) + /// The image decoder to use. + /// A new, corresponding decoded image. + public Image Decode(IImageFromStreamDecoder imageDecoder) { - encodeFn(image, this.GetStream()); - this.Width = image.Width; - this.Height = image.Height; + var image = new Image(this.Width, this.Height, this.PixelFormat); + image.DecodeFrom(this, imageDecoder); + return image; } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImagePool.cs b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImagePool.cs index 484bf9c5f..f021d8af0 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImagePool.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/EncodedImagePool.cs @@ -8,15 +8,19 @@ namespace Microsoft.Psi.Imaging /// public static class EncodedImagePool { - private static readonly SharedPool Instance = new SharedPool(() => new EncodedImage(), 10); + private static readonly KeyedSharedPool Instance = + new KeyedSharedPool(key => new EncodedImage(key.width, key.height, key.pixelFormat)); /// /// Gets or creates an encoded image from the pool. /// /// A shared encoded image from the pool. - public static Shared GetOrCreate() + /// The requested encoded image width. + /// The requested encoded image height. + /// The requested encoded image pixel format. + public static Shared GetOrCreate(int width, int height, PixelFormat pixelFormat) { - return Instance.GetOrCreate(); + return Instance.GetOrCreate((width, height, pixelFormat)); } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageCompressor.cs new file mode 100644 index 000000000..b3c68b841 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageCompressor.cs @@ -0,0 +1,51 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; + + /// + /// Defines type of compression to use when serializing out a . + /// + public enum DepthCompressionMethod + { + /// + /// Use PNG compression. + /// + Png, + + /// + /// Use no compression. + /// + None, + } + + /// + /// Defines a interface for depth image compressors. + /// + public interface IDepthImageCompressor + { + /// + /// Gets or sets the compression method used by compressor. + /// + DepthCompressionMethod DepthCompressionMethod { get; set; } + + /// + /// Serialize compressor. + /// + /// Writer to which to serialize. + /// Depth image instance to serialize. + /// Serialization context. + void Serialize(BufferWriter writer, DepthImage depthImage, SerializationContext context); + + /// + /// Deserialize compressor. + /// + /// Reader from which to deserialize. + /// Target depth image to which to deserialize. + /// Serialization context. + void Deserialize(BufferReader reader, ref DepthImage depthImage, SerializationContext context); + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageFromStreamDecoder.cs new file mode 100644 index 000000000..ef05ba717 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageFromStreamDecoder.cs @@ -0,0 +1,20 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + + /// + /// Defines a class that can decode a depth image. + /// + public interface IDepthImageFromStreamDecoder + { + /// + /// Decodes an encoded depth image from a stream into a given depth image. + /// + /// Stream containing the encoded depth image. + /// The depth image to decode into. + void DecodeFromStream(Stream stream, DepthImage depthImage); + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageToStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageToStreamEncoder.cs new file mode 100644 index 000000000..d4b8dd405 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IDepthImageToStreamEncoder.cs @@ -0,0 +1,20 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + + /// + /// Defines a class that can encode a depth image. + /// + public interface IDepthImageToStreamEncoder + { + /// + /// Encodes a depth image into a stream. + /// + /// Depth image to be encoded. + /// Stream to encode the depth image into. + void EncodeToStream(DepthImage depthImage, Stream stream); + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IImageCompressor.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IImageCompressor.cs index 8c6c597e3..71247d50f 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/IImageCompressor.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IImageCompressor.cs @@ -7,30 +7,28 @@ namespace Microsoft.Psi.Imaging using Microsoft.Psi.Serialization; /// - /// Defines type of compression to use when serializing out an Image. + /// Defines type of compression to use when serializing out a . /// public enum CompressionMethod { /// - /// Use JPEG compression + /// Use JPEG compression. /// - JPEG, + Jpeg, /// - /// Use PNG compression + /// Use PNG compression. /// - PNG, + Png, /// - /// Use no compression + /// Use no compression. /// None, } /// - /// Interface implemented by the system specific assembly. - /// For instance, Microsoft.Psi.Imaging.Windows will define - /// an ImageCompressor that implements this interfaces. + /// Defines a interface for image compressors. /// public interface IImageCompressor { @@ -43,16 +41,16 @@ public interface IImageCompressor /// Serialize compressor. /// /// Writer to which to serialize. - /// Image instance to serialize. + /// Image instance to serialize. /// Serialization context. - void Serialize(BufferWriter writer, Image instance, SerializationContext context); + void Serialize(BufferWriter writer, Image image, SerializationContext context); /// /// Deserialize compressor. /// /// Reader from which to deserialize. - /// Target image to which to deserialize. + /// Target image to which to deserialize. /// Serialization context. - void Deserialize(BufferReader reader, ref Image target, SerializationContext context); + void Deserialize(BufferReader reader, ref Image image, SerializationContext context); } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IImageFromStreamDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IImageFromStreamDecoder.cs new file mode 100644 index 000000000..fbf9cea07 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IImageFromStreamDecoder.cs @@ -0,0 +1,27 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + + /// + /// Defines a class that can decode an image. + /// + public interface IImageFromStreamDecoder + { + /// + /// Decodes an encoded image from a stream into a specified image. + /// + /// Stream containing the encoded image. + /// The image to decode into. + void DecodeFromStream(Stream stream, Image image); + + /// + /// Gets the pixel format of an encoded image from a stream. + /// + /// Stream containing the encoded image. + /// The pixel format. + PixelFormat GetPixelFormat(Stream stream); + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/IImageToStreamEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/IImageToStreamEncoder.cs new file mode 100644 index 000000000..d6205616f --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/IImageToStreamEncoder.cs @@ -0,0 +1,20 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System.IO; + + /// + /// Defines a class that can encode an image. + /// + public interface IImageToStreamEncoder + { + /// + /// Encodes an image into a stream. + /// + /// Image to be encoded. + /// Stream to encode the image into. + void EncodeToStream(Image image, Stream stream); + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/Image.cs b/Sources/Imaging/Microsoft.Psi.Imaging/Image.cs index 5f6980192..4aaa2d07e 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/Image.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/Image.cs @@ -7,28 +7,21 @@ namespace Microsoft.Psi.Imaging using System.Diagnostics; using System.Drawing; using System.Drawing.Imaging; - using System.Threading.Tasks; using Microsoft.Psi.Common; using Microsoft.Psi.Serialization; /// - /// The #Image class represents wrapper of an image in unmanaged memory. Using this class - /// it is possible as to allocate new image in unmanaged memory, as to just wrap provided - /// pointer to unmanaged memory, where an image is stored. + /// Represents an image, stored in unmanaged memory. /// + /// Using this class it is possible as to allocate a new image in unmanaged memory, + /// as to just wrap provided pointer to unmanaged memory, where an image is stored. [Serializer(typeof(Image.CustomSerializer))] - public class Image : IDisposable + public class Image : ImageBase { - private UnmanagedBuffer image; - private int width; - private int height; - private int stride; - private PixelFormat pixelFormat; - /// /// Initializes a new instance of the class. /// - /// Pointer to image data in unmanaged memory. + /// The unmanaged array containing the image. /// Image width in pixels. /// Image height in pixels. /// Image stride (line size in bytes). @@ -36,19 +29,15 @@ public class Image : IDisposable /// Using this constructor, make sure all specified image attributes are correct /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, /// this may lead to exceptions working with the unmanaged memory. - public Image(IntPtr imageData, int width, int height, int stride, PixelFormat pixelFormat) + public Image(UnmanagedBuffer unmanagedBuffer, int width, int height, int stride, PixelFormat pixelFormat) + : base(unmanagedBuffer, width, height, stride, pixelFormat) { - this.image = UnmanagedBuffer.WrapIntPtr(imageData, height * stride); - this.width = width; - this.height = height; - this.stride = stride; - this.pixelFormat = pixelFormat; } /// /// Initializes a new instance of the class. /// - /// The unmanaged array containing the image. + /// Pointer to image data in unmanaged memory. /// Image width in pixels. /// Image height in pixels. /// Image stride (line size in bytes). @@ -56,13 +45,20 @@ public Image(IntPtr imageData, int width, int height, int stride, PixelFormat pi /// Using this constructor, make sure all specified image attributes are correct /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, /// this may lead to exceptions working with the unmanaged memory. - public Image(UnmanagedBuffer image, int width, int height, int stride, PixelFormat pixelFormat) + public Image(IntPtr imageData, int width, int height, int stride, PixelFormat pixelFormat) + : base(imageData, width, height, stride, pixelFormat) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Width of image in pixels. + /// Height of image in pixels. + /// Pixel format. + public Image(int width, int height, PixelFormat pixelFormat) + : base(width, height, pixelFormat) { - this.image = image; - this.width = width; - this.height = height; - this.stride = stride; - this.pixelFormat = pixelFormat; } /// @@ -76,18 +72,7 @@ public Image(UnmanagedBuffer image, int width, int height, int stride, PixelForm /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, /// this may lead to exceptions working with the unmanaged memory. public Image(int width, int height, int stride, PixelFormat pixelFormat) - : this(UnmanagedBuffer.Allocate(height * stride), width, height, stride, pixelFormat) - { - } - - /// - /// Initializes a new instance of the class. - /// - /// Width of image in pixels. - /// Height of image in pixels. - /// Pixel format. - public Image(int width, int height, PixelFormat pixelFormat) - : this(UnmanagedBuffer.Allocate(height * width * GetBytesPerPixel(pixelFormat)), width, height, width * GetBytesPerPixel(pixelFormat), pixelFormat) + : base(width, height, stride, pixelFormat) { } @@ -95,219 +80,178 @@ public Image(int width, int height, PixelFormat pixelFormat) /// Initializes a new instance of the class. /// /// Locked bitmap data. - /// Unlike method, this constructor does not make - /// copy of managed image. This means that managed image must stay locked for the time of using the instance - /// of unamanged image. - public Image(BitmapData bitmapData) + /// Indicates whether a copy is made (default is false). + /// + /// When the parameter is false (default), the image simply wraps + /// the bitmap data. As such, the bitmap data must stay locked for the duration of using the object. + /// + /// If the parameter is set to true, a copy of the bitmap + /// data is made, and the bitmap data can be released right after the has been constructed. + /// + /// + public Image(BitmapData bitmapData, bool makeCopy = false) + : base(bitmapData, makeCopy) { - this.image = UnmanagedBuffer.WrapIntPtr(bitmapData.Scan0, bitmapData.Height * bitmapData.Stride); - this.width = bitmapData.Width; - this.height = bitmapData.Height; - this.stride = bitmapData.Stride; - this.pixelFormat = PixelFormatHelper.FromSystemPixelFormat(bitmapData.PixelFormat); } /// - /// Gets a pointer to image data in unmanaged memory. - /// - public IntPtr ImageData => this.image.Data; - - /// - /// Gets image width in pixels. - /// - public int Width => this.width; - - /// - /// Gets image height in pixels. - /// - public int Height => this.height; - - /// - /// Gets image stride (line size in bytes). + /// Creates a new from a specified bitmap. /// - public int Stride => this.stride; - - /// - /// Gets the size of the image in bytes (stride times height). - /// - public int Size => this.stride * this.height; - - /// - /// Gets the bits per pixel in the image. - /// - public int BitsPerPixel => PixelFormatHelper.GetBitsPerPixel(this.pixelFormat); + /// A bitmap to create the image from. + /// A new image, which contains a copy of the specified bitmap. + public static Image FromBitmap(Bitmap bitmap) + { + Image image = null; - /// - /// Gets image pixel format. - /// - public PixelFormat PixelFormat => this.pixelFormat; + // Make sure that the bitmap format specified is supported (not all Bitmap.PixelFormats are supported) + PixelFormatHelper.FromSystemPixelFormat(bitmap.PixelFormat); - /// - /// Allocate new image in unmanaged memory. - /// - /// Image width. - /// Image height. - /// Image pixel format. - /// Return image allocated in unmanaged memory. - /// Allocate new image with specified attributes in unmanaged memory. - /// - public static Image Create(int width, int height, PixelFormat pixelFormat) - { - int bytesPerPixel = pixelFormat.GetBitsPerPixel() / 8; + BitmapData sourceData = bitmap.LockBits( + new Rectangle(0, 0, bitmap.Width, bitmap.Height), + ImageLockMode.ReadOnly, + bitmap.PixelFormat); - // check image size - if ((width <= 0) || (height <= 0)) + try { - throw new Exception("Invalid image size specified."); + image = new Image(sourceData, true); } - - // calculate stride - int stride = width * bytesPerPixel; - - if (stride % 4 != 0) + finally { - stride += 4 - (stride % 4); + bitmap.UnlockBits(sourceData); } - // allocate memory for the image - return new Image(UnmanagedBuffer.Allocate(stride * height), width, height, stride, pixelFormat); + return image; } /// - /// Create unmanaged image from the specified managed image. + /// Copies the image contents from a specified source locked bitmap data. /// - /// Source locked image data. - /// Returns new unmanaged image, which is a copy of source managed image. - /// The method creates an exact copy of specified managed image, but allocated - /// in unmanaged memory. This means that managed image may be unlocked right after call to this - /// method. - public static Image FromManagedImage(BitmapData imageData) + /// Source locked bitmap data. + /// The method copies data from the specified bitmap into the image. + /// The image must be allocated and must have the same size as the specified + /// bitmap data. + public void CopyFrom(BitmapData bitmapData) { - PixelFormat pixelFormat = PixelFormatHelper.FromSystemPixelFormat(imageData.PixelFormat); - - // allocate memory for the image - return new Image(UnmanagedBuffer.CreateCopyFrom(imageData.Scan0, imageData.Stride * imageData.Height), imageData.Width, imageData.Height, imageData.Stride, pixelFormat); + this.UnmanagedBuffer.CopyFrom(bitmapData.Scan0, this.UnmanagedBuffer.Size); } /// - /// Create unmanaged image from the specified managed image. + /// Copies the image contents from a specified bitmap. /// - /// Source managed image. - /// Returns new unmanaged image, which is a copy of source managed image. - /// The method creates an exact copy of specified managed image, but allocated - /// in unmanaged memory. - public static Image FromManagedImage(Bitmap image) + /// A bitmap to copy from. + /// The method copies data from the specified bitmap into the image. + /// The image must be allocated and must have the same size. + public void CopyFrom(Bitmap bitmap) { - Image dstImage = null; - - // Make sure that the bitmap format specified is supported (not all Bitmap.PixelFormats are supported) - PixelFormatHelper.FromSystemPixelFormat(image.PixelFormat); - - BitmapData sourceData = image.LockBits( - new Rectangle(0, 0, image.Width, image.Height), - ImageLockMode.ReadOnly, - image.PixelFormat); - + BitmapData bitmapData = bitmap.LockBits( + new Rectangle(0, 0, this.Width, this.Height), + ImageLockMode.ReadWrite, + PixelFormatHelper.ToSystemPixelFormat(this.PixelFormat)); try { - dstImage = FromManagedImage(sourceData); + if (this.Stride != bitmapData.Stride) + { + unsafe + { + byte* src = (byte*)bitmapData.Scan0.ToPointer(); + byte* dst = (byte*)this.ImageData.ToPointer(); + for (int y = 0; y < this.Height; y++) + { + Buffer.MemoryCopy(src, dst, this.Stride, this.Stride); + src += bitmapData.Stride; + dst += this.Stride; + } + } + } + else + { + this.UnmanagedBuffer.CopyFrom(bitmapData.Scan0, this.UnmanagedBuffer.Size); + } } finally { - image.UnlockBits(sourceData); + bitmap.UnlockBits(bitmapData); } - - return dstImage; } /// - /// Function to convert RGB color into grayscale. + /// Copies the image from a specified source image of the same size. /// - /// red component (Range=0..255). - /// green component (Range=0..255). - /// Blue component (Range=0..255). - /// Grayscale value (Range=0..255). - public static byte Rgb2Gray(byte r, byte g, byte b) + /// Source image to copy the image from. + /// The method copies the current image from the specified source image. + /// The size of the images must be the same. Some differences in pixel + /// formats are allowed and the method implements a translation of pixel formats. + public void CopyFrom(Image source) { - return (byte)(((4897 * r) + (9617 * g) + (1868 * b)) >> 14); + source.CopyTo(this); } /// - /// Function to convert RGB color into grayscale. + /// Copies the image from a specified source depth image of the same size. /// - /// red component (Range=0..65535). - /// green component (Range=0..65535). - /// Blue component (Range=0..65535). - /// Grayscale value (Range=0..65535). - public static ushort Rgb2Gray(ushort r, ushort g, ushort b) + /// Source depth image to copy the image from. + /// The method copies the current image from the specified source depth image. + /// The size of the images must be the same and the method implements a translation of pixel formats. + public void CopyFrom(DepthImage source) { - return (ushort)(((4897 * r) + (9617 * g) + (1868 * b)) >> 14); + source.CopyTo(this); } /// - /// Set pallete of the 8 bpp indexed image to grayscale. + /// Decodes a specified encoded image with a specified decoder into the current image. /// - /// Image to initialize. - /// The method initializes palette of - /// Format8bppIndexed - /// image with 256 gradients of gray color. - public static void SetGrayscalePalette(Bitmap image) + /// The encoded image to decode. + /// The image decoder to use. + /// The image width, height and pixel format must match. The method should not be called concurrently. + public void DecodeFrom(EncodedImage encodedImage, IImageFromStreamDecoder imageDecoder) { - // check pixel format - if (image.PixelFormat != System.Drawing.Imaging.PixelFormat.Format8bppIndexed) - { - throw new Exception("Source image is not 8 bpp image."); - } - - // get palette - ColorPalette cp = image.Palette; - - // init palette - for (int i = 0; i < 256; i++) + if (encodedImage.Width != this.Width || encodedImage.Height != this.Height || + (encodedImage.PixelFormat != PixelFormat.Undefined && encodedImage.PixelFormat != this.PixelFormat)) { - cp.Entries[i] = Color.FromArgb(i, i, i); + throw new InvalidOperationException("Cannot decode from an encoded image that has a different width, height, or pixel format."); } - // set palette back - image.Palette = cp; + imageDecoder.DecodeFromStream(encodedImage.ToStream(), this); } /// - /// Dispose the object. + /// Encodes the image using a specified encoder. /// - /// Frees unmanaged resources used by the object. The object becomes unusable - /// after that. - /// The method needs to be called only in the case if unmanaged image was allocated - /// using method. In the case if the class instance was created using constructor, - /// this method does not free unmanaged memory. - /// - public void Dispose() + /// The image encoder to use. + /// A new, corresponding encoded image. + public EncodedImage Encode(IImageToStreamEncoder imageEncoder) { - this.image.Dispose(); - this.image = null; + var encodedImage = new EncodedImage(this.Width, this.Height, this.PixelFormat); + encodedImage.EncodeFrom(this, imageEncoder); + return encodedImage; } /// - /// Copy unmanaged image. + /// Copies the image into a specified target image of the same size. /// - /// Destination image to copy this image to. - /// The method copies current unmanaged image to the specified image. - /// Size of the destination image must be exactly the same. Some differences in pixel + /// Target image to copy this image to. + /// The method copies the current image into the specified target image. + /// The size of the images must be the same. Some differences in pixel /// formats are allowed and the method implements a translation of pixel formats. - public void CopyTo(Image destImage) + public void CopyTo(Image target) { - this.CopyTo(destImage.image.Data, destImage.Width, destImage.Height, destImage.Stride, destImage.PixelFormat); + this.CopyTo(target.ImageData, target.Width, target.Height, target.Stride, target.PixelFormat); } /// - /// Copies the psi image to a byte array buffer. + /// Copies the image into a target depth image of the same size. /// - /// The buffer to copy to. - /// The method copies current unmanaged image to the specified buffer. - /// The buffer must be allocated and must have the same size. - public void CopyTo(byte[] destinationBuffer) + /// Target depth image to copy this image to. + /// The method copies the current image into the specified depth image. + /// The size of the images must be the same, and the image must have a pixel format. + public void CopyTo(DepthImage target) { - this.image.CopyTo(destinationBuffer); + if (this.PixelFormat != PixelFormat.Gray_16bpp) + { + throw new InvalidOperationException($"The image must have the {nameof(PixelFormat.Gray_16bpp)} pixel format in order to copy it to a {nameof(DepthImage)}."); + } + + this.CopyTo(target.ImageData, target.Width, target.Height, target.Stride, target.PixelFormat); } /// @@ -321,16 +265,16 @@ public void CopyTo(byte[] destinationBuffer) /// Alpha channel's value. public void SetPixel(int x, int y, int r, int g, int b, int a) { - if (x < 0 || y < 0 || x >= (int)this.width || y >= (int)this.height) + if (x < 0 || y < 0 || x >= this.Width || y >= this.Height) { return; } unsafe { - byte* src = (byte*)this.image.Data.ToPointer(); + byte* src = (byte*)this.ImageData.ToPointer(); int pixelOffset = x * this.BitsPerPixel / 8 + y * this.Stride; - switch (this.pixelFormat) + switch (this.PixelFormat) { case PixelFormat.BGRA_32bpp: src[pixelOffset + 0] = (byte)r; @@ -351,500 +295,108 @@ public void SetPixel(int x, int y, int r, int g, int b, int a) case PixelFormat.Gray_8bpp: src[pixelOffset] = (byte)r; break; + case PixelFormat.RGBA_64bpp: + src[pixelOffset + 0] = (byte)((r >> 8) & 0xff); + src[pixelOffset + 1] = (byte)(r & 0xff); + src[pixelOffset + 2] = (byte)((g >> 8) & 0xff); + src[pixelOffset + 3] = (byte)(g & 0xff); + src[pixelOffset + 4] = (byte)((b >> 8) & 0xff); + src[pixelOffset + 5] = (byte)(b & 0xff); + src[pixelOffset + 6] = (byte)((a >> 8) & 0xff); + src[pixelOffset + 7] = (byte)(a & 0xff); + break; + case PixelFormat.Undefined: default: - throw new Exception("Unsupported type"); - } - } - } - - /// - /// Copies the psi image to an unmanaged buffer. - /// - /// The destination buffer. - /// The destination image width. - /// The destination image height. - /// The destination image stride. - /// The destination pixel format. - public void CopyTo(IntPtr destination, int width, int height, int dstStride, PixelFormat destinationFormat) - { - if ((this.width != width) || (this.height != height)) - { - throw new Exception("Destination image has different size or pixel format."); - } - - // Check if pixel formats are the same. If so, do a straight up copy - if (this.PixelFormat == destinationFormat) - { - if (this.stride == dstStride) - { - this.image.CopyTo(destination, dstStride * height); - } - else - { - unsafe - { - int copyLength = (this.stride < dstStride) ? this.stride : dstStride; - - byte* src = (byte*)this.image.Data.ToPointer(); - byte* dst = (byte*)destination.ToPointer(); - - // copy line by line - for (int i = 0; i < this.height; i++) - { - Buffer.MemoryCopy(src, dst, copyLength, copyLength); - - dst += dstStride; - src += this.stride; - } - } - } - } - else if ((this.pixelFormat == PixelFormat.BGR_24bpp) && - (destinationFormat == PixelFormat.BGRX_32bpp)) - { - unsafe - { - byte* src = (byte*)this.image.Data.ToPointer(); - byte* dst = (byte*)destination.ToPointer(); - Parallel.For(0, this.Height, i => - { - byte* srcCopy = src + (this.stride * i); - byte* dstCopy = dst + (dstStride * i); - for (int j = 0; j < this.width; j++) - { - *dstCopy++ = *srcCopy++; - *dstCopy++ = *srcCopy++; - *dstCopy++ = *srcCopy++; - *dstCopy++ = 255; - } - }); - } - } - else if ((this.pixelFormat == PixelFormat.BGRX_32bpp) && - (destinationFormat == PixelFormat.BGR_24bpp)) - { - unsafe - { - byte* src = (byte*)this.image.Data.ToPointer(); - byte* dst = (byte*)destination.ToPointer(); - Parallel.For(0, this.Height, i => - { - byte* srcCopy = src + (this.stride * i); - byte* dstCopy = dst + (dstStride * i); - for (int j = 0; j < this.width; j++) - { - *dstCopy++ = *srcCopy++; - *dstCopy++ = *srcCopy++; - *dstCopy++ = *srcCopy++; - srcCopy++; - } - }); - } - } - else if ((this.pixelFormat == PixelFormat.BGR_24bpp) && - (destinationFormat == PixelFormat.Gray_8bpp)) - { - unsafe - { - byte* src = (byte*)this.image.Data.ToPointer(); - byte* dst = (byte*)destination.ToPointer(); - - Parallel.For(0, this.Height, i => - { - byte* srcCopy = src + (this.stride * i); - byte* dstCopy = dst + (dstStride * i); - for (int j = 0; j < this.width; j++) - { - *dstCopy++ = Rgb2Gray(*srcCopy, *(srcCopy + 1), *(srcCopy + 2)); - srcCopy += 3; - } - }); - } - } - else - { - this.CopyImageSlow(this.image.Data, this.pixelFormat, destination, dstStride, destinationFormat); - } - } - - /// - /// Copies data from a byte array buffer into the psi image. - /// - /// The buffer to copy from. - /// The method copies data from the specified buffer into the unmanaged image - /// The image must be allocated and must have the same size. - public void CopyFrom(byte[] sourceBuffer) - { - this.image.CopyFrom(sourceBuffer); - } - - /// - /// Copies data from an unmanaged buffer. - /// - /// A pointer to the unmanaged buffer to copy from. - /// The method copies data from the specified buffer into the unmanaged image - /// The image must be allocated and must have the same size. - public void CopyFrom(IntPtr sourcePtr) - { - this.image.CopyFrom(sourcePtr, this.image.Size); - } - - /// - /// Copies data from an unmanaged buffer. - /// - /// A bitmap to copy from. - /// The method copies data from the specified bitmap into the unmanaged image - /// The image must be allocated and must have the same size. - public void CopyFrom(Bitmap bitmap) - { - BitmapData bitmapData = bitmap.LockBits( - new Rectangle(0, 0, this.width, this.height), - ImageLockMode.ReadWrite, - PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat)); - try - { - this.image.CopyFrom(bitmapData.Scan0, this.image.Size); - } - finally - { - bitmap.UnlockBits(bitmapData); - } - } - - /// - /// Create managed image from the unmanaged. - /// - /// Returns managed copy of the unmanaged image. - /// The method creates a managed copy of the unmanaged image with the - /// same size and pixel format (it calls specifying - /// for the makeCopy parameter). - public Bitmap ToManagedImage() - { - return this.ToManagedImage(true); - } - - /// - /// Create managed image from the unmanaged. - /// - /// Make a copy of the unmanaged image or not. - /// Returns managed copy of the unmanaged image. - /// If the is set to , then the method - /// creates a managed copy of the unmanaged image, so the managed image stays valid even when the unmanaged - /// image gets disposed. However, setting this parameter to creates a managed image which is - /// just a wrapper around the unmanaged image. So if unmanaged image is disposed, the - /// managed image becomes no longer valid and accessing it will generate an exception. - public Bitmap ToManagedImage(bool makeCopy) - { - Bitmap dstImage = null; - - try - { - if (!makeCopy) - { - dstImage = new Bitmap(this.width, this.height, this.stride, PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat), this.image.Data); - if (this.pixelFormat == PixelFormat.Gray_8bpp) - { - Image.SetGrayscalePalette(dstImage); - } + throw new ArgumentException(ExceptionDescriptionUnexpectedPixelFormat); } - else - { - // create new image of required format - dstImage = (this.pixelFormat == PixelFormat.Gray_8bpp) ? - Image.CreateGrayscaleImage(this.width, this.height) : - new Bitmap(this.width, this.height, PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat)); - - // lock destination bitmap data - BitmapData dstData = dstImage.LockBits( - new Rectangle(0, 0, this.width, this.height), - ImageLockMode.ReadWrite, - PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat)); - - int dstStride = dstData.Stride; - int lineSize = Math.Min(this.stride, dstStride); - - unsafe - { - byte* dst = (byte*)dstData.Scan0.ToPointer(); - byte* src = (byte*)this.image.Data.ToPointer(); - - if (this.stride != dstStride) - { - // copy image - for (int y = 0; y < this.height; y++) - { - Buffer.MemoryCopy(src, dst, lineSize, lineSize); - dst += dstStride; - src += this.stride; - } - } - else - { - var size = this.stride * this.height; - Buffer.MemoryCopy(src, dst, size, size); - } - } - - // unlock destination images - dstImage.UnlockBits(dstData); - } - - return dstImage; - } - catch (Exception) - { - if (dstImage != null) - { - dstImage.Dispose(); - } - - throw new Exception("The unmanaged image has some invalid properties, which results in failure of converting it to managed image."); } } /// - /// Reads image data as a series of bytes. - /// - /// Number of bytes to read. - /// Offset from start of image data. - /// Array of bytes read. - public byte[] ReadBytes(int count, int offset = 0) - { - return this.image.ReadBytes(count, offset); - } - - /// - /// Creates a copy of the image cropped to the specified dimensions. + /// Sets a pixel in the image. /// - /// The left of the region to crop. - /// The top of the region to crop. - /// The width of the region to crop. - /// The height of the region to crop. - /// The cropped image. - public Shared Crop(int left, int top, int width, int height) + /// Pixel's X coordinate. + /// Pixel's Y coordinate. + /// Gray value to set pixel to. + public void SetPixel(int x, int y, int gray) { - if ((left < 0) || (left > (this.width - 1))) + if (x < 0 || y < 0 || x >= this.Width || y >= this.Height) { - throw new ArgumentOutOfRangeException("left", "left is out of range"); - } - - if ((top < 0) || (top > (this.height - 1))) - { - throw new ArgumentOutOfRangeException("top", "top is out of range"); - } - - if ((width < 0) || ((left + width) > this.width)) - { - throw new ArgumentOutOfRangeException("width", "width is out of range"); - } - - if ((height < 0) || ((top + height) > this.height)) - { - throw new ArgumentOutOfRangeException("height", "height is out of range"); + return; } - // Cropped image will be returned as a new image - original (this) image is not modified - Shared croppedImage = ImagePool.GetOrCreate(width, height, this.pixelFormat); - Debug.Assert(croppedImage.Resource.image.Data != IntPtr.Zero, "Unexpected empty image"); unsafe { - int bytesPerPixel = this.BitsPerPixel / 8; - - // Compute the number of bytes in each line of the crop region - int copyLength = width * bytesPerPixel; - - // Start at top-left of region to crop - byte* src = (byte*)this.image.Data.ToPointer() + (top * this.stride) + (left * bytesPerPixel); - byte* dst = (byte*)croppedImage.Resource.image.Data.ToPointer(); - - // Copy line by line - for (int i = 0; i < height; i++) + byte* src = (byte*)this.ImageData.ToPointer(); + int pixelOffset = x * this.BitsPerPixel / 8 + y * this.Stride; + switch (this.PixelFormat) { - Buffer.MemoryCopy(src, dst, copyLength, copyLength); - - src += this.stride; - dst += croppedImage.Resource.stride; + case PixelFormat.BGRA_32bpp: + src[pixelOffset + 0] = (byte)gray; + src[pixelOffset + 1] = (byte)gray; + src[pixelOffset + 2] = (byte)gray; + src[pixelOffset + 3] = (byte)255; + break; + case PixelFormat.BGR_24bpp: + case PixelFormat.BGRX_32bpp: + src[pixelOffset + 0] = (byte)gray; + src[pixelOffset + 1] = (byte)gray; + src[pixelOffset + 2] = (byte)gray; + break; + case PixelFormat.Gray_16bpp: + src[pixelOffset + 0] = (byte)((gray >> 8) & 0xff); + src[pixelOffset + 1] = (byte)(gray & 0xff); + break; + case PixelFormat.Gray_8bpp: + src[pixelOffset] = (byte)gray; + break; + case PixelFormat.RGBA_64bpp: + src[pixelOffset + 0] = (byte)((gray >> 8) & 0xff); + src[pixelOffset + 1] = (byte)(gray & 0xff); + src[pixelOffset + 2] = (byte)((gray >> 8) & 0xff); + src[pixelOffset + 3] = (byte)(gray & 0xff); + src[pixelOffset + 4] = (byte)((gray >> 8) & 0xff); + src[pixelOffset + 5] = (byte)(gray & 0xff); + src[pixelOffset + 6] = (byte)255; + src[pixelOffset + 7] = (byte)255; + break; + case PixelFormat.Undefined: + default: + throw new Exception(ExceptionDescriptionUnexpectedPixelFormat); } } - - return croppedImage; - } - - private static Bitmap CreateGrayscaleImage(int width, int height) - { - // create new image - Bitmap image = new Bitmap(width, height, PixelFormatHelper.ToSystemPixelFormat(PixelFormat.Gray_8bpp)); - - // set palette to grayscale - SetGrayscalePalette(image); - - // return new image - return image; } - private static int GetBytesPerPixel(PixelFormat pixelFormat) + /// + public override ImageBase CreateEmptyOfSameSize() { - return PixelFormatHelper.GetBytesPerPixel(pixelFormat); - } - - private void CopyImageSlow(IntPtr srcBuffer, PixelFormat srcFormat, IntPtr dstBuffer, int dstStride, PixelFormat dstFormat) - { - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(srcFormat); - int dstBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(dstFormat); - Parallel.For(0, this.Height, i => - { - byte* srcCol = (byte*)srcBuffer.ToPointer() + (i * this.stride); - byte* dstCol = (byte*)dstBuffer.ToPointer() + (i * dstStride); - for (int j = 0; j < this.width; j++) - { - int red = 0; - int green = 0; - int blue = 0; - int alpha = 255; - switch (srcFormat) - { - case PixelFormat.Gray_8bpp: - red = green = blue = srcCol[0]; - break; - - case PixelFormat.Gray_16bpp: - red = green = blue = ((ushort*)srcCol)[0]; - break; - - case PixelFormat.BGR_24bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - break; - - case PixelFormat.BGRX_32bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - break; - - case PixelFormat.BGRA_32bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - alpha = srcCol[3]; - break; - - case PixelFormat.RGBA_64bpp: - red = ((ushort*)srcCol)[0]; - green = ((ushort*)srcCol)[1]; - blue = ((ushort*)srcCol)[2]; - alpha = ((ushort*)srcCol)[3]; - break; - } - - switch (dstFormat) - { - case PixelFormat.Gray_8bpp: - dstCol[0] = Rgb2Gray((byte)red, (byte)green, (byte)blue); - break; - - case PixelFormat.Gray_16bpp: - ((ushort*)dstCol)[0] = Rgb2Gray((ushort)red, (ushort)green, (ushort)blue); - break; - - case PixelFormat.BGR_24bpp: - case PixelFormat.BGRX_32bpp: - dstCol[0] = (byte)blue; - dstCol[1] = (byte)green; - dstCol[2] = (byte)red; - break; - - case PixelFormat.BGRA_32bpp: - dstCol[0] = (byte)blue; - dstCol[1] = (byte)green; - dstCol[2] = (byte)red; - dstCol[3] = (byte)alpha; - break; - - case PixelFormat.RGBA_64bpp: - ((ushort*)dstCol)[0] = (ushort)red; - ((ushort*)dstCol)[1] = (ushort)green; - ((ushort*)dstCol)[2] = (ushort)blue; - ((ushort*)dstCol)[3] = (ushort)alpha; - break; - } - - srcCol += srcBytesPerPixel; - dstCol += dstBytesPerPixel; - } - }); - } + return new Image(this.Width, this.Height, this.PixelFormat); } /// /// Custom serializer used for reading/writing images. /// - public class CustomSerializer : ISerializer + public class CustomSerializer : ImageBase.CustomSerializer { - private const int Version = 4; private static IImageCompressor imageCompressor = null; - private TypeSchema schema; /// - /// Maybe called to initialize type of compression to use. Default is no compression. + /// Configure the type of compression to use when serializing images. Default is no compression. /// - /// Compressor to be used. - public static void ConfigureCompression(IImageCompressor compressor) + /// Compressor to be used. + public static void ConfigureCompression(IImageCompressor imageCompressor) { - imageCompressor = compressor; + CustomSerializer.imageCompressor = imageCompressor; } - /// - /// Initialize custom serializer. - /// - /// Known serializers. - /// Target type schema. - /// Type schema. - public TypeSchema Initialize(KnownSerializers serializers, TypeSchema targetSchema) - { - if (targetSchema == null) - { - TypeMemberSchema[] schemaMembers = new TypeMemberSchema[6] - { - new TypeMemberSchema("compression", typeof(CompressionMethod).AssemblyQualifiedName, false), - new TypeMemberSchema("image", typeof(UnmanagedBuffer).AssemblyQualifiedName, true), - new TypeMemberSchema("width", typeof(int).AssemblyQualifiedName, true), - new TypeMemberSchema("height", typeof(int).AssemblyQualifiedName, true), - new TypeMemberSchema("stride", typeof(int).AssemblyQualifiedName, true), - new TypeMemberSchema("pixelFormat", typeof(Imaging.PixelFormat).AssemblyQualifiedName, true), - }; - var type = typeof(Imaging.Image); - var name = TypeSchema.GetContractName(type, serializers.RuntimeVersion); - this.schema = new TypeSchema(name, TypeSchema.GetId(name), type.AssemblyQualifiedName, TypeFlags.IsCollection, schemaMembers, Version); - } - else - { - this.schema = targetSchema; - } - - return this.schema; - } - - /// - /// Serialize image. - /// - /// Writer to which to serialize. - /// Image instace to serialize. - /// Serialization context. - public void Serialize(BufferWriter writer, Image instance, SerializationContext context) + /// + public override void Serialize(BufferWriter writer, Image instance, SerializationContext context) { CompressionMethod compressionMethod = (imageCompressor == null) ? CompressionMethod.None : imageCompressor.CompressionMethod; Serializer.Serialize(writer, compressionMethod, context); if (compressionMethod == CompressionMethod.None) { - Serializer.Serialize(writer, instance.image, context); - Serializer.Serialize(writer, instance.width, context); - Serializer.Serialize(writer, instance.height, context); - Serializer.Serialize(writer, instance.stride, context); - Serializer.Serialize(writer, instance.pixelFormat, context); + base.Serialize(writer, instance, context); } else { @@ -852,106 +404,24 @@ public void Serialize(BufferWriter writer, Image instance, SerializationContext } } - /// - /// Prepare target for cloning. - /// - /// Called before Clone, to ensure the target is valid. - /// Image instance from which to clone. - /// Image into which to clone. - /// Serialization context. - public void PrepareCloningTarget(Image instance, ref Image target, SerializationContext context) + /// + public override void Deserialize(BufferReader reader, ref Image target, SerializationContext context) { - if (target == null || - target.width != instance.width || - target.height != instance.height || - target.pixelFormat != instance.pixelFormat) + var compressionMethod = CompressionMethod.None; + if (this.Schema.Version >= 4) { - target?.Dispose(); - target = new Image(instance.width, instance.height, instance.pixelFormat); + Serializer.Deserialize(reader, ref compressionMethod, context); } - } - - /// - /// Clone image. - /// - /// Image instance to clone. - /// Target image into which to clone. - /// Serialization context. - public void Clone(Image instance, ref Image target, SerializationContext context) - { - Serializer.Clone(instance.image, ref target.image, context); - Serializer.Clone(instance.width, ref target.width, context); - Serializer.Clone(instance.height, ref target.height, context); - Serializer.Clone(instance.stride, ref target.stride, context); - Serializer.Clone(instance.pixelFormat, ref target.pixelFormat, context); - } - /// - /// Prepare target for deserialization. - /// - /// Called before Deserialize, to ensure the target is valid. - /// Reader being used. - /// Target image into which to deserialize. - /// Serialization context. - public void PrepareDeserializationTarget(BufferReader reader, ref Image target, SerializationContext context) - { - if (target == null) - { - target = (Image)System.Runtime.Serialization.FormatterServices.GetUninitializedObject(typeof(Image)); - } - } - - /// - /// Deserialize image. - /// - /// Buffer reader being used. - /// Target image into which to deserialize. - /// Serialization context. - public void Deserialize(BufferReader reader, ref Image target, SerializationContext context) - { - CompressionMethod methodOfCompression = CompressionMethod.None; - if (this.schema.Version >= 4) + if (compressionMethod == CompressionMethod.None) { - Serializer.Deserialize(reader, ref methodOfCompression, context); - } - - if (methodOfCompression == CompressionMethod.None) - { - Serializer.Deserialize(reader, ref target.image, context); - Serializer.Deserialize(reader, ref target.width, context); - Serializer.Deserialize(reader, ref target.height, context); - Serializer.Deserialize(reader, ref target.stride, context); - if (this.schema.Version <= 2) - { - System.Drawing.Imaging.PixelFormat pixFmt = default(System.Drawing.Imaging.PixelFormat); - Serializer.Deserialize(reader, ref pixFmt, context); - target.pixelFormat = PixelFormatHelper.FromSystemPixelFormat(pixFmt); - } - else - { - Serializer.Deserialize(reader, ref target.pixelFormat, context); - } + base.Deserialize(reader, ref target, context); } else { imageCompressor.Deserialize(reader, ref target, context); } } - - /// - /// Clear image to be reused. - /// - /// Called once the object becomes unused and can be reused as a cloning target. - /// Target image to clear. - /// Serialization context. - public void Clear(ref Image target, SerializationContext context) - { - Serializer.Clear(ref target.image, context); - Serializer.Clear(ref target.width, context); - Serializer.Clear(ref target.height, context); - Serializer.Clear(ref target.stride, context); - Serializer.Clear(ref target.pixelFormat, context); - } } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImageBase.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImageBase.cs new file mode 100644 index 000000000..6ebcd9d50 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImageBase.cs @@ -0,0 +1,671 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.Drawing; + using System.Drawing.Imaging; + using System.Threading.Tasks; + using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; + + /// + /// Represents an image, stored in unmanaged memory. + /// + /// Using this class it is possible as to allocate a new image in unmanaged memory, + /// as to just wrap provided pointer to unmanaged memory, where an image is stored. + public abstract class ImageBase : IDisposable + { + /// + /// Exception message when unexpected pixel format is encountered. + /// + public const string ExceptionDescriptionUnexpectedPixelFormat = "Unexpected pixel format"; + + /// + /// Exception message when source and destination image sizes don't match. + /// + public const string ExceptionDescriptionSourceDestImageMismatch = "Source and destination images must be the same size"; + + private UnmanagedBuffer image; + private int width; + private int height; + private int stride; + private PixelFormat pixelFormat; + + /// + /// Initializes a new instance of the class. + /// + /// The unmanaged array containing the image. + /// Image width in pixels. + /// Image height in pixels. + /// Image stride (line size in bytes). + /// Image pixel format. + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public ImageBase(UnmanagedBuffer image, int width, int height, int stride, PixelFormat pixelFormat) + { + if (pixelFormat == PixelFormat.Undefined) + { + throw new ArgumentException("Cannot create an image with an Undefined pixel format."); + } + + this.image = image; + this.width = width; + this.height = height; + this.stride = stride; + this.pixelFormat = pixelFormat; + } + + /// + /// Initializes a new instance of the class. + /// + /// Pointer to image data in unmanaged memory. + /// Image width in pixels. + /// Image height in pixels. + /// Image stride (line size in bytes). + /// Image pixel format. + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public ImageBase(IntPtr imageData, int width, int height, int stride, PixelFormat pixelFormat) + : this(UnmanagedBuffer.WrapIntPtr(imageData, height * stride), width, height, stride, pixelFormat) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Image width in pixels. + /// Image height in pixels. + /// Image stride (line size in bytes). + /// Image pixel format. + /// Using this constructor, make sure all specified image attributes are correct + /// and correspond to unmanaged memory buffer. If some attributes are specified incorrectly, + /// this may lead to exceptions working with the unmanaged memory. + public ImageBase(int width, int height, int stride, PixelFormat pixelFormat) + : this(UnmanagedBuffer.Allocate(height * stride), width, height, stride, pixelFormat) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Width of image in pixels. + /// Height of image in pixels. + /// Pixel format. + public ImageBase(int width, int height, PixelFormat pixelFormat) + : this(UnmanagedBuffer.Allocate(height * width * pixelFormat.GetBytesPerPixel()), width, height, width * pixelFormat.GetBytesPerPixel(), pixelFormat) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Locked bitmap data. + /// Indicates whether a copy is made (default is false). + /// + /// When the parameter is false (default), the image simply wraps + /// the bitmap data. As such, the bitmap data must stay locked for the duration of using the object. + /// + /// If the parameter is set to true, a copy of the bitmap + /// data is made, and the bitmap data can be released right after the has been constructed. + /// + /// + public ImageBase(BitmapData bitmapData, bool makeCopy = false) + { + this.image = makeCopy ? + UnmanagedBuffer.CreateCopyFrom(bitmapData.Scan0, bitmapData.Stride * bitmapData.Height) : + UnmanagedBuffer.WrapIntPtr(bitmapData.Scan0, bitmapData.Height * bitmapData.Stride); + this.width = bitmapData.Width; + this.height = bitmapData.Height; + this.stride = bitmapData.Stride; + this.pixelFormat = PixelFormatHelper.FromSystemPixelFormat(bitmapData.PixelFormat); + } + + /// + /// Gets a pointer to unmanaged buffer that wraps the image data in unmanaged memory. + /// + public UnmanagedBuffer UnmanagedBuffer => this.image; + + /// + /// Gets a pointer to the image data in unmanaged memory. + /// + public IntPtr ImageData => this.image.Data; + + /// + /// Gets image width in pixels. + /// + public int Width => this.width; + + /// + /// Gets image height in pixels. + /// + public int Height => this.height; + + /// + /// Gets image stride (line size in bytes). + /// + public int Stride => this.stride; + + /// + /// Gets the size of the image in bytes (stride times height). + /// + public int Size => this.stride * this.height; + + /// + /// Gets the bits per pixel in the image. + /// + public int BitsPerPixel => this.pixelFormat.GetBitsPerPixel(); + + /// + /// Gets image pixel format. + /// + public PixelFormat PixelFormat => this.pixelFormat; + + /// + /// Disposes the image. + /// + /// Frees unmanaged resources used by the object. The image becomes unusable after that. + public void Dispose() + { + this.image.Dispose(); + this.image = null; + } + + /// + /// Copies the image to a destination byte array buffer. + /// + /// The destination buffer to copy the image to. + /// The method copies current unmanaged image to the specified buffer. + /// The buffer must be allocated and must have the same size. + public void CopyTo(byte[] destinationBuffer) + { + this.image.CopyTo(destinationBuffer); + } + + /// + /// Copies the image to a destination pointer. + /// + /// The destination pointer to copy the image to. + /// The destination width. + /// The destination height. + /// The destination stride. + /// The destination pixel format. + public void CopyTo(IntPtr destination, int width, int height, int stride, PixelFormat pixelFormat) + { + if ((this.width != width) || (this.height != height)) + { + throw new InvalidOperationException("Destination image has different size."); + } + + // Check if pixel formats are the same. If so, do a straight up copy + if (this.PixelFormat == pixelFormat) + { + if (this.stride == stride) + { + this.image.CopyTo(destination, stride * height); + } + else + { + unsafe + { + int copyLength = (this.stride < stride) ? this.stride : stride; + + byte* src = (byte*)this.image.Data.ToPointer(); + byte* dst = (byte*)destination.ToPointer(); + + // copy line by line + for (int i = 0; i < this.height; i++) + { + Buffer.MemoryCopy(src, dst, copyLength, copyLength); + + dst += stride; + src += this.stride; + } + } + } + } + else if ((this.pixelFormat == PixelFormat.BGR_24bpp) && + (pixelFormat == PixelFormat.BGRX_32bpp || + pixelFormat == PixelFormat.BGRA_32bpp)) + { + unsafe + { + byte* src = (byte*)this.image.Data.ToPointer(); + byte* dst = (byte*)destination.ToPointer(); + Parallel.For(0, this.Height, i => + { + byte* srcCopy = src + (this.stride * i); + byte* dstCopy = dst + (stride * i); + for (int j = 0; j < this.width; j++) + { + *dstCopy++ = *srcCopy++; + *dstCopy++ = *srcCopy++; + *dstCopy++ = *srcCopy++; + *dstCopy++ = 255; + } + }); + } + } + else if ((this.pixelFormat == PixelFormat.BGRX_32bpp) && + (pixelFormat == PixelFormat.BGR_24bpp)) + { + unsafe + { + byte* src = (byte*)this.image.Data.ToPointer(); + byte* dst = (byte*)destination.ToPointer(); + Parallel.For(0, this.Height, i => + { + byte* srcCopy = src + (this.stride * i); + byte* dstCopy = dst + (stride * i); + for (int j = 0; j < this.width; j++) + { + *dstCopy++ = *srcCopy++; + *dstCopy++ = *srcCopy++; + *dstCopy++ = *srcCopy++; + srcCopy++; + } + }); + } + } + else if ((this.pixelFormat == PixelFormat.BGR_24bpp) && + (pixelFormat == PixelFormat.Gray_8bpp)) + { + unsafe + { + byte* src = (byte*)this.image.Data.ToPointer(); + byte* dst = (byte*)destination.ToPointer(); + + Parallel.For(0, this.Height, i => + { + byte* srcCopy = src + (this.stride * i); + byte* dstCopy = dst + (stride * i); + for (int j = 0; j < this.width; j++) + { + *dstCopy++ = Operators.Rgb2Gray(*srcCopy, *(srcCopy + 1), *(srcCopy + 2)); + srcCopy += 3; + } + }); + } + } + else + { + this.CopyImageSlow(this.image.Data, this.pixelFormat, destination, stride, pixelFormat); + } + } + + /// + /// Copies data from a source byte array buffer into the image. + /// + /// The buffer to copy the image from. + /// The method copies data from the specified buffer into the image. + /// The image must be allocated and must have the same size. + public void CopyFrom(byte[] sourceBuffer) + { + this.image.CopyFrom(sourceBuffer); + } + + /// + /// Copies data from a source pointer into the image. + /// + /// A source pointer to copy the image data from. + /// The method copies data from the specified buffer into the unmanaged image + /// The image must be allocated and must have the same size. + public void CopyFrom(IntPtr source) + { + this.image.CopyFrom(source, this.image.Size); + } + + /// + /// Creates a from the image. + /// + /// Indicates whether to make a copy of the image data or not. + /// A corresponding image. + /// If the parameter is set to , then the method + /// creates a copy of the image, so the stays valid even when the + /// image gets disposed. However, setting this parameter to creates a image which is just a wrapper around the image data. In this case, if the image is disposed, the + /// will no longer be valid and accessing it will generate an exception. + public Bitmap ToBitmap(bool makeCopy = true) + { + Bitmap bitmap = null; + + try + { + if (!makeCopy) + { + bitmap = new Bitmap(this.width, this.height, this.stride, PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat), this.image.Data); + if (this.pixelFormat == PixelFormat.Gray_8bpp) + { + Operators.SetGrayscalePalette(bitmap); + } + } + else + { + // create new image of required format + bitmap = new Bitmap(this.width, this.height, PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat)); + if (this.pixelFormat == PixelFormat.Gray_8bpp) + { + // set palette to grayscale + Operators.SetGrayscalePalette(bitmap); + } + + // lock destination bitmap data + BitmapData bitmapData = bitmap.LockBits( + new Rectangle(0, 0, this.width, this.height), + ImageLockMode.ReadWrite, + PixelFormatHelper.ToSystemPixelFormat(this.pixelFormat)); + + int bitmapStride = bitmapData.Stride; + int lineSize = Math.Min(this.stride, bitmapStride); + + unsafe + { + byte* dst = (byte*)bitmapData.Scan0.ToPointer(); + byte* src = (byte*)this.image.Data.ToPointer(); + + if (this.stride != bitmapStride) + { + // copy image + for (int y = 0; y < this.height; y++) + { + Buffer.MemoryCopy(src, dst, lineSize, lineSize); + dst += bitmapStride; + src += this.stride; + } + } + else + { + var size = this.stride * this.height; + Buffer.MemoryCopy(src, dst, size, size); + } + } + + // unlock destination images + bitmap.UnlockBits(bitmapData); + } + + return bitmap; + } + catch (Exception) + { + if (bitmap != null) + { + bitmap.Dispose(); + } + + throw new Exception("The image has some invalid properties, which caused a failure while converting it to managed image."); + } + } + + /// + /// Reads image data as a series of bytes. + /// + /// Number of bytes to read. + /// Offset from start of image data. + /// Array of bytes read. + public byte[] ReadBytes(int count, int offset = 0) + { + return this.image.ReadBytes(count, offset); + } + + /// + /// Creates an empty image of the same size. + /// + /// An empty image of the same size. + public abstract ImageBase CreateEmptyOfSameSize(); + + private void CopyImageSlow(IntPtr sourceIntPtr, PixelFormat sourceFormat, IntPtr destinationIntPtr, int destinationStride, PixelFormat destinationFormat) + { + unsafe + { + int srcBytesPerPixel = sourceFormat.GetBytesPerPixel(); + int dstBytesPerPixel = destinationFormat.GetBytesPerPixel(); + Parallel.For( + 0, + this.Height, + i => + { + byte* srcCol = (byte*)sourceIntPtr.ToPointer() + (i * this.stride); + byte* dstCol = (byte*)destinationIntPtr.ToPointer() + (i * destinationStride); + for (int j = 0; j < this.width; j++) + { + int red = 0; + int green = 0; + int blue = 0; + int alpha = 255; + switch (sourceFormat) + { + case PixelFormat.Gray_8bpp: + red = green = blue = srcCol[0]; + break; + + case PixelFormat.Gray_16bpp: + red = green = blue = ((ushort*)srcCol)[0]; + break; + + case PixelFormat.BGR_24bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + break; + + case PixelFormat.BGRX_32bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + break; + + case PixelFormat.BGRA_32bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + alpha = srcCol[3]; + break; + + case PixelFormat.RGBA_64bpp: + red = ((ushort*)srcCol)[0]; + green = ((ushort*)srcCol)[1]; + blue = ((ushort*)srcCol)[2]; + alpha = ((ushort*)srcCol)[3]; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(ExceptionDescriptionUnexpectedPixelFormat); + } + + switch (destinationFormat) + { + case PixelFormat.Gray_8bpp: + dstCol[0] = Operators.Rgb2Gray((byte)red, (byte)green, (byte)blue); + break; + + case PixelFormat.Gray_16bpp: + ((ushort*)dstCol)[0] = Operators.Rgb2Gray((ushort)red, (ushort)green, (ushort)blue); + break; + + case PixelFormat.BGR_24bpp: + case PixelFormat.BGRX_32bpp: + dstCol[0] = (byte)blue; + dstCol[1] = (byte)green; + dstCol[2] = (byte)red; + break; + + case PixelFormat.BGRA_32bpp: + dstCol[0] = (byte)blue; + dstCol[1] = (byte)green; + dstCol[2] = (byte)red; + dstCol[3] = (byte)alpha; + break; + + case PixelFormat.RGBA_64bpp: + ((ushort*)dstCol)[0] = (ushort)red; + ((ushort*)dstCol)[1] = (ushort)green; + ((ushort*)dstCol)[2] = (ushort)blue; + ((ushort*)dstCol)[3] = (ushort)alpha; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(ExceptionDescriptionUnexpectedPixelFormat); + } + + srcCol += srcBytesPerPixel; + dstCol += dstBytesPerPixel; + } + }); + } + } + + /// + /// Custom serializer used for reading/writing images. + /// + /// The type of image to custom serialize. + public abstract class CustomSerializer : ISerializer + where TImage : ImageBase + { + private const int Version = 4; + + /// + /// Gets the type schema. + /// + protected TypeSchema Schema { get; private set; } + + /// + /// Initialize custom serializer. + /// + /// Known serializers. + /// Target type schema. + /// Type schema. + public virtual TypeSchema Initialize(KnownSerializers serializers, TypeSchema targetSchema) + { + if (targetSchema == null) + { + TypeMemberSchema[] schemaMembers = new TypeMemberSchema[6] + { + new TypeMemberSchema("compression", typeof(CompressionMethod).AssemblyQualifiedName, false), + new TypeMemberSchema("image", typeof(UnmanagedBuffer).AssemblyQualifiedName, true), + new TypeMemberSchema("width", typeof(int).AssemblyQualifiedName, true), + new TypeMemberSchema("height", typeof(int).AssemblyQualifiedName, true), + new TypeMemberSchema("stride", typeof(int).AssemblyQualifiedName, true), + new TypeMemberSchema("pixelFormat", typeof(PixelFormat).AssemblyQualifiedName, true), + }; + var type = typeof(TImage); + var name = TypeSchema.GetContractName(type, serializers.RuntimeVersion); + this.Schema = new TypeSchema(name, TypeSchema.GetId(name), type.AssemblyQualifiedName, TypeFlags.IsClass, schemaMembers, Version); + } + else + { + this.Schema = targetSchema; + } + + return this.Schema; + } + + /// + /// Serialize image. + /// + /// Writer to which to serialize. + /// Image instance to serialize. + /// Serialization context. + public virtual void Serialize(BufferWriter writer, TImage instance, SerializationContext context) + { + Serializer.Serialize(writer, instance.image, context); + Serializer.Serialize(writer, instance.width, context); + Serializer.Serialize(writer, instance.height, context); + Serializer.Serialize(writer, instance.stride, context); + Serializer.Serialize(writer, instance.pixelFormat, context); + } + + /// + /// Prepare target for cloning. + /// + /// Called before Clone, to ensure the target is valid. + /// Image instance from which to clone. + /// Image into which to clone. + /// Serialization context. + public virtual void PrepareCloningTarget(TImage instance, ref TImage target, SerializationContext context) + { + if (target == null || + target.width != instance.width || + target.height != instance.height || + target.pixelFormat != instance.pixelFormat) + { + target?.Dispose(); + target = (TImage)instance.CreateEmptyOfSameSize(); + } + } + + /// + /// Clone image. + /// + /// Image instance to clone. + /// Target image into which to clone. + /// Serialization context. + public virtual void Clone(TImage instance, ref TImage target, SerializationContext context) + { + Serializer.Clone(instance.image, ref target.image, context); + Serializer.Clone(instance.width, ref target.width, context); + Serializer.Clone(instance.height, ref target.height, context); + Serializer.Clone(instance.stride, ref target.stride, context); + Serializer.Clone(instance.pixelFormat, ref target.pixelFormat, context); + } + + /// + /// Prepare target for deserialization. + /// + /// Called before Deserialize, to ensure the target is valid. + /// Reader being used. + /// Target image into which to deserialize. + /// Serialization context. + public virtual void PrepareDeserializationTarget(BufferReader reader, ref TImage target, SerializationContext context) + { + if (target == null) + { + target = (TImage)System.Runtime.Serialization.FormatterServices.GetUninitializedObject(typeof(TImage)); + } + } + + /// + /// Deserialize image. + /// + /// Buffer reader being used. + /// Target image into which to deserialize. + /// Serialization context. + public virtual void Deserialize(BufferReader reader, ref TImage target, SerializationContext context) + { + Serializer.Deserialize(reader, ref target.image, context); + Serializer.Deserialize(reader, ref target.width, context); + Serializer.Deserialize(reader, ref target.height, context); + Serializer.Deserialize(reader, ref target.stride, context); + if (this.Schema.Version <= 2) + { + System.Drawing.Imaging.PixelFormat pixFmt = default; + Serializer.Deserialize(reader, ref pixFmt, context); + target.pixelFormat = PixelFormatHelper.FromSystemPixelFormat(pixFmt); + } + else + { + Serializer.Deserialize(reader, ref target.pixelFormat, context); + } + } + + /// + /// Clear image to be reused. + /// + /// Called once the object becomes unused and can be reused as a cloning target. + /// Target image to clear. + /// Serialization context. + public virtual void Clear(ref TImage target, SerializationContext context) + { + Serializer.Clear(ref target.image, context); + Serializer.Clear(ref target.width, context); + Serializer.Clear(ref target.height, context); + Serializer.Clear(ref target.stride, context); + Serializer.Clear(ref target.pixelFormat, context); + } + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImageDecoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImageDecoder.cs new file mode 100644 index 000000000..49eb2ea77 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImageDecoder.cs @@ -0,0 +1,50 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using Microsoft.Psi; + using Microsoft.Psi.Components; + + /// + /// Component that decodes an image using a specified . + /// + public class ImageDecoder : ConsumerProducer, Shared> + { + private readonly IImageFromStreamDecoder decoder; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// The image decoder to use. + public ImageDecoder(Pipeline pipeline, IImageFromStreamDecoder decoder) + : base(pipeline) + { + this.decoder = decoder; + } + + /// + protected override void Receive(Shared sharedEncodedImage, Envelope envelope) + { + // The code below maintains back-compatibility with encoded images which did not store the pixel format + // on the instance, but only in the stream. If the pixel format is unknown, we call upon the decoder to + // retrieve the pixel format. This might be less performant, but enables decoding in the right format + // even from older versions of encoded images. + var pixelFormat = sharedEncodedImage.Resource.PixelFormat == PixelFormat.Undefined ? + this.decoder.GetPixelFormat(sharedEncodedImage.Resource.ToStream()) : sharedEncodedImage.Resource.PixelFormat; + + // If the decoder does not return a valid pixel format, we throw an exception. + if (pixelFormat == PixelFormat.Undefined) + { + throw new ArgumentException("The encoded image does not contain a supported pixel format."); + } + + using var sharedImage = ImagePool.GetOrCreate( + sharedEncodedImage.Resource.Width, sharedEncodedImage.Resource.Height, pixelFormat); + sharedImage.Resource.DecodeFrom(sharedEncodedImage.Resource, this.decoder); + this.Out.Post(sharedImage, envelope.OriginatingTime); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImageEncoder.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImageEncoder.cs new file mode 100644 index 000000000..dd3f76a63 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImageEncoder.cs @@ -0,0 +1,36 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using Microsoft.Psi; + using Microsoft.Psi.Components; + + /// + /// Component that encodes an image using a specified . + /// + public class ImageEncoder : ConsumerProducer, Shared> + { + private readonly IImageToStreamEncoder encoder; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// The image encoder to use. + public ImageEncoder(Pipeline pipeline, IImageToStreamEncoder encoder) + : base(pipeline) + { + this.encoder = encoder; + } + + /// + protected override void Receive(Shared sharedImage, Envelope e) + { + using var sharedEncodedImage = EncodedImagePool.GetOrCreate( + sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + sharedEncodedImage.Resource.EncodeFrom(sharedImage.Resource, this.encoder); + this.Out.Post(sharedEncodedImage, e.OriginatingTime); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImageExtensions.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImageExtensions.cs new file mode 100644 index 000000000..29224ece0 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImageExtensions.cs @@ -0,0 +1,1610 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.Drawing; + using System.Drawing.Imaging; + using System.Threading.Tasks; + + /// + /// Sampling mode used by various imaging operators. + /// + public enum SamplingMode + { + /// + /// Sampling mode using nearest neighbor interpolation. + /// + Point, + + /// + /// Sampling mode using bilinear interpolation. + /// + Bilinear, + + /// + /// Sampling mode using bicubic interpolation. + /// + Bicubic, + } + + /// + /// Thresholding modes. + /// + public enum Threshold + { + /// + /// Thresholds pixels such that: + /// dst(x,y) = maxvalue if (src(x,y)>threshold) + /// = 0 otherwise + /// + Binary, + + /// + /// Thresholds pixels such that: + /// dst(x,y) = 0 if (src(x,y)>threshold) + /// = maxvalue otherwise + /// + BinaryInv, + + /// + /// Thresholds pixels such that: + /// dst(x,y) = threshold if (src(x,y)>threshold) + /// = src(x,y) otherwise + /// + Truncate, + + /// + /// Thresholds pixels such that: + /// dst(x,y) = src(x,y) if (src(x,y)>threshold) + /// = 0 otherwise + /// + ToZero, + + /// + /// Thresholds pixels such that: + /// dst(x,y) = 0 if (src(x,y)>threshold) + /// = src(x,y) otherwise + /// + ToZeroInv, + } + + /// + /// Axis along which to flip an image. + /// + public enum FlipMode + { + /// + /// Leave image unflipped + /// + None, + + /// + /// Flips image along the horizontal axis + /// + AlongHorizontalAxis, + + /// + /// Flips image along the vertical axis + /// + AlongVerticalAxis, + } + + /// + /// Various imaging operators. + /// + public static partial class Operators + { + /// + /// Constant passed to ExtractChannel to indcate the red channel should be extracted. + /// + public const int ExtractRedChannel = 0; + + /// + /// Constant passed to ExtractChannel to indcate the green channel should be extracted. + /// + public const int ExtractGreenChannel = 1; + + /// + /// Constant passed to ExtractChannel to indicate the blue channel should be extracted. + /// + public const int ExtractBlueChannel = 2; + + /// + /// Constant passed to ExtractChannel to indicate the alpha channel should be extracted. + /// + public const int ExtractAlphaChannel = 3; + + /// + /// Set palette of the 8 bpp indexed image to grayscale. + /// + /// Image to initialize. + /// The method initializes palette of + /// Format8bppIndexed + /// image with 256 gradients of gray color. + public static void SetGrayscalePalette(Bitmap bitmap) + { + // check pixel format + if (bitmap.PixelFormat != System.Drawing.Imaging.PixelFormat.Format8bppIndexed) + { + throw new ArgumentException("Source image is not 8 bpp image."); + } + + // get palette + ColorPalette cp = bitmap.Palette; + + // init palette + for (int i = 0; i < 256; i++) + { + cp.Entries[i] = Color.FromArgb(i, i, i); + } + + // set palette back + bitmap.Palette = cp; + } + + /// + /// Function to convert RGB color into grayscale. + /// + /// red component (Range=0..255). + /// green component (Range=0..255). + /// Blue component (Range=0..255). + /// Grayscale value (Range=0..255). + public static byte Rgb2Gray(byte r, byte g, byte b) + { + return (byte)(((4897 * r) + (9617 * g) + (1868 * b)) >> 14); + } + + /// + /// Function to convert RGB color into grayscale. + /// + /// red component (Range=0..65535). + /// green component (Range=0..65535). + /// Blue component (Range=0..65535). + /// Grayscale value (Range=0..65535). + public static ushort Rgb2Gray(ushort r, ushort g, ushort b) + { + return (ushort)(((4897 * r) + (9617 * g) + (1868 * b)) >> 14); + } + + /// + /// Flips an image along a specified axis. + /// + /// Image to flip. + /// Axis along which to flip. + /// A new flipped image. + public static Image Flip(this Image image, FlipMode mode) + { + var destImage = new Image(image.Width, image.Height, image.PixelFormat); + image.Flip(destImage, mode); + return destImage; + } + + /// + /// Flips an image along a specified axis. + /// + /// Image to flip. + /// Destination image where to store results. + /// Axis along which to flip. + public static void Flip(this Image image, Image destImage, FlipMode mode) + { + if (image.Width != destImage.Width || image.Height != destImage.Height) + { + throw new ArgumentException("Destination image's width/height must match the source image width/height"); + } + + if (image.PixelFormat == PixelFormat.Gray_16bpp) + { + // We can't handle this through GDI. + unsafe + { + int sourceBytesPerPixel = image.PixelFormat.GetBytesPerPixel(); + int destinationBytesPerPixel = destImage.PixelFormat.GetBytesPerPixel(); + byte* sourceRow = (byte*)image.ImageData.ToPointer(); + byte* destinationRow = (byte*)destImage.ImageData.ToPointer(); + int ystep = destImage.Stride; + if (mode == FlipMode.AlongHorizontalAxis) + { + destinationRow += destImage.Stride * (image.Height - 1); + ystep = -destImage.Stride; + } + + int xstep = destinationBytesPerPixel; + int xoffset = 0; + if (mode == FlipMode.AlongVerticalAxis) + { + xoffset = destinationBytesPerPixel * (destImage.Width - 1); + xstep = -destinationBytesPerPixel; + } + + for (int i = 0; i < image.Height; i++) + { + byte* sourceColumn = sourceRow; + byte* destinationColumn = destinationRow + xoffset; + for (int j = 0; j < image.Width; j++) + { + ((ushort*)destinationColumn)[0] = ((ushort*)sourceColumn)[0]; + sourceColumn += sourceBytesPerPixel; + destinationColumn += xstep; + } + + sourceRow += image.Stride; + destinationRow += ystep; + } + } + } + else + { + // This block handles the rest of the pixel format cases + using var bitmap = new Bitmap(image.Width, image.Height, PixelFormatHelper.ToSystemPixelFormat(image.PixelFormat)); + using var graphics = Graphics.FromImage(bitmap); + switch (mode) + { + case FlipMode.AlongHorizontalAxis: + graphics.TranslateTransform(0.0f, image.Height - 1); + graphics.ScaleTransform(1.0f, -1.0f); + break; + + case FlipMode.AlongVerticalAxis: + graphics.TranslateTransform(image.Width - 1, 0.0f); + graphics.ScaleTransform(-1.0f, 1.0f); + break; + + case FlipMode.None: + break; + } + + using (var destinationImage = image.ToBitmap()) + { + graphics.DrawImage(destinationImage, new Point(0, 0)); + } + + destImage.CopyFrom(bitmap); + } + } + + /// + /// Resizes an image by the specified scale factors using the specified sampling mode. + /// + /// Image to resize. + /// Scale factor to apply in X direction. + /// Scale factor to apply in Y direction. + /// Sampling mode for sampling of pixels. + /// Returns a new image scaled by the specified scale factors. + public static Image Scale(this Image image, float scaleX, float scaleY, SamplingMode mode) + { + int scaledWidth = (int)Math.Abs(image.Width * scaleX); + int scaledHeight = (int)Math.Abs(image.Height * scaleY); + Image destImage = new Image(scaledWidth, scaledHeight, image.PixelFormat); + image.Scale(destImage, scaleX, scaleY, mode); + return destImage; + } + + /// + /// Resizes an image by the specified scale factors using the specified sampling mode. + /// + /// Image to resize. + /// Image to store scaled results. + /// Scale factor to apply in X direction. + /// Scale factor to apply in Y direction. + /// Sampling mode for sampling of pixels. + public static void Scale(this Image image, Image destImage, float scaleX, float scaleY, SamplingMode mode) + { + if (scaleX == 0.0 || scaleY == 0.0) + { + throw new System.ArgumentOutOfRangeException("Unexpected scale factors"); + } + + int scaledWidth = (int)Math.Abs(image.Width * scaleX); + int scaledHeight = (int)Math.Abs(image.Height * scaleY); + if (destImage.Width != scaledWidth || destImage.Height != scaledHeight) + { + throw new ArgumentException($"Destination image must be size={scaledWidth}x{scaledHeight}."); + } + + if (image.PixelFormat == PixelFormat.Gray_16bpp) + { + throw new System.InvalidOperationException( + "Scaling 16bpp images is not currently supported. " + + "Convert to a supported format such as color or 8bpp grayscale first."); + } + + int dstWidth = (int)(image.Width * scaleX); + int dstHeight = (int)(image.Height * scaleY); + using var bitmap = new Bitmap(dstWidth, dstHeight, PixelFormatHelper.ToSystemPixelFormat(image.PixelFormat)); + using var graphics = Graphics.FromImage(bitmap); + graphics.CompositingMode = System.Drawing.Drawing2D.CompositingMode.SourceCopy; + switch (mode) + { + case SamplingMode.Point: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighSpeed; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.NearestNeighbor; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighSpeed; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighSpeed; + break; + + case SamplingMode.Bilinear: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.Bilinear; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; + break; + + case SamplingMode.Bicubic: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; + break; + } + + graphics.ScaleTransform(scaleX, scaleY); + + using (var managedimg = image.ToBitmap()) + { + graphics.DrawImage(managedimg, new Point(0, 0)); + } + + destImage.CopyFrom(bitmap); + } + + /// + /// Resize an image. + /// + /// Image to resize. + /// Image where to store resized source image. + /// Final width of desired output. + /// Final height of desired output. + /// Method for sampling pixels when rescaling. + public static void Resize(this Image image, Image destImage, float finalWidth, float finalHeight, SamplingMode samplingMode = SamplingMode.Bilinear) + { + float scaleX = finalWidth / image.Width; + float scaleY = finalHeight / image.Height; + image.Scale(destImage, scaleX, scaleY, samplingMode); + } + + /// + /// Rotates an image. + /// + /// Image to rotate. + /// Number of degrees to rotate in counter clockwise direction. + /// Pixel resampling method. + /// Rotated image. + public static Image Rotate(this Image image, float angleInDegrees, SamplingMode mode) + { + int rotatedWidth; + int rotatedHeight; + float originx; + float originy; + DetermineRotatedWidthHeight(image.Width, image.Height, angleInDegrees, out rotatedWidth, out rotatedHeight, out originx, out originy); + Image rotatedImage = new Image(rotatedWidth, rotatedHeight, image.PixelFormat); + image.Rotate(rotatedImage, angleInDegrees, mode); + return rotatedImage; + } + + /// + /// Rotates an image. + /// + /// Image to rotate. + /// Image where to store rotated source image. + /// Number of degrees to rotate in counter clockwise direction. + /// Pixel resampling method. + public static void Rotate(this Image image, Image destImage, float angleInDegrees, SamplingMode mode) + { + int rotatedWidth; + int rotatedHeight; + float originx, originy; + DetermineRotatedWidthHeight(image.Width, image.Height, angleInDegrees, out rotatedWidth, out rotatedHeight, out originx, out originy); + using var bitmap = new Bitmap(rotatedWidth, rotatedHeight, PixelFormatHelper.ToSystemPixelFormat(image.PixelFormat)); + using var graphics = Graphics.FromImage(bitmap); + graphics.CompositingMode = System.Drawing.Drawing2D.CompositingMode.SourceCopy; + switch (mode) + { + case SamplingMode.Point: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighSpeed; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.NearestNeighbor; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighSpeed; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighSpeed; + break; + + case SamplingMode.Bilinear: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.Bilinear; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; + break; + + case SamplingMode.Bicubic: + graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; + graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic; + graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; + graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; + break; + } + + graphics.TranslateTransform(-originx, -originy); + graphics.RotateTransform(angleInDegrees); + + using (var managedimg = image.ToBitmap()) + { + graphics.DrawImage(managedimg, new Point(0, 0)); + } + + destImage.CopyFrom(bitmap); + } + + /// + /// Creates a copy of the image cropped to the specified dimensions. + /// + /// Image to crop. + /// The left of the region to crop. + /// The top of the region to crop. + /// The width of the region to crop. + /// The height of the region to crop. + /// The cropped image. + public static Image Crop(this Image image, int left, int top, int width, int height) + { + Image croppedImage = new Image(width, height, image.PixelFormat); + image.Crop(croppedImage, left, top, width, height); + return croppedImage; + } + + /// + /// Creates a copy of the image cropped to the specified dimensions. + /// + /// Image to crop. + /// Destination image that cropped area is copied to. + /// The left of the region to crop. + /// The top of the region to crop. + /// The width of the region to crop. + /// The height of the region to crop. + /// The cropped image. + public static Image Crop(this Image image, Image croppedImage, int left, int top, int width, int height) + { + if (croppedImage.Width < width) + { + throw new ArgumentOutOfRangeException("destImage.Width", "destination image width is too small"); + } + + if (croppedImage.Height < height) + { + throw new ArgumentOutOfRangeException("destImage.Height", "destination image height is too small"); + } + + if ((left < 0) || (left >= image.Width)) + { + throw new ArgumentOutOfRangeException("left", "left is out of range"); + } + + if ((top < 0) || (top >= image.Height)) + { + throw new ArgumentOutOfRangeException("top", "top is out of range"); + } + + if ((width < 0) || ((left + width) > image.Width)) + { + throw new ArgumentOutOfRangeException("width", "width is out of range"); + } + + if ((height < 0) || ((top + height) > image.Height)) + { + throw new ArgumentOutOfRangeException("height", "height is out of range"); + } + + // Cropped image will be returned as a new image - original (this) image is not modified + System.Diagnostics.Debug.Assert(croppedImage.ImageData != IntPtr.Zero, "Unexpected empty image"); + unsafe + { + int bytesPerPixel = image.BitsPerPixel / 8; + + // Compute the number of bytes in each line of the crop region + int copyLength = width * bytesPerPixel; + + // Start at top-left of region to crop + byte* src = (byte*)image.ImageData.ToPointer() + (top * image.Stride) + (left * bytesPerPixel); + byte* dst = (byte*)croppedImage.ImageData.ToPointer(); + + // Copy line by line + for (int i = 0; i < height; i++) + { + Buffer.MemoryCopy(src, dst, copyLength, copyLength); + + src += image.Stride; + dst += croppedImage.Stride; + } + } + + return croppedImage; + } + + /// + /// Creates a copy of the depth image cropped to the specified dimensions. + /// + /// Depth image to crop. + /// The left of the region to crop. + /// The top of the region to crop. + /// The width of the region to crop. + /// The height of the region to crop. + /// The cropped depth image. + public static DepthImage Crop(this DepthImage image, int left, int top, int width, int height) + { + DepthImage croppedImage = new DepthImage(width, height, image.Stride); + image.Crop(croppedImage, left, top, width, height); + return croppedImage; + } + + /// + /// Creates a copy of the image cropped to the specified dimensions. + /// + /// Image to crop. + /// Destination image that cropped area is copied to. + /// The left of the region to crop. + /// The top of the region to crop. + /// The width of the region to crop. + /// The height of the region to crop. + public static void Crop(this DepthImage image, DepthImage croppedImage, int left, int top, int width, int height) + { + if (croppedImage.Width < width) + { + throw new ArgumentOutOfRangeException("destImage.Width", "destination image width is too small"); + } + + if (croppedImage.Height < height) + { + throw new ArgumentOutOfRangeException("destImage.Height", "destination image height is too small"); + } + + if ((left < 0) || (left > (image.Width - 1))) + { + throw new ArgumentOutOfRangeException("left", "left is out of range"); + } + + if ((top < 0) || (top > (image.Height - 1))) + { + throw new ArgumentOutOfRangeException("top", "top is out of range"); + } + + if ((width < 0) || ((left + width) > image.Width)) + { + throw new ArgumentOutOfRangeException("width", "width is out of range"); + } + + if ((height < 0) || ((top + height) > image.Height)) + { + throw new ArgumentOutOfRangeException("height", "height is out of range"); + } + + // Cropped image will be returned as a new image - original (this) image is not modified + System.Diagnostics.Debug.Assert(croppedImage.ImageData != IntPtr.Zero, "Unexpected empty image"); + unsafe + { + int bytesPerPixel = image.BitsPerPixel / 8; + + // Compute the number of bytes in each line of the crop region + int copyLength = width * bytesPerPixel; + + // Start at top-left of region to crop + byte* src = (byte*)image.ImageData.ToPointer() + (top * image.Stride) + (left * bytesPerPixel); + byte* dst = (byte*)croppedImage.ImageData.ToPointer(); + + // Copy line by line + for (int i = 0; i < height; i++) + { + Buffer.MemoryCopy(src, dst, copyLength, copyLength); + + src += image.Stride; + dst += croppedImage.Stride; + } + } + } + + /// + /// Convert a depth image into a pseudo-colorized image, where more distant pixels are closer to blue, and near pixels are closer to red. + /// + /// Depth image to pseudo-colorize. + /// A tuple indicating the range (MinValue, MaxValue) of the depth values in the image. + /// The pseudo-colorized image in BGR_24bpp format. + public static Image PseudoColorize(this DepthImage depthImage, (ushort MinValue, ushort MaxValue) range) + { + var colorizedImage = new Image(depthImage.Width, depthImage.Height, PixelFormat.BGR_24bpp); + depthImage.PseudoColorize(colorizedImage, range); + return colorizedImage; + } + + /// + /// Convert a depth image into a pseudo-colorized image, where more distant pixels are closer to blue, and near pixels are closer to red. + /// + /// Depth image to pseudo-colorize. + /// Target color image. Must be in BGR_24bpp format. + /// A tuple indicating the range (MinValue, MaxValue) of the depth values in the image. + public static void PseudoColorize(this DepthImage depthImage, Image colorizedImage, (ushort MinValue, ushort MaxValue) range) + { + if (depthImage.Width != colorizedImage.Width || depthImage.Height != colorizedImage.Height) + { + throw new ArgumentException("Destination color image must have same width and height as source depth image."); + } + + if (colorizedImage.PixelFormat != PixelFormat.BGR_24bpp) + { + throw new InvalidOperationException("Only BGR 24bpp pixel format is supported for the destination color image."); + } + + unsafe + { + // Portions adapted from: + // https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/tools/k4aviewer/k4adepthpixelcolorizer.h + Parallel.For(0, depthImage.Height, iy => + { + ushort* src = (ushort*)((byte*)depthImage.ImageData.ToPointer() + (iy * depthImage.Stride)); + byte* dst = (byte*)colorizedImage.ImageData.ToPointer() + (iy * colorizedImage.Stride); + + for (int ix = 0; ix < depthImage.Width; ix++) + { + ushort depth = *src; + + if (depth == 0) + { + dst[0] = 0; + dst[1] = 0; + dst[2] = 0; + + dst += 3; + src += 1; + continue; + } + + // clamp the pixel + ushort clampedDepth = depth > range.MaxValue ? range.MaxValue : depth; + clampedDepth = depth < range.MinValue ? range.MinValue : depth; + + // get the hue + float hue = (clampedDepth - range.MinValue) / (float)(range.MaxValue - range.MinValue); + + // We want to go from blue (at 2/3 in hue space) to red (at 0 in hue space), so + // remap accordingly + // See also: https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:HSV-RGB-comparison.svg + hue = 1 - hue * .2f / .3f; + + float red = .0f; + float green = .0f; + float blue = .0f; + + if (hue < 0.25) + { + red = .0f; + green = hue / 0.25f; + blue = 1.0f; + } + else if (hue < 0.5) + { + red = .0f; + green = 1.0f; + blue = 1.0f - (hue - 0.25f) / 0.25f; + } + else if (hue < 0.75) + { + red = (hue - 0.5f) / 0.25f; + green = 1.0f; + blue = .0f; + } + else + { + red = 1.0f; + green = 1 - (hue - 0.75f) / 0.25f; + blue = .0f; + } + + dst[0] = (byte)(red * 255); + dst[1] = (byte)(green * 255); + dst[2] = (byte)(blue * 255); + + dst += 3; + src += 1; + } + }); + } + } + + /// + /// Determines the dimensions of an image after it has been rotated. + /// + /// Width (in pixels) of original image. + /// Height (in pixels) of original image. + /// Angle (in degrees) of rotation being applied. + /// Outputs the rotated image's width. + /// Outputs the rotated image's height. + /// The X coordinate of the origin after rotation (maybe negative). + /// The Y coordinate of the origin after rotation (maybe negative). + public static void DetermineRotatedWidthHeight(int imageWidth, int imageHeight, float angleInDegrees, out int rotatedWidth, out int rotatedHeight, out float originx, out float originy) + { + float ca = (float)System.Math.Cos(angleInDegrees * System.Math.PI / 180.0f); + float sa = (float)System.Math.Sin(angleInDegrees * System.Math.PI / 180.0f); + float minx = 0.0f; + float miny = 0.0f; + float maxx = 0.0f; + float maxy = 0.0f; + AddRotatedPointToBBox((float)(imageWidth - 1), 0.0f, ref minx, ref miny, ref maxx, ref maxy, ca, sa); + AddRotatedPointToBBox((float)(imageWidth - 1), (float)(imageHeight - 1), ref minx, ref miny, ref maxx, ref maxy, ca, sa); + AddRotatedPointToBBox(0.0f, (float)(imageHeight - 1), ref minx, ref miny, ref maxx, ref maxy, ca, sa); + rotatedWidth = (int)(maxx - minx + 1); + rotatedHeight = (int)(maxy - miny + 1); + originx = minx; + originy = miny; + } + + private static void AddRotatedPointToBBox(float x, float y, ref float minx, ref float miny, ref float maxx, ref float maxy, float ca, float sa) + { + float nx = (x * ca) - (y * sa); + float ny = (x * sa) + (y * ca); + if (nx < minx) + { + minx = nx; + } + + if (nx > maxx) + { + maxx = nx; + } + + if (ny < miny) + { + miny = ny; + } + + if (ny > maxy) + { + maxy = ny; + } + } + } + + /// + /// Set of operators used for drawing on an image. + /// + public static partial class Operators + { + /// + /// Draws a rectangle at the specified pixel coordinates on the image. + /// + /// Image to draw on. + /// Pixel coordinates for rectangle. + /// Color to use for drawing. + /// Width of line. + public static void DrawRectangle(this Image image, Rectangle rect, Color color, int width) + { + using Bitmap bm = image.ToBitmap(false); + using var graphics = Graphics.FromImage(bm); + using var pen = new Pen(new SolidBrush(color)); + pen.Width = width; + graphics.DrawRectangle(pen, rect); + } + + /// + /// Draws a line from point p0 to p1 in pixel coordinates on the image. + /// + /// Image to draw on. + /// Pixel coordinates for start of line. + /// Pixel coordinates for end of line. + /// Color to use for drawing. + /// Width of line. + public static void DrawLine(this Image image, Point p0, Point p1, Color color, int width) + { + using Bitmap bm = image.ToBitmap(false); + using var graphics = Graphics.FromImage(bm); + using var pen = new Pen(new SolidBrush(color)); + pen.Width = width; + graphics.DrawLine(pen, p0, p1); + } + + /// + /// Draws a circle centered at the specified pixel (p0) with the specified radius. + /// + /// Image to draw on. + /// Pixel coordinates for center of circle. + /// Radius of the circle. + /// Color to use for drawing. + /// Width of line. + public static void DrawCircle(this Image image, Point p0, int radius, Color color, int width) + { + using Bitmap bm = image.ToBitmap(false); + using var graphics = Graphics.FromImage(bm); + using var pen = new Pen(new SolidBrush(color)); + pen.Width = width; + graphics.DrawEllipse(pen, p0.X - radius, p0.Y - radius, 2 * radius, 2 * radius); + } + + /// + /// Renders text on the image at the specified pixel (p0). + /// + /// Image to draw on. + /// Text to render. + /// Pixel coordinates for center of circle. + /// Color to use when drawing text. Optional. + /// Name of font to use. Optional. + /// Size of font. Optional. + public static void DrawText(this Image image, string str, Point p0, Color color = default(Color), string font = "Arial", float fontSize = 24.0f) + { + font ??= "Arial"; + using Bitmap bm = image.ToBitmap(false); + using var graphics = Graphics.FromImage(bm); + using Font drawFont = new Font(font, fontSize); + using SolidBrush drawBrush = new SolidBrush(color); + using StringFormat drawFormat = new StringFormat(); + drawFormat.FormatFlags = 0; + graphics.DrawString(str, drawFont, drawBrush, p0.X, p0.Y, drawFormat); + } + } + + /// + /// Set of transforms for copying image data. + /// + public static partial class Operators + { + /// + /// Copies a source image into a destination image using the specified masking image. + /// See for further details. + /// + /// Source image. + /// Destination image. + /// Masking image. If null then ignored. + public static void CopyTo(this Image srcImage, Image destImage, Image maskImage) + { + if (srcImage.Width != destImage.Width || srcImage.Height != destImage.Height) + { + throw new System.Exception(Image.ExceptionDescriptionSourceDestImageMismatch); + } + + Rectangle srcRect = new Rectangle(0, 0, srcImage.Width - 1, srcImage.Height - 1); + srcImage.CopyTo(srcRect, destImage, new Point(0, 0), maskImage); + } + + /// + /// Copies a portion of the source image into a destination image. + /// See for further details. + /// + /// Source image. + /// Destination image. + /// Rectangle to copy. + public static void CopyTo(this Image srcImage, Image destImage, Rectangle rect) + { + if (srcImage.Width != destImage.Width || srcImage.Height != destImage.Height) + { + throw new ArgumentException(Image.ExceptionDescriptionSourceDestImageMismatch); + } + + srcImage.CopyTo(rect, destImage, new Point(rect.Left, rect.Right), null); + } + + /// + /// Copies a portion of a source image into a destination image. + /// See for further details. + /// + /// Source image. + /// Source rectangle to copy from. + /// Destination image. + /// Top left corner of destination image where to copy to. + public static void CopyTo(this Image srcImage, Rectangle srcRect, Image destImage, Point destTopLeftPoint) + { + if (srcImage.Width != destImage.Width || srcImage.Height != destImage.Height) + { + throw new System.Exception(Image.ExceptionDescriptionSourceDestImageMismatch); + } + + srcImage.CopyTo(srcRect, destImage, destTopLeftPoint, null); + } + + /// + /// Copies a portion of a source image into a destination image using the specified masking image. + /// Only pixels from the 'srcImage' inside the 'srcRect' are copied. If a 'maskImage' is specified + /// (it maybe null, in which case no mask is applied) then source pixels are only copied if their + /// corresponding mask pixel is > 0. The copied pixels are placed in the 'destImage' at a rectangle + /// the same size (potentially clipped by the 'destImage' boundaries) located at 'destTopLeftCorner'. + /// + /// The following picture may help clarify. In the picture '2' are pixels that are potentially copied + /// and 'x' are pixels in the 'maskImage' with values > 0. + /// + /// \verbatim + /// srcImage maskImage destImage + /// +-------------------------+ +-------------------------+ +-------------------------+ + /// | srcRect | | (srcRect) | | | + /// | +---------+ | | ........... | | | + /// | |222222222| | | . xxxx.xxxx | CopyTo | | + /// | |222222222| | + | . xxxxxx.xxxxxx | ========> | destTopLeftCorner | + /// | |222222222| | | . xxxxxxx.xxxxx | | O---------+ | + /// | +---------+ | | ........... | | | 2222| | + /// | | | | | | 222222| | + /// +-------------------------+ +-------------------------+ +-----------+---------+---+ + /// dropped pixels => . xxxxxxx. + /// due to being +.........+ + /// outside image + /// boundary + /// \endverbatim + /// . + /// + /// + /// Source image. + /// Source rectangle to copy from. + /// Destination image. + /// Top left corner of destination image where to copy to. + /// Masking image. If null then ignored. + public static void CopyTo(this Image srcImage, Rectangle srcRect, Image destImage, Point destTopLeftCorner, Image maskImage) + { + if (maskImage != null) + { + if (srcImage.Width != maskImage.Width || srcImage.Height != maskImage.Height) + { + throw new ArgumentException("Mask image size must match source image size"); + } + + if (maskImage.PixelFormat != PixelFormat.Gray_8bpp) + { + throw new ArgumentException("Mask image must be of type PixelFormat.Gray_8bpp"); + } + } + + // Clip source rectangle against source image size + int srcX = (srcRect.X < 0) ? 0 : srcRect.X; + int srcY = (srcRect.Y < 0) ? 0 : srcRect.Y; + int srcW = (srcRect.X + srcRect.Width > srcImage.Width) ? (srcImage.Width - srcRect.X) : srcRect.Width; + int srcH = (srcRect.Y + srcRect.Height > srcImage.Height) ? (srcImage.Height - srcRect.Y) : srcRect.Height; + + // Clip destination point against destination image + int dstX = (destTopLeftCorner.X < 0) ? 0 : destTopLeftCorner.X; + int dstY = (destTopLeftCorner.Y < 0) ? 0 : destTopLeftCorner.Y; + dstX = (dstX >= destImage.Width) ? destImage.Width - 1 : dstX; + dstY = (dstY >= destImage.Height) ? destImage.Height - 1 : dstY; + + // Next clip further if rect of that size would lie outside the destination image + srcW = (dstX + srcW > destImage.Width) ? (destImage.Width - dstX) : srcW; + srcH = (dstY + srcH > destImage.Height) ? (destImage.Height - dstY) : srcH; + + PixelFormat srcFormat = srcImage.PixelFormat; + PixelFormat dstFormat = destImage.PixelFormat; + System.IntPtr sourceBuffer = srcImage.ImageData; + System.IntPtr destBuffer = destImage.ImageData; + System.IntPtr maskBuffer = (maskImage != null) ? maskImage.ImageData : System.IntPtr.Zero; + unsafe + { + int srcBytesPerPixel = srcFormat.GetBytesPerPixel(); + int dstBytesPerPixel = dstFormat.GetBytesPerPixel(); + int maskBytesPerPixel = PixelFormat.Gray_8bpp.GetBytesPerPixel(); + byte* srcRow = (byte*)sourceBuffer.ToPointer() + (srcY * srcImage.Stride) + (srcX * srcBytesPerPixel); + byte* dstRow = (byte*)destBuffer.ToPointer() + (dstY * destImage.Stride) + (dstX * dstBytesPerPixel); + byte* maskRow = null; + if (maskImage != null) + { + maskRow = (byte*)maskBuffer.ToPointer() + (srcY * maskImage.Stride) + (srcX * maskBytesPerPixel); + } + + for (int i = 0; i < srcH; i++) + { + byte* srcCol = srcRow; + byte* dstCol = dstRow; + byte* maskCol = maskRow; + for (int j = 0; j < srcW; j++) + { + bool copyPixel = true; + if (maskImage != null) + { + if (*maskCol == 0) + { + copyPixel = false; + } + } + + if (copyPixel) + { + int red = 0; + int green = 0; + int blue = 0; + int alpha = 255; + switch (srcFormat) + { + case PixelFormat.Gray_8bpp: + red = green = blue = srcCol[0]; + break; + + case PixelFormat.Gray_16bpp: + red = green = blue = ((ushort*)srcCol)[0]; + break; + + case PixelFormat.BGR_24bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + break; + + case PixelFormat.BGRX_32bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + break; + + case PixelFormat.BGRA_32bpp: + blue = srcCol[0]; + green = srcCol[1]; + red = srcCol[2]; + alpha = srcCol[3]; + break; + + case PixelFormat.RGBA_64bpp: + red = ((ushort*)srcCol)[0]; + green = ((ushort*)srcCol)[1]; + blue = ((ushort*)srcCol)[2]; + alpha = ((ushort*)srcCol)[3]; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + switch (dstFormat) + { + case PixelFormat.Gray_8bpp: + dstCol[0] = Operators.Rgb2Gray((byte)red, (byte)green, (byte)blue); + break; + + case PixelFormat.Gray_16bpp: + ((ushort*)dstCol)[0] = Operators.Rgb2Gray((ushort)red, (ushort)green, (ushort)blue); + break; + + case PixelFormat.BGR_24bpp: + case PixelFormat.BGRX_32bpp: + dstCol[0] = (byte)blue; + dstCol[1] = (byte)green; + dstCol[2] = (byte)red; + dstCol[3] = 255; + break; + + case PixelFormat.BGRA_32bpp: + dstCol[0] = (byte)blue; + dstCol[1] = (byte)green; + dstCol[2] = (byte)red; + dstCol[3] = (byte)alpha; + break; + + case PixelFormat.RGBA_64bpp: + ((ushort*)dstCol)[0] = (ushort)red; + ((ushort*)dstCol)[1] = (ushort)green; + ((ushort*)dstCol)[2] = (ushort)blue; + ((ushort*)dstCol)[3] = (ushort)alpha; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + } + + srcCol += srcBytesPerPixel; + dstCol += dstBytesPerPixel; + maskCol += maskBytesPerPixel; + } + + srcRow += srcImage.Stride; + dstRow += destImage.Stride; + if (maskImage != null) + { + maskRow += maskImage.Stride; + } + } + } + } + } + + /// + /// Basic color transforms on images. + /// + public static partial class Operators + { + /// + /// Inverts each color component in an image. + /// + /// Source image to invert. + /// Returns an new image with the inverted results. + public static Image Invert(this Image srcImage) + { + Image invertedImage = new Image(srcImage.Width, srcImage.Height, srcImage.PixelFormat); + srcImage.Invert(invertedImage); + return invertedImage; + } + + /// + /// Inverts each color component in an image. + /// + /// Source image to invert. + /// Destination image where to store inverted results. + public static void Invert(this Image srcImage, Image destImage) + { + if (srcImage.Width != destImage.Width || srcImage.Height != destImage.Height) + { + throw new System.Exception(Image.ExceptionDescriptionSourceDestImageMismatch); + } + + unsafe + { + int srcBytesPerPixel = srcImage.PixelFormat.GetBytesPerPixel(); + int dstBytesPerPixel = destImage.PixelFormat.GetBytesPerPixel(); + byte* srcRow = (byte*)srcImage.ImageData.ToPointer(); + byte* dstRow = (byte*)destImage.ImageData.ToPointer(); + for (int i = 0; i < srcImage.Height; i++) + { + byte* srcCol = srcRow; + byte* dstCol = dstRow; + for (int j = 0; j < srcImage.Width; j++) + { + switch (srcImage.PixelFormat) + { + case PixelFormat.Gray_8bpp: + dstCol[0] = (byte)(255 - srcCol[0]); + break; + + case PixelFormat.Gray_16bpp: + ((ushort*)dstCol)[0] = (byte)(65535 - srcCol[0]); + break; + + case PixelFormat.BGR_24bpp: + dstCol[0] = (byte)(255 - srcCol[0]); + dstCol[1] = (byte)(255 - srcCol[1]); + dstCol[2] = (byte)(255 - srcCol[2]); + break; + + case PixelFormat.BGRX_32bpp: + dstCol[0] = (byte)(255 - srcCol[0]); + dstCol[1] = (byte)(255 - srcCol[1]); + dstCol[2] = (byte)(255 - srcCol[2]); + dstCol[3] = (byte)srcCol[3]; + break; + + case PixelFormat.BGRA_32bpp: + dstCol[0] = (byte)(255 - srcCol[0]); + dstCol[1] = (byte)(255 - srcCol[1]); + dstCol[2] = (byte)(255 - srcCol[2]); + dstCol[3] = (byte)srcCol[3]; + break; + + case PixelFormat.RGBA_64bpp: + dstCol[0] = (byte)(255 - srcCol[0]); + dstCol[1] = (byte)(255 - srcCol[1]); + dstCol[2] = (byte)(255 - srcCol[2]); + dstCol[3] = (byte)srcCol[3]; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + srcCol += srcBytesPerPixel; + dstCol += dstBytesPerPixel; + } + + srcRow += srcImage.Stride; + dstRow += destImage.Stride; + } + } + } + + /// + /// Clears each color component in an image to the specified color. + /// + /// Image to clear. + /// Color to clear to. + public static void Clear(this Image image, Color clr) + { + unsafe + { + int srcBytesPerPixel = image.PixelFormat.GetBytesPerPixel(); + byte* srcRow = (byte*)image.ImageData.ToPointer(); + for (int i = 0; i < image.Height; i++) + { + byte* srcCol = srcRow; + for (int j = 0; j < image.Width; j++) + { + switch (image.PixelFormat) + { + case PixelFormat.Gray_8bpp: + srcCol[0] = Operators.Rgb2Gray(clr.R, clr.G, clr.B); + break; + + case PixelFormat.Gray_16bpp: + ((ushort*)srcCol)[0] = Operators.Rgb2Gray((ushort)clr.R, (ushort)clr.G, (ushort)clr.B); + break; + + case PixelFormat.BGR_24bpp: + srcCol[2] = clr.R; + srcCol[1] = clr.G; + srcCol[0] = clr.B; + break; + + case PixelFormat.BGRX_32bpp: + srcCol[2] = clr.R; + srcCol[1] = clr.G; + srcCol[0] = clr.B; + srcCol[3] = 255; + break; + + case PixelFormat.BGRA_32bpp: + srcCol[2] = clr.R; + srcCol[1] = clr.G; + srcCol[0] = clr.B; + srcCol[3] = clr.A; + break; + + case PixelFormat.RGBA_64bpp: + ((ushort*)srcCol)[3] = (ushort)((clr.R << 8) | clr.R); + ((ushort*)srcCol)[2] = (ushort)((clr.G << 8) | clr.G); + ((ushort*)srcCol)[1] = (ushort)((clr.B << 8) | clr.B); + ((ushort*)srcCol)[0] = (ushort)((clr.A << 8) | clr.A); + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + srcCol += srcBytesPerPixel; + } + + srcRow += image.Stride; + } + } + } + + /// + /// Extracts a single channel from the image and returns it as a gray scale image. + /// + /// Image to extract from. + /// Index of channel to extract from. This should be one of the following: ExtractRedChannel, ExtractGreenChannel, ExtractBlueChannel, or ExtractAlphaChannel. + /// Returns a new grayscale image containing the color from the specified channel in the original source image. + public static Image ExtractChannel(this Image image, int channel) + { + Image destImage = new Image(image.Width, image.Height, PixelFormat.Gray_8bpp); + image.ExtractChannel(destImage, channel); + return destImage; + } + + /// + /// Extracts a single channel from the image and returns it as a gray scale image. + /// + /// Image to extract from. + /// Image to write results to. + /// Index of channel to extract from. This should be one of the following: ExtractRedChannel, ExtractGreenChannel, ExtractBlueChannel, or ExtractAlphaChannel. + public static void ExtractChannel(this Image image, Image destImage, int channel) + { + if (image.Width != destImage.Width || image.Height != destImage.Height) + { + throw new InvalidOperationException(Image.ExceptionDescriptionSourceDestImageMismatch); + } + + if (destImage.PixelFormat != PixelFormat.Gray_8bpp) + { + throw new ArgumentException("Destination must be of pixel format type: Gray_8bpp."); + } + + if (image.PixelFormat != PixelFormat.BGRA_32bpp && + image.PixelFormat != PixelFormat.BGRX_32bpp && + image.PixelFormat != PixelFormat.BGR_24bpp) + { + throw new InvalidOperationException("Extract only supports the following pixel formats: BGRA_32bpp, BGRX_32bpp, and BGR_24bpp"); + } + + if (channel < 0 || + (image.PixelFormat != PixelFormat.BGR_24bpp && channel > 3) || + (image.PixelFormat == PixelFormat.BGR_24bpp && channel > 2)) + { + throw new ArgumentException("Unsupported channel"); + } + + unsafe + { + int srcBytesPerPixel = image.PixelFormat.GetBytesPerPixel(); + int dstBytesPerPixel = PixelFormat.Gray_8bpp.GetBytesPerPixel(); + byte* srcRow = (byte*)image.ImageData.ToPointer(); + byte* dstRow = (byte*)destImage.ImageData.ToPointer(); + for (int i = 0; i < image.Height; i++) + { + byte* srcCol = srcRow; + byte* dstCol = dstRow; + for (int j = 0; j < image.Width; j++) + { + dstCol[0] = srcCol[channel]; + srcCol += srcBytesPerPixel; + dstCol += dstBytesPerPixel; + } + + srcRow += image.Stride; + dstRow += destImage.Stride; + } + } + } + } + + /// + /// Imaging math operators. + /// + public static partial class Operators + { + /// + /// Performs per channel thresholding on the image. + /// + /// Image to be thresholded. + /// Threshold value. + /// Maximum value. + /// Type of thresholding to perform. + /// The thresholded image. + public static Image Threshold(this Image image, int threshold, int maxvalue, Threshold type) + { + Image thresholdedImage = new Image(image.Width, image.Height, image.PixelFormat); + image.Threshold(thresholdedImage, threshold, maxvalue, type); + return thresholdedImage; + } + + /// + /// Performs per channel thresholding on the image. + /// + /// Image to be thresholded. + /// Destination image where thresholded results are stored. + /// Threshold value. + /// Maximum value. + /// Type of thresholding to perform. + public static void Threshold(this Image srcImage, Image destImage, int threshold, int maxvalue, Threshold type) + { + unsafe + { + int bytesPerPixel = srcImage.PixelFormat.GetBytesPerPixel(); + byte* srcRow = (byte*)srcImage.ImageData.ToPointer(); + byte* dstRow = (byte*)destImage.ImageData.ToPointer(); + for (int i = 0; i < srcImage.Height; i++) + { + byte* srcCol = srcRow; + byte* dstCol = dstRow; + for (int j = 0; j < srcImage.Width; j++) + { + int r = 0, g = 0, b = 0, a = 255; + switch (srcImage.PixelFormat) + { + case PixelFormat.BGRA_32bpp: + b = srcCol[0]; + g = srcCol[1]; + r = srcCol[2]; + a = srcCol[3]; + break; + + case PixelFormat.BGRX_32bpp: + b = srcCol[0]; + g = srcCol[1]; + r = srcCol[2]; + break; + + case PixelFormat.BGR_24bpp: + b = srcCol[0]; + g = srcCol[1]; + r = srcCol[2]; + break; + + case PixelFormat.Gray_16bpp: + r = g = b = a = ((ushort*)srcCol)[0]; + break; + + case PixelFormat.Gray_8bpp: + r = g = b = a = srcCol[0]; + break; + + case PixelFormat.RGBA_64bpp: + r = ((ushort*)srcCol)[0]; + g = ((ushort*)srcCol)[1]; + b = ((ushort*)srcCol)[2]; + a = ((ushort*)srcCol)[3]; + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + switch (type) + { + case Imaging.Threshold.Binary: + r = (r > threshold) ? maxvalue : 0; + g = (g > threshold) ? maxvalue : 0; + b = (b > threshold) ? maxvalue : 0; + a = (a > threshold) ? maxvalue : 0; + break; + + case Imaging.Threshold.BinaryInv: + r = (r > threshold) ? 0 : maxvalue; + g = (g > threshold) ? 0 : maxvalue; + b = (b > threshold) ? 0 : maxvalue; + a = (a > threshold) ? 0 : maxvalue; + break; + + case Imaging.Threshold.Truncate: + r = (r > threshold) ? threshold : r; + g = (g > threshold) ? threshold : g; + b = (b > threshold) ? threshold : b; + a = (a > threshold) ? threshold : a; + break; + + case Imaging.Threshold.ToZero: + r = (r > threshold) ? r : 0; + g = (g > threshold) ? g : 0; + b = (b > threshold) ? b : 0; + a = (a > threshold) ? a : 0; + break; + + case Imaging.Threshold.ToZeroInv: + r = (r > threshold) ? 0 : r; + g = (g > threshold) ? 0 : g; + b = (b > threshold) ? 0 : b; + a = (a > threshold) ? 0 : a; + break; + + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + switch (destImage.PixelFormat) + { + case PixelFormat.BGRA_32bpp: + dstCol[0] = (byte)b; + dstCol[1] = (byte)g; + dstCol[2] = (byte)r; + dstCol[3] = (byte)a; + break; + + case PixelFormat.BGRX_32bpp: + dstCol[0] = (byte)b; + dstCol[1] = (byte)g; + dstCol[2] = (byte)r; + dstCol[3] = (byte)a; + break; + + case PixelFormat.BGR_24bpp: + dstCol[0] = (byte)b; + dstCol[1] = (byte)g; + dstCol[2] = (byte)r; + break; + + case PixelFormat.Gray_16bpp: + ((ushort*)srcCol)[0] = (ushort)r; + break; + + case PixelFormat.Gray_8bpp: + srcCol[0] = (byte)r; + break; + + case PixelFormat.RGBA_64bpp: + ((ushort*)srcCol)[0] = (ushort)r; + ((ushort*)srcCol)[1] = (ushort)g; + ((ushort*)srcCol)[2] = (ushort)b; + ((ushort*)srcCol)[3] = (ushort)a; + break; + + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + srcCol += bytesPerPixel; + dstCol += bytesPerPixel; + } + + srcRow += srcImage.Stride; + dstRow += destImage.Stride; + } + } + } + + /// + /// Computes the absolute difference between two images. + /// + /// First image. + /// Second image. + /// Difference image. + public static Image AbsDiff(this Image imageA, Image imageB) + { + Image diffImage = new Image(imageA.Width, imageA.Height, imageA.PixelFormat); + imageA.AbsDiff(imageB, diffImage); + return diffImage; + } + + /// + /// Computes the absolute difference between two images. + /// + /// First image. + /// Second image. + /// Destination image where to store difference image. + public static void AbsDiff(this Image imageA, Image imageB, Image destImage) + { + if (imageA.Width != imageB.Width || imageA.Height != imageB.Height || imageA.PixelFormat != imageB.PixelFormat) + { + throw new ArgumentException("Images sizes/types don't match"); + } + + unsafe + { + int bytesPerPixel = imageA.PixelFormat.GetBytesPerPixel(); + byte* srcRowA = (byte*)imageA.ImageData.ToPointer(); + byte* srcRowB = (byte*)imageB.ImageData.ToPointer(); + byte* dstRow = (byte*)destImage.ImageData.ToPointer(); + for (int i = 0; i < imageA.Height; i++) + { + byte* srcColA = srcRowA; + byte* srcColB = srcRowB; + byte* dstCol = dstRow; + int delta0, delta1, delta2, delta3; + for (int j = 0; j < imageA.Width; j++) + { + switch (imageA.PixelFormat) + { + case PixelFormat.BGRA_32bpp: + delta0 = srcColA[0] - srcColB[0]; + delta1 = srcColA[1] - srcColB[1]; + delta2 = srcColA[2] - srcColB[2]; + delta3 = srcColA[3] - srcColB[3]; + dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); + dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); + dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); + dstCol[3] = (byte)((delta3 < 0) ? -delta3 : delta3); + break; + + case PixelFormat.BGRX_32bpp: + delta0 = srcColA[0] - srcColB[0]; + delta1 = srcColA[1] - srcColB[1]; + delta2 = srcColA[2] - srcColB[2]; + dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); + dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); + dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); + dstCol[3] = 255; + break; + + case PixelFormat.BGR_24bpp: + delta0 = srcColA[0] - srcColB[0]; + delta1 = srcColA[1] - srcColB[1]; + delta2 = srcColA[2] - srcColB[2]; + dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); + dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); + dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); + break; + + case PixelFormat.Gray_16bpp: + delta0 = ((ushort*)srcColA)[0] - ((ushort*)srcColB)[0]; + ((ushort*)dstCol)[0] = (ushort)((delta0 < 0) ? -delta0 : delta0); + break; + + case PixelFormat.Gray_8bpp: + delta0 = srcColA[0] - srcColB[0]; + dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); + break; + + case PixelFormat.RGBA_64bpp: + delta0 = (ushort)(((ushort*)srcColA)[0] - ((ushort*)srcColB)[0]); + delta1 = (ushort)(((ushort*)srcColA)[1] - ((ushort*)srcColB)[1]); + delta2 = (ushort)(((ushort*)srcColA)[2] - ((ushort*)srcColB)[2]); + delta3 = (ushort)(((ushort*)srcColA)[3] - ((ushort*)srcColB)[3]); + ((ushort*)dstCol)[0] = (ushort)((delta0 < 0) ? -delta0 : delta0); + ((ushort*)dstCol)[1] = (ushort)((delta1 < 0) ? -delta1 : delta1); + ((ushort*)dstCol)[2] = (ushort)((delta2 < 0) ? -delta2 : delta2); + ((ushort*)dstCol)[3] = (ushort)((delta3 < 0) ? -delta3 : delta3); + break; + + case PixelFormat.Undefined: + default: + throw new ArgumentException(Image.ExceptionDescriptionUnexpectedPixelFormat); + } + + srcColA += bytesPerPixel; + srcColB += bytesPerPixel; + dstCol += bytesPerPixel; + } + + srcRowA += imageA.Stride; + srcRowB += imageB.Stride; + dstRow += destImage.Stride; + } + } + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImagePool.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImagePool.cs index f28486280..181659fc0 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/ImagePool.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImagePool.cs @@ -3,7 +3,7 @@ namespace Microsoft.Psi.Imaging { - using System; + using System.Drawing; using System.Drawing.Imaging; /// @@ -12,7 +12,7 @@ namespace Microsoft.Psi.Imaging public static class ImagePool { private static readonly KeyedSharedPool Instance = - new KeyedSharedPool(key => Image.Create(key.width, key.height, key.format)); + new KeyedSharedPool(key => new Image(key.width, key.height, key.format)); /// /// Gets or creates an image from the pool. @@ -29,29 +29,26 @@ public static Shared GetOrCreate(int width, int height, PixelFormat pixel /// /// Gets or creates an image from the pool and initializes it with a managed Bitmap object. /// - /// A bitmap from which to copy the image data. - /// A shared image from the pool containing a copy of the image data from . - public static Shared GetOrCreate(System.Drawing.Bitmap image) + /// A bitmap from which to copy the image data. + /// A shared image from the pool containing a copy of the image data from . + public static Shared GetOrCreateFromBitmap(Bitmap bitmap) { - BitmapData sourceData = image.LockBits( - new System.Drawing.Rectangle(0, 0, image.Width, image.Height), + BitmapData sourceData = bitmap.LockBits( + new Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.ReadOnly, - image.PixelFormat); - Shared sharedImg = null; + bitmap.PixelFormat); + Shared sharedImage = null; try { - sharedImg = GetOrCreate(image.Width, image.Height, PixelFormatHelper.FromSystemPixelFormat(image.PixelFormat)); - unsafe - { - Buffer.MemoryCopy(sourceData.Scan0.ToPointer(), sharedImg.Resource.ImageData.ToPointer(), sourceData.Stride * sourceData.Height, sourceData.Stride * sourceData.Height); - } + sharedImage = GetOrCreate(bitmap.Width, bitmap.Height, PixelFormatHelper.FromSystemPixelFormat(bitmap.PixelFormat)); + sharedImage.Resource.CopyFrom(sourceData); } finally { - image.UnlockBits(sourceData); + bitmap.UnlockBits(sourceData); } - return sharedImg; + return sharedImage; } } } diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ImageTransformer.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ImageTransformer.cs index 2d9681210..25d84d551 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/ImageTransformer.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ImageTransformer.cs @@ -9,17 +9,18 @@ namespace Microsoft.Psi.Imaging /// /// Defines the delegate used to perform an image transformation. /// - /// Source image to be transformed. - /// Destination for transformed image. - public delegate void TransformDelegate(Image src, Image dest); + /// Source image to be transformed. + /// Destination for transformed image. + public delegate void TransformDelegate(Image source, Image destination); /// /// Component that transforms an image given a specified transformer. /// public class ImageTransformer : ConsumerProducer, Shared> { - private TransformDelegate transformer; - private PixelFormat pixelFormat; + private readonly TransformDelegate transformer; + private readonly PixelFormat pixelFormat; + private System.Func> sharedImageAllocator; /// /// Initializes a new instance of the class. @@ -27,11 +28,14 @@ public class ImageTransformer : ConsumerProducer, Shared> /// Pipeline this component is a part of. /// Function for transforming the source image. /// Pixel format for destination image. - public ImageTransformer(Pipeline pipeline, TransformDelegate transformer, PixelFormat pixelFormat) + /// Optional image allocator for creating new shared image. + public ImageTransformer(Pipeline pipeline, TransformDelegate transformer, PixelFormat pixelFormat, System.Func> sharedImageAllocator = null) : base(pipeline) { this.transformer = transformer; this.pixelFormat = pixelFormat; + sharedImageAllocator ??= (width, height, pixelFormat) => ImagePool.GetOrCreate(width, height, pixelFormat); + this.sharedImageAllocator = sharedImageAllocator; } /// @@ -41,11 +45,9 @@ public ImageTransformer(Pipeline pipeline, TransformDelegate transformer, PixelF /// Pipeline sample information. protected override void Receive(Shared sharedImage, Envelope e) { - using (var psiImageDest = ImagePool.GetOrCreate(sharedImage.Resource.Width, sharedImage.Resource.Height, this.pixelFormat)) - { - this.transformer(sharedImage.Resource, psiImageDest.Resource); - this.Out.Post(psiImageDest, e.OriginatingTime); - } + using var sharedResultImage = this.sharedImageAllocator (sharedImage.Resource.Width, sharedImage.Resource.Height, this.pixelFormat); + this.transformer(sharedImage.Resource, sharedResultImage.Resource); + this.Out.Post(sharedResultImage, e.OriginatingTime); } } } \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/Microsoft.Psi.Imaging.csproj b/Sources/Imaging/Microsoft.Psi.Imaging/Microsoft.Psi.Imaging.csproj index 6ab1e2d36..ca1933750 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/Microsoft.Psi.Imaging.csproj +++ b/Sources/Imaging/Microsoft.Psi.Imaging/Microsoft.Psi.Imaging.csproj @@ -31,6 +31,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/Operators.cs b/Sources/Imaging/Microsoft.Psi.Imaging/Operators.cs deleted file mode 100644 index 795eb213f..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging/Operators.cs +++ /dev/null @@ -1,1127 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -#pragma warning disable SA1402 // File may only contain a single class -#pragma warning disable SA1649 // File name must match first type name - -namespace Microsoft.Psi.Imaging -{ - using System.Drawing; - - /// - /// Sampling mode used by various imaging operators. - /// - public enum SamplingMode - { - /// - /// Sampling mode using nearest neighbor interpolation. - /// - Point, - - /// - /// Sampling mode using bilinear interpolation. - /// - Bilinear, - - /// - /// Sampling mode using bicubic interpolation. - /// - Bicubic, - } - - /// - /// Thresholding modes. - /// - public enum Threshold - { - /// - /// Thresholds pixels such that: - /// dst(x,y) = maxvalue if (src(x,y)>threshold) - /// = 0 otherwise - /// - Binary, - - /// - /// Thresholds pixels such that: - /// dst(x,y) = 0 if (src(x,y)>threshold) - /// = maxvalue otherwise - /// - BinaryInv, - - /// - /// Thresholds pixels such that: - /// dst(x,y) = threshold if (src(x,y)>threshold) - /// = src(x,y) otherwise - /// - Truncate, - - /// - /// Thresholds pixels such that: - /// dst(x,y) = src(x,y) if (src(x,y)>threshold) - /// = 0 otherwise - /// - ToZero, - - /// - /// Thresholds pixels such that: - /// dst(x,y) = 0 if (src(x,y)>threshold) - /// = src(x,y) otherwise - /// - ToZeroInv, - } - - /// - /// Axis along which to flip an image. - /// - public enum FlipMode - { - /// - /// Leave image unflipped - /// - None, - - /// - /// Flips image along the horizontal axis - /// - AlongHorizontalAxis, - - /// - /// Flips image along the vertical axis - /// - AlongVerticalAxis, - } - - /// - /// Various imaging operators. - /// - public static partial class Operators - { - /// - /// Flips an image along a specified axis. - /// - /// Image to flip. - /// Axis along which to flip. - /// A new flipped image. - public static Shared Flip(this Image image, FlipMode mode) - { - if (image.PixelFormat == PixelFormat.Gray_16bpp) - { - // We can't handle this through GDI. - Shared dstImage = ImagePool.GetOrCreate(image.Width, image.Height, image.PixelFormat); - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(image.PixelFormat); - int dstBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(dstImage.Resource.PixelFormat); - byte* srcRow = (byte*)image.ImageData.ToPointer(); - byte* dstRow = (byte*)dstImage.Resource.ImageData.ToPointer(); - int ystep = dstImage.Resource.Stride; - if (mode == FlipMode.AlongHorizontalAxis) - { - dstRow += dstImage.Resource.Stride * (image.Height - 1); - ystep = -dstImage.Resource.Stride; - } - - int xstep = dstBytesPerPixel; - int xoffset = 0; - if (mode == FlipMode.AlongVerticalAxis) - { - xoffset = dstBytesPerPixel * (dstImage.Resource.Width - 1); - xstep = -dstBytesPerPixel; - } - - for (int i = 0; i < image.Height; i++) - { - byte* srcCol = srcRow; - byte* dstCol = dstRow + xoffset; - for (int j = 0; j < image.Width; j++) - { - ((ushort*)dstCol)[0] = ((ushort*)srcCol)[0]; - srcCol += srcBytesPerPixel; - dstCol += xstep; - } - - srcRow += image.Stride; - dstRow += ystep; - } - } - - return dstImage; - } - else - { - using (var bitmap = new Bitmap(image.Width, image.Height)) - { - using (var graphics = Graphics.FromImage(bitmap)) - { - switch (mode) - { - case FlipMode.AlongHorizontalAxis: - graphics.TranslateTransform(0.0f, image.Height - 1); - graphics.ScaleTransform(1.0f, -1.0f); - break; - - case FlipMode.AlongVerticalAxis: - graphics.TranslateTransform(image.Width - 1, 0.0f); - graphics.ScaleTransform(-1.0f, 1.0f); - break; - } - - using (var dstimage = image.ToManagedImage()) - { - graphics.DrawImage(dstimage, new Point(0, 0)); - } - - return ImagePool.GetOrCreate(bitmap); - } - } - } - } - - /// - /// Resizes an image by the specified scale factors using the specified sampling mode. - /// - /// Image to resize. - /// Scale factor to apply in X direction. - /// Scale factor to apply in Y direction. - /// Sampling mode for sampling of pixels. - /// Returns a new image scaled by the specified scale factors. - public static Shared Scale(this Image image, float scaleX, float scaleY, SamplingMode mode) - { - if (scaleX == 0.0 || scaleY == 0.0) - { - throw new System.Exception("Unexpected scale factors"); - } - - if (image.PixelFormat == PixelFormat.Gray_16bpp) - { - throw new System.NotSupportedException( - "Scaling 16bpp images is not currently supported. " + - "Convert to a supported format such as color or 8bpp grayscale first."); - } - - int dstWidth = (int)(image.Width * scaleX); - int dstHeight = (int)(image.Height * scaleY); - using (var bitmap = new Bitmap(dstWidth, dstHeight)) - { - using (var graphics = Graphics.FromImage(bitmap)) - { - graphics.CompositingMode = System.Drawing.Drawing2D.CompositingMode.SourceCopy; - switch (mode) - { - case SamplingMode.Point: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighSpeed; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.NearestNeighbor; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighSpeed; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighSpeed; - break; - - case SamplingMode.Bilinear: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.Bilinear; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; - break; - - case SamplingMode.Bicubic: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; - break; - } - - graphics.ScaleTransform(scaleX, scaleY); - - using (var managedimg = image.ToManagedImage()) - { - graphics.DrawImage(managedimg, new Point(0, 0)); - } - - return ImagePool.GetOrCreate(bitmap); - } - } - } - - /// - /// Rotates an image. - /// - /// Image to rotate. - /// Number of degrees to rotate in counter clockwise direction. - /// Pixel resampling method. - /// Rotated image. - public static Shared Rotate(this Image image, float angleInDegrees, SamplingMode mode) - { - float ca = (float)System.Math.Cos(angleInDegrees * System.Math.PI / 180.0f); - float sa = (float)System.Math.Sin(angleInDegrees * System.Math.PI / 180.0f); - float minx = 0.0f; - float miny = 0.0f; - float maxx = 0.0f; - float maxy = 0.0f; - float x = image.Width - 1; - float y = 0.0f; - float nx = (x * ca) - (y * sa); - float ny = (x * sa) + (y * ca); - if (nx < minx) - { - minx = nx; - } - - if (nx > maxx) - { - maxx = nx; - } - - if (ny < miny) - { - miny = ny; - } - - if (ny > maxy) - { - maxy = ny; - } - - x = image.Width - 1; - y = image.Height - 1; - nx = (x * ca) - (y * sa); - ny = (x * sa) + (y * ca); - if (nx < minx) - { - minx = nx; - } - - if (nx > maxx) - { - maxx = nx; - } - - if (ny < miny) - { - miny = ny; - } - - if (ny > maxy) - { - maxy = ny; - } - - x = 0.0f; - y = image.Height - 1; - nx = (x * ca) - (y * sa); - ny = (x * sa) + (y * ca); - if (nx < minx) - { - minx = nx; - } - - if (nx > maxx) - { - maxx = nx; - } - - if (ny < miny) - { - miny = ny; - } - - if (ny > maxy) - { - maxy = ny; - } - - int dstWidth = (int)(maxx - minx + 1); - int dstHeight = (int)(maxy - miny + 1); - using (var bitmap = new Bitmap(dstWidth, dstHeight)) - { - using (var graphics = Graphics.FromImage(bitmap)) - { - graphics.CompositingMode = System.Drawing.Drawing2D.CompositingMode.SourceCopy; - switch (mode) - { - case SamplingMode.Point: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighSpeed; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.NearestNeighbor; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighSpeed; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighSpeed; - break; - - case SamplingMode.Bilinear: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.Bilinear; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; - break; - - case SamplingMode.Bicubic: - graphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality; - graphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic; - graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality; - graphics.PixelOffsetMode = System.Drawing.Drawing2D.PixelOffsetMode.HighQuality; - break; - } - - graphics.TranslateTransform(-minx, -miny); - graphics.RotateTransform(angleInDegrees); - - using (var managedimg = image.ToManagedImage()) - { - graphics.DrawImage(managedimg, new Point(0, 0)); - } - - return ImagePool.GetOrCreate(bitmap); - } - } - } - } - - /// - /// Set of operators used for drawing on an image. - /// - public static partial class Operators - { - /// - /// Draws a rectangle at the specified pixel coordinates on the image. - /// - /// Image to draw on. - /// Pixel coordinates for rectangle. - /// Color to use for drawing. - /// Width of line. - public static void DrawRectangle(this Image image, Rectangle rect, Color color, int width) - { - using (Bitmap bm = image.ToManagedImage(false)) - { - using (var graphics = Graphics.FromImage(bm)) - { - using (var pen = new Pen(new SolidBrush(color))) - { - pen.Width = width; - graphics.DrawRectangle(pen, rect); - } - } - } - } - - /// - /// Draws a line from point p0 to p1 in pixel coordinates on the image. - /// - /// Image to draw on. - /// Pixel coordinates for start of line. - /// Pixel coordinates for end of line. - /// Color to use for drawing. - /// Width of line. - public static void DrawLine(this Image image, Point p0, Point p1, Color color, int width) - { - using (Bitmap bm = image.ToManagedImage(false)) - { - using (var graphics = Graphics.FromImage(bm)) - { - using (var pen = new Pen(new SolidBrush(color))) - { - pen.Width = width; - graphics.DrawLine(pen, p0, p1); - } - } - } - } - - /// - /// Draws a circle centered at the specified pixel (p0) with the specified radius. - /// - /// Image to draw on. - /// Pixel coordinates for center of circle. - /// Radius of the circle. - /// Color to use for drawing. - /// Width of line. - public static void DrawCircle(this Image image, Point p0, int radius, Color color, int width) - { - using (Bitmap bm = image.ToManagedImage(false)) - { - using (var graphics = Graphics.FromImage(bm)) - { - using (var pen = new Pen(new SolidBrush(color))) - { - pen.Width = width; - graphics.DrawEllipse(pen, p0.X - radius, p0.Y - radius, 2 * radius, 2 * radius); - } - } - } - } - - /// - /// Renders text on the image at the specified pixel (p0). - /// - /// Image to draw on. - /// Text to render. - /// Pixel coordinates for center of circle. - public static void DrawText(this Image image, string str, Point p0) - { - using (Bitmap bm = image.ToManagedImage(false)) - { - using (var graphics = Graphics.FromImage(bm)) - { - using (Font drawFont = new Font("Arial", 24)) - { - using (SolidBrush drawBrush = new SolidBrush(Color.Black)) - { - using (StringFormat drawFormat = new StringFormat()) - { - drawFormat.FormatFlags = 0; - graphics.DrawString(str, drawFont, drawBrush, p0.X, p0.Y, drawFormat); - } - } - } - } - } - } - } - - /// - /// Set of transforms for copying image data. - /// - public static partial class Operators - { - /// - /// Copies a source image into a destination image using the specified masking image. - /// Only pixels in the source image whose corresponding mask image pixels are > 0 - /// are copied to the destination image. - /// - /// Source image. - /// Destination image. - /// Masking image. - public static void CopyTo(this Image srcImage, Image dstImage, Image maskImage) - { - if (srcImage.Width != dstImage.Width || srcImage.Height != dstImage.Height) - { - throw new System.Exception("Source and destination images must be the same size"); - } - - Rectangle srcRect = new Rectangle(0, 0, srcImage.Width - 1, srcImage.Height - 1); - Rectangle dstRect = new Rectangle(0, 0, dstImage.Width - 1, dstImage.Height - 1); - srcImage.CopyTo(srcRect, dstImage, dstRect, maskImage); - } - - /// - /// Copies a source image into a destination image using the specified masking image. - /// Only pixels in the source image whose corresponding mask image pixels are > 0 - /// are copied to the destination image. Only pixels from the srcRect are copied - /// to the destination rect. - /// - /// Source image. - /// Destination image. - /// Rectangle to copy. - public static void CopyTo(this Image srcImage, Image dstImage, Rectangle rect) - { - if (srcImage.Width != dstImage.Width || srcImage.Height != dstImage.Height) - { - throw new System.Exception("Source and destination images must be the same size"); - } - - srcImage.CopyTo(rect, dstImage, rect, null); - } - - /// - /// Copies a source image into a destination image using the specified masking image. - /// Only pixels in the source image whose corresponding mask image pixels are > 0 - /// are copied to the destination image. Only pixels from the srcRect are copied - /// to the destination rect. - /// - /// Source image. - /// Source rectangle to copy from. - /// Destination image. - /// Destunatuin rectangle to copy to. - public static void CopyTo(this Image srcImage, Rectangle srcRect, Image dstImage, Rectangle dstRect) - { - if (srcImage.Width != dstImage.Width || srcImage.Height != dstImage.Height) - { - throw new System.Exception("Source and destination images must be the same size"); - } - - srcImage.CopyTo(srcRect, dstImage, dstRect, null); - } - - /// - /// Copies a source image into a destination image using the specified masking image. - /// Only pixels in the source image whose corresponding mask image pixels are > 0 - /// are copied to the destination image. Only pixels from the srcRect are copied - /// to the destination rect. - /// - /// Source image. - /// Source rectangle to copy from. - /// Destination image. - /// Destination rectangle to copy to. - /// Masking image. - public static void CopyTo(this Image srcImage, Rectangle srcRect, Image dstImage, Rectangle dstRect, Image maskImage) - { - if (srcRect.Width != dstRect.Width || srcRect.Height != dstRect.Height) - { - throw new System.Exception("Source and destination rectangles sizes must match"); - } - - if (maskImage != null) - { - if (srcImage.Width != maskImage.Width || srcImage.Height != maskImage.Height) - { - throw new System.Exception("Mask image size must match source image size"); - } - - if (maskImage.PixelFormat != PixelFormat.Gray_8bpp) - { - throw new System.Exception("Mask image must be of type PixelFormat.Gray_8bpp"); - } - } - - PixelFormat srcFormat = srcImage.PixelFormat; - PixelFormat dstFormat = dstImage.PixelFormat; - System.IntPtr srcBuffer = srcImage.ImageData; - System.IntPtr dstBuffer = dstImage.ImageData; - System.IntPtr maskBuffer = (maskImage != null) ? maskImage.ImageData : System.IntPtr.Zero; - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(srcFormat); - int dstBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(dstFormat); - int maskBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(PixelFormat.Gray_8bpp); - byte* srcRow = (byte*)srcBuffer.ToPointer() + (srcRect.Y * srcImage.Stride) + (srcRect.X * srcBytesPerPixel); - byte* dstRow = (byte*)dstBuffer.ToPointer() + (dstRect.Y * dstImage.Stride) + (dstRect.X * dstBytesPerPixel); - byte* maskRow = null; - if (maskImage != null) - { - maskRow = (byte*)maskBuffer.ToPointer() + (srcRect.Y * maskImage.Stride) + (srcRect.X * maskBytesPerPixel); - } - - for (int i = 0; i < srcRect.Height; i++) - { - byte* srcCol = srcRow; - byte* dstCol = dstRow; - byte* maskCol = maskRow; - for (int j = 0; j < srcRect.Width; j++) - { - bool copyPixel = true; - if (maskImage != null) - { - if (*maskCol == 0) - { - copyPixel = false; - } - } - - if (copyPixel) - { - int red = 0; - int green = 0; - int blue = 0; - int alpha = 255; - switch (srcFormat) - { - case PixelFormat.Gray_8bpp: - red = green = blue = srcCol[0]; - break; - - case PixelFormat.Gray_16bpp: - red = green = blue = ((ushort*)srcCol)[0]; - break; - - case PixelFormat.BGR_24bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - break; - - case PixelFormat.BGRX_32bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - break; - - case PixelFormat.BGRA_32bpp: - blue = srcCol[0]; - green = srcCol[1]; - red = srcCol[2]; - alpha = srcCol[3]; - break; - - case PixelFormat.RGBA_64bpp: - red = ((ushort*)srcCol)[0]; - green = ((ushort*)srcCol)[1]; - blue = ((ushort*)srcCol)[2]; - alpha = ((ushort*)srcCol)[3]; - break; - } - - switch (dstFormat) - { - case PixelFormat.Gray_8bpp: - dstCol[0] = Image.Rgb2Gray((byte)red, (byte)green, (byte)blue); - break; - - case PixelFormat.Gray_16bpp: - ((ushort*)dstCol)[0] = Image.Rgb2Gray((ushort)red, (ushort)green, (ushort)blue); - break; - - case PixelFormat.BGR_24bpp: - case PixelFormat.BGRX_32bpp: - dstCol[0] = (byte)blue; - dstCol[1] = (byte)green; - dstCol[2] = (byte)red; - break; - - case PixelFormat.BGRA_32bpp: - dstCol[0] = (byte)blue; - dstCol[1] = (byte)green; - dstCol[2] = (byte)red; - dstCol[3] = (byte)alpha; - break; - - case PixelFormat.RGBA_64bpp: - ((ushort*)dstCol)[0] = (ushort)red; - ((ushort*)dstCol)[1] = (ushort)green; - ((ushort*)dstCol)[2] = (ushort)blue; - ((ushort*)dstCol)[3] = (ushort)alpha; - break; - } - } - - srcCol += srcBytesPerPixel; - dstCol += dstBytesPerPixel; - maskCol += maskBytesPerPixel; - } - - srcRow += srcImage.Stride; - dstRow += dstImage.Stride; - if (maskImage != null) - { - maskRow += maskImage.Stride; - } - } - } - } - } - - /// - /// Basic color transforms on images. - /// - public static partial class Operators - { - /// - /// Inverts an image. - /// - /// Image to invert. - /// Returns the inverted image. - public static Shared Invert(this Image image) - { - Shared dstImage = ImagePool.GetOrCreate(image.Width, image.Height, image.PixelFormat); - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(image.PixelFormat); - int dstBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(dstImage.Resource.PixelFormat); - byte* srcRow = (byte*)image.ImageData.ToPointer(); - byte* dstRow = (byte*)dstImage.Resource.ImageData.ToPointer(); - for (int i = 0; i < image.Height; i++) - { - byte* srcCol = srcRow; - byte* dstCol = dstRow; - for (int j = 0; j < image.Width; j++) - { - switch (image.PixelFormat) - { - case PixelFormat.Gray_8bpp: - dstCol[0] = (byte)(255 - srcCol[0]); - break; - - case PixelFormat.Gray_16bpp: - ((ushort*)dstCol)[0] = (byte)(65535 - srcCol[0]); - break; - - case PixelFormat.BGR_24bpp: - dstCol[0] = (byte)(255 - srcCol[0]); - dstCol[1] = (byte)(255 - srcCol[1]); - dstCol[2] = (byte)(255 - srcCol[2]); - break; - - case PixelFormat.BGRX_32bpp: - dstCol[0] = (byte)(255 - srcCol[0]); - dstCol[1] = (byte)(255 - srcCol[1]); - dstCol[2] = (byte)(255 - srcCol[2]); - break; - - case PixelFormat.BGRA_32bpp: - dstCol[0] = (byte)(255 - srcCol[0]); - dstCol[1] = (byte)(255 - srcCol[1]); - dstCol[2] = (byte)(255 - srcCol[2]); - dstCol[3] = (byte)(255 - srcCol[3]); - break; - } - - srcCol += srcBytesPerPixel; - dstCol += dstBytesPerPixel; - } - - srcRow += image.Stride; - dstRow += dstImage.Resource.Stride; - } - } - - return dstImage; - } - - /// - /// Clears an image. - /// - /// Image to clear. - /// Color to clear to. - public static void Clear(this Image image, Color clr) - { - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(image.PixelFormat); - byte* srcRow = (byte*)image.ImageData.ToPointer(); - for (int i = 0; i < image.Height; i++) - { - byte* srcCol = srcRow; - for (int j = 0; j < image.Width; j++) - { - switch (image.PixelFormat) - { - case PixelFormat.Gray_8bpp: - srcCol[0] = Image.Rgb2Gray(clr.R, clr.G, clr.B); - break; - - case PixelFormat.Gray_16bpp: - ((ushort*)srcCol)[0] = Image.Rgb2Gray((ushort)clr.R, (ushort)clr.G, (ushort)clr.B); - break; - - case PixelFormat.BGR_24bpp: - srcCol[2] = clr.R; - srcCol[1] = clr.G; - srcCol[0] = clr.B; - break; - - case PixelFormat.BGRX_32bpp: - srcCol[2] = clr.R; - srcCol[1] = clr.G; - srcCol[0] = clr.B; - break; - - case PixelFormat.BGRA_32bpp: - srcCol[3] = clr.R; - srcCol[2] = clr.G; - srcCol[1] = clr.B; - srcCol[0] = clr.A; - break; - } - - srcCol += srcBytesPerPixel; - } - - srcRow += image.Stride; - } - } - } - - /// - /// Extracts a single channel from the image and returns it as a gray scale image. - /// - /// Image to extract from. - /// Index of channel to extract from. - /// Returns a new grayscale image containing the color from the specified channel in the original source image. - public static Shared ExtractChannel(this Image image, int channel /* 0=red, 1=green, 2=blue, 3=alpha */) - { - if (image.PixelFormat != PixelFormat.BGRA_32bpp && - image.PixelFormat != PixelFormat.BGRX_32bpp && - image.PixelFormat != PixelFormat.BGR_24bpp) - { - throw new System.Exception("Extract only supports the following pixel formats: BGRA_32bpp, BGRX_32bpp, and BGR_24bpp"); - } - - if (channel < 0 || - (image.PixelFormat != PixelFormat.BGR_24bpp && channel > 3) || - (image.PixelFormat == PixelFormat.BGR_24bpp && channel > 2)) - { - throw new System.Exception("Unsupported channel"); - } - - Shared dstImage = ImagePool.GetOrCreate(image.Width, image.Height, PixelFormat.Gray_8bpp); - unsafe - { - int srcBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(image.PixelFormat); - int dstBytesPerPixel = PixelFormatHelper.GetBytesPerPixel(PixelFormat.Gray_8bpp); - byte* srcRow = (byte*)image.ImageData.ToPointer(); - byte* dstRow = (byte*)dstImage.Resource.ImageData.ToPointer(); - for (int i = 0; i < image.Height; i++) - { - byte* srcCol = srcRow; - byte* dstCol = dstRow; - for (int j = 0; j < image.Width; j++) - { - dstCol[0] = srcCol[channel]; - srcCol += srcBytesPerPixel; - dstCol += dstBytesPerPixel; - } - - srcRow += image.Stride; - dstRow += dstImage.Resource.Stride; - } - } - - return dstImage; - } - } - - /// - /// Imaging math operators. - /// - public static partial class Operators - { - /// - /// Performs per channel thresholding on the image. - /// - /// Image to be thresholded. - /// Threshold value. - /// Maximum value. - /// Type of thresholding to perform. - /// The thresholded image. - public static Shared Threshold(this Image image, int threshold, int maxvalue, Threshold type) - { - Shared dstImage = ImagePool.GetOrCreate(image.Width, image.Height, image.PixelFormat); - - unsafe - { - int bytesPerPixel = PixelFormatHelper.GetBytesPerPixel(image.PixelFormat); - byte* srcRow = (byte*)image.ImageData.ToPointer(); - byte* dstRow = (byte*)dstImage.Resource.ImageData.ToPointer(); - for (int i = 0; i < image.Height; i++) - { - byte* srcCol = srcRow; - byte* dstCol = dstRow; - for (int j = 0; j < image.Width; j++) - { - int r = 0, g = 0, b = 0, a = 0; - switch (image.PixelFormat) - { - case PixelFormat.BGRA_32bpp: - b = srcCol[0]; - g = srcCol[1]; - r = srcCol[2]; - a = srcCol[3]; - break; - - case PixelFormat.BGRX_32bpp: - b = srcCol[0]; - g = srcCol[1]; - r = srcCol[2]; - break; - - case PixelFormat.BGR_24bpp: - b = srcCol[0]; - g = srcCol[1]; - r = srcCol[2]; - break; - - case PixelFormat.Gray_16bpp: - r = g = b = a = ((ushort*)srcCol)[0]; - break; - - case PixelFormat.Gray_8bpp: - r = g = b = a = srcCol[0]; - break; - - case PixelFormat.RGBA_64bpp: - r = ((ushort*)srcCol)[0]; - g = ((ushort*)srcCol)[1]; - b = ((ushort*)srcCol)[2]; - a = ((ushort*)srcCol)[3]; - break; - - default: - break; - } - - switch (type) - { - case Imaging.Threshold.Binary: - r = (r > threshold) ? maxvalue : 0; - g = (g > threshold) ? maxvalue : 0; - b = (b > threshold) ? maxvalue : 0; - a = (a > threshold) ? maxvalue : 0; - break; - - case Imaging.Threshold.BinaryInv: - r = (r > threshold) ? 0 : maxvalue; - g = (g > threshold) ? 0 : maxvalue; - b = (b > threshold) ? 0 : maxvalue; - a = (a > threshold) ? 0 : maxvalue; - break; - - case Imaging.Threshold.Truncate: - r = (r > threshold) ? threshold : r; - g = (g > threshold) ? threshold : g; - b = (b > threshold) ? threshold : b; - a = (a > threshold) ? threshold : a; - break; - - case Imaging.Threshold.ToZero: - r = (r > threshold) ? r : 0; - g = (g > threshold) ? g : 0; - b = (b > threshold) ? b : 0; - a = (a > threshold) ? a : 0; - break; - - case Imaging.Threshold.ToZeroInv: - r = (r > threshold) ? 0 : r; - g = (g > threshold) ? 0 : g; - b = (b > threshold) ? 0 : b; - a = (a > threshold) ? 0 : a; - break; - } - - switch (image.PixelFormat) - { - case PixelFormat.BGRA_32bpp: - dstCol[0] = (byte)b; - dstCol[1] = (byte)g; - dstCol[2] = (byte)r; - dstCol[3] = (byte)a; - break; - - case PixelFormat.BGRX_32bpp: - dstCol[0] = (byte)b; - dstCol[1] = (byte)g; - dstCol[2] = (byte)r; - break; - - case PixelFormat.BGR_24bpp: - dstCol[0] = (byte)b; - dstCol[1] = (byte)g; - dstCol[2] = (byte)r; - break; - - case PixelFormat.Gray_16bpp: - ((ushort*)srcCol)[0] = (ushort)r; - break; - - case PixelFormat.Gray_8bpp: - srcCol[0] = (byte)r; - break; - - case PixelFormat.RGBA_64bpp: - ((ushort*)srcCol)[0] = (ushort)r; - ((ushort*)srcCol)[1] = (ushort)g; - ((ushort*)srcCol)[2] = (ushort)b; - ((ushort*)srcCol)[3] = (ushort)a; - break; - - default: - break; - } - - srcCol += bytesPerPixel; - dstCol += bytesPerPixel; - } - - srcRow += image.Stride; - dstRow += dstImage.Resource.Stride; - } - } - - return dstImage; - } - - /// - /// Computes the absolute difference between two images. - /// - /// First image. - /// Second image. - /// Difference image. - public static Shared AbsDiff(this Image imageA, Image imageB) - { - if (imageA.Width != imageB.Width || imageA.Height != imageB.Height || imageA.PixelFormat != imageB.PixelFormat) - { - throw new System.Exception("Images sizes/types don't match"); - } - - Shared dstImage = ImagePool.GetOrCreate(imageA.Width, imageA.Height, imageA.PixelFormat); - - unsafe - { - int bytesPerPixel = PixelFormatHelper.GetBytesPerPixel(imageA.PixelFormat); - byte* srcRowA = (byte*)imageA.ImageData.ToPointer(); - byte* srcRowB = (byte*)imageB.ImageData.ToPointer(); - byte* dstRow = (byte*)dstImage.Resource.ImageData.ToPointer(); - for (int i = 0; i < imageA.Height; i++) - { - byte* srcColA = srcRowA; - byte* srcColB = srcRowB; - byte* dstCol = dstRow; - int delta0, delta1, delta2, delta3; - for (int j = 0; j < imageA.Width; j++) - { - switch (imageA.PixelFormat) - { - case PixelFormat.BGRA_32bpp: - delta0 = srcColA[0] - srcColB[0]; - delta1 = srcColA[1] - srcColB[1]; - delta2 = srcColA[2] - srcColB[2]; - delta3 = srcColA[3] - srcColB[3]; - dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); - dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); - dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); - dstCol[3] = (byte)((delta3 < 0) ? -delta3 : delta3); - break; - - case PixelFormat.BGRX_32bpp: - delta0 = srcColA[0] - srcColB[0]; - delta1 = srcColA[1] - srcColB[1]; - delta2 = srcColA[2] - srcColB[2]; - dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); - dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); - dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); - break; - - case PixelFormat.BGR_24bpp: - delta0 = srcColA[0] - srcColB[0]; - delta1 = srcColA[1] - srcColB[1]; - delta2 = srcColA[2] - srcColB[2]; - dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); - dstCol[1] = (byte)((delta1 < 0) ? -delta1 : delta1); - dstCol[2] = (byte)((delta2 < 0) ? -delta2 : delta2); - break; - - case PixelFormat.Gray_16bpp: - delta0 = ((ushort*)srcColA)[0] - ((ushort*)srcColB)[0]; - ((ushort*)dstCol)[0] = (ushort)((delta0 < 0) ? -delta0 : delta0); - break; - - case PixelFormat.Gray_8bpp: - delta0 = srcColA[0] - srcColB[0]; - dstCol[0] = (byte)((delta0 < 0) ? -delta0 : delta0); - break; - - case PixelFormat.RGBA_64bpp: - delta0 = (ushort)(((ushort*)srcColA)[0] - ((ushort*)srcColB)[0]); - delta1 = (ushort)(((ushort*)srcColA)[1] - ((ushort*)srcColB)[1]); - delta2 = (ushort)(((ushort*)srcColA)[2] - ((ushort*)srcColB)[2]); - delta3 = (ushort)(((ushort*)srcColA)[3] - ((ushort*)srcColB)[3]); - ((ushort*)dstCol)[0] = (ushort)((delta0 < 0) ? -delta0 : delta0); - ((ushort*)dstCol)[1] = (ushort)((delta1 < 0) ? -delta1 : delta1); - ((ushort*)dstCol)[2] = (ushort)((delta2 < 0) ? -delta2 : delta2); - ((ushort*)dstCol)[3] = (ushort)((delta3 < 0) ? -delta3 : delta3); - break; - - default: - throw new System.Exception("Unexpected image format"); - } - - srcColA += bytesPerPixel; - srcColB += bytesPerPixel; - dstCol += bytesPerPixel; - } - - srcRowA += imageA.Stride; - srcRowB += imageB.Stride; - dstRow += dstImage.Resource.Stride; - } - } - - return dstImage; - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormat.cs b/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormat.cs deleted file mode 100644 index fb429bf7a..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormat.cs +++ /dev/null @@ -1,202 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -#pragma warning disable SA1402 // File may only contain a single class -#pragma warning disable SA1649 // File name must match first type name - -namespace Microsoft.Psi.Imaging -{ - using System; - - /// - /// PixelFormat defines. - /// - public enum PixelFormat - { - /// - /// Used when the pixel format isn't defined. - /// - Undefined, - - /// - /// Defines an grayscale image where each pixel is 8 bits. - /// - Gray_8bpp, - - /// - /// Defines an grayscale image where each pixel is 16 bits. - /// - Gray_16bpp, - - /// - /// Defines an color image format where each red/green/blue component is 8 bits. - /// The byte order in memory is: bb gg rr. - /// - BGR_24bpp, - - /// - /// Defines an color image format where each red/green/blue component is 8 bits. - /// The byte order in memory is: bb gg rr xx. - /// - BGRX_32bpp, - - /// - /// Defines an color image format where each red/green/blue/alpha component is 8 bits. - /// The byte order in memory is: bb gg rr aa. - /// - BGRA_32bpp, - - /// - /// Defines an color image format where each red/green/blue/alpha component is 16 bits. - /// The byte order in memory is: rrrr gggg bbbb aaaa. - /// - RGBA_64bpp, - } - - /// - /// Defines a set of extensions for getting info about a PixelFormat. - /// - public static class PixelFormatExtensions - { - /// - /// Returns the number of bits per pixel for a given pixel format. - /// - /// Pixel format for which to find bits per pixel. - /// Number of bits per pixel for the given pixel format. - public static int GetBitsPerPixel(this PixelFormat pixelFormat) - { - return PixelFormatHelper.GetBytesPerPixel(pixelFormat) * 8; - } - - /// - /// Returns the number of bytes per pixel for a given pixel format. - /// - /// Pixel format for which to find bytes per pixel. - /// Number of bytes per pixel for the given pixel format. - public static int GetBytesPerPixel(this PixelFormat pixelFormat) - { - return PixelFormatHelper.GetBytesPerPixel(pixelFormat); - } - } - - /// - /// Set of static functions for manipulating pixel formats. - /// - public static class PixelFormatHelper - { - /// - /// Converts from a system pixel format into a Psi.Imaging pixel format. - /// - /// System pixel format to be converted. - /// Psi.Imaging pixel format that matches the specified system pixel format. - public static PixelFormat FromSystemPixelFormat(System.Drawing.Imaging.PixelFormat pf) - { - if (pf == System.Drawing.Imaging.PixelFormat.Format24bppRgb) - { - return PixelFormat.BGR_24bpp; - } - - if (pf == System.Drawing.Imaging.PixelFormat.Format32bppRgb) - { - return PixelFormat.BGRX_32bpp; - } - - if (pf == System.Drawing.Imaging.PixelFormat.Format8bppIndexed) - { - return PixelFormat.Gray_8bpp; - } - - if (pf == System.Drawing.Imaging.PixelFormat.Format16bppGrayScale) - { - return PixelFormat.Gray_16bpp; - } - - if (pf == System.Drawing.Imaging.PixelFormat.Format32bppArgb) - { - return PixelFormat.BGRA_32bpp; - } - - if (pf == System.Drawing.Imaging.PixelFormat.Format64bppArgb) - { - return PixelFormat.RGBA_64bpp; - } - - throw new Exception("Unsupported pixel format"); - } - - /// - /// Converts from a Psi.Imaging PixelFormat to a System.Drawing.Imaging.PixelFormat. - /// - /// Pixel format to convert. - /// The system pixel format that corresponds to the Psi.Imaging pixel format. - public static System.Drawing.Imaging.PixelFormat ToSystemPixelFormat(PixelFormat pf) - { - switch (pf) - { - case PixelFormat.BGR_24bpp: - return System.Drawing.Imaging.PixelFormat.Format24bppRgb; - - case PixelFormat.BGRX_32bpp: - return System.Drawing.Imaging.PixelFormat.Format32bppRgb; - - case PixelFormat.Gray_8bpp: - return System.Drawing.Imaging.PixelFormat.Format8bppIndexed; - - case PixelFormat.Gray_16bpp: - return System.Drawing.Imaging.PixelFormat.Format16bppGrayScale; - - case PixelFormat.BGRA_32bpp: - return System.Drawing.Imaging.PixelFormat.Format32bppArgb; - - case PixelFormat.RGBA_64bpp: - return System.Drawing.Imaging.PixelFormat.Format64bppArgb; - - default: - throw new Exception("Unknown pixel format?!"); - } - } - - /// - /// Returns number of bits/pixel for the specified pixel format. - /// - /// Pixel format for which to detemine number of bits/pxiel. - /// Number of bits per pixel in specified format. - public static int GetBitsPerPixel(PixelFormat pixelFormat) - { - return GetBytesPerPixel(pixelFormat) * 8; - } - - /// - /// Returns number of bytes/pixel for the specified pixel format. - /// - /// Pixel format for which to determine number of bytes. - /// Number of bytes in each pixel of the specified format. - public static int GetBytesPerPixel(PixelFormat pixelFormat) - { - switch (pixelFormat) - { - case PixelFormat.Gray_8bpp: - return 1; - - case PixelFormat.Gray_16bpp: - return 2; - - case PixelFormat.BGR_24bpp: - return 3; - - case PixelFormat.BGRX_32bpp: - case PixelFormat.BGRA_32bpp: - return 4; - - case PixelFormat.RGBA_64bpp: - return 8; - - case PixelFormat.Undefined: - return 0; - - default: - throw new Exception("Unknown pixel format"); - } - } - } -} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatExtensions.cs b/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatExtensions.cs new file mode 100644 index 000000000..5a76fce2c --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatExtensions.cs @@ -0,0 +1,96 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + /// + /// Defines the various pixel formats supported by the type. + /// + public enum PixelFormat + { + /// + /// Used when the pixel format isn't defined. + /// + Undefined, + + /// + /// Defines an grayscale image where each pixel is 8 bits. + /// + Gray_8bpp, + + /// + /// Defines an grayscale image where each pixel is 16 bits. + /// + Gray_16bpp, + + /// + /// Defines an color image format where each red/green/blue component is 8 bits. + /// The byte order in memory is: bb gg rr. + /// + BGR_24bpp, + + /// + /// Defines an color image format where each red/green/blue component is 8 bits. + /// The byte order in memory is: bb gg rr xx. + /// + BGRX_32bpp, + + /// + /// Defines an color image format where each red/green/blue/alpha component is 8 bits. + /// The byte order in memory is: bb gg rr aa. + /// + BGRA_32bpp, + + /// + /// Defines an color image format where each red/green/blue/alpha component is 16 bits. + /// The byte order in memory is: rrrr gggg bbbb aaaa. + /// + RGBA_64bpp, + } + + /// + /// Defines a set of extensions for getting info about a PixelFormat. + /// + public static class PixelFormatExtensions + { + /// + /// Returns the number of bits per pixel for a given pixel format. + /// + /// Pixel format for which to find bits per pixel. + /// Number of bits per pixel for the given pixel format. + public static int GetBitsPerPixel(this PixelFormat pixelFormat) + { + return PixelFormatHelper.GetBytesPerPixel(pixelFormat) * 8; + } + + /// + /// Returns the number of bytes per pixel for a given pixel format. + /// + /// Pixel format for which to find bytes per pixel. + /// Number of bytes per pixel for the given pixel format. + public static int GetBytesPerPixel(this PixelFormat pixelFormat) + { + return PixelFormatHelper.GetBytesPerPixel(pixelFormat); + } + + /// + /// Returns the pixel format correspond to the specified pixel format. + /// + /// The pixel format. + /// The corresponding pixel format. + public static PixelFormat ToPsiPixelFormat(this System.Drawing.Imaging.PixelFormat pixelFormat) + { + return PixelFormatHelper.FromSystemPixelFormat(pixelFormat); + } + + /// + /// Returns the pixel format correspond to the specified pixel format. + /// + /// The pixel format. + /// The corresponding pixel format. + public static System.Drawing.Imaging.PixelFormat ToSystemPixelFormat(this PixelFormat pixelFormat) + { + return PixelFormatHelper.ToSystemPixelFormat(pixelFormat); + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatHelper.cs b/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatHelper.cs new file mode 100644 index 000000000..497e17440 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/PixelFormatHelper.cs @@ -0,0 +1,108 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + + /// + /// Set of static functions for manipulating pixel formats. + /// + internal static class PixelFormatHelper + { + /// + /// Converts from a system pixel format into a Psi.Imaging pixel format. + /// + /// System pixel format to be converted. + /// Psi.Imaging pixel format that matches the specified system pixel format. + internal static PixelFormat FromSystemPixelFormat(System.Drawing.Imaging.PixelFormat pixelFormat) + { + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format24bppRgb) + { + return PixelFormat.BGR_24bpp; + } + + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format32bppRgb) + { + return PixelFormat.BGRX_32bpp; + } + + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format8bppIndexed) + { + return PixelFormat.Gray_8bpp; + } + + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format16bppGrayScale) + { + return PixelFormat.Gray_16bpp; + } + + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format32bppArgb) + { + return PixelFormat.BGRA_32bpp; + } + + if (pixelFormat == System.Drawing.Imaging.PixelFormat.Format64bppArgb) + { + return PixelFormat.RGBA_64bpp; + } + + throw new NotSupportedException($"The {pixelFormat} pixel format is not currently supported by {nameof(Microsoft.Psi.Imaging)}."); + } + + /// + /// Converts from a Psi.Imaging PixelFormat to a System.Drawing.Imaging.PixelFormat. + /// + /// Pixel format to convert. + /// The system pixel format that corresponds to the Psi.Imaging pixel format. + internal static System.Drawing.Imaging.PixelFormat ToSystemPixelFormat(PixelFormat pixelFormat) + { + return pixelFormat switch + { + PixelFormat.BGR_24bpp => System.Drawing.Imaging.PixelFormat.Format24bppRgb, + PixelFormat.BGRX_32bpp => System.Drawing.Imaging.PixelFormat.Format32bppRgb, + PixelFormat.Gray_8bpp => System.Drawing.Imaging.PixelFormat.Format8bppIndexed, + PixelFormat.Gray_16bpp => System.Drawing.Imaging.PixelFormat.Format16bppGrayScale, + PixelFormat.BGRA_32bpp => System.Drawing.Imaging.PixelFormat.Format32bppArgb, + PixelFormat.RGBA_64bpp => System.Drawing.Imaging.PixelFormat.Format64bppArgb, + PixelFormat.Undefined => + throw new InvalidOperationException( + $"Cannot convert {nameof(PixelFormat.Undefined)} pixel format to {nameof(System.Drawing.Imaging.PixelFormat)}."), + _ => throw new Exception("Unknown pixel format."), + }; + } + + /// + /// Returns number of bytes/pixel for the specified pixel format. + /// + /// Pixel format for which to determine number of bytes. + /// Number of bytes in each pixel of the specified format. + internal static int GetBytesPerPixel(PixelFormat pixelFormat) + { + switch (pixelFormat) + { + case PixelFormat.Gray_8bpp: + return 1; + + case PixelFormat.Gray_16bpp: + return 2; + + case PixelFormat.BGR_24bpp: + return 3; + + case PixelFormat.BGRX_32bpp: + case PixelFormat.BGRA_32bpp: + return 4; + + case PixelFormat.RGBA_64bpp: + return 8; + + case PixelFormat.Undefined: + return 0; + + default: + throw new ArgumentException("Unknown pixel format"); + } + } + } +} diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/PsiImaging.cs b/Sources/Imaging/Microsoft.Psi.Imaging/PsiImaging.cs deleted file mode 100644 index afee2dd68..000000000 --- a/Sources/Imaging/Microsoft.Psi.Imaging/PsiImaging.cs +++ /dev/null @@ -1,162 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Imaging -{ - using System; - using System.Drawing; - using System.Linq; - - /// - /// Implements stream operator methods for Imaging. - /// - public static partial class Operators - { - /// - /// Converts the source image to a different pixel format. - /// - /// The source stream. - /// The pixel format to convert to. - /// An optional delivery policy. - /// The resulting stream. - public static IProducer> Convert(this IProducer> source, PixelFormat pixelFormat, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ToPixelFormat(source.Out.Pipeline, pixelFormat), deliveryPolicy); - } - - /// - /// Converts an image to a different pixel format using the specified transformer. - /// - /// Source image to compress. - /// Method for converting an image sample. - /// Pixel format to use for converted image. - /// An optional delivery policy. - /// Returns a producer that generates the transformed images. - public static IProducer> Transform(this IProducer> source, TransformDelegate transformer, PixelFormat pixelFormat, DeliveryPolicy> deliveryPolicy = null) - { - return source.PipeTo(new ImageTransformer(source.Out.Pipeline, transformer, pixelFormat), deliveryPolicy); - } - - /// - /// Crops an image using the specified rectangle. - /// - /// Source of image and rectangle samples. - /// An optional delivery policy. - /// Returns a producer generating new cropped image samples. - public static IProducer> Crop(this IProducer<(Shared, Rectangle)> source, DeliveryPolicy<(Shared, Rectangle)> deliveryPolicy = null) - { - return source.Process<(Shared, Rectangle), Shared>( - (rectWithImage, env, e) => - { - using (var croppedImage = rectWithImage.Item1.Resource.Crop( - rectWithImage.Item2.Left, - rectWithImage.Item2.Top, - rectWithImage.Item2.Width, - rectWithImage.Item2.Height)) - { - e.Post(croppedImage, env.OriginatingTime); - } - }, deliveryPolicy); - } - - /// - /// Converts an image to grayscale. - /// - /// Image producer to use as source images. - /// An optional delivery policy. - /// Producers of grayscale images. - public static IProducer> ToGray(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) - { - return Convert(source, PixelFormat.Gray_8bpp, deliveryPolicy); - } - - /// - /// Resizes an image. - /// - /// Image to scale. - /// Final width of desired output. - /// Final height of desired output. - /// An optional delivery policy. - /// Returns a producer that generates resized images. - public static IProducer> Resize(this IProducer> source, float finalWidth, float finalHeight, DeliveryPolicy> deliveryPolicy = null) - { - return source.Process, Shared>( - (image, env, emitter) => - { - float scaleX = finalWidth / image.Resource.Width; - float scaleY = finalHeight / image.Resource.Height; - using (var resizedImage = image.Resource.Scale(scaleX, scaleY, SamplingMode.Bilinear)) - { - emitter.Post(resizedImage, env.OriginatingTime); - } - }, deliveryPolicy); - } - - /// - /// Flips an image about the horizontal or vertical axis. - /// - /// Image to flip. - /// Axis about which to flip. - /// An optional delivery policy. - /// A producer that generates flip images. - public static IProducer> Flip(this IProducer> source, FlipMode mode, DeliveryPolicy> deliveryPolicy = null) - { - if (mode == FlipMode.None) - { - // just post original image in the case of a no-op - return source.Process, Shared>( - (image, env, emitter) => emitter.Post(image, env.OriginatingTime), deliveryPolicy); - } - else - { - return source.Process, Shared>( - (image, env, emitter) => - { - using (var flippedImage = image.Resource.Flip(mode)) - { - emitter.Post(flippedImage, env.OriginatingTime); - } - }, deliveryPolicy); - } - } - - /// - /// Computes the absolute difference between two images. - /// - /// Images to diff. - /// An optional delivery policy. - /// Producer that returns the difference image. - public static IProducer> AbsDiff(this IProducer<(Shared, Shared)> sources, DeliveryPolicy<(Shared, Shared)> deliveryPolicy = null) - { - return sources.Process, Shared>, Shared>( - (images, env, e) => - { - using (var destImage = images.Item1.Resource.AbsDiff(images.Item2.Resource)) - { - e.Post(destImage, env.OriginatingTime); - } - }, deliveryPolicy); - } - - /// - /// Thresholds the image. See Threshold for what modes of thresholding are available. - /// - /// Images to threshold. - /// Threshold value. - /// Maximum value. - /// Type of thresholding to perform. - /// An optional delivery policy. - /// Producer that returns the difference image. - public static IProducer> Threshold(this IProducer> image, int threshold, int maxvalue, Threshold thresholdType, DeliveryPolicy> deliveryPolicy = null) - { - return image.Process, Shared>( - (srcimage, env, e) => - { - using (var destImage = srcimage.Resource.Threshold(threshold, maxvalue, thresholdType)) - { - e.Post(destImage, env.OriginatingTime); - } - }, deliveryPolicy); - } - } -} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/StreamOperators.cs b/Sources/Imaging/Microsoft.Psi.Imaging/StreamOperators.cs new file mode 100644 index 000000000..14b7c5b12 --- /dev/null +++ b/Sources/Imaging/Microsoft.Psi.Imaging/StreamOperators.cs @@ -0,0 +1,587 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Imaging +{ + using System; + using System.Drawing; + using Microsoft.Psi.Components; + + /// + /// Implements operators for processing streams of images. + /// + public static partial class Operators + { + /// + /// Converts a stream of images into a stream of depth images. + /// + /// A producer of images. + /// An optional delivery policy. + /// Optional image allocator for creating new shared depth image. + /// A corresponding stream of depth images. + /// The images in the source stream need to be in format. + public static IProducer> ToDepthImage(this IProducer> source, DeliveryPolicy> deliveryPolicy = null, Func> sharedDepthImageAllocator = null) + { + sharedDepthImageAllocator ??= DepthImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var sharedDepthImage = sharedDepthImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height); + sharedDepthImage.Resource.CopyFrom(sharedImage.Resource); + emitter.Post(sharedDepthImage, envelope.OriginatingTime); + }, + deliveryPolicy); + } + + /// + /// Converts a stream of depth images into a stream of format images. + /// + /// A producer of depth images. + /// An optional delivery policy. + /// Optional image allocator for creating new shared depth image. + /// A corresponding stream of images. + public static IProducer> ToImage(this IProducer> source, DeliveryPolicy> deliveryPolicy = null, Func> sharedDepthImageAllocator = null) + { + sharedDepthImageAllocator ??= (width, height) => ImagePool.GetOrCreate(width, height, PixelFormat.Gray_16bpp); + return source.Process, Shared>( + (sharedDepthImage, envelope, emitter) => + { + using var sharedImage = sharedDepthImageAllocator(sharedDepthImage.Resource.Width, sharedDepthImage.Resource.Height); + sharedImage.Resource.CopyFrom(sharedDepthImage.Resource); + emitter.Post(sharedImage, envelope.OriginatingTime); + }, + deliveryPolicy); + } + + /// + /// Converts the source image to a different pixel format. + /// + /// The source stream. + /// The pixel format to convert to. + /// An optional delivery policy. + /// Optional image allocator for creating new shared image. + /// The resulting stream. + public static IProducer> ToPixelFormat(this IProducer> source, PixelFormat pixelFormat, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + return source.PipeTo(new ToPixelFormat(source.Out.Pipeline, pixelFormat, sharedImageAllocator), deliveryPolicy); + } + + /// + /// Converts a shared image to a different pixel format using the specified transformer. + /// + /// Source image to compress. + /// Method for converting an image sample. + /// Pixel format to use for converted image. + /// An optional delivery policy. + /// Optional image allocator for creating new shared image. + /// Returns a producer that generates the transformed images. + public static IProducer> Transform(this IProducer> source, TransformDelegate transformer, PixelFormat pixelFormat, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + return source.PipeTo(new ImageTransformer(source.Out.Pipeline, transformer, pixelFormat, sharedImageAllocator), deliveryPolicy); + } + + /// + /// Crops a shared depth image using the specified rectangle. + /// + /// Source of image and rectangle samples. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer generating new cropped image samples. + public static IProducer> Crop(this IProducer<(Shared, Rectangle)> source, DeliveryPolicy<(Shared, Rectangle)> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process<(Shared, Rectangle), Shared>( + (tupleOfSharedImageAndRectangle, envelope, emitter) => + { + using var croppedSharedImage = sharedImageAllocator(tupleOfSharedImageAndRectangle.Item2.Width, tupleOfSharedImageAndRectangle.Item2.Height, tupleOfSharedImageAndRectangle.Item1.Resource.PixelFormat); + tupleOfSharedImageAndRectangle.Item1.Resource.Crop( + croppedSharedImage.Resource, + tupleOfSharedImageAndRectangle.Item2.Left, + tupleOfSharedImageAndRectangle.Item2.Top, + tupleOfSharedImageAndRectangle.Item2.Width, + tupleOfSharedImageAndRectangle.Item2.Height); + emitter.Post(croppedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Crops a shared image using the specified rectangle. + /// + /// Source of image and rectangle samples. + /// An optional delivery policy. + /// Optional image allocator to create new shared depth image. + /// Returns a producer generating new cropped image samples. + public static IProducer> Crop(this IProducer<(Shared, Rectangle)> source, DeliveryPolicy<(Shared, Rectangle)> deliveryPolicy = null, Func> sharedDepthImageAllocator = null) + { + sharedDepthImageAllocator ??= DepthImagePool.GetOrCreate; + return source.Process<(Shared, Rectangle), Shared>( + (tupleOfSharedImageAndRectangle, envelope, emitter) => + { + using var croppedSharedImage = sharedDepthImageAllocator(tupleOfSharedImageAndRectangle.Item2.Width, tupleOfSharedImageAndRectangle.Item2.Height); + tupleOfSharedImageAndRectangle.Item1.Resource.Crop( + croppedSharedImage.Resource, + tupleOfSharedImageAndRectangle.Item2.Left, + tupleOfSharedImageAndRectangle.Item2.Top, + tupleOfSharedImageAndRectangle.Item2.Width, + tupleOfSharedImageAndRectangle.Item2.Height); + emitter.Post(croppedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Convert a producer of depth images into pseudo-colorized images, where more distant pixels are blue, and closer pixels are reddish. + /// + /// Source producer of depth images. + /// A tuple indicating the range (MinValue, MaxValue) of the depth values in the image. + /// An optional delivery policy. + /// Optional image allocator to create new shared images (in format). + /// A producer of pseudo-colorized images. + public static IProducer> PseudoColorize( + this IProducer> source, + (ushort MinValue, ushort MaxValue) range, + DeliveryPolicy> deliveryPolicy = null, + Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= (width, height) => ImagePool.GetOrCreate(width, height, PixelFormat.BGR_24bpp); + return source.Process, Shared>( + (sharedDepthImage, envelope, emitter) => + { + using var colorizedImage = sharedImageAllocator(sharedDepthImage.Resource.Width, sharedDepthImage.Resource.Height); + sharedDepthImage.Resource.PseudoColorize(colorizedImage.Resource, range); + emitter.Post(colorizedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Converts a shared image to grayscale. + /// + /// Image producer to use as source images. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producers of grayscale images. + public static IProducer> ToGray(this IProducer> source, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + return source.ToPixelFormat(PixelFormat.Gray_8bpp, deliveryPolicy, sharedImageAllocator); + } + + /// + /// Resizes a shared image. + /// + /// Image to scale. + /// Final width of desired output. + /// Final height of desired output. + /// Method for sampling pixels when rescaling. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates resized images. + public static IProducer> Resize(this IProducer> source, float finalWidth, float finalHeight, SamplingMode samplingMode = SamplingMode.Bilinear, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var resizedSharedImage = sharedImageAllocator((int)finalWidth, (int)finalHeight, sharedImage.Resource.PixelFormat); + sharedImage.Resource.Resize(resizedSharedImage.Resource, finalWidth, finalHeight, samplingMode); + emitter.Post(resizedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Scales a shared by the specified scale factors. + /// + /// Image to scale. + /// Scale factor for X. + /// Scale factor for Y. + /// Method for sampling pixels when rescaling. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates resized images. + public static IProducer> Scale(this IProducer> source, float scaleX, float scaleY, SamplingMode samplingMode = SamplingMode.Bilinear, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + int finalWidth = (int)(sharedImage.Resource.Width * scaleX); + int finalHeight = (int)(sharedImage.Resource.Height * scaleY); + using var scaledSharedImage = sharedImageAllocator(finalWidth, finalHeight, sharedImage.Resource.PixelFormat); + sharedImage.Resource.Scale(scaledSharedImage.Resource, scaleX, scaleY, samplingMode); + emitter.Post(scaledSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Flips a shared image about the horizontal or vertical axis. + /// + /// Image to flip. + /// Axis about which to flip. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// A producer that generates flip images. + public static IProducer> Flip(this IProducer> source, FlipMode mode, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + if (mode == FlipMode.None) + { + // just post original image in the case of a no-op + return source; + } + else + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var flippedSharedImage = sharedImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + sharedImage.Resource.Flip(flippedSharedImage.Resource, mode); + emitter.Post(flippedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + } + + /// + /// Scales a shared image by the specified scale factors. + /// + /// Image to scale. + /// Angle for rotation specified in degrees. + /// Sampling mode to use when sampling pixels. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates rotated images. + public static IProducer> Rotate(this IProducer> source, float angleInDegrees, SamplingMode samplingMode, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + DetermineRotatedWidthHeight( + sharedImage.Resource.Width, + sharedImage.Resource.Height, + angleInDegrees, + out int rotatedWidth, + out int rotateHeight, + out float originx, + out float originy); + using var rotatedSharedImage = sharedImageAllocator(rotatedWidth, rotateHeight, sharedImage.Resource.PixelFormat); + rotatedSharedImage.Resource.Clear(Color.Black); + sharedImage.Resource.Rotate(rotatedSharedImage.Resource, angleInDegrees, samplingMode); + emitter.Post(rotatedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Draws a rectangle over a shared image. + /// + /// Image to draw rectangle on. + /// Pixel coordinates for rectangle. + /// Color to use when drawing the rectangle. + /// Line width (in pixels) of each side of the rectangle. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates images overdrawn with a rectangle. + public static IProducer> DrawRectangle(this IProducer> source, Rectangle rect, Color color, int width, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var drawRectSharedImage = sharedImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + drawRectSharedImage.Resource.CopyFrom(sharedImage.Resource); + drawRectSharedImage.Resource.DrawRectangle(rect, color, width); + emitter.Post(drawRectSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Draws a line over a shared image. + /// + /// Image to draw line on. + /// Pixel coordinates for one end of the line. + /// Pixel coordinates for the other end of the line. + /// Color to use when drawing the line. + /// Line width (in pixels). + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates images overdrawn with a line. + public static IProducer> DrawLine(this IProducer> source, Point p0, Point p1, Color color, int width, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var drawLineSharedImage = sharedImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + drawLineSharedImage.Resource.CopyFrom(sharedImage.Resource); + drawLineSharedImage.Resource.DrawLine(p0, p1, color, width); + emitter.Post(drawLineSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Draws a circle over a shared image. + /// + /// Image to draw circle on. + /// Center of circle (in pixels). + /// Radius of circle (in pixels). + /// Color to use when drawing the circle. + /// Line width (in pixels). + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates images overdrawn with a circle. + public static IProducer> DrawCircle(this IProducer> source, Point p0, int radius, Color color, int width, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var drawCircleSharedImage = sharedImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + drawCircleSharedImage.Resource.CopyFrom(sharedImage.Resource); + drawCircleSharedImage.Resource.DrawCircle(p0, radius, color, width); + emitter.Post(drawCircleSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Draws a piece of text over a shared image. + /// + /// Image to draw text on. + /// Text to render. + /// Coordinates for start of text (in pixels). + /// Color to use while drawing text. + /// Name of font to use. Optional. + /// Size of font. Optional. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Returns a producer that generates images overdrawn with text. + public static IProducer> DrawText(this IProducer> source, string text, Point p0, Color color, string font = null, float fontSize = 24.0f, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sharedImage, envelope, emitter) => + { + using var drawTextSharedImage = sharedImageAllocator(sharedImage.Resource.Width, sharedImage.Resource.Height, sharedImage.Resource.PixelFormat); + drawTextSharedImage.Resource.CopyFrom(sharedImage.Resource); + drawTextSharedImage.Resource.DrawText(text, p0, color, font, fontSize); + emitter.Post(drawTextSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Inverts each color channel in a shared image. + /// + /// Images to invert. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producer that returns the inverted image. + public static IProducer> Invert(this IProducer> source, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sourceImage, envelope, emitter) => + { + using var invertedSharedImage = sharedImageAllocator(sourceImage.Resource.Width, sourceImage.Resource.Height, sourceImage.Resource.PixelFormat); + sourceImage.Resource.Invert(invertedSharedImage.Resource); + emitter.Post(invertedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Clears a shared image to the specified color. + /// + /// Images to clear. + /// Color to set image to. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producer that returns the cleared image. + public static IProducer> Clear(this IProducer> source, Color clr, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sourceImage, envelope, emitter) => + { + using var clearedSharedImage = sharedImageAllocator(sourceImage.Resource.Width, sourceImage.Resource.Height, sourceImage.Resource.PixelFormat); + clearedSharedImage.Resource.Clear(clr); + emitter.Post(clearedSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Extracts a color channel from a shared image. Returned image is of type Gray_8bpp. + /// + /// Images to extract channel from. + /// Index of which channel to extract. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producer that returns the extracted channel as a gray scale image. + public static IProducer> ExtractChannel(this IProducer> source, int channel, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return source.Process, Shared>( + (sourceImage, envelope, emitter) => + { + using var channelSharedImage = sharedImageAllocator(sourceImage.Resource.Width, sourceImage.Resource.Height, PixelFormat.Gray_8bpp); + sourceImage.Resource.ExtractChannel(channelSharedImage.Resource, channel); + emitter.Post(channelSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Computes the absolute difference between two images. + /// + /// Images to diff. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producer that returns the difference image. + public static IProducer> AbsDiff(this IProducer<(Shared, Shared)> sources, DeliveryPolicy<(Shared, Shared)> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return sources.Process<(Shared, Shared), Shared>( + (tupleOfSharedImages, envelope, emitter) => + { + using var absdiffSharedImage = sharedImageAllocator(tupleOfSharedImages.Item1.Resource.Width, tupleOfSharedImages.Item1.Resource.Height, tupleOfSharedImages.Item1.Resource.PixelFormat); + tupleOfSharedImages.Item1.Resource.AbsDiff(tupleOfSharedImages.Item2.Resource, absdiffSharedImage.Resource); + emitter.Post(absdiffSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Thresholds the image. See Threshold for what modes of thresholding are available. + /// + /// Images to threshold. + /// Threshold value. + /// Maximum value. + /// Type of thresholding to perform. + /// An optional delivery policy. + /// Optional image allocator to create new shared image. + /// Producer that returns the difference image. + public static IProducer> Threshold(this IProducer> image, int threshold, int maxvalue, Threshold thresholdType, DeliveryPolicy> deliveryPolicy = null, Func> sharedImageAllocator = null) + { + sharedImageAllocator ??= ImagePool.GetOrCreate; + return image.Process, Shared>( + (sharedSourceImage, envelope, emitter) => + { + using var thresholdSharedImage = sharedImageAllocator(sharedSourceImage.Resource.Width, sharedSourceImage.Resource.Height, sharedSourceImage.Resource.PixelFormat); + sharedSourceImage.Resource.Threshold(thresholdSharedImage.Resource, threshold, maxvalue, thresholdType); + emitter.Post(thresholdSharedImage, envelope.OriginatingTime); + }, deliveryPolicy); + } + + /// + /// Encodes a shared image using a specified encoder component. + /// + /// A producer of images to encode. + /// Constructor function that returns an encoder component given a pipeline. + /// An optional delivery policy. + /// A producer that generates the encoded images. + public static IProducer> Encode( + this IProducer> source, + Func, Shared>> encoderConstructor, + DeliveryPolicy> deliveryPolicy = null) + { + return source.PipeTo(encoderConstructor(source.Out.Pipeline), deliveryPolicy); + } + + /// + /// Encodes a shared image using a specified image-to-stream encoder. + /// + /// A producer of images to encode. + /// The image-to-stream encoder to use when encoding images. + /// An optional delivery policy. + /// A producer that generates the encoded images. + public static IProducer> Encode( + this IProducer> source, + IImageToStreamEncoder encoder, + DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(p => new ImageEncoder(p, encoder), deliveryPolicy); + } + + /// + /// Decodes an encoded image using a specified decoder component. + /// + /// A producer of images to decode. + /// Constructor function that returns a decoder component given a pipeline. + /// An optional delivery policy. + /// A producer that generates the decoded images. + public static IProducer> Decode( + this IProducer> source, + Func, Shared>> decoderConstructor, + DeliveryPolicy> deliveryPolicy = null) + { + return source.PipeTo(decoderConstructor(source.Out.Pipeline), deliveryPolicy); + } + + /// + /// Decodes an encoded image using a specified image-from-stream decoder. + /// + /// A producer of images to decode. + /// The image-from-stream decoder to use when decoding images. + /// An optional delivery policy. + /// A producer that generates the decoded images. + public static IProducer> Decode( + this IProducer> source, + IImageFromStreamDecoder decoder, + DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(p => new ImageDecoder(p, decoder), deliveryPolicy); + } + + /// + /// Encodes a depth image using a specified encoder component. + /// + /// A producer of depth images to encode. + /// Constructor function that returns an encoder component given a pipeline. + /// An optional delivery policy. + /// A producer that generates the encoded depth images. + public static IProducer> Encode( + this IProducer> source, + Func, Shared>> encoderConstructor, + DeliveryPolicy> deliveryPolicy = null) + { + return source.PipeTo(encoderConstructor(source.Out.Pipeline), deliveryPolicy); + } + + /// + /// Encodes a depth image using a specified depth-image-to-stream encoder. + /// + /// A producer of depth images to encode. + /// The depth image encoder to use. + /// An optional delivery policy. + /// A producer that generates the encoded depth images. + public static IProducer> Encode( + this IProducer> source, + IDepthImageToStreamEncoder encoder, + DeliveryPolicy> deliveryPolicy = null) + { + return source.Encode(p => new DepthImageEncoder(p, encoder), deliveryPolicy); + } + + /// + /// Decodes a depth image using a specified decoder component. + /// + /// A producer of depth images to decode. + /// Constructor function that returns a decoder component given a pipeline. + /// An optional delivery policy. + /// A producer that generates the decoded depth images. + public static IProducer> Decode( + this IProducer> source, + Func, Shared>> decoderConstructor, + DeliveryPolicy> deliveryPolicy = null) + { + return source.PipeTo(decoderConstructor(source.Out.Pipeline), deliveryPolicy); + } + + /// + /// Decodes a depth image using a specified depth-image-from-stream decoder. + /// + /// A producer of depth images to decode. + /// The depth image decoder to use. + /// An optional delivery policy. + /// A producer that generates the decoded depth images. + public static IProducer> Decode( + this IProducer> source, + IDepthImageFromStreamDecoder decoder, + DeliveryPolicy> deliveryPolicy = null) + { + return source.Decode(p => new DepthImageDecoder(p, decoder), deliveryPolicy); + } + } +} \ No newline at end of file diff --git a/Sources/Imaging/Microsoft.Psi.Imaging/ToPixelFormat.cs b/Sources/Imaging/Microsoft.Psi.Imaging/ToPixelFormat.cs index cbf96b5ab..7ea31b0ea 100644 --- a/Sources/Imaging/Microsoft.Psi.Imaging/ToPixelFormat.cs +++ b/Sources/Imaging/Microsoft.Psi.Imaging/ToPixelFormat.cs @@ -7,21 +7,25 @@ namespace Microsoft.Psi.Imaging using Microsoft.Psi.Components; /// - /// Pipeline component that converts an Image to a different format. + /// Pipeline component that converts an shared Image to a different format. /// internal class ToPixelFormat : ConsumerProducer, Shared> { - private PixelFormat pixelFormat; + private readonly PixelFormat pixelFormat; + private System.Func> sharedImageAllocator; /// /// Initializes a new instance of the class. /// - /// The pipline. - /// The pixel format to conver to. - internal ToPixelFormat(Pipeline pipeline, PixelFormat pixelFormat) + /// The pipeline. + /// The pixel format to convert to. + /// Optional image allocator for creating new shared image. + internal ToPixelFormat(Pipeline pipeline, PixelFormat pixelFormat, System.Func> sharedImageAllocator = null) : base(pipeline) { this.pixelFormat = pixelFormat; + sharedImageAllocator ??= (width, height, pixelFormat) => ImagePool.GetOrCreate(width, height, pixelFormat); + this.sharedImageAllocator = sharedImageAllocator; } /// @@ -38,11 +42,9 @@ protected override void Receive(Shared sharedImage, Envelope e) } else { - using (var image = ImagePool.GetOrCreate(sharedImage.Resource.Width, sharedImage.Resource.Height, this.pixelFormat)) - { - sharedImage.Resource.CopyTo(image.Resource); - this.Out.Post(image, e.OriginatingTime); - } + using var image = this.sharedImageAllocator (sharedImage.Resource.Width, sharedImage.Resource.Height, this.pixelFormat); + sharedImage.Resource.CopyTo(image.Resource); + this.Out.Post(image, e.OriginatingTime); } } } diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/ImageTester.cs b/Sources/Imaging/Test.Psi.Imaging.Windows/ImageTester.cs index 8774de392..dae1a2917 100644 --- a/Sources/Imaging/Test.Psi.Imaging.Windows/ImageTester.cs +++ b/Sources/Imaging/Test.Psi.Imaging.Windows/ImageTester.cs @@ -6,22 +6,234 @@ namespace Test.Psi.Imaging { using System; - using System.IO; - using System.Windows.Media.Imaging; using Microsoft.Psi; + using Microsoft.Psi.Common; using Microsoft.Psi.Imaging; + using Microsoft.Psi.Serialization; using Microsoft.VisualStudio.TestTools.UnitTesting; [TestClass] public class ImageTester { - private Image testImage = Image.FromManagedImage(Properties.Resources.TestImage); - private Image testImage_0_0_200_100 = Image.FromManagedImage(Properties.Resources.TestImage_Crop_0_0_200_100); - private Image testImage_153_57_103_199 = Image.FromManagedImage(Properties.Resources.TestImage_Crop_153_57_103_199); - private Image testImage_73_41_59_37 = Image.FromManagedImage(Properties.Resources.TestImage_Crop_73_41_59_37); - private Image testImage_50_25 = Image.FromManagedImage(Properties.Resources.TestImage_Scale_50_25); - private Image testImage_150_125 = Image.FromManagedImage(Properties.Resources.TestImage_Scale_150_125); - private Image testImage_25_200 = Image.FromManagedImage(Properties.Resources.TestImage_Scale_25_200); + private Image testImage = Image.FromBitmap(Properties.Resources.TestImage); + private Image testImage2 = Image.FromBitmap(Properties.Resources.TestImage2); + private Image testImage2_Threshold = Image.FromBitmap(Properties.Resources.TestImage2_Threshold); + private Image testImage2_RedChannel = Image.FromBitmap(Properties.Resources.TestImage2_RedChannel); + private Image testImage2_GreenChannel = Image.FromBitmap(Properties.Resources.TestImage2_GreenChannel); + private Image testImage2_BlueChannel = Image.FromBitmap(Properties.Resources.TestImage2_BlueChannel); + private Image testImage2_CopyImage = Image.FromBitmap(Properties.Resources.TestImage2_CopyImage); + private Image testImage2_Invert = Image.FromBitmap(Properties.Resources.TestImage2_Invert); + private Image testImage2_Mask = Image.FromBitmap(Properties.Resources.TestImage2_Mask); + private Image testImage2_FlipHoriz = Image.FromBitmap(Properties.Resources.TestImage2_FlipHoriz); + private Image testImage2_FlipVert = Image.FromBitmap(Properties.Resources.TestImage2_FlipVert); + private Image testImage2_Rotate_Neg10 = Image.FromBitmap(Properties.Resources.TestImage2_Rotate_Neg10); + private Image testImage2_Rotate_110 = Image.FromBitmap(Properties.Resources.TestImage2_Rotate_110); + private Image testImage2_DrawRect = Image.FromBitmap(Properties.Resources.TestImage2_DrawRect); + private Image testImage2_DrawLine = Image.FromBitmap(Properties.Resources.TestImage2_DrawLine); + private Image testImage2_DrawCircle = Image.FromBitmap(Properties.Resources.TestImage2_DrawCircle); + private Image testImage2_DrawText = Image.FromBitmap(Properties.Resources.TestImage2_DrawText); + private Image testImage2_AbsDiff = Image.FromBitmap(Properties.Resources.TestImage2_AbsDiff); + private Image testImage_0_0_200_100 = Image.FromBitmap(Properties.Resources.TestImage_Crop_0_0_200_100); + private Image testImage_153_57_103_199 = Image.FromBitmap(Properties.Resources.TestImage_Crop_153_57_103_199); + private Image testImage_73_41_59_37 = Image.FromBitmap(Properties.Resources.TestImage_Crop_73_41_59_37); + private Image testImage_50_25_Cubic = Image.FromBitmap(Properties.Resources.TestImage_Scale_50_25_Cubic); + private Image testImage_150_125_Point = Image.FromBitmap(Properties.Resources.TestImage_Scale_150_125_Point); + private Image testImage_25_200_Linear = Image.FromBitmap(Properties.Resources.TestImage_Scale_25_200_Linear); + + [TestMethod] + [Timeout(60000)] + public void Image_FlipViaOperator() + { + using (var pipeline = Pipeline.Create("FlipViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Flip(FlipMode.None).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2, img.Resource); + }); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Flip(FlipMode.AlongHorizontalAxis).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_FlipHoriz, img.Resource); + }); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Flip(FlipMode.AlongVerticalAxis).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_FlipVert, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_RotateViaOperator() + { + using (var pipeline = Pipeline.Create("RotateViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Rotate(-10.0f, SamplingMode.Point).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_Rotate_Neg10, img.Resource); + }); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Rotate(110.0f, SamplingMode.Point).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_Rotate_110, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_DrawRectangleViaOperator() + { + using (var pipeline = Pipeline.Create("DrawRectangleViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).DrawRectangle(new System.Drawing.Rectangle(20, 20, 255, 255), System.Drawing.Color.White, 3).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_DrawRect, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_DrawLineViaOperator() + { + using (var pipeline = Pipeline.Create("DrawLineViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).DrawLine(new System.Drawing.Point(0, 0), new System.Drawing.Point(255, 255), System.Drawing.Color.White, 3).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_DrawLine, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_DrawCircleViaOperator() + { + using (var pipeline = Pipeline.Create("DrawCircleViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).DrawCircle(new System.Drawing.Point(250, 250), 100, System.Drawing.Color.White, 3).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_DrawCircle, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_DrawTextViaOperator() + { + using (var pipeline = Pipeline.Create("DrawTextViaOperator")) + { + using (var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat)) + { + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).DrawText("Testing", new System.Drawing.Point(100, 100), System.Drawing.Color.White).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_DrawText, img.Resource); + }); + pipeline.Run(); + } + } + } + + [TestMethod] + [Timeout(60000)] + public void Image_CopyImage() + { + using var destImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat); + destImage.Resource.Clear(System.Drawing.Color.Black); + System.Drawing.Rectangle rect = new System.Drawing.Rectangle(50, 300, 100, 255); + this.testImage2.CopyTo(rect, destImage.Resource, new System.Drawing.Point(-10, 0), this.testImage2_Mask); + this.AssertAreImagesEqual(this.testImage2_CopyImage, destImage.Resource); + } + + [TestMethod] + [Timeout(60000)] + public void Image_Invert() + { + using var pipeline = Pipeline.Create("ImageInvert"); + using var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat); + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Invert().Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_Invert, img.Resource); + }); + pipeline.Run(); + } + + [TestMethod] + [Timeout(60000)] + public void Image_AbsDiff() + { + using var pipeline = Pipeline.Create("ImageAbsDiff"); + using var sharedImage = ImagePool.GetOrCreate(this.testImage2_DrawCircle.Width, this.testImage2_DrawCircle.Height, this.testImage2_DrawCircle.PixelFormat); + this.testImage2_DrawCircle.CopyTo(sharedImage.Resource); + using var sharedImage2 = ImagePool.GetOrCreate(this.testImage2_DrawRect.Width, this.testImage2_DrawRect.Height, this.testImage2_DrawRect.PixelFormat); + this.testImage2_DrawRect.CopyTo(sharedImage2.Resource); + Generators.Sequence(pipeline, new[] { (sharedImage, sharedImage2) }, default, null, keepOpen: false).AbsDiff().Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_AbsDiff, img.Resource); + }); + pipeline.Run(); + } + + [TestMethod] + [Timeout(60000)] + public void Image_Threshold() + { + using var pipeline = Pipeline.Create("ImageThreshold"); + using var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat); + this.testImage2.CopyTo(sharedImage.Resource); + Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false).Threshold(10, 170, Threshold.Binary).Do((img) => + { + this.AssertAreImagesEqual(this.testImage2_Threshold, img.Resource); + }); + pipeline.Run(); + } + + [TestMethod] + [Timeout(60000)] + public void Image_ExtractChannels() + { + using var pipeline = Pipeline.Create("ImageExtractChannel"); + using var sharedImage = ImagePool.GetOrCreate(this.testImage2.Width, this.testImage2.Height, this.testImage2.PixelFormat); + this.testImage2.CopyTo(sharedImage.Resource); + var seq = Generators.Sequence(pipeline, new[] { sharedImage }, default, null, keepOpen: false); + var rchannel = seq.ExtractChannel(0); + var gchannel = seq.ExtractChannel(1); + var bchannel = seq.ExtractChannel(2); + rchannel.Join(gchannel.Join(bchannel)).Do((imgs) => + { + this.AssertAreImagesEqual(this.testImage2_RedChannel, imgs.Item1.Resource); + this.AssertAreImagesEqual(this.testImage2_GreenChannel, imgs.Item2.Resource); + this.AssertAreImagesEqual(this.testImage2_BlueChannel, imgs.Item3.Resource); + }); + pipeline.Run(); + } [TestMethod] [Timeout(60000)] @@ -97,25 +309,25 @@ public void Image_Crop() // Crop the entire image region (a no-op) and verify that the original image is preserved using (var croppedImage = this.testImage.Crop(0, 0, this.testImage.Width, this.testImage.Height)) { - this.AssertAreImagesEqual(this.testImage, croppedImage.Resource); + this.AssertAreImagesEqual(this.testImage, croppedImage); } // Crop an upper-left region and verify using (var croppedImage = this.testImage.Crop(0, 0, 200, 100)) { - this.AssertAreImagesEqual(this.testImage_0_0_200_100, croppedImage.Resource); + this.AssertAreImagesEqual(this.testImage_0_0_200_100, croppedImage); } // Crop a lower-right region and verify using (var croppedImage = this.testImage.Crop(153, 57, 103, 199)) { - this.AssertAreImagesEqual(this.testImage_153_57_103_199, croppedImage.Resource); + this.AssertAreImagesEqual(this.testImage_153_57_103_199, croppedImage); } // Crop an interior region and verify using (var croppedImage = this.testImage.Crop(73, 41, 59, 37)) { - this.AssertAreImagesEqual(this.testImage_73_41_59_37, croppedImage.Resource); + this.AssertAreImagesEqual(this.testImage_73_41_59_37, croppedImage); } } @@ -124,9 +336,9 @@ public void Image_Crop() public void Image_CropDifferentRegions() { // Crop a slightly different interior region of the same size and verify that the data is different (as a sanity check) - using (var sharedCroppedImage = this.testImage.Crop(74, 42, 59, 37)) + using (var croppedImage = this.testImage.Crop(74, 42, 59, 37)) { - var croppedImage_74_42_59_37 = sharedCroppedImage.Resource; + var croppedImage_74_42_59_37 = croppedImage; CollectionAssert.AreNotEqual( this.testImage_73_41_59_37.ReadBytes(this.testImage_73_41_59_37.Size), croppedImage_74_42_59_37.ReadBytes(croppedImage_74_42_59_37.Size)); @@ -138,11 +350,9 @@ public void Image_CropDifferentRegions() public void EncodeImage() { // Crop a slightly different interior region of the same size and verify that the data is different (as a sanity check) - EncodedImage encImg = new EncodedImage(); - ImageEncoder.EncodeFrom(encImg, this.testImage, new PngBitmapEncoder()); - Image target = new Image(this.testImage.Width, this.testImage.Height, this.testImage.Stride, this.testImage.PixelFormat); - ImageDecoder.DecodeTo(encImg, target); - this.AssertAreImagesEqual(this.testImage, target); + var encodedImage = this.testImage.Encode(new ImageToPngStreamEncoder()); + var decodedImage = encodedImage.Decode(new ImageFromStreamDecoder()); + this.AssertAreImagesEqual(this.testImage, decodedImage); } [TestMethod] @@ -150,13 +360,10 @@ public void EncodeImage() public void EncodeImageJpg() { // Crop a slightly different interior region of the same size and verify that the data is different (as a sanity check) - EncodedImage encImg = new EncodedImage(); - ImageEncoder.EncodeFrom(encImg, this.testImage, new JpegBitmapEncoder()); - Image target = new Image(this.testImage.Width, this.testImage.Height, this.testImage.Stride, this.testImage.PixelFormat); - ImageDecoder.DecodeTo(encImg, target); - Image target2 = new Image(this.testImage.Width, this.testImage.Height, this.testImage.Stride, this.testImage.PixelFormat); - ImageDecoder.DecodeTo(encImg, target2); - this.AssertAreImagesEqual(target, target2); + var encodedImage = this.testImage.Encode(new ImageToJpegStreamEncoder()); + var decodedImage = encodedImage.Decode(new ImageFromStreamDecoder()); + var decodedImage2 = encodedImage.Decode(new ImageFromStreamDecoder()); + this.AssertAreImagesEqual(decodedImage, decodedImage2); } [TestMethod] @@ -164,32 +371,102 @@ public void EncodeImageJpg() public void Image_Scale() { // Scale using nearest-neighbor - this.AssertAreImagesEqual(this.testImage_50_25, this.testImage.Scale(0.5f, 0.25f, SamplingMode.Point).Resource); + this.AssertAreImagesEqual(this.testImage_50_25_Cubic, this.testImage.Scale( + (float)this.testImage_50_25_Cubic.Width / (float)this.testImage.Width, + (float)this.testImage_50_25_Cubic.Height / (float)this.testImage.Height, + SamplingMode.Bicubic)); // Scale using bilinear - this.AssertAreImagesEqual(this.testImage_150_125, this.testImage.Scale(1.5f, 1.25f, SamplingMode.Bilinear).Resource); + this.AssertAreImagesEqual(this.testImage_150_125_Point, this.testImage.Scale( + (float)this.testImage_150_125_Point.Width / (float)this.testImage.Width, + (float)this.testImage_150_125_Point.Height / (float)this.testImage.Height, + SamplingMode.Point)); // Scale using bicubic - this.AssertAreImagesEqual(this.testImage_25_200, this.testImage.Scale(0.25f, 2.0f, SamplingMode.Bicubic).Resource); + this.AssertAreImagesEqual(this.testImage_25_200_Linear, this.testImage.Scale( + (float)this.testImage_25_200_Linear.Width / (float)this.testImage.Width, + (float)this.testImage_25_200_Linear.Height / (float)this.testImage.Height, + SamplingMode.Bilinear)); // Attempt to scale 16bpp grayscale - should throw NotSupportedException - var depthImage = Image.Create(100, 100, PixelFormat.Gray_16bpp); - Assert.ThrowsException(() => depthImage.Scale(0.5f, 0.5f, SamplingMode.Point)); + var depthImage = new Image(100, 100, PixelFormat.Gray_16bpp); + Assert.ThrowsException(() => depthImage.Scale(0.5f, 0.5f, SamplingMode.Point)); } - private void AssertAreImagesEqual(Image referenceImage, Image subjectImage) + [TestMethod] + [Timeout(60000)] + public void Image_Serialize() { + // create the serialization context + var knownSerializers = new KnownSerializers(); + var context = new SerializationContext(knownSerializers); + + // serialize the image + var writer = new BufferWriter(this.testImage.Size); + Serializer.Serialize(writer, this.testImage, context); + + // verify the image type schema + string contract = TypeSchema.GetContractName(typeof(Image), knownSerializers.RuntimeVersion); + Assert.IsTrue(knownSerializers.Schemas.ContainsKey(contract)); + + // deserialize the image and verify the data + Image targetImage = null; + var reader = new BufferReader(writer.Buffer); + Serializer.Deserialize(reader, ref targetImage, context); + this.AssertAreImagesEqual(this.testImage, targetImage); + } + + [TestMethod] + [Timeout(60000)] + public void DepthImage_Serialize() + { + // generate a "depth image" for testing + var testImage16bpp = new Image(this.testImage.Width, this.testImage.Height, PixelFormat.Gray_16bpp); + this.testImage.CopyTo(testImage16bpp); + var testDepthImage = DepthImage.CreateFrom(testImage16bpp.ToBitmap()); + + // create the serialization context + var knownSerializers = new KnownSerializers(); + var context = new SerializationContext(knownSerializers); + + // serialize the image + var writer = new BufferWriter(testDepthImage.Size); + Serializer.Serialize(writer, testDepthImage, context); + + // verify the image type schema + string contract = TypeSchema.GetContractName(typeof(DepthImage), knownSerializers.RuntimeVersion); + Assert.IsTrue(knownSerializers.Schemas.ContainsKey(contract)); + + // deserialize the image and verify the data + DepthImage targetDepthImage = null; + var reader = new BufferReader(writer.Buffer); + Serializer.Deserialize(reader, ref targetDepthImage, context); + this.AssertAreImagesEqual(testDepthImage, targetDepthImage); + } + + private void AssertAreImagesEqual(ImageBase referenceImage, ImageBase subjectImage) + { + Assert.AreEqual(referenceImage.GetType(), subjectImage.GetType()); Assert.AreEqual(referenceImage.PixelFormat, subjectImage.PixelFormat); Assert.AreEqual(referenceImage.Width, subjectImage.Width); Assert.AreEqual(referenceImage.Height, subjectImage.Height); - Assert.AreEqual(referenceImage.Size, subjectImage.Size); // compare one line of the image at a time since a stride may contain padding bytes for (int line = 0; line < referenceImage.Height; line++) { - CollectionAssert.AreEqual( - referenceImage.ReadBytes(referenceImage.Width * referenceImage.BitsPerPixel / 8, line * referenceImage.Stride), - subjectImage.ReadBytes(subjectImage.Width * subjectImage.BitsPerPixel / 8, line * subjectImage.Stride)); + var refbytes = referenceImage.ReadBytes(referenceImage.Width * referenceImage.BitsPerPixel / 8, line * referenceImage.Stride); + var subjbytes = subjectImage.ReadBytes(subjectImage.Width * subjectImage.BitsPerPixel / 8, line * subjectImage.Stride); + if (referenceImage.PixelFormat == PixelFormat.BGRX_32bpp) + { + // BGRX images can have any value for they alpha channel. Here we normalize them to 255 + for (int i = 0; i < referenceImage.Width; i++) + { + refbytes[4 * i + 3] = 255; + subjbytes[4 * i + 3] = 255; + } + } + + CollectionAssert.AreEqual(refbytes, subjbytes); } } } diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/AssemblyInfo.cs b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/AssemblyInfo.cs index afbcb7fab..6f4b42ca8 100644 --- a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/AssemblyInfo.cs +++ b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/AssemblyInfo.cs @@ -10,6 +10,6 @@ [assembly: AssemblyCopyright("Copyright (c) Microsoft Corporation. All rights reserved.")] [assembly: ComVisible(false)] [assembly: Guid("191df615-3d8f-45a3-b763-dd4a604a712a")] -[assembly: AssemblyVersion("0.11.82.2")] -[assembly: AssemblyFileVersion("0.11.82.2")] -[assembly: AssemblyInformationalVersion("0.11.82.2-beta")] +[assembly: AssemblyVersion("0.12.53.2")] +[assembly: AssemblyFileVersion("0.12.53.2")] +[assembly: AssemblyInformationalVersion("0.12.53.2-beta")] diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.Designer.cs b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.Designer.cs index d8bc8e872..4a3928b52 100644 --- a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.Designer.cs +++ b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.Designer.cs @@ -103,9 +103,9 @@ internal class Resources { /// /// Looks up a localized resource of type System.Drawing.Bitmap. /// - internal static System.Drawing.Bitmap TestImage_Scale_150_125 { + internal static System.Drawing.Bitmap TestImage_Scale_150_125_Point { get { - object obj = ResourceManager.GetObject("TestImage_Scale_150_125", resourceCulture); + object obj = ResourceManager.GetObject("TestImage_Scale_150_125_Point", resourceCulture); return ((System.Drawing.Bitmap)(obj)); } } @@ -113,9 +113,9 @@ internal class Resources { /// /// Looks up a localized resource of type System.Drawing.Bitmap. /// - internal static System.Drawing.Bitmap TestImage_Scale_25_200 { + internal static System.Drawing.Bitmap TestImage_Scale_25_200_Linear { get { - object obj = ResourceManager.GetObject("TestImage_Scale_25_200", resourceCulture); + object obj = ResourceManager.GetObject("TestImage_Scale_25_200_Linear", resourceCulture); return ((System.Drawing.Bitmap)(obj)); } } @@ -123,9 +123,212 @@ internal class Resources { /// /// Looks up a localized resource of type System.Drawing.Bitmap. /// - internal static System.Drawing.Bitmap TestImage_Scale_50_25 { + internal static System.Drawing.Bitmap TestImage_Scale_50_25_Cubic { get { - object obj = ResourceManager.GetObject("TestImage_Scale_50_25", resourceCulture); + object obj = ResourceManager.GetObject("TestImage_Scale_50_25_Cubic", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2 + { + get + { + object obj = ResourceManager.GetObject("TestImage2", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_Threshold + { + get + { + object obj = ResourceManager.GetObject("TestImage2_Threshold", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_RedChannel + { + get + { + object obj = ResourceManager.GetObject("TestImage2_RedChannel", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_GreenChannel + { + get + { + object obj = ResourceManager.GetObject("TestImage2_GreenChannel", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_BlueChannel + { + get + { + object obj = ResourceManager.GetObject("TestImage2_BlueChannel", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_Mask + { + get + { + object obj = ResourceManager.GetObject("TestImage2_Mask", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_FlipHoriz + { + get + { + object obj = ResourceManager.GetObject("TestImage2_FlipHoriz", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_CopyImage + { + get + { + object obj = ResourceManager.GetObject("TestImage2_CopyImage", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_Invert + { + get + { + object obj = ResourceManager.GetObject("TestImage2_Invert", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_FlipVert + { + get + { + object obj = ResourceManager.GetObject("TestImage2_FlipVert", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_Rotate_Neg10 + { + get + { + object obj = ResourceManager.GetObject("TestImage2_Rotate_Neg10", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_Rotate_110 + { + get + { + object obj = ResourceManager.GetObject("TestImage2_Rotate_110", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_DrawRect + { + get + { + object obj = ResourceManager.GetObject("TestImage2_DrawRect", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_DrawLine + { + get + { + object obj = ResourceManager.GetObject("TestImage2_DrawLine", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_DrawCircle + { + get + { + object obj = ResourceManager.GetObject("TestImage2_DrawCircle", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_DrawText + { + get + { + object obj = ResourceManager.GetObject("TestImage2_DrawText", resourceCulture); + return ((System.Drawing.Bitmap)(obj)); + } + } + /// + /// Looks up a localized resource of type System.Drawing.Bitmap. + /// + internal static System.Drawing.Bitmap TestImage2_AbsDiff + { + get + { + object obj = ResourceManager.GetObject("TestImage2_AbsDiff", resourceCulture); return ((System.Drawing.Bitmap)(obj)); } } diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.resx b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.resx index 254f4fd72..232ccfaed 100644 --- a/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.resx +++ b/Sources/Imaging/Test.Psi.Imaging.Windows/Properties/Resources.resx @@ -121,6 +121,60 @@ ..\Resources\TestImage.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + ..\Resources\TestImage2.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_RedChannel.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_GreenChannel.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_BlueChannel.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_FlipHoriz.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_AbsDiff.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_FlipVert.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Rotate_Neg10.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Rotate_110.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_DrawRect.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_DrawLine.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_DrawCircle.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_DrawText.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_CopyImage.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Invert.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Threshold.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Mask.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\Resources\TestImage2_Mask.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + ..\Resources\TestImage_Crop_0_0_200_100.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a @@ -130,13 +184,13 @@ ..\Resources\testimage_crop_73_41_59_37.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - ..\Resources\TestImage_Scale_150_125.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + ..\Resources\TestImage_Scale_150_125_Point.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - ..\Resources\TestImage_Scale_25_200.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + ..\Resources\TestImage_Scale_25_200_Linear.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - ..\Resources\TestImage_Scale_50_25.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + ..\Resources\TestImage_Scale_50_25_Cubic.bmp;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - \ No newline at end of file + diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2.bmp new file mode 100644 index 000000000..313db424e Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_AbsDiff.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_AbsDiff.bmp new file mode 100644 index 000000000..f85462b00 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_AbsDiff.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_BlueChannel.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_BlueChannel.bmp new file mode 100644 index 000000000..50df048ad Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_BlueChannel.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_CopyImage.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_CopyImage.bmp new file mode 100644 index 000000000..f75e500bf Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_CopyImage.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawCircle.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawCircle.bmp new file mode 100644 index 000000000..a6b0aad08 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawCircle.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawLine.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawLine.bmp new file mode 100644 index 000000000..339b519ac Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawLine.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawRect.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawRect.bmp new file mode 100644 index 000000000..a46af5aa4 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawRect.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawText.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawText.bmp new file mode 100644 index 000000000..87d85fee9 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_DrawText.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipHoriz.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipHoriz.bmp new file mode 100644 index 000000000..64501cf36 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipHoriz.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipVert.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipVert.bmp new file mode 100644 index 000000000..2c144f49e Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_FlipVert.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_GreenChannel.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_GreenChannel.bmp new file mode 100644 index 000000000..c1bd3a95b Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_GreenChannel.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Invert.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Invert.bmp new file mode 100644 index 000000000..e2244a183 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Invert.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask-6-0.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask-6-0.bmp new file mode 100644 index 000000000..2eb83aaf6 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask-6-0.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask.bmp new file mode 100644 index 000000000..8fca6f938 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Mask.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_RedChannel.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_RedChannel.bmp new file mode 100644 index 000000000..cf6548ac6 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_RedChannel.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_110.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_110.bmp new file mode 100644 index 000000000..f4f15b377 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_110.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_Neg10.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_Neg10.bmp new file mode 100644 index 000000000..11f1d39c2 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Rotate_Neg10.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold-2-0.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold-2-0.bmp new file mode 100644 index 000000000..7d41dc09f Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold-2-0.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp new file mode 100644 index 000000000..7d41dc09f Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp-4-0.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp-4-0.bmp new file mode 100644 index 000000000..7d41dc09f Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage2_Threshold.bmp-4-0.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125.bmp deleted file mode 100644 index 864d75d75..000000000 Binary files a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125.bmp and /dev/null differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125_Point.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125_Point.bmp new file mode 100644 index 000000000..fd58f44cd Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_150_125_Point.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200.bmp deleted file mode 100644 index 0f8b72b1e..000000000 Binary files a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200.bmp and /dev/null differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200_Linear.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200_Linear.bmp new file mode 100644 index 000000000..5a7a4b19d Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_25_200_Linear.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25.bmp deleted file mode 100644 index 648bd5d29..000000000 Binary files a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25.bmp and /dev/null differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25_Cubic.bmp b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25_Cubic.bmp new file mode 100644 index 000000000..a2ee05ed2 Binary files /dev/null and b/Sources/Imaging/Test.Psi.Imaging.Windows/Resources/TestImage_Scale_50_25_Cubic.bmp differ diff --git a/Sources/Imaging/Test.Psi.Imaging.Windows/Test.Psi.Imaging.Windows.csproj b/Sources/Imaging/Test.Psi.Imaging.Windows/Test.Psi.Imaging.Windows.csproj index f90860ef1..88961e75a 100644 --- a/Sources/Imaging/Test.Psi.Imaging.Windows/Test.Psi.Imaging.Windows.csproj +++ b/Sources/Imaging/Test.Psi.Imaging.Windows/Test.Psi.Imaging.Windows.csproj @@ -100,17 +100,23 @@ ResXFileCodeGenerator Resources.Designer.cs + Designer + + 2.9.8 + runtime; build; native; contentfiles; analyzers + all + - 2.1.0 + 2.1.1 - 2.1.0 + 2.1.1 1.1.118 @@ -137,13 +143,26 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/FaceRecognizer.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/FaceRecognizer.cs index 3f8bcf374..9cafd07af 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/FaceRecognizer.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/FaceRecognizer.cs @@ -1,124 +1,124 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.CognitiveServices.Face -{ - using System; +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.CognitiveServices.Face +{ + using System; using System.Collections.Generic; using System.Drawing.Imaging; - using System.IO; - using System.Linq; - using System.Threading.Tasks; - using Microsoft.Azure.CognitiveServices.Vision.Face; - using Microsoft.Azure.CognitiveServices.Vision.Face.Models; - using Microsoft.Psi; - using Microsoft.Psi.Components; - using Microsoft.Psi.Imaging; - - /// - /// Component that performs face recognition via Microsoft Cognitive Services Face API. - /// - /// The component takes in a stream of images and produces a stream of messages containing detected faces and candidate identities of each - /// person in the image. A Microsoft Cognitive Services Face API - /// subscription key is required to use this component. In addition, a person group needs to be created ahead of time, and the id of the person group - /// passed to the component via the configuration. For more information, and to see how to create person groups, see the full direct API for. - /// Microsoft Cognitive Services Face API - /// - public sealed class FaceRecognizer : AsyncConsumerProducer, IList>>, IDisposable - { + using System.IO; + using System.Linq; + using System.Threading.Tasks; + using Microsoft.Azure.CognitiveServices.Vision.Face; + using Microsoft.Azure.CognitiveServices.Vision.Face.Models; + using Microsoft.Psi; + using Microsoft.Psi.Components; + using Microsoft.Psi.Imaging; + + /// + /// Component that performs face recognition via Microsoft Cognitive Services Face API. + /// + /// The component takes in a stream of images and produces a stream of messages containing detected faces and candidate identities of each + /// person in the image. A Microsoft Cognitive Services Face API + /// subscription key is required to use this component. In addition, a person group needs to be created ahead of time, and the id of the person group + /// passed to the component via the configuration. For more information, and to see how to create person groups, see the full direct API for. + /// Microsoft Cognitive Services Face API + /// + public sealed class FaceRecognizer : AsyncConsumerProducer, IList>>, IDisposable + { /// /// Empty results. - /// - private static readonly IList> Empty = new IList<(string, double)>[0]; - - /// - /// The configuration to use for this component. - /// - private readonly FaceRecognizerConfiguration configuration; - - /// - /// The client that communicates with the cloud image analyzer service. - /// - private FaceClient client = null; - - /// - /// The group of persons from the cognitive services API. - /// - private Dictionary people = null; - - /// - /// Initializes a new instance of the class. - /// - /// The pipeline to add the component to. - /// The component configuration. - public FaceRecognizer(Pipeline pipeline, FaceRecognizerConfiguration configuration) - : base(pipeline) - { - this.configuration = configuration; - this.RateLimitExceeded = pipeline.CreateEmitter(this, nameof(this.RateLimitExceeded)); - this.client = new FaceClient(new ApiKeyServiceClientCredentials(this.configuration.SubscriptionKey)) - { - Endpoint = this.configuration.Endpoint, - }; - this.client.PersonGroupPerson.ListAsync(this.configuration.PersonGroupId).ContinueWith(list => - { - this.people = list.Result.ToDictionary(p => p.PersonId); - }).Wait(); - } - - /// - /// Gets a stream that indicates the rate limit for the service calls was exceeded. - /// - /// A value of true is posted on the stream each time the rate limit for calling Cognitive Services FACE API is exceeded. - public Emitter RateLimitExceeded { get; } - - /// - /// Disposes the face recognizer component. - /// - public void Dispose() - { - this.client.Dispose(); - this.client = null; - } - - /// - protected override async Task ReceiveAsync(Shared data, Envelope e) - { - using (Stream imageFileStream = new MemoryStream()) - { - try - { - // convert image to a Stream and send to Cog Services - data.Resource.ToManagedImage(false).Save(imageFileStream, ImageFormat.Jpeg); - imageFileStream.Seek(0, SeekOrigin.Begin); - - var detected = (await this.client.Face.DetectWithStreamAsync(imageFileStream, recognitionModel: this.configuration.RecognitionModelName)).Select(d => d.FaceId.Value).ToList(); - - // Identify each face - if (detected.Count > 0) - { - var identified = await this.client.Face.IdentifyAsync(detected, this.configuration.PersonGroupId); - var results = identified.Select(p => (IList<(string, double)>)p.Candidates.Select(c => (this.people[c.PersonId].Name, c.Confidence)).ToList()).ToList(); - this.Out.Post(results, e.OriginatingTime); - } + /// + private static readonly IList> Empty = new IList<(string, double)>[0]; + + /// + /// The configuration to use for this component. + /// + private readonly FaceRecognizerConfiguration configuration; + + /// + /// The client that communicates with the cloud image analyzer service. + /// + private FaceClient client = null; + + /// + /// The group of persons from the cognitive services API. + /// + private Dictionary people = null; + + /// + /// Initializes a new instance of the class. + /// + /// The pipeline to add the component to. + /// The component configuration. + public FaceRecognizer(Pipeline pipeline, FaceRecognizerConfiguration configuration) + : base(pipeline) + { + this.configuration = configuration; + this.RateLimitExceeded = pipeline.CreateEmitter(this, nameof(this.RateLimitExceeded)); + this.client = new FaceClient(new ApiKeyServiceClientCredentials(this.configuration.SubscriptionKey)) + { + Endpoint = this.configuration.Endpoint, + }; + this.client.PersonGroupPerson.ListAsync(this.configuration.PersonGroupId).ContinueWith(list => + { + this.people = list.Result.ToDictionary(p => p.PersonId); + }).Wait(); + } + + /// + /// Gets a stream that indicates the rate limit for the service calls was exceeded. + /// + /// A value of true is posted on the stream each time the rate limit for calling Cognitive Services FACE API is exceeded. + public Emitter RateLimitExceeded { get; } + + /// + /// Disposes the face recognizer component. + /// + public void Dispose() + { + this.client.Dispose(); + this.client = null; + } + + /// + protected override async Task ReceiveAsync(Shared data, Envelope e) + { + using (Stream imageFileStream = new MemoryStream()) + { + try + { + // convert image to a Stream and send to Cog Services + data.Resource.ToBitmap(false).Save(imageFileStream, ImageFormat.Jpeg); + imageFileStream.Seek(0, SeekOrigin.Begin); + + var detected = (await this.client.Face.DetectWithStreamAsync(imageFileStream, recognitionModel: this.configuration.RecognitionModelName)).Select(d => d.FaceId.Value).ToList(); + + // Identify each face + if (detected.Count > 0) + { + var identified = await this.client.Face.IdentifyAsync(detected, this.configuration.PersonGroupId); + var results = identified.Select(p => (IList<(string, double)>)p.Candidates.Select(c => (this.people[c.PersonId].Name, c.Confidence)).ToList()).ToList(); + this.Out.Post(results, e.OriginatingTime); + } else { this.Out.Post(Empty, e.OriginatingTime); - } - } - catch (APIErrorException exception) - { - // swallow exceptions unless it's a rate limit exceeded - if (exception.Body.Error.Code == "RateLimitExceeded") - { - this.RateLimitExceeded.Post(true, e.OriginatingTime); - } - else - { - throw exception; - } - } - } - } - } + } + } + catch (APIErrorException exception) + { + // swallow exceptions unless it's a rate limit exceeded + if (exception.Body.Error.Code == "RateLimitExceeded") + { + this.RateLimitExceeded.Post(true, e.OriginatingTime); + } + else + { + throw; + } + } + } + } + } } \ No newline at end of file diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/Microsoft.Psi.CognitiveServices.Face.csproj b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/Microsoft.Psi.CognitiveServices.Face.csproj index 4a50350df..7e8fc470f 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/Microsoft.Psi.CognitiveServices.Face.csproj +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/Microsoft.Psi.CognitiveServices.Face.csproj @@ -3,7 +3,7 @@ netstandard2.0 true true - false + ../../../../Build/Microsoft.Psi.ruleset Provides components for using Microsoft's Cognitive Services Face API. @@ -21,6 +21,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/PersonGroupTasks.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/PersonGroupTasks.cs index d0dd918a9..c3b4271d5 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/PersonGroupTasks.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Face/PersonGroupTasks.cs @@ -62,7 +62,7 @@ public static void TestPersonGroup(string subscriptionKey, string endpoint, stri { var files = Generators.Sequence(pipeline, Directory.GetFiles(directory), TimeSpan.FromTicks(1)); files - .Select(file => ImagePool.GetOrCreate(new Bitmap(File.OpenRead(file)))) + .Select(file => ImagePool.GetOrCreateFromBitmap(new Bitmap(File.OpenRead(file)))) .RecognizeFace(new FaceRecognizerConfiguration(subscriptionKey, endpoint, groupId)) .Join(files) .Do(x => diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/Microsoft.Psi.CognitiveServices.Language.Windows.csproj b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/Microsoft.Psi.CognitiveServices.Language.Windows.csproj index 6e4e62b2a..832076518 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/Microsoft.Psi.CognitiveServices.Language.Windows.csproj +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/Microsoft.Psi.CognitiveServices.Language.Windows.csproj @@ -25,6 +25,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/PersonalityChat.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/PersonalityChat.cs index 828d60c9d..1ae55002c 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/PersonalityChat.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language.Windows/PersonalityChat.cs @@ -9,7 +9,7 @@ namespace Microsoft.Psi.CognitiveServices.Language using Microsoft.Psi.Components; /// - /// Component that generates dialogue responses to textual inputs using the Microsoft Cognitive Services Personality Chat API. + /// Component that generates dialog responses to textual inputs using the Microsoft Cognitive Services Personality Chat API. /// /// /// The component takes in text-based phrases or utterances on its input stream and generates diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language/Microsoft.Psi.CognitiveServices.Language.csproj b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language/Microsoft.Psi.CognitiveServices.Language.csproj index 54aeb27f6..8123146b1 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language/Microsoft.Psi.CognitiveServices.Language.csproj +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Language/Microsoft.Psi.CognitiveServices.Language.csproj @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizer.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizer.cs index 0988f179f..5beaef4e8 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizer.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizer.cs @@ -26,7 +26,7 @@ namespace Microsoft.Psi.CognitiveServices.Speech /// public sealed class AzureSpeechRecognizer : AsyncConsumerProducer, IStreamingSpeechRecognitionResult>, IDisposable { - // For cancelling any pending recognition tasks before disposal + // For canceling any pending recognition tasks before disposal private readonly CancellationTokenSource cancellationTokenSource = new CancellationTokenSource(); /// @@ -295,7 +295,7 @@ private void OnResponseReceivedHandler(object sender, SpeechResponseEventArgs e) // get the current (oldest) recognition task from the queue if (!this.pendingRecognitionTasks.TryPeek(out var currentRecognitionTask)) { - // This proabaly means that we have just received an end-of-dictation response which normally + // This probably means that we have just received an end-of-dictation response which normally // arrives after a successful recognition result, so we would have already completed the // recognition task. Hence we just ignore the response. return; @@ -343,7 +343,7 @@ private void OnPartialResponseReceivedHandler(object sender, PartialSpeechRespon // update the in-progress recognition task currentRecognitionTask.AppendResult(e.PartialResult); - // Since this is a partial response, VAD may not yet have signalled the end of speech + // Since this is a partial response, VAD may not yet have signaled the end of speech // so just use the last audio packet time (which will probably be ahead). this.PostWithOriginatingTimeConsistencyCheck(this.PartialRecognitionResults, currentRecognitionTask.BuildPartialSpeechRecognitionResult(), originatingTime); diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizerConfiguration.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizerConfiguration.cs index 5604ae6a5..029e9e9cf 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizerConfiguration.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/AzureSpeechRecognizerConfiguration.cs @@ -19,12 +19,6 @@ public sealed class AzureSpeechRecognizerConfiguration /// public AzureSpeechRecognizerConfiguration() { - this.Language = "en-us"; - this.SubscriptionKey = null; // This must be set to the key associated with your account - this.Region = null; // This must be set to the region associated to the key - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -35,7 +29,7 @@ public AzureSpeechRecognizerConfiguration() /// this defaults to "en-us" (U.S. English). Other supported locales include "en-gb", /// "de-de", "es-es", "fr-fr", "it-it" and "zh-cn". /// - public string Language { get; set; } + public string Language { get; set; } = "en-us"; /// /// Gets or sets the subscription key. @@ -47,15 +41,15 @@ public AzureSpeechRecognizerConfiguration() /// https://azure.microsoft.com/en-us/try/cognitive-services/ for more information on obtaining a /// subscription. /// - public string SubscriptionKey { get; set; } + public string SubscriptionKey { get; set; } = null; /// - /// Gets or sets the expected input format of the audio stream. + /// Gets the expected input format of the audio stream. /// /// /// Currently, the only supported input audio format is 16000 Hz, 1-channel, 16-bit PCM. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; } = WaveFormat.Create16kHz1Channel16BitPcm(); /// /// Gets or sets the region. @@ -63,6 +57,6 @@ public AzureSpeechRecognizerConfiguration() /// /// This is the region that is associated to the subscription key. /// - public string Region { get; set; } + public string Region { get; set; } = null; } } diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizer.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizer.cs index c9fc200b1..312bb8344 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizer.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizer.cs @@ -36,7 +36,7 @@ namespace Microsoft.Psi.CognitiveServices.Speech [Obsolete("The Bing Speech service will be retired soon. Please use the AzureSpeechRecognizer instead.", false)] public sealed class BingSpeechRecognizer : AsyncConsumerProducer, IStreamingSpeechRecognitionResult>, ISourceComponent, IDisposable { - // For cancelling any pending recognition tasks before disposal + // For canceling any pending recognition tasks before disposal private readonly CancellationTokenSource cancellationTokenSource = new CancellationTokenSource(); /// @@ -357,7 +357,7 @@ private void OnPartialResponseReceivedHandler(object sender, PartialSpeechRespon this.lastPartialResult = e.PartialResult.Text; var result = this.BuildPartialSpeechRecognitionResult(e.PartialResult.Text); - // Since this is a partial response, VAD may not yet have signalled the end of speech + // Since this is a partial response, VAD may not yet have signaled the end of speech // so just use the last audio packet time (which will probably be ahead). this.PostWithOriginatingTimeConsistencyCheck(this.PartialRecognitionResults, result, this.lastAudioOriginatingTime); diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizerConfiguration.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizerConfiguration.cs index df8a4799a..82ba94cbd 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizerConfiguration.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/BingSpeechRecognizerConfiguration.cs @@ -29,12 +29,6 @@ public sealed class BingSpeechRecognizerConfiguration /// public BingSpeechRecognizerConfiguration() { - this.Language = "en-us"; - this.RecognitionMode = SpeechRecognitionMode.Interactive; - this.SubscriptionKey = null; // This must be set to the key associated with your account - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -45,7 +39,7 @@ public BingSpeechRecognizerConfiguration() /// this defaults to "en-us" (U.S. English). Other supported locales include "en-gb", /// "de-de", "es-es", "fr-fr", "it-it" and "zh-cn". /// - public string Language { get; set; } + public string Language { get; set; } = "en-us"; /// /// Gets or sets the speech recognition mode. @@ -54,7 +48,7 @@ public BingSpeechRecognizerConfiguration() /// The speech recognition mode must be one of the values defined in the enumeration /// . The default value is . /// - public SpeechRecognitionMode RecognitionMode { get; set; } + public SpeechRecognitionMode RecognitionMode { get; set; } = SpeechRecognitionMode.Interactive; /// /// Gets or sets the subscription key. @@ -71,14 +65,14 @@ public BingSpeechRecognizerConfiguration() /// instead. You may obtain a subscription key for the Azure Speech service here: /// https://azure.microsoft.com/en-us/try/cognitive-services/?api=speech-services. /// - public string SubscriptionKey { get; set; } + public string SubscriptionKey { get; set; } = null; /// - /// Gets or sets the expected input format of the audio stream. + /// Gets the expected input format of the audio stream. /// /// /// Currently, the only supported input audio format is 16000 Hz, 1-channel, 16-bit PCM. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Microsoft.Psi.CognitiveServices.Speech.csproj b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Microsoft.Psi.CognitiveServices.Speech.csproj index aaaed584b..01f39a098 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Microsoft.Psi.CognitiveServices.Speech.csproj +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Microsoft.Psi.CognitiveServices.Speech.csproj @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechRecognitionClient.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechRecognitionClient.cs index 4751b9613..8e0e3b153 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechRecognitionClient.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechRecognitionClient.cs @@ -57,12 +57,12 @@ public SpeechRecognitionClient(SpeechRecognitionMode recognitionMode, string lan } /// - /// An event that is raised when a final speech recognition response has beeen received. + /// An event that is raised when a final speech recognition response has been received. /// public event EventHandler OnResponseReceived; /// - /// An event that is raised when a partial speech recognition response has beeen received. + /// An event that is raised when a partial speech recognition response has been received. /// public event EventHandler OnPartialResponseReceived; @@ -76,6 +76,7 @@ public void Dispose() { this.authentication.Dispose(); this.semaphore.Dispose(); + this.webSocket.Dispose(); } /// diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechStartDetectedMessage.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechStartDetectedMessage.cs index e546f8c30..9e06f9819 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechStartDetectedMessage.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/Service/SpeechStartDetectedMessage.cs @@ -17,7 +17,7 @@ public SpeechStartDetectedMessage() } /// - /// Gets or sets the the offset (in 100-nanosecond units) when speech was detected in the + /// Gets or sets the offset (in 100-nanosecond units) when speech was detected in the /// audio stream, relative to the start of the stream. /// public int Offset { get; set; } diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/ImageAnalyzer.cs b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/ImageAnalyzer.cs index e5ff9b586..ba740e9f8 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/ImageAnalyzer.cs +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/ImageAnalyzer.cs @@ -242,7 +242,7 @@ private async Task ReceiveAsync(Shared data, Envelope e) using (Stream imageFileStream = new MemoryStream()) { // convert image to a stream and send to service - data.Resource.ToManagedImage(false).Save(imageFileStream, System.Drawing.Imaging.ImageFormat.Jpeg); + data.Resource.ToBitmap(false).Save(imageFileStream, System.Drawing.Imaging.ImageFormat.Jpeg); imageFileStream.Seek(0, SeekOrigin.Begin); try { diff --git a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/Microsoft.Psi.CognitiveServices.Vision.csproj b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/Microsoft.Psi.CognitiveServices.Vision.csproj index 845c32f04..5dbdf378d 100644 --- a/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/Microsoft.Psi.CognitiveServices.Vision.csproj +++ b/Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Vision/Microsoft.Psi.CognitiveServices.Vision.csproj @@ -28,6 +28,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Language/Test.Psi.CognitiveServices.Language.csproj b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Language/Test.Psi.CognitiveServices.Language.csproj index 672c17311..c48f69423 100644 --- a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Language/Test.Psi.CognitiveServices.Language.csproj +++ b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Language/Test.Psi.CognitiveServices.Language.csproj @@ -1,8 +1,8 @@ - + Exe - netcoreapp2.0 + netcoreapp3.1 false Test.Psi.CognitiveServices.Language.ConsoleMain @@ -30,10 +30,14 @@ - + + all + runtime; build; native; contentfiles; analyzers + + - - + + diff --git a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Speech/Test.Psi.CognitiveServices.Speech.csproj b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Speech/Test.Psi.CognitiveServices.Speech.csproj index fd899f1a0..3adb40897 100644 --- a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Speech/Test.Psi.CognitiveServices.Speech.csproj +++ b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Speech/Test.Psi.CognitiveServices.Speech.csproj @@ -1,8 +1,8 @@ - + Exe - netcoreapp2.0 + netcoreapp3.1 false Test.Psi.CognitiveServices.Speech.ConsoleMain @@ -21,7 +21,7 @@ - + @@ -30,10 +30,14 @@ - + + all + runtime; build; native; contentfiles; analyzers + + - - + + diff --git a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Vision.Windows/Test.Psi.CognitiveServices.Vision.Windows.csproj b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Vision.Windows/Test.Psi.CognitiveServices.Vision.Windows.csproj index 2b38a5f0d..ada852c6a 100644 --- a/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Vision.Windows/Test.Psi.CognitiveServices.Vision.Windows.csproj +++ b/Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Vision.Windows/Test.Psi.CognitiveServices.Vision.Windows.csproj @@ -1,20 +1,17 @@  net472 - false - false + ../../../../Build/Test.Psi.ruleset Exe - ../../../../Build/Test.Psi.ruleset true x64 - ../../../../Build/Test.Psi.ruleset true x64 @@ -26,6 +23,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all diff --git a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/Microsoft.Psi.MicrosoftSpeech.Windows.csproj b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/Microsoft.Psi.MicrosoftSpeech.Windows.csproj index 0772c30da..e3fe2e295 100644 --- a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/Microsoft.Psi.MicrosoftSpeech.Windows.csproj +++ b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/Microsoft.Psi.MicrosoftSpeech.Windows.csproj @@ -24,6 +24,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizer.cs b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizer.cs index b93969e6d..1e7b0fb32 100644 --- a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizer.cs +++ b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizer.cs @@ -199,7 +199,7 @@ private enum EmitterGroup public Emitter AudioLevelUpdated { get; } /// - /// Gets the output stream of emulate recognize completed completed events. + /// Gets the output stream of emulate recognize completed events. /// public Emitter EmulateRecognizeCompleted { get; } @@ -273,6 +273,7 @@ public void Dispose() // Free any other managed objects here. this.recognizeCompleteManualReset.Dispose(); this.recognizeCompleteManualReset = null; + this.inputAudioStream.Dispose(); } /// diff --git a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizerConfiguration.cs b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizerConfiguration.cs index 024a8bf39..6f3795deb 100644 --- a/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizerConfiguration.cs +++ b/Sources/Integrations/MicrosoftSpeech/Microsoft.Psi.MicrosoftSpeech.Windows/MicrosoftSpeechRecognizerConfiguration.cs @@ -21,12 +21,6 @@ public sealed class MicrosoftSpeechRecognizerConfiguration /// public MicrosoftSpeechRecognizerConfiguration() { - this.Language = "en-us"; - this.Grammars = null; - this.BufferLengthInMs = 1000; - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -41,7 +35,7 @@ public MicrosoftSpeechRecognizerConfiguration() /// in a particular language. For a list of supported Runtime Languages and to download them, see /// http://go.microsoft.com/fwlink/?LinkID=223569. /// - public string Language { get; set; } + public string Language { get; set; } = "en-us"; /// /// Gets or sets the list of grammar files. @@ -53,7 +47,7 @@ public MicrosoftSpeechRecognizerConfiguration() /// At least one grammar is required in order to use the . /// [XmlElement("Grammar")] - public GrammarInfo[] Grammars { get; set; } + public GrammarInfo[] Grammars { get; set; } = null; /// /// Gets or sets the length of the recognizer's input stream buffer in milliseconds. @@ -65,31 +59,31 @@ public MicrosoftSpeechRecognizerConfiguration() /// on the length of audio to buffer in milliseconds and the audio input format. By default, a 1000 ms /// buffer is used. It is safe to leave this value unchanged. /// - public int BufferLengthInMs { get; set; } + public int BufferLengthInMs { get; set; } = 1000; /// /// Gets or sets the number of milliseconds during which the internal speech detection /// engine accepts input containing only silence before making a state transition. /// - public int InitialSilenceTimeoutMs { get; set; } + public int InitialSilenceTimeoutMs { get; set; } = 0; /// /// Gets or sets the number of milliseconds during which the internal speech detection /// engine accepts input containing only background noise before making a state transition. /// - public int BabbleTimeoutMs { get; set; } + public int BabbleTimeoutMs { get; set; } = 0; /// /// Gets or sets the number of milliseconds of silence that the internal speech detection /// engine will accept at the end of unambiguous input before making a state transition. /// - public int EndSilenceTimeoutMs { get; set; } + public int EndSilenceTimeoutMs { get; set; } = 150; /// /// Gets or sets the number of milliseconds of silence that the internal speech detection /// engine will accept at the end of ambiguous input before making a state transition. /// - public int EndSilenceTimeoutAmbiguousMs { get; set; } + public int EndSilenceTimeoutAmbiguousMs { get; set; } = 500; /// /// Gets or sets the expected input format of the audio stream. @@ -100,6 +94,6 @@ public MicrosoftSpeechRecognizerConfiguration() /// static methods to create the appropriate /// object. If not specified, a default value of 16000 Hz is assumed. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Integrations/ROS/Microsoft.Psi.ROS/Microsoft.Psi.ROS.fsproj b/Sources/Integrations/ROS/Microsoft.Psi.ROS/Microsoft.Psi.ROS.fsproj index 7f6dcfdfa..d19c22864 100644 --- a/Sources/Integrations/ROS/Microsoft.Psi.ROS/Microsoft.Psi.ROS.fsproj +++ b/Sources/Integrations/ROS/Microsoft.Psi.ROS/Microsoft.Psi.ROS.fsproj @@ -41,6 +41,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Integrations/ROS/Microsoft.Psi.ROS/RosMessage.fs b/Sources/Integrations/ROS/Microsoft.Psi.ROS/RosMessage.fs index 24c0f77ab..dd7af8465 100644 --- a/Sources/Integrations/ROS/Microsoft.Psi.ROS/RosMessage.fs +++ b/Sources/Integrations/ROS/Microsoft.Psi.ROS/RosMessage.fs @@ -22,8 +22,8 @@ module RosMessage = type RosFieldVal = | BoolVal of bool // introduced in ROS 0.9 - | Int8Val of int8 // depricated alias: byte - | UInt8Val of uint8 // depricated alias: char + | Int8Val of int8 // deprecated alias: byte + | UInt8Val of uint8 // deprecated alias: char | Int16Val of int16 | UInt16Val of uint16 | Int32Val of int32 diff --git a/Sources/Integrations/ROS/Microsoft.Psi.ROS/XmlRpc.fs b/Sources/Integrations/ROS/Microsoft.Psi.ROS/XmlRpc.fs index 7b48cbeff..bc8d08bed 100644 --- a/Sources/Integrations/ROS/Microsoft.Psi.ROS/XmlRpc.fs +++ b/Sources/Integrations/ROS/Microsoft.Psi.ROS/XmlRpc.fs @@ -211,11 +211,11 @@ module XmlRpc = (new XmlRpcReader(client.GetResponse().GetResponseStream())).ReadMethodResponse() // An XmlRpc "server" listens for requests on a particular TCP port, waiting for HTTP POSTs - // I didn't like how HttpListener requires admin privilages and restricts listening blanketly on a port + // I didn't like how HttpListener requires admin privileges and restricts listening blanketly on a port // Instead we handle a very minimal HTTP protocol right here // A port is assigned and returned. // Your callback is called with method name and args and is expected to return a value (XmlRpcValue) or throw - // Again, no attemt is made here to parse the method call or validate arguments, etc. + // Again, no attempt is made here to parse the method call or validate arguments, etc. let xmlRpcListen (callback: string -> XmlRpcValue list -> XmlRpcValue) = let listener = new TcpListener(IPAddress.Any, 0) listener.Start() diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyListVisualizationObject.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyListVisualizationObject.cs new file mode 100644 index 000000000..bdf6de19c --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyListVisualizationObject.cs @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect.Visualization +{ + using System.Collections.Generic; + using Microsoft.Psi.AzureKinect; + using Microsoft.Psi.Visualization.VisualizationObjects; + + /// + /// Implements a visualization object for a list of Azure Kinect bodies. + /// + [VisualizationObject("Visualize Azure Kinect Bodies")] + public class AzureKinectBodyListVisualizationObject : ModelVisual3DVisualizationObjectEnumerable> + { + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyVisualizationObject.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyVisualizationObject.cs new file mode 100644 index 000000000..dcfab7d65 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/AzureKinectBodyVisualizationObject.cs @@ -0,0 +1,284 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect.Visualization +{ + using System; + using System.ComponentModel; + using System.Runtime.Serialization; + using System.Windows; + using System.Windows.Media; + using HelixToolkit.Wpf; + using Microsoft.Azure.Kinect.BodyTracking; + using Microsoft.Psi.AzureKinect; + using Microsoft.Psi.Visualization.VisualizationObjects; + using Xceed.Wpf.Toolkit.PropertyGrid.Attributes; + using Win3D = System.Windows.Media.Media3D; + + /// + /// Implements a visualization object for Azure Kinect bodies. + /// + [VisualizationObject("Visualize Azure Kinect Body")] + public class AzureKinectBodyVisualizationObject : ModelVisual3DVisualizationObject + { + private readonly BillboardTextVisual3D billboard; + private readonly UpdatableVisual3DDictionary visualJoints; + private readonly UpdatableVisual3DDictionary<(JointId ChildJoint, JointId ParentJoint), PipeVisual3D> visualBones; + + private Color color = Colors.White; + private double inferredJointsOpacity = 30; + private double thicknessMm = 30; + private bool showBillboard = false; + private int polygonResolution = 6; + private double billboardHeightCm = 100; + + /// + /// Initializes a new instance of the class. + /// + public AzureKinectBodyVisualizationObject() + { + this.visualJoints = new UpdatableVisual3DDictionary(null); + this.visualBones = new UpdatableVisual3DDictionary<(JointId ChildJoint, JointId ParentJoint), PipeVisual3D>(null); + + this.billboard = new BillboardTextVisual3D() + { + Background = Brushes.Gray, + Foreground = new SolidColorBrush(Colors.White), + Padding = new Thickness(5), + }; + + this.UpdateVisibility(); + } + + /// + /// Gets or sets the color. + /// + [DataMember] + [Description("Color of the body.")] + public Color Color + { + get { return this.color; } + set { this.Set(nameof(this.Color), ref this.color, value); } + } + + /// + /// Gets or sets the inferred joints opacity. + /// + [DataMember] + [Description("Opacity for rendering inferred joints and bones.")] + public double InferredJointsOpacity + { + get { return this.inferredJointsOpacity; } + set { this.Set(nameof(this.InferredJointsOpacity), ref this.inferredJointsOpacity, value); } + } + + /// + /// Gets or sets the thickness. + /// + [DataMember] + [DisplayName("Thickness (mm)")] + [Description("Diameter of bones and radius of joints (mm).")] + public double ThicknessMm + { + get { return this.thicknessMm; } + set { this.Set(nameof(this.ThicknessMm), ref this.thicknessMm, value); } + } + + /// + /// Gets or sets a value indicating whether to show a billboard with information about the body. + /// + [DataMember] + [PropertyOrder(0)] + [Description("Show a billboard with information about the body.")] + public bool ShowBillboard + { + get { return this.showBillboard; } + set { this.Set(nameof(this.ShowBillboard), ref this.showBillboard, value); } + } + + /// + /// Gets or sets the height at which to draw the billboard (cm). + /// + [DataMember] + [PropertyOrder(1)] + [DisplayName("Billboard Height (cm)")] + [Description("Height at which to draw the billboard (cm).")] + public double BillboardHeightCm + { + get { return this.billboardHeightCm; } + set { this.Set(nameof(this.BillboardHeightCm), ref this.billboardHeightCm, value); } + } + + /// + /// Gets or sets the number of divisions to use when rendering polygons for joints and bones. + /// + [DataMember] + [Description("Level of resolution at which to render joint and bone polygons (minimum value is 3).")] + public int PolygonResolution + { + get { return this.polygonResolution; } + set { this.Set(nameof(this.PolygonResolution), ref this.polygonResolution, value < 3 ? 3 : value); } + } + + /// + public override void UpdateData() + { + if (this.CurrentData != null) + { + this.UpdateVisuals(); + } + + this.UpdateVisibility(); + } + + /// + public override void NotifyPropertyChanged(string propertyName) + { + if (propertyName == nameof(this.Color) || + propertyName == nameof(this.InferredJointsOpacity) || + propertyName == nameof(this.ThicknessMm) || + propertyName == nameof(this.PolygonResolution)) + { + this.UpdateVisuals(); + } + else if (propertyName == nameof(this.ShowBillboard)) + { + this.UpdateBillboardVisibility(); + } + else if (propertyName == nameof(this.BillboardHeightCm)) + { + this.UpdateBillboard(); + } + else if (propertyName == nameof(this.Visible)) + { + this.UpdateVisibility(); + } + } + + private void UpdateVisuals() + { + this.visualJoints.BeginUpdate(); + this.visualBones.BeginUpdate(); + + if (this.CurrentData != null) + { + var trackedEntitiesBrush = new SolidColorBrush(this.Color); + var untrackedEntitiesBrush = new SolidColorBrush( + Color.FromArgb( + (byte)(Math.Max(0, Math.Min(100, this.InferredJointsOpacity)) * 2.55), + this.Color.R, + this.Color.G, + this.Color.B)); + + // update the joints + foreach (var jointType in this.CurrentData.Joints.Keys) + { + var jointState = this.CurrentData.Joints[jointType].Confidence; + var visualJoint = this.visualJoints[jointType]; + visualJoint.BeginEdit(); + var isTracked = jointState == JointConfidenceLevel.High || jointState == JointConfidenceLevel.Medium; + var visible = jointState != JointConfidenceLevel.None && (isTracked || this.InferredJointsOpacity > 0); + + if (visible) + { + var jointPosition = this.CurrentData.Joints[jointType].Pose.Origin; + + if (visualJoint.Radius != this.ThicknessMm / 1000.0) + { + visualJoint.Radius = this.ThicknessMm / 1000.0; + } + + var fill = isTracked ? trackedEntitiesBrush : untrackedEntitiesBrush; + if (visualJoint.Fill != fill) + { + visualJoint.Fill = fill; + } + + visualJoint.Transform = new Win3D.TranslateTransform3D(jointPosition.X, jointPosition.Y, jointPosition.Z); + + visualJoint.PhiDiv = this.PolygonResolution; + visualJoint.ThetaDiv = this.PolygonResolution; + + visualJoint.Visible = true; + } + else + { + visualJoint.Visible = false; + } + + visualJoint.EndEdit(); + } + + // update the bones + foreach (var bone in AzureKinectBody.Bones) + { + var parentState = this.CurrentData.Joints[bone.ParentJoint].Confidence; + var childState = this.CurrentData.Joints[bone.ChildJoint].Confidence; + var parentIsTracked = parentState == JointConfidenceLevel.High || parentState == JointConfidenceLevel.Medium; + var childIsTracked = childState == JointConfidenceLevel.High || childState == JointConfidenceLevel.Medium; + var isTracked = parentIsTracked && childIsTracked; + var visible = parentState != JointConfidenceLevel.None && childState != JointConfidenceLevel.None && (isTracked || this.InferredJointsOpacity > 0); + var visualBone = this.visualBones[bone]; + visualBone.BeginEdit(); + if (visible) + { + if (visualBone.Diameter != this.ThicknessMm / 1000.0) + { + visualBone.Diameter = this.ThicknessMm / 1000.0; + } + + var joint1Position = this.visualJoints[bone.ParentJoint].Transform.Value; + var joint2Position = this.visualJoints[bone.ChildJoint].Transform.Value; + + visualBone.Point1 = new Win3D.Point3D(joint1Position.OffsetX, joint1Position.OffsetY, joint1Position.OffsetZ); + visualBone.Point2 = new Win3D.Point3D(joint2Position.OffsetX, joint2Position.OffsetY, joint2Position.OffsetZ); + + var fill = isTracked ? trackedEntitiesBrush : untrackedEntitiesBrush; + if (visualBone.Fill != fill) + { + visualBone.Fill = fill; + } + + visualBone.ThetaDiv = this.PolygonResolution; + + visualBone.Visible = true; + } + else + { + visualBone.Visible = false; + } + + visualBone.EndEdit(); + } + + // set billboard position + this.UpdateBillboard(); + } + + this.visualJoints.EndUpdate(); + this.visualBones.EndUpdate(); + } + + private void UpdateBillboard() + { + if (this.CurrentData != null) + { + var origin = this.CurrentData.Joints[JointId.Pelvis].Pose.Origin; + this.billboard.Position = new Win3D.Point3D(origin.X, origin.Y, origin.Z + (this.BillboardHeightCm / 100.0)); + this.billboard.Text = this.CurrentData.ToString(); + } + } + + private void UpdateVisibility() + { + this.UpdateChildVisibility(this.visualJoints, this.Visible && this.CurrentData != default); + this.UpdateChildVisibility(this.visualBones, this.Visible && this.CurrentData != default); + this.UpdateBillboardVisibility(); + } + + private void UpdateBillboardVisibility() + { + this.UpdateChildVisibility(this.billboard, this.Visible && this.CurrentData != default && this.ShowBillboard); + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/Microsoft.Psi.AzureKinect.Visualization.Windows.x64.csproj b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/Microsoft.Psi.AzureKinect.Visualization.Windows.x64.csproj new file mode 100644 index 000000000..22e05b1b8 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/Microsoft.Psi.AzureKinect.Visualization.Windows.x64.csproj @@ -0,0 +1,45 @@ + + + net472 + Microsoft.Psi.AzureKinect.Visualization.Windows.x64 + Microsoft.Psi.AzureKinect.Visualization + x64 + ../../../Build/Microsoft.Psi.ruleset + + + DEBUG;TRACE + bin\Debug\net472\Microsoft.Psi.AzureKinect.Visualization.Windows.x64.xml + true + + + + bin\Release\net472\Microsoft.Psi.AzureKinect.Visualization.Windows.x64.xml + true + + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/stylecop.json b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/stylecop.json new file mode 100644 index 000000000..6f09427eb --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.Visualization/stylecop.json @@ -0,0 +1,16 @@ +{ + // ACTION REQUIRED: This file was automatically added to your project, but it + // will not take effect until additional steps are taken to enable it. See the + // following page for additional information: + // + // https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/EnableConfiguration.md + + "$schema": "https://raw.githubusercontent.com/DotNetAnalyzers/StyleCopAnalyzers/master/StyleCop.Analyzers/StyleCop.Analyzers/Settings/stylecop.schema.json", + "settings": { + "documentationRules": { + "companyName": "Microsoft Corporation", + "copyrightText": "Copyright (c) Microsoft Corporation. All rights reserved.\nLicensed under the MIT license.", + "xmlHeader": false + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBody.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBody.cs new file mode 100644 index 000000000..563fd3171 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBody.cs @@ -0,0 +1,162 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System.Collections.Generic; + using System.Numerics; + using MathNet.Numerics.LinearAlgebra; + using MathNet.Spatial.Euclidean; + using Microsoft.Azure.Kinect.BodyTracking; + + /// + /// Represents a body detected by the Azure Kinect. + /// + public class AzureKinectBody + { + private static readonly CoordinateSystem KinectBasis = new CoordinateSystem(default, UnitVector3D.ZAxis, UnitVector3D.XAxis.Negate(), UnitVector3D.YAxis.Negate()); + private static readonly CoordinateSystem KinectBasisInverted = KinectBasis.Invert(); + + /// + /// Initializes a new instance of the class. + /// + public AzureKinectBody() + { + for (int i = 0; i < Skeleton.JointCount; i++) + { + this.Joints.Add((JointId)i, (null, JointConfidenceLevel.None)); + } + } + + /// + /// Gets the bone relationships. + /// + /// + /// Bone connections defined here: https://docs.microsoft.com/en-us/azure/Kinect-dk/body-joints. + /// + public static List<(JointId ChildJoint, JointId ParentJoint)> Bones { get; } = new List<(JointId, JointId)> + { + // Spine + (JointId.SpineNavel, JointId.Pelvis), + (JointId.SpineChest, JointId.SpineNavel), + (JointId.Neck, JointId.SpineChest), + + // Left arm + (JointId.ClavicleLeft, JointId.SpineChest), + (JointId.ShoulderLeft, JointId.ClavicleLeft), + (JointId.ElbowLeft, JointId.ShoulderLeft), + (JointId.WristLeft, JointId.ElbowLeft), + (JointId.HandLeft, JointId.WristLeft), + (JointId.HandTipLeft, JointId.HandLeft), + (JointId.ThumbLeft, JointId.WristLeft), + + // Right arm + (JointId.ClavicleRight, JointId.SpineChest), + (JointId.ShoulderRight, JointId.ClavicleRight), + (JointId.ElbowRight, JointId.ShoulderRight), + (JointId.WristRight, JointId.ElbowRight), + (JointId.HandRight, JointId.WristRight), + (JointId.HandTipRight, JointId.HandRight), + (JointId.ThumbRight, JointId.WristRight), + + // Left leg + (JointId.HipLeft, JointId.Pelvis), + (JointId.KneeLeft, JointId.HipLeft), + (JointId.AnkleLeft, JointId.KneeLeft), + (JointId.FootLeft, JointId.AnkleLeft), + + // Right leg + (JointId.HipRight, JointId.Pelvis), + (JointId.KneeRight, JointId.HipRight), + (JointId.AnkleRight, JointId.KneeRight), + (JointId.FootRight, JointId.AnkleRight), + + // Head + (JointId.Head, JointId.Neck), + (JointId.Nose, JointId.Head), + (JointId.EyeLeft, JointId.Head), + (JointId.EarLeft, JointId.Head), + (JointId.EyeRight, JointId.Head), + (JointId.EarRight, JointId.Head), + }; + + /// + /// Gets the joint information. + /// + public Dictionary Joints { get; } = new Dictionary(); + + /// + /// Gets the body's tracking ID. + /// + public uint TrackingId { get; private set; } + + /// + /// Copies new joint and tracking information for this body from a Microsoft.Azure.Kinect body instance. + /// + /// The Microsoft.Azure.Kinect body instance to populate this body from. + public void CopyFrom(Body body) + { + this.TrackingId = body.Id; + + for (int i = 0; i < Skeleton.JointCount; i++) + { + var joint = body.Skeleton.GetJoint(i); + var position = joint.Position; + var orientation = joint.Quaternion; + var confidence = joint.ConfidenceLevel; + this.Joints[(JointId)i] = (this.CreateCoordinateSystem(position, orientation), confidence); + } + } + + /// + public override string ToString() + { + return $"ID: {this.TrackingId}"; + } + + private CoordinateSystem CreateCoordinateSystem(Vector3 position, System.Numerics.Quaternion orientation) + { + // Convert the quaternion to a rotation matrix (System.Numerics) + var jointRotation = Matrix4x4.CreateFromQuaternion(orientation); + + // In the System.Numerics.Matrix4x4 joint rotation above, the axes of rotation are defined as follows: + // X: [M11; M12; M13] + // Y: [M21; M22; M23] + // Z: [M31; M32; M33] + // However, joint rotation axes are defined differently from the Azure Kinect sensor axes, as defined here: + // https://docs.microsoft.com/en-us/azure/Kinect-dk/body-joints + // and here: + // https://docs.microsoft.com/en-us/azure/Kinect-dk/coordinate-systems + // Joint Axes: + // X + // | Y + // | / + // | / + // Z <----+ + // Azure Kinect Axes: + // Z + // / + // / + // +---->X + // | + // | + // | + // Y + // Therefore we first create a transformation matrix in Azure Kinect basis by converting axes: + // X (Azure Kinect) = -Z (Joint) + // Y (Azure Kinect) = -X (Joint) + // Z (Azure Kinect) = Y (Joint) + // and converting from millimeters to meters. + var transformationMatrix = Matrix.Build.DenseOfArray(new double[,] + { + { -jointRotation.M31, -jointRotation.M11, jointRotation.M21, position.X / 1000.0 }, + { -jointRotation.M32, -jointRotation.M12, jointRotation.M22, position.Y / 1000.0 }, + { -jointRotation.M33, -jointRotation.M13, jointRotation.M23, position.Z / 1000.0 }, + { 0, 0, 0, 1 }, + }); + + // Finally, convert from Azure Kinect's basis to MathNet's basis: + return new CoordinateSystem(KinectBasisInverted * transformationMatrix * KinectBasis); + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTracker.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTracker.cs new file mode 100644 index 000000000..e45be0574 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTracker.cs @@ -0,0 +1,133 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System; + using System.Collections.Generic; + using Microsoft.Azure.Kinect.BodyTracking; + using Microsoft.Azure.Kinect.Sensor; + using Microsoft.Psi; + using Microsoft.Psi.Components; + using DepthImage = Microsoft.Psi.Imaging.DepthImage; + using Image = Microsoft.Psi.Imaging.Image; + + /// + /// Component that performs body tracking from the depth/IR images captured by the Azure Kinect sensor. + /// + /// It is important that Depth and IR images do *not* go through lossy encoding/decoding (e.g., JPEG) + /// before arriving at the tracker. Unencoded or non-lossy (e.g. PNG) encoding are okay. + public sealed class AzureKinectBodyTracker : ConsumerProducer<(Shared Depth, Shared IR), List>, IDisposable + { + private readonly AzureKinectBodyTrackerConfiguration configuration; + private readonly List currentBodies = new List(); + private readonly Capture capture = new Capture(); + + private Tracker tracker = null; + private byte[] depthBytes = null; + private byte[] infraredBytes = null; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// An optional configuration to use for the body tracker. + public AzureKinectBodyTracker(Pipeline pipeline, AzureKinectBodyTrackerConfiguration configuration = null) + : base(pipeline) + { + this.configuration = configuration ?? new AzureKinectBodyTrackerConfiguration(); + this.AzureKinectSensorCalibration = pipeline.CreateReceiver(this, this.ReceiveCalibration, nameof(this.AzureKinectSensorCalibration)); + } + + /// + /// Gets the receiver for sensor calibration needed to initialize the tracker. + /// + public Receiver AzureKinectSensorCalibration { get; } + + /// + public void Dispose() + { + if (this.tracker != null) + { + this.tracker.Dispose(); + this.tracker = null; + } + + this.capture?.Dispose(); + } + + /// + protected override void Receive((Shared Depth, Shared IR) depthAndIRImages, Envelope envelope) + { + if (this.tracker != null) + { + // Allocate depth and IR image buffers. + if (this.capture.Depth == null || this.capture.IR == null || this.depthBytes == null || this.infraredBytes == null) + { + this.capture.Depth = new Azure.Kinect.Sensor.Image(ImageFormat.Depth16, depthAndIRImages.Depth.Resource.Width, depthAndIRImages.Depth.Resource.Height); + this.capture.IR = new Azure.Kinect.Sensor.Image(ImageFormat.IR16, depthAndIRImages.IR.Resource.Width, depthAndIRImages.IR.Resource.Height); + this.depthBytes = new byte[depthAndIRImages.Depth.Resource.Size]; + this.infraredBytes = new byte[depthAndIRImages.IR.Resource.Size]; + } + + // Copy the depth image data. + depthAndIRImages.Depth.Resource.CopyTo(this.depthBytes); + var depthMemory = new Memory(this.depthBytes); + depthMemory.CopyTo(this.capture.Depth.Memory); + + // Copy the IR image data. + depthAndIRImages.IR.Resource.CopyTo(this.infraredBytes); + var infraredMemory = new Memory(this.infraredBytes); + infraredMemory.CopyTo(this.capture.IR.Memory); + + // Call the body tracker. + this.tracker.EnqueueCapture(this.capture); + using (var bodyFrame = this.tracker.PopResult(false)) + { + // Parse the output into a list of KinectBody's to post + if (bodyFrame != null) + { + uint bodyIndex = 0; + while (bodyIndex < bodyFrame.NumberOfBodies) + { + if (bodyIndex >= this.currentBodies.Count) + { + this.currentBodies.Add(new AzureKinectBody()); + } + + this.currentBodies[(int)bodyIndex].CopyFrom(bodyFrame.GetBody(bodyIndex)); + bodyIndex++; + } + + this.currentBodies.RemoveRange((int)bodyIndex, this.currentBodies.Count - (int)bodyIndex); + } + else + { + this.currentBodies.Clear(); + } + } + + this.Out.Post(this.currentBodies, envelope.OriginatingTime); + } + } + + private void InitializeTracker(Calibration calibration) + { + this.tracker = Tracker.Create(calibration, new TrackerConfiguration() + { + SensorOrientation = this.configuration.SensorOrientation, + ProcessingMode = this.configuration.CpuOnlyMode ? TrackerProcessingMode.Cpu : TrackerProcessingMode.Gpu, + }); + + this.tracker.SetTemporalSmooting(this.configuration.TemporalSmoothing); + } + + private void ReceiveCalibration(Calibration calibration, Envelope envelope) + { + if (this.tracker == null) + { + this.InitializeTracker(calibration); + } + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTrackerConfiguration.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTrackerConfiguration.cs new file mode 100644 index 000000000..6b05c4fcb --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectBodyTrackerConfiguration.cs @@ -0,0 +1,35 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using Microsoft.Azure.Kinect.BodyTracking; + + /// + /// Represents the Azure Kinect body tracker configuration. + /// + public class AzureKinectBodyTrackerConfiguration + { + /// + /// Gets or sets the temporal smoothing to use across frames for the body tracker. + /// + /// + /// Set between 0 (no smoothing) and 1 (full smoothing). Less smoothing will increase + /// the responsiveness of the detected skeletons but will cause more positional and + /// orientational jitters. + /// + public float TemporalSmoothing { get; set; } = 0.5f; + + /// + /// Gets or sets a value indicating whether to perform body tracking computation only + /// on the CPU. + /// + /// If false, the tracker requires CUDA hardware and drivers. + public bool CpuOnlyMode { get; set; } = false; + + /// + /// Gets or sets the sensor orientation used by body tracking. + /// + public SensorOrientation SensorOrientation { get; set; } = SensorOrientation.Default; + } +} diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectCore.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectCore.cs new file mode 100644 index 000000000..95d7cc0a3 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectCore.cs @@ -0,0 +1,461 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System; + using System.Collections.Generic; + using System.ComponentModel; + using System.Diagnostics; + using System.Threading; + using System.Threading.Tasks; + using MathNet.Numerics.LinearAlgebra; + using MathNet.Spatial.Euclidean; + using Microsoft.Azure.Kinect.Sensor; + using Microsoft.Psi; + using Microsoft.Psi.Calibration; + using Microsoft.Psi.Components; + using Microsoft.Psi.DeviceManagement; + using Microsoft.Psi.Imaging; + using Image = Microsoft.Psi.Imaging.Image; + + /// + /// Component that captures core sensor streams (color, depth, IR, and IMU) from the Azure Kinect device. + /// + internal sealed class AzureKinectCore : ISourceComponent, IDisposable + { + private readonly Pipeline pipeline; + private readonly AzureKinectSensorConfiguration configuration; + + /// + /// The underlying Azure Kinect device. + /// + private Device device = null; + private Thread captureThread = null; + private Thread imuSampleThread = null; + private bool shutdown = false; + + private int colorImageWidth; + private int colorImageHeight; + private int depthImageWidth; + private int depthImageHeight; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline this component is a part of. + /// Configuration to use for the device. + public AzureKinectCore(Pipeline pipeline, AzureKinectSensorConfiguration config = null) + { + this.pipeline = pipeline; + this.configuration = config ?? new AzureKinectSensorConfiguration(); + + if (this.configuration.OutputColor) + { + if (this.configuration.ColorResolution == ColorResolution.Off) + { + throw new ArgumentException("Invalid configuration: Cannot output color stream when color resolution is set to Off."); + } + + if (this.configuration.ColorFormat != ImageFormat.ColorBGRA32) + { + throw new NotImplementedException("Invalid configuration: Psi so far only supports BGRA32 pixel format for the AzureKinect color camera"); + } + } + + if (this.configuration.OutputDepth) + { + if (this.configuration.DepthMode == DepthMode.Off) + { + throw new ArgumentException("Invalid configuration: Cannot output depth stream when depth mode is set to Off."); + } + + if (this.configuration.DepthMode == DepthMode.PassiveIR) + { + throw new ArgumentException("Invalid configuration: Cannot output depth stream when depth mode is set to PassiveIR."); + } + } + + if (this.configuration.OutputInfrared && this.configuration.DepthMode == DepthMode.Off) + { + throw new ArgumentException("Invalid configuration: Cannot output IR stream when depth mode is set to Off. Try DepthMode=PassiveIR if the intent is to capture only IR."); + } + + this.DepthImage = pipeline.CreateEmitter>(this, nameof(this.DepthImage)); + this.InfraredImage = pipeline.CreateEmitter>(this, nameof(this.InfraredImage)); + this.ColorImage = pipeline.CreateEmitter>(this, nameof(this.ColorImage)); + this.Imu = pipeline.CreateEmitter(this, nameof(this.Imu)); + this.DepthDeviceCalibrationInfo = pipeline.CreateEmitter(this, nameof(this.DepthDeviceCalibrationInfo)); + this.AzureKinectSensorCalibration = pipeline.CreateEmitter(this, nameof(this.AzureKinectSensorCalibration)); + this.FrameRate = pipeline.CreateEmitter(this, nameof(this.FrameRate)); + this.Temperature = pipeline.CreateEmitter(this, nameof(this.Temperature)); + this.DepthAndIRImages = pipeline.CreateEmitter<(Shared, Shared)>(this, nameof(this.DepthAndIRImages)); + + this.DetermineImageDimensions(); + } + + /// + /// Gets the current image from the color camera. + /// + public Emitter> ColorImage { get; private set; } + + /// + /// Gets the current infrared image. + /// + public Emitter> InfraredImage { get; private set; } + + /// + /// Gets the current depth image. + /// + public Emitter> DepthImage { get; private set; } + + /// + /// Gets the current frames-per-second actually achieved. + /// + public Emitter FrameRate { get; private set; } + + /// + /// Gets the current IMU sample. + /// + public Emitter Imu { get; private set; } + + /// + /// Gets the Azure Kinect depth device calibration information as an object (see Microsoft.Psi.Calibration). + /// + public Emitter DepthDeviceCalibrationInfo { get; private set; } + + /// + /// Gets the underlying device calibration (provided directly by Azure Kinect SDK and required by the body tracker). + /// + public Emitter AzureKinectSensorCalibration { get; private set; } + + /// + /// Gets the Kinect's temperature in degrees Celsius. + /// + public Emitter Temperature { get; private set; } + + /// + /// Gets both the depth and IR images together (required by the body tracker). + /// + internal Emitter<(Shared Depth, Shared IR)> DepthAndIRImages { get; private set; } + + /// + /// Returns the number of Kinect for Azure devices available on the system. + /// + /// Number of available devices. + public static int GetInstalledCount() + { + return Device.GetInstalledCount(); + } + + /// + public void Dispose() + { + if (this.device != null) + { + this.device.Dispose(); + this.device = null; + } + } + + /// + public void Start(Action notifyCompletionTime) + { + // notify that this is an infinite source component + notifyCompletionTime(DateTime.MaxValue); + + this.device = Device.Open(this.configuration.DeviceIndex); + this.device.StartCameras(new DeviceConfiguration() + { + ColorFormat = this.configuration.ColorFormat, + ColorResolution = this.configuration.ColorResolution, + DepthMode = this.configuration.DepthMode, + CameraFPS = this.configuration.CameraFPS, + SynchronizedImagesOnly = this.configuration.SynchronizedImagesOnly, + }); + + this.captureThread = new Thread(new ThreadStart(this.CaptureThreadProc)); + this.captureThread.Start(); + + if (this.configuration.OutputImu) + { + this.device.StartImu(); + this.imuSampleThread = new Thread(new ThreadStart(this.ImuSampleThreadProc)); + this.imuSampleThread.Start(); + } + } + + /// + public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) + { + this.shutdown = true; + TimeSpan waitTime = TimeSpan.FromSeconds(1); + if (this.captureThread != null && this.captureThread.Join(waitTime) != true) + { + this.captureThread.Abort(); + } + + this.device.StopCameras(); + + if (this.configuration.OutputImu) + { + if (this.imuSampleThread != null && this.imuSampleThread.Join(waitTime) != true) + { + this.imuSampleThread.Abort(); + } + + this.device.StopImu(); + } + + notifyCompleted(); + } + + private void CaptureThreadProc() + { + if (this.configuration.ColorResolution == ColorResolution.Off && + this.configuration.DepthMode == DepthMode.Off) + { + return; + } + + var colorImageFormat = PixelFormat.BGRA_32bpp; + var infraredImageFormat = PixelFormat.Gray_16bpp; + + var calibrationPosted = false; + + Stopwatch sw = new Stopwatch(); + int frameCount = 0; + sw.Start(); + + while (this.device != null && !this.shutdown) + { + if (this.configuration.OutputCalibration && !calibrationPosted) + { + // Compute and post the device's calibration object. + var currentTime = this.pipeline.GetCurrentTime(); + var calibration = this.device.GetCalibration(); + + if (calibration != null) + { + this.AzureKinectSensorCalibration.Post(calibration, currentTime); + + var colorExtrinsics = calibration.ColorCameraCalibration.Extrinsics; + var colorIntrinsics = calibration.ColorCameraCalibration.Intrinsics; + var depthIntrinsics = calibration.DepthCameraCalibration.Intrinsics; + + if (colorIntrinsics.Type == CalibrationModelType.Rational6KT || depthIntrinsics.Type == CalibrationModelType.Rational6KT) + { + throw new Exception("Calibration output not permitted for deprecated internal Azure Kinect cameras. Only Brown_Conrady calibration supported."); + } + else if (colorIntrinsics.Type != CalibrationModelType.BrownConrady || depthIntrinsics.Type != CalibrationModelType.BrownConrady) + { + throw new Exception("Calibration output only supported for Brown_Conrady model."); + } + else + { + Matrix colorCameraMatrix = Matrix.Build.Dense(3, 3); + colorCameraMatrix[0, 0] = colorIntrinsics.Parameters[2]; + colorCameraMatrix[1, 1] = colorIntrinsics.Parameters[3]; + colorCameraMatrix[0, 2] = colorIntrinsics.Parameters[0]; + colorCameraMatrix[1, 2] = colorIntrinsics.Parameters[1]; + colorCameraMatrix[2, 2] = 1; + Matrix depthCameraMatrix = Matrix.Build.Dense(3, 3); + depthCameraMatrix[0, 0] = depthIntrinsics.Parameters[2]; + depthCameraMatrix[1, 1] = depthIntrinsics.Parameters[3]; + depthCameraMatrix[0, 2] = depthIntrinsics.Parameters[0]; + depthCameraMatrix[1, 2] = depthIntrinsics.Parameters[1]; + depthCameraMatrix[2, 2] = 1; + Matrix depthToColorMatrix = Matrix.Build.Dense(4, 4); + for (int i = 0; i < 3; i++) + { + for (int j = 0; j < 3; j++) + { + // The AzureKinect SDK assumes that vectors are row vectors, while the MathNet SDK assumes + // column vectors, so we need to flip them here. + depthToColorMatrix[i, j] = colorExtrinsics.Rotation[(j * 3) + i]; + } + } + + depthToColorMatrix[3, 0] = colorExtrinsics.Translation[0]; + depthToColorMatrix[3, 1] = colorExtrinsics.Translation[1]; + depthToColorMatrix[3, 2] = colorExtrinsics.Translation[2]; + depthToColorMatrix[3, 3] = 1.0; + var metersToMillimeters = Matrix.Build.Dense(4, 4); + metersToMillimeters[0, 0] = 1000.0; + metersToMillimeters[1, 1] = 1000.0; + metersToMillimeters[2, 2] = 1000.0; + metersToMillimeters[3, 3] = 1.0; + var millimetersToMeters = Matrix.Build.Dense(4, 4); + millimetersToMeters[0, 0] = 1.0 / 1000.0; + millimetersToMeters[1, 1] = 1.0 / 1000.0; + millimetersToMeters[2, 2] = 1.0 / 1000.0; + millimetersToMeters[3, 3] = 1.0; + depthToColorMatrix = (metersToMillimeters * depthToColorMatrix * millimetersToMeters).Transpose(); + + double[] colorRadialDistortion = new double[6] + { + colorIntrinsics.Parameters[4], + colorIntrinsics.Parameters[5], + colorIntrinsics.Parameters[6], + colorIntrinsics.Parameters[7], + colorIntrinsics.Parameters[8], + colorIntrinsics.Parameters[9], + }; + double[] colorTangentialDistortion = new double[2] { colorIntrinsics.Parameters[13], colorIntrinsics.Parameters[12] }; + double[] depthRadialDistortion = new double[6] + { + depthIntrinsics.Parameters[4], + depthIntrinsics.Parameters[5], + depthIntrinsics.Parameters[6], + depthIntrinsics.Parameters[7], + depthIntrinsics.Parameters[8], + depthIntrinsics.Parameters[9], + }; + double[] depthTangentialDistortion = new double[2] { depthIntrinsics.Parameters[13], depthIntrinsics.Parameters[12] }; + + // Azure Kinect uses a basis under the hood that assumes Forward=Z, Right=X, Down=Y. + var kinectBasis = new CoordinateSystem(default, UnitVector3D.ZAxis, UnitVector3D.XAxis.Negate(), UnitVector3D.YAxis.Negate()); + + var cameraCalibration = new DepthDeviceCalibrationInfo( + calibration.ColorCameraCalibration.ResolutionWidth, + calibration.ColorCameraCalibration.ResolutionHeight, + colorCameraMatrix, + colorRadialDistortion, + colorTangentialDistortion, + kinectBasis.Invert() * depthToColorMatrix * kinectBasis, + calibration.DepthCameraCalibration.ResolutionWidth, + calibration.DepthCameraCalibration.ResolutionHeight, + depthCameraMatrix, + depthRadialDistortion, + depthTangentialDistortion, + CoordinateSystem.CreateIdentity(4)); + + this.DepthDeviceCalibrationInfo.Post(cameraCalibration, currentTime); + } + + calibrationPosted = true; + } + } + + // Wait for a capture on a thread pool thread + using var capture = this.device.GetCapture(this.configuration.DeviceCaptureTimeout); + if (capture != null) + { + var currentTime = this.pipeline.GetCurrentTime(); + + if (this.configuration.OutputColor && capture.Color != null) + { + using var sharedColorImage = ImagePool.GetOrCreate(this.colorImageWidth, this.colorImageHeight, colorImageFormat); + sharedColorImage.Resource.CopyFrom(capture.Color.Memory.ToArray()); + this.ColorImage.Post(sharedColorImage, currentTime); + } + + Shared sharedIRImage = null; + Shared sharedDepthImage = null; + + if (this.configuration.OutputInfrared && capture.IR != null) + { + sharedIRImage = ImagePool.GetOrCreate(this.depthImageWidth, this.depthImageHeight, infraredImageFormat); + sharedIRImage.Resource.CopyFrom(capture.IR.Memory.ToArray()); + this.InfraredImage.Post(sharedIRImage, currentTime); + } + + if (this.configuration.OutputDepth && capture.Depth != null) + { + sharedDepthImage = DepthImagePool.GetOrCreate(this.depthImageWidth, this.depthImageHeight); + sharedDepthImage.Resource.CopyFrom(capture.Depth.Memory.ToArray()); + this.DepthImage.Post(sharedDepthImage, currentTime); + + if (sharedIRImage != null) + { + this.DepthAndIRImages.Post((sharedDepthImage, sharedIRImage), currentTime); + } + } + + sharedIRImage?.Dispose(); + sharedDepthImage?.Dispose(); + + this.Temperature.Post(capture.Temperature, currentTime); + + ++frameCount; + if (sw.Elapsed > this.configuration.FrameRateReportingFrequency) + { + this.FrameRate.Post((double)frameCount / sw.Elapsed.TotalSeconds, currentTime); + frameCount = 0; + sw.Restart(); + } + } + } + } + + private void ImuSampleThreadProc() + { + while (this.device != null && !this.shutdown) + { + this.Imu.Post(this.device.GetImuSample(TimeSpan.MaxValue), this.pipeline.GetCurrentTime()); + } + } + + private void DetermineImageDimensions() + { + // Initialize the color image width and height based on config + switch (this.configuration.ColorResolution) + { + case ColorResolution.R720p: + this.colorImageWidth = 1280; + this.colorImageHeight = 720; + break; + case ColorResolution.R1080p: + this.colorImageWidth = 1920; + this.colorImageHeight = 1080; + break; + case ColorResolution.R1440p: + this.colorImageWidth = 2560; + this.colorImageHeight = 1440; + break; + case ColorResolution.R1536p: + this.colorImageWidth = 2048; + this.colorImageHeight = 1536; + break; + case ColorResolution.R2160p: + this.colorImageWidth = 3840; + this.colorImageHeight = 2160; + break; + case ColorResolution.R3072p: + this.colorImageWidth = 4096; + this.colorImageHeight = 3072; + break; + case ColorResolution.Off: + break; + default: + throw new InvalidEnumArgumentException($"Unexpected Azure Kinect Color Resolution: {this.configuration.ColorResolution}"); + } + + // Initialize the depth image width and height based on config + switch (this.configuration.DepthMode) + { + case DepthMode.NFOV_2x2Binned: + this.depthImageWidth = 320; + this.depthImageHeight = 288; + break; + case DepthMode.NFOV_Unbinned: + this.depthImageWidth = 640; + this.depthImageHeight = 576; + break; + case DepthMode.WFOV_2x2Binned: + this.depthImageWidth = 512; + this.depthImageHeight = 512; + break; + case DepthMode.WFOV_Unbinned: + case DepthMode.PassiveIR: + this.depthImageWidth = 1024; + this.depthImageHeight = 1024; + break; + case DepthMode.Off: + break; + default: + throw new InvalidEnumArgumentException($"Unexpected Azure Kinect Depth Mode: {this.configuration.DepthMode}"); + } + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectExtensions.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectExtensions.cs new file mode 100644 index 000000000..f8d6ff2e2 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectExtensions.cs @@ -0,0 +1,33 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System; + using Microsoft.Azure.Kinect.Sensor; + + /// + /// Helper class with extension methods for Azure Kinect data types. + /// + public static class AzureKinectExtensions + { + /// + /// Gets the range of pixel values for a specified depth mode. + /// + /// The depth mode. + /// A tuple indicating the range of pixel values, in millimeters. + public static (ushort MinValue, ushort MaxValue) GetRange(this DepthMode depthMode) + { + // Using the same values as in: + // https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/tools/k4aviewer/k4astaticimageproperties.h + return depthMode switch + { + DepthMode.NFOV_2x2Binned => (500, 5800), + DepthMode.NFOV_Unbinned => (500, 4000), + DepthMode.WFOV_2x2Binned => (250, 3000), + DepthMode.WFOV_Unbinned => (250, 2500), + _ => throw new Exception("Invalid depth mode."), + }; + } + } +} diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensor.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensor.cs new file mode 100644 index 000000000..f65f88797 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensor.cs @@ -0,0 +1,243 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System; + using System.Collections.Generic; + using Microsoft.Azure.Kinect.Sensor; + using Microsoft.Psi; + using Microsoft.Psi.Calibration; + using Microsoft.Psi.DeviceManagement; + using DepthImage = Microsoft.Psi.Imaging.DepthImage; + using Image = Microsoft.Psi.Imaging.Image; + + /// + /// Component that captures all sensor streams and tracked bodies from the Azure Kinect device. + /// + public class AzureKinectSensor : Subpipeline + { + private static List allDevices = null; + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline to add this component to. + /// Configuration to use for the sensor. + /// An optional default delivery policy for the subpipeline (defaults is LatestMessage). + /// An optional delivery policy for sending the depth-and-IR images stream to the body tracker (default is LatestMessage). + public AzureKinectSensor( + Pipeline pipeline, + AzureKinectSensorConfiguration configuration = null, + DeliveryPolicy defaultDeliveryPolicy = null, + DeliveryPolicy bodyTrackerDeliveryPolicy = null) + : base(pipeline, nameof(AzureKinectSensor), defaultDeliveryPolicy ?? DeliveryPolicy.LatestMessage) + { + if (configuration == null) + { + configuration = new AzureKinectSensorConfiguration(); + } + + if (configuration.BodyTrackerConfiguration != null) + { + if (!configuration.OutputCalibration) + { + throw new Exception($"The body tracker requires that the {nameof(AzureKinectSensor)} component must be configured to output calibration."); + } + + if (!configuration.OutputInfrared || !configuration.OutputDepth) + { + throw new Exception($"The body tracker requires that the {nameof(AzureKinectSensor)} component must be configured to output both Depth and IR streams."); + } + } + + var azureKinectCore = new AzureKinectCore(this, configuration); + + // Connect the sensor streams + this.ColorImage = azureKinectCore.ColorImage.BridgeTo(pipeline, nameof(this.ColorImage)).Out; + this.Imu = azureKinectCore.Imu.BridgeTo(pipeline, nameof(this.Imu)).Out; + this.DepthImage = azureKinectCore.DepthImage.BridgeTo(pipeline, nameof(this.DepthImage)).Out; + this.InfraredImage = azureKinectCore.InfraredImage.BridgeTo(pipeline, nameof(this.InfraredImage)).Out; + this.FrameRate = azureKinectCore.FrameRate.BridgeTo(pipeline, nameof(this.FrameRate)).Out; + this.Temperature = azureKinectCore.Temperature.BridgeTo(pipeline, nameof(this.Temperature)).Out; + this.DepthDeviceCalibrationInfo = azureKinectCore.DepthDeviceCalibrationInfo.BridgeTo(pipeline, nameof(this.DepthDeviceCalibrationInfo)).Out; + this.AzureKinectSensorCalibration = azureKinectCore.AzureKinectSensorCalibration.BridgeTo(pipeline, nameof(this.AzureKinectSensorCalibration)).Out; + + // Pipe captures and calibration to the body tracker + if (configuration.BodyTrackerConfiguration != null) + { + var bodyTracker = new AzureKinectBodyTracker(this, configuration.BodyTrackerConfiguration); + azureKinectCore.DepthAndIRImages.PipeTo(bodyTracker, bodyTrackerDeliveryPolicy ?? DeliveryPolicy.LatestMessage); + azureKinectCore.AzureKinectSensorCalibration.PipeTo(bodyTracker.AzureKinectSensorCalibration, DeliveryPolicy.Unlimited); + this.Bodies = bodyTracker.BridgeTo(pipeline, nameof(this.Bodies)).Out; + } + else + { + // create unused emitter to allow wiring while OutputBodies=false + this.Bodies = pipeline.CreateEmitter>(this, nameof(this.Bodies)); + } + } + + /// + /// Gets a list of all available capture devices. + /// + public static IEnumerable AllDevices + { + get + { + if (allDevices == null) + { + allDevices = new List(); + int numDevices = Device.GetInstalledCount(); + for (int i = 0; i < numDevices; i++) + { + CameraDeviceInfo di = new CameraDeviceInfo + { + FriendlyName = $"AzureKinect-{i}", + DeviceName = $"AzureKinect-{i}", + DeviceType = "AzureKinect", + DeviceId = i, + }; + Device dev; + try + { + dev = Device.Open(i); + } + catch (Exception) + { + continue; + } + + di.SerialNumber = dev.SerialNum; + dev.Dispose(); + di.Sensors = new List(); + for (int k = 0; k < 3; k++) + { + CameraDeviceInfo.Sensor sensor = new CameraDeviceInfo.Sensor(); + uint[,] resolutions = null; + switch (k) + { + case 0: // color mode + sensor.Type = CameraDeviceInfo.Sensor.SensorType.Color; + resolutions = new uint[,] + { + { 1280, 720, 1 }, + { 1920, 1080, 1 }, + { 2560, 1440, 1 }, + { 3840, 2160, 1 }, + { 2048, 1536, 1 }, + { 4096, 3072, 0 }, + }; + break; + case 1: // depth mode + sensor.Type = CameraDeviceInfo.Sensor.SensorType.Depth; + resolutions = new uint[,] + { + { 640, 576, 1 }, + { 320, 288, 1 }, + { 512, 512, 1 }, + { 1024, 1024, 0 }, + }; + break; + case 2: // IR mode + sensor.Type = CameraDeviceInfo.Sensor.SensorType.IR; + resolutions = new uint[,] + { + { 1024, 1024, 1 }, + }; + break; + } + + sensor.Modes = new List(); + uint[] frameRates = { 30, 15, 5 }; + for (int j = 0; j < resolutions.Length / 3; j++) + { + foreach (var fr in frameRates) + { + if (fr == 30 && resolutions[j, 2] == 0) + { + continue; // Mode doesn't support 30fps + } + + CameraDeviceInfo.Sensor.ModeInfo mi = new CameraDeviceInfo.Sensor.ModeInfo + { + Format = Imaging.PixelFormat.BGRA_32bpp, + FrameRateNumerator = fr, + FrameRateDenominator = 1, + ResolutionWidth = resolutions[j, 0], + ResolutionHeight = resolutions[j, 1], + }; + sensor.Modes.Add(mi); + } + } + + di.Sensors.Add(sensor); + } + + allDevices.Add(di); + } + } + + return allDevices; + } + } + + // Note: the following emitters mirror those in AzureKinectSensorCore + + /// + /// Gets the current image from the color camera. + /// + public Emitter> ColorImage { get; private set; } + + /// + /// Gets the current infrared image. + /// + public Emitter> InfraredImage { get; private set; } + + /// + /// Gets the current depth image. + /// + public Emitter> DepthImage { get; private set; } + + /// + /// Gets the current frames-per-second actually achieved. + /// + public Emitter FrameRate { get; private set; } + + /// + /// Gets the current IMU sample. + /// + public Emitter Imu { get; private set; } + + /// + /// Gets the Azure Kinect's depth device calibration information. + /// + public Emitter DepthDeviceCalibrationInfo { get; private set; } + + /// + /// Gets the underlying device calibration (provided by Azure Kinect SDK). + /// + public Emitter AzureKinectSensorCalibration { get; private set; } + + /// + /// Gets the Kinect's temperature in degrees Celsius. + /// + public Emitter Temperature { get; private set; } + + // Note: the following emitter mirrors that in AzureKinectBodyTracker + + /// + /// Gets the emitter of lists of currently tracked bodies. + /// + public Emitter> Bodies { get; private set; } + + /// + /// Returns the number of Kinect for Azure devices available on the system. + /// + /// Number of available devices. + public static int GetInstalledCount() + { + return Device.GetInstalledCount(); + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensorConfiguration.cs b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensorConfiguration.cs new file mode 100644 index 000000000..28259eb86 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/AzureKinectSensorConfiguration.cs @@ -0,0 +1,87 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.AzureKinect +{ + using System; + using Microsoft.Azure.Kinect.Sensor; + + /// + /// Represents the Azure Kinect configuration. + /// + public class AzureKinectSensorConfiguration + { + /// + /// Gets or sets the index of the device to open. + /// + public int DeviceIndex { get; set; } = 0; // K4A_DEVICE_DEFAULT = 0 + + /// + /// Gets the color image format, i.e. . + /// + /// This property does not have a setter because currently + /// the component only supports the + /// . + public ImageFormat ColorFormat { get; } = ImageFormat.ColorBGRA32; + + /// + /// Gets or sets the resolution of the color camera. + /// + public ColorResolution ColorResolution { get; set; } = ColorResolution.R1080p; + + /// + /// Gets or sets the depth camera mode. + /// + public DepthMode DepthMode { get; set; } = DepthMode.NFOV_Unbinned; + + /// + /// Gets or sets the desired frame rate. + /// + public FPS CameraFPS { get; set; } = FPS.FPS30; + + /// + /// Gets or sets a value indicating whether color and depth captures should be strictly synchronized. + /// + public bool SynchronizedImagesOnly { get; set; } = true; + + /// + /// Gets or sets a value indicating whether the color stream is emitted. + /// + public bool OutputColor { get; set; } = true; + + /// + /// Gets or sets a value indicating whether the depth stream is emitted. + /// + public bool OutputDepth { get; set; } = true; + + /// + /// Gets or sets a value indicating whether the infrared stream is emitted. + /// + public bool OutputInfrared { get; set; } = true; + + /// + /// Gets or sets a value indicating whether to use the Azure Kinect's IMU. + /// + public bool OutputImu { get; set; } = false; + + /// + /// Gets or sets a value indicating whether the Azure Kinect outputs its calibration settings. + /// + public bool OutputCalibration { get; set; } = true; + + /// + /// Gets or sets the body tracker configuration. + /// + public AzureKinectBodyTrackerConfiguration BodyTrackerConfiguration { get; set; } = null; + + /// + /// Gets or sets the timeout used for device capture. + /// + public TimeSpan DeviceCaptureTimeout { get; set; } = TimeSpan.FromMinutes(1); + + /// + /// Gets or sets the frequency at which frame rate is reported on the FrameRate emitter. + /// + public TimeSpan FrameRateReportingFrequency { get; set; } = TimeSpan.FromSeconds(2); + } +} diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.csproj b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.csproj new file mode 100644 index 000000000..8f4d75a9f --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.csproj @@ -0,0 +1,49 @@ + + + netstandard2.0 + x64 + Microsoft.Psi.AzureKinect + Provides APIs and components for using Microsoft Azure Kinect sensor. + true + Microsoft.Psi.AzureKinect.x64.nuspec + configuration=$(Configuration);version=$(Version) + ../../../Build/Microsoft.Psi.ruleset + + + x64 + DEBUG;TRACE + true + bin\x64\Debug\netstandard2.0\Microsoft.Psi.AzureKinect.x64.xml + + + x64 + bin\x64\Release\netstandard2.0\Microsoft.Psi.AzureKinect.x64.xml + true + + + + + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.nuspec b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.nuspec new file mode 100644 index 000000000..372b59b4d --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/Microsoft.Psi.AzureKinect.x64.nuspec @@ -0,0 +1,35 @@ + + + + Microsoft.Psi.AzureKinect.x64 + $version$ + Microsoft + Microsoft + true + LICENSE.txt + https://github.com/microsoft/psi/wiki + Provides APIs and components for using the Microsoft Azure Kinect sensor. + © Microsoft Corporation. All rights reserved. + Psi Microsoft + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/build.sh b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/build.sh new file mode 100644 index 000000000..956d2ae29 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/build.sh @@ -0,0 +1,3 @@ +#!/usr/bin/env bash + +dotnet build ./Microsoft.Psi.AzureKinect.x64.csproj diff --git a/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/stylecop.json b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/stylecop.json new file mode 100644 index 000000000..75ffdcd86 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.AzureKinect.x64/stylecop.json @@ -0,0 +1,16 @@ +{ + // ACTION REQUIRED: This file was automatically added to your project, but it + // will not take effect until additional steps are taken to enable it. See the + // following page for additional information: + // + // https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/EnableConfiguration.md + + "$schema": "https://raw.githubusercontent.com/DotNetAnalyzers/StyleCopAnalyzers/master/StyleCop.Analyzers/StyleCop.Analyzers/Settings/stylecop.schema.json", + "settings": { + "documentationRules": { + "companyName": "Microsoft Corporation", + "copyrightText": "Copyright (c) Microsoft Corporation. All rights reserved.\nLicensed under the MIT license.", + "xmlHeader": false + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFace.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFace.cs index 084b75ce7..595b66290 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFace.cs +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFace.cs @@ -38,7 +38,7 @@ public class KinectFace public Dictionary FacePointsInColorSpace { get; set; } /// - /// Gets or sets a list of points for each face point. Points are defined in pixels relative to the infared image. + /// Gets or sets a list of points for each face point. Points are defined in pixels relative to the infrared image. /// public Dictionary FacePointsInInfraredSpace { get; set; } diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFaceDetector.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFaceDetector.cs index 85a9172da..cf924467d 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFaceDetector.cs +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/KinectFaceDetector.cs @@ -15,9 +15,9 @@ namespace Microsoft.Psi.Kinect.Face /// public class KinectFaceDetector : IKinectFaceDetector, ISourceComponent, IDisposable { - private Kinect.KinectSensor kinectSensor = null; - private KinectFaceDetectorConfiguration configuration = null; - private Pipeline pipeline; + private readonly KinectSensor kinectSensor; + private readonly KinectFaceDetectorConfiguration configuration; + private readonly Pipeline pipeline; private FaceFrameReader[] faceFrameReaders = null; private FaceFrameSource[] faceFrameSources = null; @@ -33,7 +33,7 @@ public class KinectFaceDetector : IKinectFaceDetector, ISourceComponent, IDispos /// Pipeline this sensor is a part of. /// Psi Kinect device from which we get our associated bodies. /// Configuration to use. - public KinectFaceDetector(Pipeline pipeline, Kinect.KinectSensor kinectSensor, KinectFaceDetectorConfiguration configuration = null) + public KinectFaceDetector(Pipeline pipeline, KinectSensor kinectSensor, KinectFaceDetectorConfiguration configuration = null) { this.pipeline = pipeline; this.configuration = configuration ?? new KinectFaceDetectorConfiguration(); diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/Microsoft.Psi.Kinect.Face.Windows.x64.csproj b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/Microsoft.Psi.Kinect.Face.Windows.x64.csproj index 16d67c351..f465470ed 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/Microsoft.Psi.Kinect.Face.Windows.x64.csproj +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Face.Windows.x64/Microsoft.Psi.Kinect.Face.Windows.x64.csproj @@ -28,9 +28,13 @@ + + all + runtime; build; native; contentfiles; analyzers + - - + + diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyListVisualizationObject.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyListVisualizationObject.cs new file mode 100644 index 000000000..5062b23e4 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyListVisualizationObject.cs @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Kinect.Visualization +{ + using System.Collections.Generic; + using Microsoft.Psi.Kinect; + using Microsoft.Psi.Visualization.VisualizationObjects; + + /// + /// Represents a visualization object for Azure Kinect bodies. + /// + [VisualizationObject("Visualize Kinect Bodies")] + public class KinectBodyListVisualizationObject : ModelVisual3DVisualizationObjectEnumerable> + { + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyVisualizationObject.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyVisualizationObject.cs new file mode 100644 index 000000000..fda5b0401 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/KinectBodyVisualizationObject.cs @@ -0,0 +1,284 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Kinect.Visualization +{ + using System; + using System.ComponentModel; + using System.Runtime.Serialization; + using System.Windows; + using System.Windows.Media; + using HelixToolkit.Wpf; + using Microsoft.Kinect; + using Microsoft.Psi.Kinect; + using Microsoft.Psi.Visualization.VisualizationObjects; + using Xceed.Wpf.Toolkit.PropertyGrid.Attributes; + using Win3D = System.Windows.Media.Media3D; + + /// + /// Represents a visualization object for Kinect bodies. + /// + [VisualizationObject("Visualize Kinect Body")] + public class KinectBodyVisualizationObject : ModelVisual3DVisualizationObject + { + private readonly BillboardTextVisual3D billboard; + private readonly UpdatableVisual3DDictionary visualJoints; + private readonly UpdatableVisual3DDictionary<(JointType ChildJoint, JointType ParentJoint), PipeVisual3D> visualBones; + + private Color color = Colors.White; + private double inferredJointsOpacity = 30; + private double thicknessMm = 30; + private bool showBillboard = false; + private int polygonResolution = 6; + private double billboardHeightCm = 100; + + /// + /// Initializes a new instance of the class. + /// + public KinectBodyVisualizationObject() + { + this.visualJoints = new UpdatableVisual3DDictionary(null); + this.visualBones = new UpdatableVisual3DDictionary<(JointType ChildJoint, JointType ParentJoint), PipeVisual3D>(null); + + this.billboard = new BillboardTextVisual3D() + { + Background = Brushes.Gray, + Foreground = new SolidColorBrush(Colors.White), + Padding = new Thickness(5), + }; + + this.UpdateVisibility(); + } + + /// + /// Gets or sets the color. + /// + [DataMember] + [Description("Color of the body.")] + public Color Color + { + get { return this.color; } + set { this.Set(nameof(this.Color), ref this.color, value); } + } + + /// + /// Gets or sets the inferred joints opacity. + /// + [DataMember] + [Description("Opacity for rendering inferred joints and bones.")] + public double InferredJointsOpacity + { + get { return this.inferredJointsOpacity; } + set { this.Set(nameof(this.InferredJointsOpacity), ref this.inferredJointsOpacity, value); } + } + + /// + /// Gets or sets the thickness. + /// + [DataMember] + [DisplayName("Thickness (mm)")] + [Description("Diameter of bones and radius of joints (mm).")] + public double ThicknessMm + { + get { return this.thicknessMm; } + set { this.Set(nameof(this.ThicknessMm), ref this.thicknessMm, value); } + } + + /// + /// Gets or sets a value indicating whether to show a billboard with information about the body. + /// + [DataMember] + [PropertyOrder(0)] + [Description("Show a billboard with information about the body.")] + public bool ShowBillboard + { + get { return this.showBillboard; } + set { this.Set(nameof(this.ShowBillboard), ref this.showBillboard, value); } + } + + /// + /// Gets or sets the height at which to draw the billboard (cm). + /// + [DataMember] + [PropertyOrder(1)] + [DisplayName("Billboard Height (cm)")] + [Description("Height at which to draw the billboard (cm).")] + public double BillboardHeightCm + { + get { return this.billboardHeightCm; } + set { this.Set(nameof(this.BillboardHeightCm), ref this.billboardHeightCm, value); } + } + + /// + /// Gets or sets the number of divisions to use when rendering polygons for joints and bones. + /// + [DataMember] + [Description("Level of resolution at which to render joint and bone polygons (minimum value is 3).")] + public int PolygonResolution + { + get { return this.polygonResolution; } + set { this.Set(nameof(this.PolygonResolution), ref this.polygonResolution, value < 3 ? 3 : value); } + } + + /// + public override void UpdateData() + { + if (this.CurrentData != null) + { + this.UpdateVisuals(); + } + + this.UpdateVisibility(); + } + + /// + public override void NotifyPropertyChanged(string propertyName) + { + if (propertyName == nameof(this.Color) || + propertyName == nameof(this.InferredJointsOpacity) || + propertyName == nameof(this.ThicknessMm) || + propertyName == nameof(this.PolygonResolution)) + { + this.UpdateVisuals(); + } + else if (propertyName == nameof(this.ShowBillboard)) + { + this.UpdateBillboardVisibility(); + } + else if (propertyName == nameof(this.BillboardHeightCm)) + { + this.UpdateBillboard(); + } + else if (propertyName == nameof(this.Visible)) + { + this.UpdateVisibility(); + } + } + + private void UpdateVisuals() + { + this.visualJoints.BeginUpdate(); + this.visualBones.BeginUpdate(); + + if (this.CurrentData != null) + { + var trackedEntitiesBrush = new SolidColorBrush(this.Color); + var untrackedEntitiesBrush = new SolidColorBrush( + Color.FromArgb( + (byte)(Math.Max(0, Math.Min(100, this.InferredJointsOpacity)) * 2.55), + this.Color.R, + this.Color.G, + this.Color.B)); + + // update the joints + foreach (var jointType in this.CurrentData.Joints.Keys) + { + var jointState = this.CurrentData.Joints[jointType].TrackingState; + var visualJoint = this.visualJoints[jointType]; + visualJoint.BeginEdit(); + var isTracked = jointState == TrackingState.Tracked; + var visible = jointState != TrackingState.NotTracked && (isTracked || this.InferredJointsOpacity > 0); + + if (visible) + { + var jointPosition = this.CurrentData.Joints[jointType].Pose.Origin; + + if (visualJoint.Radius != this.ThicknessMm / 1000.0) + { + visualJoint.Radius = this.ThicknessMm / 1000.0; + } + + var fill = isTracked ? trackedEntitiesBrush : untrackedEntitiesBrush; + if (visualJoint.Fill != fill) + { + visualJoint.Fill = fill; + } + + visualJoint.Transform = new Win3D.TranslateTransform3D(jointPosition.X, jointPosition.Y, jointPosition.Z); + + visualJoint.PhiDiv = this.PolygonResolution; + visualJoint.ThetaDiv = this.PolygonResolution; + + visualJoint.Visible = true; + } + else + { + visualJoint.Visible = false; + } + + visualJoint.EndEdit(); + } + + // update the bones + foreach (var bone in KinectBody.Bones) + { + var parentState = this.CurrentData.Joints[bone.ParentJoint].TrackingState; + var childState = this.CurrentData.Joints[bone.ChildJoint].TrackingState; + var parentIsTracked = parentState == TrackingState.Tracked; + var childIsTracked = childState == TrackingState.Tracked; + var isTracked = parentIsTracked && childIsTracked; + var visible = parentState != TrackingState.NotTracked && childState != TrackingState.NotTracked && (isTracked || this.InferredJointsOpacity > 0); + var visualBone = this.visualBones[bone]; + visualBone.BeginEdit(); + if (visible) + { + if (visualBone.Diameter != this.ThicknessMm / 1000.0) + { + visualBone.Diameter = this.ThicknessMm / 1000.0; + } + + var joint1Position = this.visualJoints[bone.ParentJoint].Transform.Value; + var joint2Position = this.visualJoints[bone.ChildJoint].Transform.Value; + + visualBone.Point1 = new Win3D.Point3D(joint1Position.OffsetX, joint1Position.OffsetY, joint1Position.OffsetZ); + visualBone.Point2 = new Win3D.Point3D(joint2Position.OffsetX, joint2Position.OffsetY, joint2Position.OffsetZ); + + var fill = isTracked ? trackedEntitiesBrush : untrackedEntitiesBrush; + if (visualBone.Fill != fill) + { + visualBone.Fill = fill; + } + + visualBone.ThetaDiv = this.PolygonResolution; + + visualBone.Visible = true; + } + else + { + visualBone.Visible = false; + } + + visualBone.EndEdit(); + } + + // set billboard position + this.UpdateBillboard(); + } + + this.visualJoints.EndUpdate(); + this.visualBones.EndUpdate(); + } + + private void UpdateBillboard() + { + if (this.CurrentData != null) + { + var origin = this.CurrentData.Joints[JointType.SpineBase].Pose.Origin; + this.billboard.Position = new Win3D.Point3D(origin.X, origin.Y, origin.Z + (this.BillboardHeightCm / 100.0)); + this.billboard.Text = this.CurrentData.ToString(); + } + } + + private void UpdateVisibility() + { + this.UpdateChildVisibility(this.visualJoints, this.Visible && this.CurrentData != default); + this.UpdateChildVisibility(this.visualBones, this.Visible && this.CurrentData != default); + this.UpdateBillboardVisibility(); + } + + private void UpdateBillboardVisibility() + { + this.UpdateChildVisibility(this.billboard, this.Visible && this.CurrentData != default && this.ShowBillboard); + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/Microsoft.Psi.Kinect.Visualization.Windows.csproj b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/Microsoft.Psi.Kinect.Visualization.Windows.csproj new file mode 100644 index 000000000..8883063d6 --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/Microsoft.Psi.Kinect.Visualization.Windows.csproj @@ -0,0 +1,46 @@ + + + net472 + Microsoft.Psi.Kinect.Visualization.Windows + Microsoft.Psi.Kinect.Visualization + AnyCPU + ../../../Build/Microsoft.Psi.ruleset + Provides visualizers for Kinect v2. + + + DEBUG;TRACE + bin\Debug\net472\Microsoft.Psi.Kinect.Visualization.Windows.xml + true + + + + bin\Release\net472\Microsoft.Psi.Kinect.Visualization.Windows.xml + true + + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/stylecop.json b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/stylecop.json new file mode 100644 index 000000000..6f09427eb --- /dev/null +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Visualization.Windows/stylecop.json @@ -0,0 +1,16 @@ +{ + // ACTION REQUIRED: This file was automatically added to your project, but it + // will not take effect until additional steps are taken to enable it. See the + // following page for additional information: + // + // https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/EnableConfiguration.md + + "$schema": "https://raw.githubusercontent.com/DotNetAnalyzers/StyleCopAnalyzers/master/StyleCop.Analyzers/StyleCop.Analyzers/Settings/stylecop.schema.json", + "settings": { + "documentationRules": { + "companyName": "Microsoft Corporation", + "copyrightText": "Copyright (c) Microsoft Corporation. All rights reserved.\nLicensed under the MIT license.", + "xmlHeader": false + } + } +} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthExtensions.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthExtensions.cs deleted file mode 100644 index 6855d0b5e..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthExtensions.cs +++ /dev/null @@ -1,83 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System.IO; - using System.IO.Compression; - using Microsoft.Psi.Imaging; - - /// - /// Define set of extensions for dealing with depth maps. - /// - public static class DepthExtensions - { - /// - /// Simple producer for converting from depth map to colored version of depth map. - /// - /// Depth image to convert. - /// An optional delivery policy. - /// Returns colored representation of the depth map. - public static IProducer> ToColor(this IProducer> depthImage, DeliveryPolicy> deliveryPolicy = null) - { - return depthImage.PipeTo(new DepthToColorConverter(depthImage.Out.Pipeline), deliveryPolicy); - } - - /// - /// Creates a gzipped byte array of the depth image. - /// - /// Depth image to compress. - /// An optional delivery policy. - /// Byte array containing the compressed depth map. - public static IProducer GZipCompressDepthImage(this IProducer> depthImage, DeliveryPolicy> deliveryPolicy = null) - { - var memoryStream = new MemoryStream(); - var memoryStreamLo = new MemoryStream(); - var memoryStreamHi = new MemoryStream(); - byte[] buffer = null; - return depthImage.Select( - image => - { - if (buffer == null) - { - buffer = new byte[image.Resource.Size]; - } - - image.Resource.CopyTo(buffer); - memoryStream.Seek(0, SeekOrigin.Begin); - using (var compressedStream = new GZipStream(memoryStream, CompressionLevel.Optimal, true)) - { - compressedStream.Write(buffer, 0, buffer.Length); - } - - var output = new byte[memoryStream.Position]; - memoryStream.Seek(0, SeekOrigin.Begin); - memoryStream.Read(output, 0, output.Length); - return output; - }, deliveryPolicy); - } - - /// - /// Uncompressed a depth map that was previously compressed with GZip. - /// - /// Byte array of compressed depth values. - /// An optional delivery policy. - /// Uncompressed depth map as an image. - public static IProducer> GZipUncompressDepthImage(this IProducer compressedDepthBytes, DeliveryPolicy deliveryPolicy = null) - { - var buffer = new byte[424 * 512 * 2]; - return compressedDepthBytes.Select( - bytes => - { - using (var compressedStream = new GZipStream(new MemoryStream(bytes), CompressionMode.Decompress)) - { - compressedStream.Read(buffer, 0, buffer.Length); - } - - var psiImage = ImagePool.GetOrCreate(512, 424, PixelFormat.Gray_16bpp); - psiImage.Resource.CopyFrom(buffer); - return psiImage; - }, deliveryPolicy); - } - } -} diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthToColorConverter.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthToColorConverter.cs deleted file mode 100644 index ebdee19df..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/DepthToColorConverter.cs +++ /dev/null @@ -1,76 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System; - using System.Threading.Tasks; - using Microsoft.Psi; - using Microsoft.Psi.Components; - using Microsoft.Psi.Imaging; - - /// - /// DepthToColorConverter defines a component for converting a depth image from the Kinect - /// into a color image (where more distant objects are blue, and closer objects are reddish). - /// - public class DepthToColorConverter : ConsumerProducer, Shared> - { - /// - /// Initializes a new instance of the class. - /// - /// Pipeline this component is a part of. - public DepthToColorConverter(Pipeline pipeline) - : base(pipeline) - { - } - - /// - /// Pipeline callback for converting depth image to colored image. - /// - /// Depth image. - /// Pipeline information about current depthImage sample. - protected override void Receive(Shared depthImage, Envelope e) - { - using (var colorImageDest = ImagePool.GetOrCreate(depthImage.Resource.Width, depthImage.Resource.Height, Imaging.PixelFormat.BGR_24bpp)) - { - unsafe - { - ushort maxDepth = ushort.MaxValue; - ushort minDepth = 0; - - Parallel.For(0, depthImage.Resource.Height, iy => - { - ushort* src = (ushort*)((byte*)depthImage.Resource.ImageData.ToPointer() + (iy * depthImage.Resource.Stride)); - byte* dst = (byte*)colorImageDest.Resource.ImageData.ToPointer() + (iy * colorImageDest.Resource.Stride); - - for (int ix = 0; ix < depthImage.Resource.Width; ix++) - { - ushort depth = *src; - - // short adaptation - int normalizedDepth = (depth >= minDepth && depth <= maxDepth) ? (depth * 1024 / 8000) : 0; - dst[0] = (byte)this.Saturate(384 - (int)Math.Abs(normalizedDepth - 256)); - dst[1] = (byte)this.Saturate(384 - (int)Math.Abs(normalizedDepth - 512)); - dst[2] = (byte)this.Saturate(384 - (int)Math.Abs(normalizedDepth - 768)); - - dst += 3; - src += 1; - } - }); - } - - this.Out.Post(colorImageDest, e.OriginatingTime); - } - } - - /// - /// Simple saturate (clamp between 0..255) helper. - /// - /// Value to saturate. - /// Clamped color value. - private float Saturate(int value) - { - return Math.Max(0f, Math.Min(255, value)); - } - } -} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/IKinectSensor.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/IKinectSensor.cs deleted file mode 100644 index 9d8c62cdb..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/IKinectSensor.cs +++ /dev/null @@ -1,62 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System.Collections.Generic; - using Microsoft.Psi; - using Microsoft.Psi.Audio; - using Microsoft.Psi.Calibration; - using Microsoft.Psi.Imaging; - - /// - /// IKinectSensor defines the interface used to interact with the Kinect. - /// - public interface IKinectSensor - { - /// - /// Gets an emitter that emits a stream of KinectBody samples. - /// - Emitter> Bodies { get; } - - /// - /// Gets an emitter that emits a stream of image samples for the Kinect's color camera. - /// - Emitter> ColorImage { get; } - - /// - /// Gets an emitter that emits a stream of image samples for the Kinect's depth camera. - /// - Emitter> DepthImage { get; } - - /// - /// Gets an emitter that emits a stream of image samples for the Kinect's infrared feed. - /// - Emitter> InfraredImage { get; } - - /// - /// Gets an emitter that emits a stream of image samples for the Kinect's long exposure infrared feed. - /// - Emitter> LongExposureInfraredImage { get; } - - /// - /// Gets an emitter that emits a stream of depth device calibration info objects for the Kinect. - /// - Emitter DepthDeviceCalibrationInfo { get; } - - /// - /// Gets an emitter that emits a stream of AudioBuffer samples from the Kinect. - /// - Emitter Audio { get; } - - /// - /// Gets an emitter that emits a stream of KinectAudioBeamInfo samples from the Kinect. - /// - Emitter AudioBeamInfo { get; } - - /// - /// Gets an emitter that emits a stream of IList.ulong samples from the Kinect. - /// - Emitter> AudioBodyCorrelations { get; } - } -} \ No newline at end of file diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectBody.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectBody.cs index 2cd585fbb..313623257 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectBody.cs +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectBody.cs @@ -4,6 +4,9 @@ namespace Microsoft.Psi.Kinect { using System.Collections.Generic; + using System.Numerics; + using MathNet.Numerics.LinearAlgebra; + using MathNet.Spatial.Euclidean; using Microsoft.Kinect; /// @@ -11,76 +14,181 @@ namespace Microsoft.Psi.Kinect /// public class KinectBody { - private static int jointCount = Body.JointCount; + private static readonly CoordinateSystem KinectBasis = new CoordinateSystem(default, UnitVector3D.ZAxis, UnitVector3D.XAxis, UnitVector3D.YAxis); + private static readonly CoordinateSystem KinectBasisInverse = KinectBasis.Invert(); /// - /// Gets the number of joints in body's skeleton. + /// Gets the bone relationships. /// - public int JointCount => jointCount; + public static List<(JointType ChildJoint, JointType ParentJoint)> Bones { get; } = new List<(JointType, JointType)> + { + // Spine and head + (JointType.SpineMid, JointType.SpineBase), + (JointType.SpineShoulder, JointType.SpineMid), + (JointType.Neck, JointType.SpineShoulder), + (JointType.Head, JointType.Neck), + + // Left arm + (JointType.ShoulderLeft, JointType.SpineShoulder), + (JointType.ElbowLeft, JointType.ShoulderLeft), + (JointType.WristLeft, JointType.ElbowLeft), + (JointType.HandLeft, JointType.WristLeft), + (JointType.HandTipLeft, JointType.HandLeft), + (JointType.ThumbLeft, JointType.WristLeft), + + // Right arm + (JointType.ShoulderRight, JointType.SpineShoulder), + (JointType.ElbowRight, JointType.ShoulderRight), + (JointType.WristRight, JointType.ElbowRight), + (JointType.HandRight, JointType.WristRight), + (JointType.HandTipRight, JointType.HandRight), + (JointType.ThumbRight, JointType.WristRight), + + // Left leg + (JointType.HipLeft, JointType.SpineBase), + (JointType.KneeLeft, JointType.HipLeft), + (JointType.AnkleLeft, JointType.KneeLeft), + (JointType.FootLeft, JointType.AnkleLeft), + + // Right leg + (JointType.HipRight, JointType.SpineBase), + (JointType.KneeRight, JointType.HipRight), + (JointType.AnkleRight, JointType.KneeRight), + (JointType.FootRight, JointType.AnkleRight), + }; /// - /// Gets or sets the clipped edges. + /// Gets the clipped edges. /// - public FrameEdges ClippedEdges { get; set; } + public FrameEdges ClippedEdges { get; private set; } /// /// Gets or sets the floor's clip plane. /// - public Vector4 FloorClipPlane { get; set; } + public Microsoft.Kinect.Vector4 FloorClipPlane { get; set; } /// - /// Gets or sets confidence in position/pose of left hand. + /// Gets confidence in position/pose of left hand. /// - public TrackingConfidence HandLeftConfidence { get; set; } + public TrackingConfidence HandLeftConfidence { get; private set; } /// - /// Gets or sets state of left hand. + /// Gets state of left hand. /// - public HandState HandLeftState { get; set; } + public HandState HandLeftState { get; private set; } /// - /// Gets or sets confidence in position/pose of right hand. + /// Gets confidence in position/pose of right hand. /// - public TrackingConfidence HandRightConfidence { get; set; } + public TrackingConfidence HandRightConfidence { get; private set; } /// - /// Gets or sets state of right hand. + /// Gets state of right hand. /// - public HandState HandRightState { get; set; } + public HandState HandRightState { get; private set; } /// - /// Gets or sets a value indicating whether the body is restricted. + /// Gets a value indicating whether the body is restricted. /// - public bool IsRestricted { get; set; } + public bool IsRestricted { get; private set; } /// - /// Gets or sets a value indicating whether the body is tracked. + /// Gets a value indicating whether the body is tracked. /// - public bool IsTracked { get; set; } + public bool IsTracked { get; private set; } /// - /// Gets or sets the joint orientations. + /// Gets the joint information. /// - public Dictionary JointOrientations { get; set; } + public Dictionary Joints { get; } = + new Dictionary(); /// - /// Gets or sets the joints. + /// Gets the lean point. /// - public Dictionary Joints { get; set; } + public Point2D Lean { get; private set; } /// - /// Gets or sets the lean point. + /// Gets the lean tracking state. /// - public PointF Lean { get; set; } + public TrackingState LeanTrackingState { get; private set; } /// - /// Gets or sets the lean tracking state. + /// Gets the body's tracking ID. /// - public TrackingState LeanTrackingState { get; set; } + public ulong TrackingId { get; private set; } /// - /// Gets or sets the body's tracking ID. + /// Populate this body representation with new joint and tracking information. /// - public ulong TrackingId { get; set; } + /// The body from the Kinect sensor from which to populate this body with information. + public void UpdateFrom(Body body) + { + this.TrackingId = body.TrackingId; + this.ClippedEdges = body.ClippedEdges; + this.HandLeftConfidence = body.HandLeftConfidence; + this.HandLeftState = body.HandLeftState; + this.HandRightConfidence = body.HandRightConfidence; + this.HandRightState = body.HandRightState; + this.IsRestricted = body.IsRestricted; + this.IsTracked = body.IsTracked; + this.Lean = new Point2D(body.Lean.X, body.Lean.Y); + this.LeanTrackingState = body.LeanTrackingState; + + foreach (var jointType in body.Joints.Keys) + { + var joint = body.Joints[jointType]; + + CoordinateSystem kinectJointCS; + + if (body.JointOrientations.ContainsKey(jointType)) + { + kinectJointCS = this.CreateCoordinateSystem(joint.Position, body.JointOrientations[jointType].Orientation); + } + else + { + kinectJointCS = this.CreateCoordinateSystem(joint.Position, default); + } + + this.Joints[jointType] = (kinectJointCS, joint.TrackingState); + } + } + + /// + public override string ToString() + { + return $"ID: {this.TrackingId}"; + } + + private CoordinateSystem CreateCoordinateSystem(CameraSpacePoint position, Microsoft.Kinect.Vector4 quaternion) + { + Matrix kinectJointMatrix; + + if (quaternion == default) + { + kinectJointMatrix = Matrix.Build.DenseOfArray(new double[,] + { + { 1, 0, 0, position.X }, + { 0, 1, 0, position.Y }, + { 0, 0, 1, position.Z }, + { 0, 0, 0, 1 }, + }); + } + else + { + // Convert the quaternion into a System.Numerics matrix, and then create a MathNet matrix while including the position. + var jointRotation = Matrix4x4.CreateFromQuaternion(new System.Numerics.Quaternion(quaternion.X, quaternion.Y, quaternion.Z, quaternion.W)); + kinectJointMatrix = Matrix.Build.DenseOfArray(new double[,] + { + { jointRotation.M11, jointRotation.M21, jointRotation.M31, position.X }, + { jointRotation.M12, jointRotation.M22, jointRotation.M32, position.Y }, + { jointRotation.M13, jointRotation.M23, jointRotation.M33, position.Z }, + { 0, 0, 0, 1 }, + }); + } + + // Convert from Kinect to MathNet basis. + return new CoordinateSystem(KinectBasisInverse * kinectJointMatrix * KinectBasis); + } } } diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectExtensions.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectExtensions.cs deleted file mode 100644 index c4acc6ea9..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectExtensions.cs +++ /dev/null @@ -1,472 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System; - using System.Collections.Generic; - using System.IO; - using System.IO.Compression; - using System.Linq; - using MathNet.Numerics.LinearAlgebra; - using MathNet.Spatial.Euclidean; - using Microsoft.Kinect; - using Microsoft.Psi.Calibration; - using Microsoft.Psi.Imaging; - - /// - /// Implements stream operator methods for Kinect. - /// - public static class KinectExtensions - { - private static readonly CoordinateSystem KinectBasis = new CoordinateSystem(default, UnitVector3D.ZAxis, UnitVector3D.XAxis, UnitVector3D.YAxis); - private static readonly CoordinateSystem KinectBasisInverse = KinectBasis.Invert(); - - /// - /// Returns the position of a given joint in the body. - /// - /// The stream of kinect body. - /// The type of joint. - /// An optional delivery policy. - /// The joint position as a 3D point in Kinect camera space. - public static IProducer GetJointPosition(this IProducer source, JointType jointType, DeliveryPolicy deliveryPolicy = null) - { - return source.Select(kb => kb.Joints[jointType].Position.ToPoint3D(), deliveryPolicy); - } - - /// - /// Projects set of 2D image points into 3D. - /// - /// Tuple of depth image, list of points to project, and calibration information. - /// An optional delivery policy. - /// Returns a producer that generates a list of corresponding 3D points in Kinect camera space. - public static IProducer> ProjectTo3D( - this IProducer<(Shared, List, IDepthDeviceCalibrationInfo)> source, DeliveryPolicy<(Shared, List, IDepthDeviceCalibrationInfo)> deliveryPolicy = null) - { - var projectTo3D = new ProjectTo3D(source.Out.Pipeline); - source.PipeTo(projectTo3D, deliveryPolicy); - return projectTo3D; - } - - /// - /// Transforms a Kinect CameraSpacePoint to another coordinate system. - /// - /// Coordinate system to transform to. - /// Point in Kinect camera space. - /// Tranformed point in Kinect camera space. - public static CameraSpacePoint Transform(this CoordinateSystem cs, CameraSpacePoint cameraSpacePoint) - { - return cs.Transform(cameraSpacePoint.ToPoint3D()).ToCameraSpacePoint(); - } - - /// - /// Rebases the kinect body in a specified coordinate system. - /// - /// Body to transform. - /// Coordinate system to transform to. - /// The body rebased in the specified coordinate system. - /// The method rebases all the joints, including position and orientation, in the specified coordinate system. - public static KinectBody Rebase(this KinectBody kinectBody, CoordinateSystem cs) - { - var transformed = kinectBody.DeepClone(); - - foreach (JointType jointType in Enum.GetValues(typeof(JointType))) - { - // Create a CoordinateSystem from the joint - CoordinateSystem kinectJointCS = transformed.GetJointCoordinateSystem(jointType); - - // Transform by the given coordinate system - kinectJointCS = kinectJointCS.TransformBy(cs); - - // Convert position back to camera space point - if (transformed.Joints.ContainsKey(jointType)) - { - var joint = transformed.Joints[jointType]; - joint.Position = kinectJointCS.Origin.ToCameraSpacePoint(); - transformed.Joints[jointType] = joint; - } - - // Convert rotation back to Kinect joint orientation - if (transformed.JointOrientations.ContainsKey(jointType)) - { - var rot = kinectJointCS.FromMathNetBasis().GetRotationSubMatrix(); - var q = MatrixToQuaternion(rot); - JointOrientation jointOrientation = transformed.JointOrientations[jointType]; - jointOrientation.Orientation.X = q.X; - jointOrientation.Orientation.Y = q.Y; - jointOrientation.Orientation.Z = q.Z; - jointOrientation.Orientation.W = q.W; - transformed.JointOrientations[jointType] = jointOrientation; - } - } - - return transformed; - } - - /// - /// Creates a coordinate system from a given Kinect joint. - /// - /// Kinect body containing the joint. - /// Which joint to create a coordinate system from. - /// Coordinate system capturing the given joint's orientation and position. - public static CoordinateSystem GetJointCoordinateSystem(this KinectBody kinectBody, JointType jointType) - { - if (!kinectBody.Joints.ContainsKey(jointType)) - { - throw new Exception($"Cannot create a coordinate system out of non-existent joint: {jointType}"); - } - - // Create a CoordinateSystem, starting from the Kinect's defined basis - CoordinateSystem kinectJointCS = new CoordinateSystem(); - - // Get the orientation as a rotation - if (kinectBody.JointOrientations.ContainsKey(jointType)) - { - var jointOrientation = kinectBody.JointOrientations[jointType].Orientation; - kinectJointCS = kinectJointCS.SetRotationSubMatrix(QuaternionToMatrix(jointOrientation)).ToMathNetBasis(); - } - - // Get the position as a translation - var cameraSpacePoint = kinectBody.Joints[jointType].Position; - return kinectJointCS.SetTranslation(cameraSpacePoint.ToPoint3D().ToVector3D()); - } - - /// - /// Transforms the specified 3D point into a 2D point via the specified calibration. - /// - /// A stream of tuples containing the 3D point and calibration inforamtion. - /// An optional delivery policy. - /// A producer that generates the 2D transformed points. - public static IProducer ToColorSpace(this IProducer<(Point3D, IDepthDeviceCalibrationInfo)> source, DeliveryPolicy<(Point3D, IDepthDeviceCalibrationInfo)> deliveryPolicy = null) - { - return source.Select( - m => - { - var (point3D, calibration) = m; - if (calibration != default(IDepthDeviceCalibrationInfo)) - { - return calibration.ToColorSpace(point3D); - } - else - { - return default(Point2D?); - } - }, - deliveryPolicy).Where(p => p.HasValue, DeliveryPolicy.SynchronousOrThrottle).Select(p => p.Value, DeliveryPolicy.SynchronousOrThrottle); - } - - /// - /// Converts points in from Kinect color space into 2D points. - /// - /// A stream of points in color space. - /// An optional delivery policy. - /// A producer that generates transformed 2D points. - public static IProducer ToPoint2D(this IProducer source, DeliveryPolicy deliveryPolicy = null) - { - return source.NullableSelect(p => new Point2D(p.X, p.Y), deliveryPolicy); - } - - /// - /// Converts points in from Kinect color space into 2D points. - /// - /// A stream of points in color space. - /// An optional delivery policy. - /// A producer that generates transformed 2D points. - public static IProducer ToPoint2D(this IProducer source, DeliveryPolicy deliveryPolicy = null) - { - return source.Select(p => new Point2D(p.X, p.Y), deliveryPolicy); - } - - /// - /// Returns the coordinate system corresponding to a tracked joint from the kinect body. - /// - /// The stream of Kinect body. - /// The joint to return. - /// An optional delivery policy. - /// A producer that generates the coordinate system for the specified joint if the joint is tracked. If the joint is not tracked, no message is posted on the return stream. - public static IProducer GetTrackedJointPosition(this IProducer source, JointType jointType, DeliveryPolicy deliveryPolicy = null) - { - return source.GetTrackedJointPositionOrDefault(jointType, deliveryPolicy).Where(cs => cs != null, DeliveryPolicy.SynchronousOrThrottle); - } - - /// - /// Returns the coordinate system corresponding to a tracked joint from the kinect body, or null if the specified joint is not currently tracked. - /// - /// The stream of Kinect body. - /// The joint to return. - /// An optional delivery policy. - /// A producer that generates the coordinate system for the specified joint if the joint is tracker, or null otherwise. - public static IProducer GetTrackedJointPositionOrDefault(this IProducer source, JointType jointType, DeliveryPolicy deliveryPolicy = null) - { - return source.Select( - body => - { - var joint = body.Joints.Values.FirstOrDefault(j => j.JointType == jointType && j.TrackingState == TrackingState.Tracked); - if (joint == default(Joint)) - { - return null; - } - else - { - var jointOrientation = body.JointOrientations.Values.FirstOrDefault(j => j.JointType == jointType); - var quaternion = new Quaternion(jointOrientation.Orientation.W, jointOrientation.Orientation.X, jointOrientation.Orientation.Y, jointOrientation.Orientation.Z); - var euler = quaternion.ToEulerAngles(); - var cs = CoordinateSystem.Rotation(euler.Gamma, euler.Beta, euler.Alpha); - return cs.TransformBy(CoordinateSystem.Translation(new Vector3D(joint.Position.X, joint.Position.Y, joint.Position.Z))).ToMathNetBasis(); - } - }, deliveryPolicy); - } - - /// - /// Converts a point from Kinect camera space to a 3D MathNet point. - /// - /// The Kinect camera space point to convert. - /// The corresponding 3D point (in MathNet basis). - public static Point3D ToPoint3D(this CameraSpacePoint cameraSpacePoint) - { - return new Point3D(cameraSpacePoint.Z, cameraSpacePoint.X, cameraSpacePoint.Y); - } - - /// - /// Converts points from Kinect camera space to 3D MathNet points. - /// - /// Stream of Kinect camera space points. - /// An optional delivery policy. - /// A producer that generates the corresponding 3D points (in MathNet basis). - public static IProducer ToPoint3D(this IProducer source, DeliveryPolicy deliveryPolicy = null) - { - return source.Select(p => ToPoint3D(p), deliveryPolicy); - } - - /// - /// Converts a 3D MathNet point to a Kinect camera space point. - /// - /// The 3D point to convert. - /// The corresponding Kinect camera space point. - public static CameraSpacePoint ToCameraSpacePoint(this Point3D point) - { - return new CameraSpacePoint() - { - X = (float)point.Y, - Y = (float)point.Z, - Z = (float)point.X, - }; - } - - /// - /// Converts 3D points to Kinect camera space points. - /// - /// Stream of 3D points to convert. - /// An optional delivery policy. - /// A producer that generates the corresponding Kinect camera space points. - public static IProducer ToCameraSpacePoint(this IProducer point, DeliveryPolicy deliveryPolicy = null) - { - return point.Select(p => ToCameraSpacePoint(p), deliveryPolicy); - } - - /// - /// Compresses a list of Kinect camera space points. - /// - /// Stream of list of Kinect camera space points. - /// An optional delivery policy. - /// A producer that generates a compressed byte array representation of the given Kinect camera space points. - public static IProducer GZipCompressImageProjection(this IProducer cameraSpacePoints, DeliveryPolicy deliveryPolicy = null) - { - var memoryStream = new MemoryStream(); - var memoryStreamLo = new MemoryStream(); - var memoryStreamHi = new MemoryStream(); - byte[] buffer = null; - return cameraSpacePoints.Select( - pointArray => - { - var flatPointList = new List(); - foreach (var cp in pointArray) - { - flatPointList.Add(cp.X); - flatPointList.Add(cp.Y); - flatPointList.Add(cp.Z); - } - - if (buffer == null) - { - buffer = new byte[flatPointList.Count * 4]; - } - - Buffer.BlockCopy(flatPointList.ToArray(), 0, buffer, 0, buffer.Length); - memoryStream.Seek(0, SeekOrigin.Begin); - using (var compressedStream = new GZipStream(memoryStream, CompressionLevel.Optimal, true)) - { - compressedStream.Write(buffer, 0, buffer.Length); - } - - var output = new byte[memoryStream.Position]; - memoryStream.Seek(0, SeekOrigin.Begin); - memoryStream.Read(output, 0, output.Length); - return output; - }, deliveryPolicy); - } - - /// - /// Uncompresses a list of Kinect 3D points. - /// - /// A stream containing a compressed representation of Kinect camera space points. - /// An optional delivery policy. - /// A producer that generates the corresponding list of 3D points. - public static IProducer GZipUncompressImageProjection(this IProducer compressedBytes, DeliveryPolicy deliveryPolicy = null) - { - var buffer = new byte[1920 * 1080 * 12]; - return compressedBytes.Select( - bytes => - { - using (var compressedStream = new GZipStream(new MemoryStream(bytes), CompressionMode.Decompress)) - { - compressedStream.Read(buffer, 0, buffer.Length); - } - - var floatArray = new float[buffer.Length / 4]; - Buffer.BlockCopy(buffer, 0, floatArray, 0, buffer.Length); - - List pointList = new List(); - for (int i = 0; i < floatArray.Length; i += 3) - { - pointList.Add(new CameraSpacePoint() - { - X = floatArray[i], - Y = floatArray[i + 1], - Z = floatArray[i + 2], - }); - } - - return pointList.ToArray(); - }, deliveryPolicy); - } - - /// - /// Converts a rotation matrix to a quaternion. - /// - /// Rotation matrix to convert. - /// Quaternion that represents the rotation. - // Derived from: - // http://www.euclideanspace.com/maths/geometry/rotations/conversions/matrixToQuaternion/index.htm - public static Vector4 MatrixToQuaternion(Matrix matrix) - { - Vector4 v; - float trace = (float)(1.0 + matrix[0, 0] + matrix[1, 1] + matrix[2, 2]); - if (trace > 0) - { - var s = Math.Sqrt(trace) * 2.0; - v.X = (float)((matrix[2, 1] - matrix[1, 2]) / s); - v.Y = (float)((matrix[0, 2] - matrix[2, 0]) / s); - v.Z = (float)((matrix[1, 0] - matrix[0, 1]) / s); - v.W = (float)(s / 4.0); - } - else if ((matrix[0, 0] > matrix[1, 1]) && (matrix[0, 0] > matrix[2, 2])) - { - var s = Math.Sqrt(1.0 + matrix[0, 0] - matrix[1, 1] - matrix[2, 2]) * 2.0; - v.X = (float)(s / 4.0); - v.Y = (float)((matrix[0, 1] + matrix[1, 0]) / s); - v.Z = (float)((matrix[0, 2] + matrix[2, 0]) / s); - v.W = (float)((matrix[2, 1] - matrix[1, 2]) / s); - } - else if (matrix[1, 1] > matrix[2, 2]) - { - var s = Math.Sqrt(1.0 + matrix[1, 1] - matrix[0, 0] - matrix[2, 2]) * 2.0; - v.X = (float)((matrix[0, 1] + matrix[1, 0]) / s); - v.Y = (float)(s / 4.0); - v.Z = (float)((matrix[1, 2] + matrix[2, 1]) / s); - v.W = (float)((matrix[0, 2] - matrix[2, 0]) / s); - } - else - { - var s = Math.Sqrt(1.0 + matrix[2, 2] - matrix[0, 0] - matrix[1, 1]) * 2.0; - v.X = (float)((matrix[0, 2] + matrix[2, 0]) / s); - v.Y = (float)((matrix[1, 2] + matrix[2, 1]) / s); - v.Z = (float)(s / 4.0); - v.W = (float)((matrix[1, 0] - matrix[0, 1]) / s); - } - - return v; - } - - /// - /// Converts a quaternion into an axis/angle representation. - /// - /// Quaternion to convert. - /// Axis angle representation corresponding to the quaternion. - internal static Vector4 QuaternionAsAxisAngle(Vector4 quaternion) - { - Vector4 v; - float len = (float)Math.Sqrt(quaternion.X * quaternion.X + quaternion.Y * quaternion.Y + quaternion.Z * quaternion.Z); - v.X = quaternion.X / len; - v.Y = quaternion.Y / len; - v.Z = quaternion.Z / len; - v.W = 2.0f * (float)Math.Atan2(len, quaternion.W); - return v; - } - - /// - /// Converts a quaternion into a matrix. - /// - /// Quaternion to convert. - /// Rotation matrix corresponding to the quaternion. - internal static Matrix QuaternionToMatrix(Vector4 quaternion) - { - var s = (float)Math.Sqrt(quaternion.X * quaternion.X + quaternion.Y * quaternion.Y + quaternion.Z * quaternion.Z + quaternion.W * quaternion.W); - - if (s <= float.Epsilon) - { - return CoordinateSystem.CreateIdentity(3); - } - - quaternion.X /= s; - quaternion.Y /= s; - quaternion.Z /= s; - quaternion.W /= s; - var qij = quaternion.X * quaternion.Y; - var qik = quaternion.X * quaternion.Z; - var qir = quaternion.X * quaternion.W; - var qjk = quaternion.Y * quaternion.Z; - var qjr = quaternion.Y * quaternion.W; - var qkr = quaternion.Z * quaternion.W; - var qii = quaternion.X * quaternion.X; - var qjj = quaternion.Y * quaternion.Y; - var qkk = quaternion.Z * quaternion.Z; - var a00 = 1.0 - 2.0 * (qjj + qkk); - var a11 = 1.0 - 2.0 * (qii + qkk); - var a22 = 1.0 - 2.0 * (qii + qjj); - var a01 = 2.0 * (qij - qkr); - var a10 = 2.0 * (qij + qkr); - var a02 = 2.0 * (qik + qjr); - var a20 = 2.0 * (qik - qjr); - var a12 = 2.0 * (qjk - qir); - var a21 = 2.0 * (qjk + qir); - double[] data = - { - a00, a01, a02, - a10, a11, a12, - a20, a21, a22, - }; - return MathNet.Numerics.LinearAlgebra.CreateMatrix.Dense(3, 3, data); - } - - /// - /// Convert a given coordinate system from Kinect basis (Forward=Z, Left=X, Up=Y) to MathNet's asssumed basis (Forward=X, Left=Y, Up=Z). - /// - /// Input coordinate system in Kinect basis. - /// The coordinate system transformed into MathNet's basis. - internal static CoordinateSystem ToMathNetBasis(this CoordinateSystem cs) - { - return new CoordinateSystem(KinectBasisInverse * cs * KinectBasis); - } - - /// - /// Convert a given coordinate system to Kinect basis (Forward=Z, Left=X, Up=Y) from MathNet's asssumed basis (Forward=X, Left=Y, Up=Z). - /// - /// Input coordinate system in MathNet basis. - /// The coordinate system transformed into Kinect's basis. - internal static CoordinateSystem FromMathNetBasis(this CoordinateSystem cs) - { - return new CoordinateSystem(KinectBasis * cs * KinectBasisInverse); - } - } -} diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectInternalCalibration.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectInternalCalibration.cs index 1b8ab0306..e8074bdcf 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectInternalCalibration.cs +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectInternalCalibration.cs @@ -8,7 +8,9 @@ namespace Microsoft.Psi.Kinect using System; using System.Collections.Generic; using System.Xml.Serialization; + using MathNet.Numerics.LinearAlgebra; using Microsoft.Kinect; + using Microsoft.Psi.Calibration; internal class KinectInternalCalibration { @@ -17,94 +19,21 @@ internal class KinectInternalCalibration public const int colorImageWidth = 1920; public const int colorImageHeight = 1080; - public Matrix colorCameraMatrix = new Matrix(3, 3); - public Matrix colorLensDistortion = new Matrix(5, 1); - public Matrix depthCameraMatrix = new Matrix(3, 3); - public Matrix depthLensDistortion = new Matrix(5, 1); - public Matrix depthToColorTransform = new Matrix(4, 4); + public Matrix colorCameraMatrix = Matrix.Build.Dense(3, 3); + public Vector colorLensDistortion = Vector.Build.Dense(5); + public Matrix depthCameraMatrix = Matrix.Build.Dense(3, 3); + public Vector depthLensDistortion = Vector.Build.Dense(5); + public Matrix depthToColorTransform = Matrix.Build.Dense(4, 4); [XmlIgnoreAttribute] public bool silent = true; - /// - /// Converts a camera space point (3D) to the corresponding color space point (2D-RGB camera) - /// - /// The color space point to convert - /// The corresponding camera space point - public ColorSpacePoint ToColorSpacePoint(CameraSpacePoint cameraSpacePoint) - { - var colorPoint = new Matrix(4, 1); - var depthPoint = new Matrix(4, 1); - depthPoint[0] = (double)cameraSpacePoint.X; - depthPoint[1] = (double)cameraSpacePoint.Y; - depthPoint[2] = (double)cameraSpacePoint.Z; - depthPoint[3] = 1; - - colorPoint.Mult(this.depthToColorTransform, depthPoint); - - double colorX, colorY; - Project(this.colorCameraMatrix, this.colorLensDistortion, colorPoint[0], colorPoint[1], colorPoint[2], out colorX, out colorY); - ColorSpacePoint colorSpacePoint; - colorSpacePoint.X = (float)colorX; - colorSpacePoint.Y = (float)(colorImageHeight - colorY); - return colorSpacePoint; - } - - public void ToCameraSpacePoint(ColorSpacePoint colorSpacePoint, out CameraSpacePoint cameraOrigin3D, out CameraSpacePoint point3D) - { - Matrix c2d = new Matrix(); - c2d.Inverse(depthToColorTransform); - - var cameraOrigin = new Matrix(4, 1); - cameraOrigin[0] = 0; - cameraOrigin[1] = 0; - cameraOrigin[2] = 0; - cameraOrigin[3] = 1; - var cameraOriginIn3DSpace = new Matrix(4, 1); - cameraOriginIn3DSpace.Mult(c2d, cameraOrigin); - cameraOrigin3D.X = (float)cameraOriginIn3DSpace[0]; - cameraOrigin3D.Y = (float)cameraOriginIn3DSpace[1]; - cameraOrigin3D.Z = (float)cameraOriginIn3DSpace[2]; - - var pointInImage = new Matrix(4, 1); - double undistx; - double undisty; - Undistort(colorCameraMatrix, colorLensDistortion, colorSpacePoint.X, (colorImageHeight - colorSpacePoint.Y), out undistx, out undisty); - pointInImage[0] = undistx; - pointInImage[1] = undisty; - pointInImage[2] = 1; - pointInImage[3] = 1; - - var pointIn3DSpace = new Matrix(4, 1); - pointIn3DSpace.Mult(c2d, pointInImage); - point3D.X = (float)pointIn3DSpace[0]; - point3D.Y = (float)pointIn3DSpace[1]; - point3D.Z = (float)pointIn3DSpace[2]; - } - - - /// - /// Converts a cameras space point to a depth space point - /// - /// The color space point to convert - /// The corresponding camera space point - public DepthSpacePoint ToDepthSpacePoint(CameraSpacePoint cameraSpacePoint) - { - double depthX, depthY; - Project(this.depthCameraMatrix, this.depthLensDistortion, cameraSpacePoint.X, cameraSpacePoint.Y, cameraSpacePoint.Z, out depthX, out depthY); - DepthSpacePoint depthSpacePoint; - depthSpacePoint.X = (float)depthX; - depthSpacePoint.Y = (float)(depthImageHeight - depthY); - - return depthSpacePoint; - } - - public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSensor) + internal void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSensor) { var stopWatch = new System.Diagnostics.Stopwatch(); stopWatch.Start(); - var objectPoints1 = new List(); + var objectPoints1 = new List>(); var colorPoints1 = new List(); var depthPoints1 = new List(); @@ -119,7 +48,7 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen kinectCameraPoint.Z = z; // use SDK's projection - // adjust Y to make RH cooridnate system that is a projection of Kinect 3D points + // adjust Y to make RH coordinate system that is a projection of Kinect 3D points var kinectColorPoint = kinectSensor.CoordinateMapper.MapCameraPointToColorSpace(kinectCameraPoint); kinectColorPoint.Y = colorImageHeight - kinectColorPoint.Y; var kinectDepthPoint = kinectSensor.CoordinateMapper.MapCameraPointToDepthSpace(kinectCameraPoint); @@ -131,7 +60,7 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen (kinectDepthPoint.Y >= 0) && (kinectDepthPoint.Y < depthImageHeight)) { n++; - var objectPoint = new Matrix(3, 1); + var objectPoint = Vector.Build.Dense(3); objectPoint[0] = kinectCameraPoint.X; objectPoint[1] = kinectCameraPoint.Y; objectPoint[2] = kinectCameraPoint.Z; @@ -152,39 +81,38 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen } } - colorCameraMatrix[0, 0] = 1000; //fx - colorCameraMatrix[1, 1] = 1000; //fy - colorCameraMatrix[0, 2] = colorImageWidth / 2; //cx - colorCameraMatrix[1, 2] = colorImageHeight / 2; //cy - colorCameraMatrix[2, 2] = 1; + this.colorCameraMatrix[0, 0] = 1000; //fx + this.colorCameraMatrix[1, 1] = 1000; //fy + this.colorCameraMatrix[0, 2] = colorImageWidth / 2; //cx + this.colorCameraMatrix[1, 2] = colorImageHeight / 2; //cy + this.colorCameraMatrix[2, 2] = 1; - var rotation = new Matrix(3, 1); - var translation = new Matrix(3, 1); - var colorError = CalibrateColorCamera(objectPoints1, colorPoints1, colorCameraMatrix, colorLensDistortion, rotation, translation, silent); - var rotationMatrix = Orientation.Rodrigues(rotation); + var rotation = Vector.Build.Dense(3); + var translation = Vector.Build.Dense(3); + var colorError = CalibrateColorCamera(objectPoints1, colorPoints1, colorCameraMatrix, colorLensDistortion, rotation, translation, this.silent); + var rotationMatrix = RotationExtensions.AxisAngleToMatrix(rotation); - depthToColorTransform = Matrix.Identity(4, 4); + this.depthToColorTransform = Matrix.Build.DenseIdentity(4, 4); for (int i = 0; i < 3; i++) { - depthToColorTransform[i, 3] = translation[i]; + this.depthToColorTransform[i, 3] = translation[i]; for (int j = 0; j < 3; j++) - depthToColorTransform[i, j] = rotationMatrix[i, j]; + this.depthToColorTransform[i, j] = rotationMatrix[i, j]; } - depthCameraMatrix[0, 0] = 360; //fx - depthCameraMatrix[1, 1] = 360; //fy - depthCameraMatrix[0, 2] = depthImageWidth / 2; //cx - depthCameraMatrix[1, 2] = depthImageHeight / 2; //cy - depthCameraMatrix[2, 2] = 1; + this.depthCameraMatrix[0, 0] = 360; //fx + this.depthCameraMatrix[1, 1] = 360; //fy + this.depthCameraMatrix[0, 2] = depthImageWidth / 2.0; //cx + this.depthCameraMatrix[1, 2] = depthImageHeight / 2.0; //cy + this.depthCameraMatrix[2, 2] = 1; var depthError = CalibrateDepthCamera(objectPoints1, depthPoints1, depthCameraMatrix, depthLensDistortion, silent); // check projections double depthProjectionError = 0; double colorProjectionError = 0; - var color = new Matrix(4, 1); - var testObjectPoint4 = new Matrix(4, 1); + var testObjectPoint4 = Vector.Build.Dense(4); for (int i = 0; i < n; i++) { var testObjectPoint = objectPoints1[i]; @@ -206,8 +134,8 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen testObjectPoint4[2] = testObjectPoint[2]; testObjectPoint4[3] = 1; - color.Mult(depthToColorTransform, testObjectPoint4); - color.Scale(1.0 / color[3]); // not necessary for this transform + var color = depthToColorTransform * testObjectPoint4; + color *= (1.0 / color[3]); // not necessary for this transform double colorU, colorV; Project(colorCameraMatrix, colorLensDistortion, color[0], color[1], color[2], out colorU, out colorV); @@ -221,7 +149,7 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen stopWatch.Stop(); - if (!silent) + if (!this.silent) { Console.WriteLine("FakeCalibration :"); Console.WriteLine("n = " + n); @@ -239,7 +167,7 @@ public void RecoverCalibrationFromSensor(Microsoft.Kinect.KinectSensor kinectSen } } - public static void Project(Matrix cameraMatrix, Matrix distCoeffs, double x, double y, double z, out double u, out double v) + private static void Project(Matrix cameraMatrix, Vector distCoeffs, double x, double y, double z, out double u, out double v) { double xp = x / z; double yp = y / z; @@ -259,7 +187,7 @@ public static void Project(Matrix cameraMatrix, Matrix distCoeffs, double x, dou v = fy * ypp + cy; } - public static void Undistort(Matrix cameraMatrix, Matrix distCoeffs, double xin, double yin, out double xout, out double yout) + private static void Undistort(Matrix cameraMatrix, Vector distCoeffs, double xin, double yin, out double xout, out double yout) { float fx = (float)cameraMatrix[0, 0]; float fy = (float)cameraMatrix[1, 1]; @@ -269,7 +197,7 @@ public static void Undistort(Matrix cameraMatrix, Matrix distCoeffs, double xin, Undistort(fx, fy, cx, cy, kappa, xin, yin, out xout, out yout); } - public static void Undistort(float fx, float fy, float cx, float cy, float[] kappa, double xin, double yin, out double xout, out double yout) + private static void Undistort(float fx, float fy, float cx, float cy, float[] kappa, double xin, double yin, out double xout, out double yout) { // maps coords in undistorted image (xin, yin) to coords in distorted image (xout, yout) double x = (xin - cx) / fx; @@ -303,38 +231,14 @@ public static void Undistort(float fx, float fy, float cx, float cy, float[] kap yout = y / factor; } - public Microsoft.Kinect.PointF[] ComputeDepthFrameToCameraSpaceTable() - { - float fx = (float)depthCameraMatrix[0, 0]; - float fy = (float)depthCameraMatrix[1, 1]; - float cx = (float)depthCameraMatrix[0, 2]; - float cy = (float)depthCameraMatrix[1, 2]; - float[] kappa = new float[] { (float)depthLensDistortion[0], (float)depthLensDistortion[1] }; - - var table = new Microsoft.Kinect.PointF[depthImageWidth * depthImageHeight]; - - for (int framey = 0; framey < depthImageHeight; framey++) - for (int framex = 0; framex < depthImageWidth; framex++) - { - double xout, yout; - Undistort(fx, fy, cx, cy, kappa, framex, (depthImageHeight - framey), out xout, out yout); - - var point = new Microsoft.Kinect.PointF(); - point.X = (float)xout; - point.Y = (float)yout; - table[depthImageWidth * framey + framex] = point; - } - return table; - } - - static double CalibrateDepthCamera(List worldPoints, List imagePoints, Matrix cameraMatrix, Matrix distCoeffs, bool silent) + private static double CalibrateDepthCamera(List> worldPoints, List imagePoints, Matrix cameraMatrix, Vector distCoeffs, bool silent = true) { int nPoints = worldPoints.Count; // pack parameters into vector // parameters: fx, fy, cx, cy, k1, k2 = 6 parameters int nParameters = 6; - var parameters = new Matrix(nParameters, 1); + var parameters = Vector.Build.Dense(nParameters); { int pi = 0; @@ -349,9 +253,9 @@ static double CalibrateDepthCamera(List worldPoints, List p) { - var fvec = new Matrix(nValues, 1); + var fvec = Vector.Build.Dense(nValues); // unpack parameters int pi = 0; @@ -362,13 +266,13 @@ static double CalibrateDepthCamera(List worldPoints, List.Build.DenseIdentity(3, 3); K[0, 0] = fx; K[1, 1] = fy; K[0, 2] = cx; K[1, 2] = cy; - var d = Matrix.Zero(5, 1); + var d = Vector.Build.Dense(5, 0); d[0] = k1; d[1] = k2; @@ -420,23 +324,23 @@ static double CalibrateDepthCamera(List worldPoints, List worldPoints, List imagePoints, Matrix cameraMatrix, Matrix distCoeffs, Matrix rotation, Matrix translation, bool silent) + private static double CalibrateColorCamera(List> worldPoints, List imagePoints, Matrix cameraMatrix, Vector distCoeffs, Vector rotation, Vector translation, bool silent = true) { int nPoints = worldPoints.Count; { - Matrix R, t; + Matrix R; + Vector t; DLT(cameraMatrix, distCoeffs, worldPoints, imagePoints, out R, out t); - var r = Orientation.RotationVector(R); - rotation.Copy(r); - translation.Copy(t); + var r = RotationExtensions.MatrixToAxisAngle(R); + r.CopyTo(rotation); + t.CopyTo(translation); } // pack parameters into vector // parameters: fx, fy, cx, cy, k1, k2, + 3 for rotation, 3 translation = 12 int nParameters = 12; - var parameters = new Matrix(nParameters, 1); - + var parameters = Vector.Build.Dense(nParameters); { int pi = 0; parameters[pi++] = cameraMatrix[0, 0]; // fx @@ -457,9 +361,9 @@ static double CalibrateColorCamera(List worldPoints, List p) { - var fvec = new Matrix(nValues, 1); + var fvec = Vector.Build.Dense(nValues); // unpack parameters int pi = 0; @@ -471,38 +375,34 @@ static double CalibrateColorCamera(List worldPoints, List.Build.DenseIdentity(3, 3); K[0, 0] = fx; K[1, 1] = fy; K[0, 2] = cx; K[1, 2] = cy; - var d = Matrix.Zero(5, 1); + var d = Vector.Build.Dense(5, 0); d[0] = k1; d[1] = k2; - var r = new Matrix(3, 1); + var r = Vector.Build.Dense(3); r[0] = p[pi++]; r[1] = p[pi++]; r[2] = p[pi++]; - var t = new Matrix(3, 1); + var t = Vector.Build.Dense(3); t[0] = p[pi++]; t[1] = p[pi++]; t[2] = p[pi++]; - var R = Orientation.Rodrigues(r); - - - - var x = new Matrix(3, 1); + var R = RotationExtensions.AxisAngleToMatrix(r); int fveci = 0; for (int i = 0; i < worldPoints.Count; i++) { // transform world point to local camera coordinates - x.Mult(R, worldPoints[i]); - x.Add(t); + var x = R * worldPoints[i]; + x += t; // fvec_i = y_i - f(x_i) double u, v; @@ -559,11 +459,11 @@ static double CalibrateColorCamera(List worldPoints, List worldPoints, List imagePoints, out Matrix R, out Matrix t) + private static void DLT(Matrix cameraMatrix, VectordistCoeffs, List> worldPoints, List imagePoints, out Matrix R, out Vectort) { int n = worldPoints.Count; - var A = Matrix.Zero(2 * n, 12); + var A = Matrix.Build.Dense(2 * n, 12); for (int j = 0; j < n; j++) { @@ -597,54 +497,52 @@ static void DLT(Matrix cameraMatrix, Matrix distCoeffs, List worldPoints } // Pcolumn is the eigenvector of ATA with the smallest eignvalue - var Pcolumn = new Matrix(12, 1); + var Pcolumn = Vector.Build.Dense(12); { - var ATA = new Matrix(12, 12); - ATA.MultATA(A, A); - - var V = new Matrix(12, 12); - var ww = new Matrix(12, 1); - ATA.Eig(V, ww); - - Pcolumn.CopyCol(V, 0); + var ATA = A.TransposeThisAndMultiply(A); + ATA.Evd().EigenVectors.Column(0).CopyTo(Pcolumn); } // reshape into 3x4 projection matrix - var P = new Matrix(3, 4); - P.Reshape(Pcolumn); + var P = Matrix.Build.Dense(3, 4); + { + for (int i = 0; i < 3; i++) + { + for (int j = 0; j < 4; j++) + { + P[i, j] = Pcolumn[i*4 + j]; + } + } + } - R = new Matrix(3, 3); + R = Matrix.Build.Dense(3, 3); for (int i = 0; i < 3; i++) for (int j = 0; j < 3; j++) R[i, j] = P[i, j]; - if (R.Det3x3() < 0) + if (R.Determinant() < 0) { - R.Scale(-1); - P.Scale(-1); + R *= -1; + P *= -1; } // orthogonalize R { - var U = new Matrix(3, 3); - var V = new Matrix(3, 3); - var ww = new Matrix(3, 1); - R.SVD(U, ww, V); - R.MultAAT(U, V); + var svd = R.Svd(); + R = svd.U * svd.VT; } // determine scale factor - var RP = new Matrix(3, 3); + var RP = Matrix.Build.Dense(3, 3); for (int i = 0; i < 3; i++) for (int j = 0; j < 3; j++) RP[i, j] = P[i, j]; - double s = RP.Norm() / R.Norm(); + double s = RP.L2Norm() / R.L2Norm(); - t = new Matrix(3, 1); + t = Vector.Build.Dense(3); for (int i = 0; i < 3; i++) t[i] = P[i, 3]; - t.Scale(1.0 / s); + t *= (1.0 / s); } - } } diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectSensor.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectSensor.cs index a67973d24..cebf2bc5d 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectSensor.cs +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/KinectSensor.cs @@ -6,7 +6,6 @@ namespace Microsoft.Psi.Kinect using System; using System.Collections.Generic; using System.Linq; - using MathNet.Numerics.LinearAlgebra; using MathNet.Spatial.Euclidean; using Microsoft.Kinect; using Microsoft.Psi; @@ -19,7 +18,7 @@ namespace Microsoft.Psi.Kinect /// /// Component that captures and streams information (images, depth, audio, bodies, etc.) from a Kinect One (v2) sensor. /// - public class KinectSensor : IKinectSensor, ISourceComponent, IDisposable + public class KinectSensor : ISourceComponent, IDisposable { private static List allDevices = null; private static WaveFormat audioFormat = WaveFormat.Create16kHz1ChannelIeeeFloat(); @@ -36,7 +35,6 @@ public class KinectSensor : IKinectSensor, ISourceComponent, IDisposable private IList bodies = null; private List kinectBodies = null; - private int trackedBodies = 0; private byte[] audioBuffer = null; private IList bodyTrackingIds = null; private bool disposed = false; @@ -74,7 +72,7 @@ private KinectSensor(Pipeline pipeline) this.Bodies = pipeline.CreateEmitter>(this, nameof(this.Bodies)); this.ColorImage = pipeline.CreateEmitter>(this, nameof(this.ColorImage)); this.RGBDImage = pipeline.CreateEmitter>(this, nameof(this.RGBDImage)); - this.DepthImage = pipeline.CreateEmitter>(this, nameof(this.DepthImage)); + this.DepthImage = pipeline.CreateEmitter>(this, nameof(this.DepthImage)); this.InfraredImage = pipeline.CreateEmitter>(this, nameof(this.InfraredImage)); this.LongExposureInfraredImage = pipeline.CreateEmitter>(this, nameof(this.LongExposureInfraredImage)); this.DepthDeviceCalibrationInfo = pipeline.CreateEmitter(this, nameof(this.DepthDeviceCalibrationInfo)); @@ -101,6 +99,7 @@ public static IEnumerable AllDevices di.FriendlyName = $"Kinect-v2"; var kinectSensor = Microsoft.Kinect.KinectSensor.GetDefault(); di.DeviceName = kinectSensor.UniqueKinectId; + di.DeviceId = 0; kinectSensor?.Close(); di.SerialNumber = string.Empty; di.Sensors = new List(); @@ -174,9 +173,9 @@ public static IEnumerable AllDevices public Emitter> RGBDImage { get; private set; } /// - /// Gets the current image from the depth camera. + /// Gets the current depth image from the depth camera. /// - public Emitter> DepthImage { get; private set; } + public Emitter> DepthImage { get; private set; } /// /// Gets the current image from the infrared camera. @@ -234,8 +233,9 @@ public void Dispose() { if (!this.disposed) { - this.multiFrameReader?.Dispose(); - this.audioBeamFrameReader?.Dispose(); + // Cast to IDisposable to suppress false CA2213 warning + ((IDisposable)this.multiFrameReader)?.Dispose(); + ((IDisposable)this.audioBeamFrameReader)?.Dispose(); this.kinectSensor?.Close(); this.disposed = true; } @@ -324,34 +324,6 @@ private void CoordinateMapper_CoordinateMappingChanged(object sender, Coordinate var kinectInternalCalibration = new KinectInternalCalibration(); kinectInternalCalibration.RecoverCalibrationFromSensor(this.kinectSensor); - Matrix colorCameraMatrix = Matrix.Build.Dense(3, 3); - Matrix depthCameraMatrix = Matrix.Build.Dense(3, 3); - for (int i = 0; i < 3; i++) - { - for (int j = 0; j < 3; j++) - { - colorCameraMatrix[i, j] = kinectInternalCalibration.colorCameraMatrix[i, j]; - depthCameraMatrix[i, j] = kinectInternalCalibration.depthCameraMatrix[i, j]; - } - } - - Vector colorLensDistortion = Vector.Build.Dense(5); - Vector depthLensDistortion = Vector.Build.Dense(5); - for (int i = 0; i < 5; i++) - { - colorLensDistortion[i] = kinectInternalCalibration.colorLensDistortion[i]; - depthLensDistortion[i] = kinectInternalCalibration.depthLensDistortion[i]; - } - - Matrix depthToColorTransform = Matrix.Build.Dense(4, 4); - for (int i = 0; i < 4; i++) - { - for (int j = 0; j < 4; j++) - { - depthToColorTransform[i, j] = kinectInternalCalibration.depthToColorTransform[i, j]; - } - } - // Kinect uses a basis under the hood that assumes Forward=Z, Left=X, Up=Y. var kinectBasis = new CoordinateSystem(default, UnitVector3D.ZAxis, UnitVector3D.XAxis, UnitVector3D.YAxis); @@ -363,13 +335,13 @@ private void CoordinateMapper_CoordinateMappingChanged(object sender, Coordinate this.depthDeviceCalibrationInfo = new DepthDeviceCalibrationInfo( this.kinectSensor.ColorFrameSource.FrameDescription.Width, this.kinectSensor.ColorFrameSource.FrameDescription.Height, - colorCameraMatrix, + kinectInternalCalibration.colorCameraMatrix, colorRadialDistortion, colorTangentialDistortion, - kinectBasis.Invert() * depthToColorTransform * kinectBasis, // Convert to MathNet + kinectBasis.Invert() * kinectInternalCalibration.depthToColorTransform * kinectBasis, // Convert to MathNet basis this.kinectSensor.DepthFrameSource.FrameDescription.Width, this.kinectSensor.DepthFrameSource.FrameDescription.Height, - depthCameraMatrix, + kinectInternalCalibration.depthCameraMatrix, depthRadialDistortion, depthTangentialDistortion, CoordinateSystem.CreateIdentity(4)); @@ -437,7 +409,7 @@ private void DepthFrameReader_FrameArrived(DepthFrame depthFrame) FrameDescription depthFrameDescription = depthFrame.FrameDescription; using (KinectBuffer depthBuffer = depthFrame.LockImageBuffer()) { - using (var dest = ImagePool.GetOrCreate(depthFrameDescription.Width, depthFrameDescription.Height, Imaging.PixelFormat.Gray_16bpp)) + using (var dest = DepthImagePool.GetOrCreate(depthFrameDescription.Width, depthFrameDescription.Height)) { depthFrame.CopyFrameDataToIntPtr(dest.Resource.ImageData, (uint)(depthFrameDescription.Width * depthFrameDescription.Height * 2)); var time = this.pipeline.GetCurrentTimeFromElapsedTicks(depthFrame.RelativeTime.Ticks); @@ -564,12 +536,12 @@ private void BodyFrameReader_FrameArrived(BodyFrameReference bodyFrameReference) bodyFrame.GetAndRefreshBodyData(this.bodies); // compute the number of tracked bodies - this.trackedBodies = this.bodies.Count(b => b.IsTracked); + var numTrackedBodies = this.bodies.Count(b => b.IsTracked); - if (this.kinectBodies == null || this.kinectBodies.Count != this.trackedBodies) + if (this.kinectBodies == null || this.kinectBodies.Count != numTrackedBodies) { - this.kinectBodies = new List(this.trackedBodies); - for (int i = 0; i < this.trackedBodies; i++) + this.kinectBodies = new List(numTrackedBodies); + for (int i = 0; i < numTrackedBodies; i++) { this.kinectBodies.Add(new KinectBody()); } @@ -581,20 +553,8 @@ private void BodyFrameReader_FrameArrived(BodyFrameReference bodyFrameReference) { if (this.bodies[i].IsTracked) { + this.kinectBodies[ti].UpdateFrom(this.bodies[i]); this.kinectBodies[ti].FloorClipPlane = bodyFrame.FloorClipPlane; - this.kinectBodies[ti].ClippedEdges = this.bodies[i].ClippedEdges; - this.kinectBodies[ti].HandLeftConfidence = this.bodies[i].HandLeftConfidence; - this.kinectBodies[ti].HandLeftState = this.bodies[i].HandLeftState; - this.kinectBodies[ti].HandRightConfidence = this.bodies[i].HandRightConfidence; - this.kinectBodies[ti].HandRightState = this.bodies[i].HandRightState; - this.kinectBodies[ti].IsRestricted = this.bodies[i].IsRestricted; - this.kinectBodies[ti].IsRestricted = this.bodies[i].IsRestricted; - this.kinectBodies[ti].IsTracked = this.bodies[i].IsTracked; - this.kinectBodies[ti].JointOrientations = this.CloneDictionary(this.bodies[i].JointOrientations); - this.kinectBodies[ti].Joints = this.CloneDictionary(this.bodies[i].Joints); - this.kinectBodies[ti].Lean = this.bodies[i].Lean; - this.kinectBodies[ti].LeanTrackingState = this.bodies[i].LeanTrackingState; - this.kinectBodies[ti].TrackingId = this.bodies[i].TrackingId; ti++; } } diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Matrix.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Matrix.cs deleted file mode 100644 index 30ba78f04..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Matrix.cs +++ /dev/null @@ -1,1420 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System; - using System.Threading.Tasks; - using MathNet.Numerics.LinearAlgebra; - -#pragma warning disable SA1600 - - /// - /// Defines a Matrix class. - /// - internal class Matrix - { - private static double z0; - private static double z1; - private static bool generate = false; - private static Random random = new Random(); - - private int m; - private int n; - private int mn; - private double[] data; - - /// - /// Initializes a new instance of the class. - /// - public Matrix() - { - } - - /// - /// Initializes a new instance of the class. - /// - /// Number of rows in matrix. - /// Number of columns in matrix. - public Matrix(int m, int n) - { - this.m = m; - this.n = n; - this.mn = this.m * this.n; - this.data = new double[this.mn]; - } - - /// - /// Initializes a new instance of the class. - /// - /// Matrix to copy from. - public Matrix(Matrix mat) - { - this.m = mat.m; - this.n = mat.n; - this.mn = this.m * this.n; - this.data = new double[this.mn]; - this.Copy(mat); - } - - /// - /// Gets or sets a column of values in the matrix. - /// - public double[][] ValuesByColumn - { - get - { - double[][] mat = new double[this.n][]; - - for (int j = 0; j < this.n; j++) - { - mat[j] = new double[this.m]; - } - - for (int i = 0; i < this.m; i++) - { - for (int j = 0; j < this.n; j++) - { - mat[j][i] = this[i, j]; - } - } - - return mat; - } - - set - { - double[][] mat = value; - this.n = mat.Length; - this.m = mat[0].Length; - this.mn = this.m * this.n; - this.data = new double[this.mn]; - for (int i = 0; i < this.m; i++) - { - for (int j = 0; j < this.n; j++) - { - this[i, j] = mat[j][i]; - } - } - } - } - - /// - /// Gets number of rows in the matrix. - /// - public int Rows - { - get { return this.m; } - } - - /// - /// Gets number of columns in the matrix. - /// - public int Cols - { - get { return this.n; } - } - - /// - /// Gets the total number of elements in the matrix. - /// - public int Size - { - get { return this.mn; } - } - - /// - /// Indexer into the matrix. - /// - /// Row to access. - /// Column to access. - /// The entry at specified row/column. - public double this[int i, int j] - { - get { return this.data[(i * this.n) + j]; } - set { this.data[(i * this.n) + j] = value; } - } - - /// - /// Indexer that treats the matrix as a flat array. - /// - /// Index to access. - /// Value at Ith element in the matrix. - public double this[int i] - { - get { return this.data[i]; } - set { this.data[i] = value; } - } - - /// - /// Returns an identity matrix. - /// - /// Number of rows in matrix. - /// Number of columns in matrix. - /// New identity matrix of size MxN. - public static Matrix Identity(int m, int n) - { - var mat = new Matrix(m, n); - mat.Identity(); - return mat; - } - - /// - /// Returns an zero matrix. - /// - /// Number of rows in matrix. - /// Number of columns in matrix. - /// New zero matrix of size MxN. - public static Matrix Zero(int m, int n) - { - var mat = new Matrix(m, n); - mat.Zero(); - return mat; - } - - /// - /// Copies a submatrix from matA into matB. - /// - /// Matrix to copy from. - /// Row offset to start copying from. - /// Column offset to start copying from. - /// Number of rows to copy. - /// Number of columns to copy. - /// Matrix to copy to. - /// Row offset to copy to. - /// Column offset to copy to. - public static void CopyRange(Matrix matA, int ai, int aj, int m, int n, Matrix matB, int bi, int bj) - { - for (int i = 0; i < m; i++) - { - for (int j = 0; j < n; j++) - { - matB[bi + i, bj + j] = matA[ai + i, aj + j]; - } - } - } - - /// - /// Copies a single row from matA to matB. - /// - /// Matrix to copy from. - /// Row to copy. - /// Matrix to copy to. - public static void CopyRow(Matrix matA, int row, Matrix matB) - { - for (int j = 0; j < matA.n; j++) - { - matB.data[j] = matA[row, j]; - } - } - - /// - /// Copies a single column from matA to matB. - /// - /// Matrix to copy from. - /// Column to copy. - /// Matrix to copy to. - public static void CopyCol(Matrix matA, int col, Matrix matB) - { - for (int i = 0; i < matA.m; i++) - { - matB.data[i] = matA[i, col]; - } - } - - /// - /// Copies the diagonal from matA to matB. - /// - /// Matrix to copy from. - /// Matrix to copy to. - public static void CopyDiag(Matrix matA, Matrix matB) - { - int maxd = (matA.m > matA.n) ? matA.m : matA.n; - for (int i = 0; i < maxd; i++) - { - matB.data[i] = matA[i, i]; - } - } - - // equals - public static bool Equals(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - if (matA.data[i] != matB.data[i]) - { - return false; - } - } - - return true; - } - - public static void Reshape(Matrix matA, Matrix matB) - { - int k = 0; - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < matA.n; j++) - { - matB.data[k++] = matA[i, j]; - } - } - } - - // change shape - public static void Transpose(Matrix matA, Matrix matB) - { - if (matA != matB) - { - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < matA.n; j++) - { - matB[j, i] = matA[i, j]; - } - } - } - else - { // must be square - double s; - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < i; j++) - { - s = matA[i, j]; - matA[i, j] = matA[j, i]; - matA[j, i] = s; - } - } - } - } - - // matrix-scalar ops - public static void Identity(Matrix matA) - { - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < matA.n; j++) - { - if (i == j) - { - matA[i, j] = 1.0; - } - else - { - matA[i, j] = 0.0; - } - } - } - } - - public static void Set(Matrix matA, double c) - { - for (int i = 0; i < matA.mn; i++) - { - matA.data[i] = c; - } - } - - public static void Pow(Matrix matA, double c, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = Math.Pow(matA.data[i], c); - } - } - - public static void Exp(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = Math.Exp(matA.data[i]); - } - } - - public static void Log(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = Math.Log(matA.data[i]); - } - } - - public static void Abs(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = Math.Abs(matA.data[i]); - } - } - - public static void Add(Matrix matA, double c, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = c + matA.data[i]; - } - } - - public static void Scale(Matrix matA, double c, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = c * matA.data[i]; - } - } - - public static void ScaleAdd(Matrix matA, double c, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] += c * matA.data[i]; - } - } - - public static void Reciprocal(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.mn; i++) - { - matB.data[i] = 1.0 / matA.data[i]; - } - } - - public static void Bound(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.mn; i++) - { - if (matC.data[i] < matA.data[i]) - { - matC.data[i] = matA.data[i]; - } - - if (matC.data[i] > matB.data[i]) - { - matC.data[i] = matB.data[i]; - } - } - } - - public static void Add(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.Size; i++) - { - matC.data[i] = matA.data[i] + matB.data[i]; - } - } - - public static void Sub(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.mn; i++) - { - matC.data[i] = matA.data[i] - matB.data[i]; - } - } - - public static void ElemMult(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.mn; i++) - { - matC.data[i] = matA.data[i] * matB.data[i]; - } - } - - public static void Divide(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.mn; i++) - { - matC.data[i] = matA.data[i] / matB.data[i]; - } - } - - public static double Dot(Matrix matA, Matrix matB) - { - double sum = 0.0; - for (int i = 0; i < matA.mn; i++) - { - sum += matA.data[i] * matB.data[i]; - } - - return sum; - } - - public static void Outer(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matC.m; i++) - { - for (int j = 0; j < matC.n; j++) - { - matC[i, j] = matA.data[i] * matB.data[j]; - } - } - } - - public static void Cross(Matrix matA, Matrix matB, Matrix matC) - { - matC.data[0] = (matA.data[1] * matB.data[2]) - (matA.data[2] * matB.data[1]); - matC.data[1] = (matA.data[2] * matB.data[0]) - (matA.data[0] * matB.data[2]); - matC.data[2] = (matA.data[0] * matB.data[1]) - (matA.data[1] * matB.data[0]); - } - - public static void Mult(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < matB.n; j++) - { - double sum = 0; - for (int k = 0; k < matA.n; k++) - { - sum += matA[i, k] * matB[k, j]; - } - - matC[i, j] = sum; - } - } - } - - public static void MultATA(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.n; i++) - { // matA.m - for (int j = 0; j < matB.n; j++) - { - double sum = 0; - for (int k = 0; k < matA.m; k++) - { // matA.n - sum += matA[k, i] * matB[k, j]; - } - - matC[i, j] = sum; - } - } - } - - public static void MultATAParallel(Matrix matA, Matrix matB, Matrix matC) - { - Parallel.For(0, matA.n, i => - { - for (int j = 0; j < matB.n; j++) - { - double sum = 0; - for (int k = 0; k < matA.m; k++) - { // matA.n - sum += matA[k, i] * matB[k, j]; - } - - matC[i, j] = sum; - } - }); - } - - public static Matrix ToMathNet(Matrix matA) - { - var matB = Matrix.Build.Dense(matA.Rows, matA.Cols); - for (int i = 0; i < matA.Rows; i++) - { - for (int j = 0; j < matA.Cols; j++) - { - matB[i, j] = matA[i, j]; - } - } - - return matB; - } - - public static void FromMathNet(Matrix matA, Matrix matB) - { - for (int i = 0; i < matA.RowCount; i++) - { - for (int j = 0; j < matA.ColumnCount; j++) - { - matB[i, j] = matA[i, j]; - } - } - } - - public static void FromMathNet(Vector matA, Matrix matB) - { - for (int i = 0; i < matA.Count; i++) - { - matB[i] = matA[i]; - } - } - - public static double Det3x3(Matrix matA) - { - double a = matA[0, 0]; - double b = matA[0, 1]; - double c = matA[0, 2]; - double d = matA[1, 0]; - double e = matA[1, 1]; - double f = matA[1, 2]; - double g = matA[2, 0]; - double h = matA[2, 1]; - double i = matA[2, 2]; - - return ((a * e * i) + (b * f * g) + (c * d * h)) - ((c * e * g) + (b * d * i) + (a * f * h)); - } - - public static void LeastSquares(Matrix x, Matrix matA, Matrix matB) - { - // use svd - // for overdetermined systems A*x = b - // x = V * diag(1/wj) * U T * b - // NRC p. 66 - int m = matA.m; - int n = matA.n; - - Matrix matU = new Matrix(m, n); - Matrix matV = new Matrix(n, n); - Matrix w = new Matrix(n, 1); - Matrix matW = new Matrix(n, n); - matA.SVD(matU, w, matV); - w.Reciprocal(); - matW.Diag(w); - - Matrix matM = new Matrix(n, n); - matM.Mult(matV, matW); - - Matrix matN = new Matrix(n, m); - matN.MultAAT(matM, matU); - - x.Mult(matN, matB); - } - - public static void Rot2D(Matrix matA, double theta) - { - // clockwise rotation - double s = Math.Sin(theta); - double c = Math.Cos(theta); - matA[0, 0] = c; - matA[1, 0] = s; - matA[0, 1] = -s; - matA[1, 1] = c; - } - - public static void RotEuler2Matrix(double x, double y, double z, Matrix matA) - { - double s1 = Math.Sin(x); - double s2 = Math.Sin(y); - double s3 = Math.Sin(z); - double c1 = Math.Cos(x); - double c2 = Math.Cos(y); - double c3 = Math.Cos(z); - - matA[0, 0] = c3 * c2; - matA[0, 1] = (-s3 * c1) + (c3 * s2 * s1); - matA[0, 2] = (s3 * s1) + (c3 * s2 * c1); - matA[1, 0] = s3 * c2; - matA[1, 1] = (c3 * c1) + (s3 * s2 * s1); - matA[1, 2] = (-c3 * s1) + (s3 * s2 * c1); - matA[2, 0] = -s2; - matA[2, 1] = c2 * s1; - matA[2, 2] = c2 * c1; - } - - public static void RotFromTo2Quat(Matrix x, Matrix y, Matrix q) - { - Matrix axis = new Matrix(3, 1); - axis.Cross(y, x); - axis.Normalize(); - - double angle = Math.Acos(x.Dot(y)); - double s = Math.Sin(angle / 2.0); - - q[0] = axis[0] * s; - q[1] = axis[1] * s; - q[2] = axis[2] * s; - q[3] = Math.Cos(angle / 2.0); - } - - public static void RotQuat2Matrix(Matrix q, Matrix matA) - { - double x = q[0]; - double y = q[1]; - double z = q[2]; - double w = q[3]; - - // Watt and Watt p. 363 - double s = 2.0 / Math.Sqrt((x * x) + (y * y) + (z * z) + (w * w)); - - double xs = x * s; - double ys = y * s; - double zs = z * s; - double wx = w * xs; - double wy = w * ys; - double wz = w * zs; - double xx = x * xs; - double xy = x * ys; - double xz = x * zs; - double yy = y * ys; - double yz = y * zs; - double zz = z * zs; - - matA[0, 0] = 1 - (yy + zz); - matA[0, 1] = xy + wz; - matA[0, 2] = xz - wy; - - matA[1, 0] = xy - wz; - matA[1, 1] = 1 - (xx + zz); - matA[1, 2] = yz + wx; - - matA[2, 0] = xz + wy; - matA[2, 1] = yz - wx; - matA[2, 2] = 1 - (xx + yy); - } - - public static void RotAxisAngle2Quat(Matrix axis, double angle, Matrix q) - { - q[0] = axis[0] * Math.Sin(angle / 2.0); - q[1] = axis[1] * Math.Sin(angle / 2.0); - q[2] = axis[2] * Math.Sin(angle / 2.0); - q[3] = Math.Cos(angle / 2.0); - } - - public static void RotMatrix2Quat(Matrix matA, Matrix q) - { - // Watt and Watt p. 362 - double trace = matA[0, 0] + matA[1, 1] + matA[2, 2] + 1.0; - q[3] = Math.Sqrt(trace); - - q[0] = (matA[2, 1] - matA[1, 2]) / (4 * q[3]); - q[1] = (matA[0, 2] - matA[2, 0]) / (4 * q[3]); - q[2] = (matA[1, 0] - matA[0, 1]) / (4 * q[3]); - - // not tested - } - - public static void RotMatrix2Euler(Matrix matA, ref double x, ref double y, ref double z) - { - y = -Math.Asin(matA[2, 0]); - double c = Math.Cos(y); - - double cost3 = matA[0, 0] / c; - double sint3 = matA[1, 0] / c; - z = Math.Atan2(sint3, cost3); - - double sint1 = matA[2, 1] / c; - double cost1 = matA[2, 2] / c; - x = Math.Atan2(sint1, cost1); - } - - public static void QuatMult(Matrix matA, Matrix matB, Matrix matC) - { - Matrix v1 = new Matrix(3, 1); - Matrix v2 = new Matrix(3, 1); - Matrix v3 = new Matrix(3, 1); - - v1[0] = matA[0]; - v1[1] = matA[1]; - v1[2] = matA[2]; - double s1 = matA[3]; - - v2[0] = matB[0]; - v2[1] = matB[1]; - v2[2] = matB[2]; - double s2 = matB[3]; - - v3.Cross(v1, v2); - - matC[0] = (s1 * v2[0]) + (s2 * v1[0]) + v3[0]; - matC[1] = (s1 * v2[1]) + (s2 * v1[1]) + v3[1]; - matC[2] = (s1 * v2[2]) + (s2 * v1[2]) + v3[2]; - matC[3] = (s1 * s2) - v1.Dot(v2); - } - - public static void QuatInvert(Matrix matA, Matrix matB) - { - matB[0] = -matA[0]; - matB[1] = -matA[1]; - matB[2] = -matA[2]; - matB[3] = matA[3]; // w - } - - public static void QuatRot(Matrix q, Matrix x, Matrix y) - { - // p. 361 Watt and Watt - Matrix p = new Matrix(4, 1); - p[0] = x[0]; - p[1] = x[1]; - p[2] = x[2]; - p[3] = 0.0; - - Matrix q1 = new Matrix(4, 1); - Matrix q2 = new Matrix(4, 1); - Matrix qi = new Matrix(4, 1); - - qi.QuatInvert(q); - - q1.QuatMult(q, p); - q2.QuatMult(q1, qi); - - y[0] = q2[0]; - y[1] = q2[1]; - y[2] = q2[2]; - } - - public static double L1distance(Matrix matA, Matrix matB) - { - double s = 0.0; - double d; - for (int i = 0; i < matA.mn; i++) - { - d = matA.data[i] - matB.data[i]; - s += Math.Abs(d); - } - - return s; - } - - public static double L2distance(Matrix matA, Matrix matB) - { - double s = 0.0; - double d; - for (int i = 0; i < matA.mn; i++) - { - d = matA.data[i] - matB.data[i]; - s += d * d; - } - - return Math.Sqrt(s); - } - - public static void Normalize(Matrix matA, Matrix matB) - { - matB.Scale(matA, 1.0 / matA.Norm()); - } - - public static Matrix GaussianSample(int m, int n) - { - var matA = new Matrix(m, n); - for (int i = 0; i < m; i++) - { - for (int j = 0; j < n; j++) - { - matA[i, j] = NextGaussianSample(0, 1); - } - } - - return matA; - } - - public static Matrix GaussianSample(Matrix mu, double sigma) - { - int m = mu.Rows; - int n = mu.Cols; - - var matA = new Matrix(m, n); - for (int i = 0; i < m; i++) - { - for (int j = 0; j < n; j++) - { - matA[i, j] = NextGaussianSample(mu[i, j], sigma); - } - } - - return matA; - } - - public static double NextGaussianSample(double mu, double sigma) - { - // Box-Muller transform - const double epsilon = double.MinValue; - const double tau = 2.0 * Math.PI; - - generate = !generate; - if (!generate) - { - return (z1 * sigma) + mu; - } - - double u1, u2; - do - { - u1 = random.NextDouble(); - u2 = random.NextDouble(); - } - while (u1 <= epsilon); - - z0 = Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Cos(tau * u2); - z1 = Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Sin(tau * u2); - return (z0 * sigma) + mu; - } - - public static void MultAAT(Matrix matA, Matrix matB, Matrix matC) - { - for (int i = 0; i < matA.m; i++) - { - for (int j = 0; j < matB.m; j++) - { - double sum = 0; - for (int k = 0; k < matA.n; k++) - { - sum += matA[i, k] * matB[j, k]; - } - - matC[i, j] = sum; - } - } - } - - public float[] AsFloatArray() - { - float[] array = new float[this.mn]; - for (int i = 0; i < this.mn; i++) - { - array[i] = (float)this.data[i]; - } - - return array; - } - - // copy - public void Copy(Matrix mat) - { - for (int i = 0; i < this.m; i++) - { - for (int j = 0; j < this.n; j++) - { - this[i, j] = mat[i, j]; - } - } - } - - public void Copy(int bi, int bj, Matrix mat) - { - CopyRange(mat, 0, 0, mat.Rows, mat.Cols, this, bi, bj); - } - - public void Copy(int bi, int bj, Matrix mat, int ai, int aj, int rows, int cols) - { - CopyRange(mat, ai, aj, rows, cols, this, bi, bj); - } - - public void CopyRow(Matrix matA, int row) - { - CopyRow(matA, row, this); - } - - public void CopyCol(Matrix matA, int col) - { - CopyCol(matA, col, this); - } - - public void CopyDiag(Matrix matA) - { - CopyDiag(matA, this); - } - - public void Diag(Matrix matA, Matrix d) - { - matA.Zero(); - for (int i = 0; i < matA.m; i++) - { - matA[i, i] = d[i]; - } - } - - public void Diag(Matrix d) - { - this.Diag(this, d); - } - - public bool Equals(Matrix matA) - { - return Equals(matA, this); - } - - public void Transpose(Matrix matA) - { - Transpose(matA, this); - } - - public void Transpose() - { - Transpose(this, this); - } - - public void Reshape(Matrix matA) - { - Reshape(matA, this); - } - - public void Identity() - { - Identity(this); - } - - public void Set(double c) - { - Set(this, c); - } - - public void Zero() - { - this.Set(0.0); - } - - public void Randomize() - { - System.Random rnd = new System.Random(); - for (int i = 0; i < this.mn; i++) - { - this.data[i] = rnd.NextDouble(); - } - } - - public void Linspace(double x0, double x1) - { - double dx = (x1 - x0) / (double)(this.mn - 1); - for (int i = 0; i < this.mn; i++) - { - this.data[i] = x0 + (dx * i); - } - } - - public void Pow(Matrix matA, double c) - { - Pow(matA, c, this); - } - - public void Pow(double c) - { - Pow(this, c, this); - } - - public void Exp(Matrix matA) - { - Exp(matA, this); - } - - public void Exp() - { - Exp(this, this); - } - - public void Log(Matrix matA) - { - Log(matA, this); - } - - public void Log() - { - Log(this, this); - } - - public void Abs(Matrix matA) - { - Abs(matA, this); - } - - public void Abs() - { - Abs(this, this); - } - - public void Add(Matrix matA, double c) - { - Add(matA, c, this); - } - - public void Add(double c) - { - Add(this, c, this); - } - - public void Scale(Matrix matA, double c) - { - Scale(matA, c, this); - } - - public void Scale(double c) - { - Scale(this, c, this); - } - - public void ScaleAdd(Matrix matA, double c) - { - ScaleAdd(matA, c, this); - } - - public void ScaleAdd(double c) - { - ScaleAdd(this, c, this); - } - - public void Reciprocal(Matrix matA) - { - Reciprocal(matA, this); - } - - public void Reciprocal() - { - Reciprocal(this, this); - } - - // limits data between elements of A and B - public void Bound(Matrix matA, Matrix matB) - { - Bound(matA, matB, this); - } - - // matrix-matrix elementwise ops - public void Add(Matrix matA, Matrix matB) - { - Add(matA, matB, this); - } - - public void Add(Matrix matB) - { - Add(this, matB, this); - } - - public void Sub(Matrix matA, Matrix matB) - { - Sub(matA, matB, this); - } - - public void Sub(Matrix matB) - { - Sub(this, matB, this); - } - - public void ElemMult(Matrix matA, Matrix matB) - { - ElemMult(matA, matB, this); - } - - public void ElemMult(Matrix matB) - { - ElemMult(this, matB, this); - } - - public void Divide(Matrix matA, Matrix matB) - { - Divide(matA, matB, this); - } - - public void Divide(Matrix matB) - { - Divide(this, matB, this); - } - - // vector ops - public double Dot(Matrix matB) - { - return Dot(this, matB); - } - - public void Outer(Matrix matA, Matrix matB) - { - Outer(matA, matB, this); - } - - public void Cross(Matrix matA, Matrix matB) - { - Cross(matA, matB, this); - } - - // matrix-matrix ops - public void Mult(Matrix matA, Matrix matB) - { - Mult(matA, matB, this); - } - - public void MultAAT(Matrix matA, Matrix matB) - { - MultAAT(matA, matB, this); - } - - public void MultATA(Matrix matA, Matrix matB) - { - MultATA(matA, matB, this); - } - - public void MultATAParallel(Matrix matA, Matrix matB) - { - MultATAParallel(matA, matB, this); - } - - public void Inverse(Matrix matA) - { - // invert A and store in this - var anet = ToMathNet(matA); - var inverse = anet.Inverse(); - FromMathNet(inverse, this); - } - - public double Det3x3() - { - return Det3x3(this); - } - - public void Eig(Matrix v, Matrix d) - { - var evd = ToMathNet(this).Evd(); - FromMathNet(evd.EigenVectors, v); - for (int i = 0; i < this.Rows; i++) - { - d[i] = evd.D[i, i]; - } - } - - public void Eig2x2(Matrix matA, Matrix v, Matrix matD) - { - double a = matA[0, 0]; - double b = matA[0, 1]; - double c = matA[1, 0]; - double d = matA[1, 1]; - - // solve det(A - l*I) = 0 for eigenvalues l - double s = Math.Sqrt(((a + d) * (a + d)) + (4 * ((b * c) - (a * d)))); - matD[0] = (a + d + s) / 2; - matD[1] = (a + d - s) / 2; - - // solve for eigenvectors v in (A - l*I)*v = 0 for each eigenvalue - // set v1 = 1.0 - double v0, n; - - // first eigenvector - v0 = (matD[0] - d) / c; - n = Math.Sqrt((v0 * v0) + 1); - - v[0, 0] = v0 / n; - v[1, 0] = 1.0 / n; - - // second eigenvector - v0 = (matD[1] - d) / c; - n = Math.Sqrt((v0 * v0) + 1); - - v[0, 1] = v0 / n; - v[1, 1] = 1.0 / n; - } - - public void Eig2x2(Matrix v, Matrix d) - { - this.Eig2x2(this, v, d); - } - - public void SVD(Matrix matU, Matrix w, Matrix matV) - { - var svd = ToMathNet(this).Svd(); - FromMathNet(svd.U, matU); - FromMathNet(svd.S, w); - FromMathNet(svd.VT.Transpose(), matV); - } - - public void LeastSquares(Matrix matA, Matrix matB) - { - LeastSquares(this, matA, matB); - } - - // rotation conversions - public void Rot2D(double theta) - { - Rot2D(this, theta); - } - - public void RotEuler2Matrix(double x, double y, double z) - { - RotEuler2Matrix(x, y, z, this); - } - - public void RotFromTo2Quat(Matrix x, Matrix y) - { - RotFromTo2Quat(x, y, this); - } - - public void RotQuat2Matrix(Matrix q) - { - RotQuat2Matrix(q, this); - } - - public void RotAxisAngle2Quat(Matrix axis, double angle) - { - RotAxisAngle2Quat(axis, angle, this); - } - - public void RotMatrix2Quat(Matrix matA) - { - RotMatrix2Quat(matA, this); - } - - public void RotMatrix2Euler(ref double x, ref double y, ref double z) - { - RotMatrix2Euler(this, ref x, ref y, ref z); - } - - // quaternion ops; quat is ((X, Y, Z), W) - public void QuatMult(Matrix matA, Matrix matB) - { - QuatMult(matA, matB, this); - } - - public void QuatInvert(Matrix matA) - { - QuatInvert(matA, this); - } - - public void QuatInvert() - { - QuatInvert(this, this); - } - - public void QuatRot(Matrix q, Matrix x) - { - QuatRot(q, x, this); - } - - // norms - public double Minimum(ref int argmin) - { - double min = this.data[0]; - int mini = 0; - for (int i = 1; i < this.mn; i++) - { - if (this.data[i] < min) - { - min = this.data[i]; - mini = i; - } - } - - argmin = mini; - return min; - } - - public double Maximum(ref int argmax) - { - double max = this.data[0]; - int maxi = 0; - for (int i = 1; i < this.mn; i++) - { - if (this.data[i] > max) - { - max = this.data[i]; - maxi = i; - } - } - - argmax = maxi; - return max; - } - - public double Norm() - { - double sum = 0; - for (int i = 0; i < this.mn; i++) - { - sum += this.data[i] * this.data[i]; - } - - return Math.Sqrt(sum); - } - - public double Sum() - { - double sum = 0; - for (int i = 0; i < this.mn; i++) - { - sum += this.data[i]; - } - - return sum; - } - - public double SumSquares() - { - double sum = 0; - for (int i = 0; i < this.mn; i++) - { - sum += this.data[i] * this.data[i]; - } - - return sum; - } - - public double Product() - { - double product = 1.0; - for (int i = 0; i < this.mn; i++) - { - product *= this.data[i]; - } - - return product; - } - - public double L1distance(Matrix matA) - { - return L1distance(matA, this); - } - - public double L2distance(Matrix matA) - { - return L2distance(matA, this); - } - - public void Normalize(Matrix matA) - { - Normalize(matA, this); - } - - public void Normalize() - { - Normalize(this, this); - } - - public double Magnitude() - { - double sum = 0; - for (int i = 0; i < this.mn; i++) - { - sum += this.data[i] * this.data[i]; - } - - return sum; - } - - public void NormalizeRows() - { - double sum; - for (int i = 0; i < this.m; i++) - { - sum = 0; - for (int j = 0; j < this.n; j++) - { - sum += this[i, j]; - } - - for (int j = 0; j < this.n; j++) - { - this[i, j] = this[i, j] / sum; - } - } - } - - public override string ToString() - { - string s = string.Empty; - - for (int i = 0; i < this.m; i++) - { - for (int j = 0; j < this.n; j++) - { - s += this[i, j].ToString(); - if (j < this.n - 1) - { - s += ", \t"; - } - } - - s += " \r\n"; - } - - return s; - } - } -#pragma warning restore SA1600 -} diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Microsoft.Psi.Kinect.Windows.csproj b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Microsoft.Psi.Kinect.Windows.csproj index 1a776b4b3..e1ff74d33 100644 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Microsoft.Psi.Kinect.Windows.csproj +++ b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Microsoft.Psi.Kinect.Windows.csproj @@ -28,8 +28,12 @@ - - + + + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Orientation.cs b/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Orientation.cs deleted file mode 100644 index 173acf99d..000000000 --- a/Sources/Kinect/Microsoft.Psi.Kinect.Windows/Orientation.cs +++ /dev/null @@ -1,171 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Kinect -{ - using System; - -#pragma warning disable SA1600 - internal class Orientation - { - public static Matrix Rodrigues(Matrix r) - { - // where r rotation vector, theta = norm(r), M = skew(r/theta) - // R = I + sin(theta) M + (1-cos(theta)) M M - double theta = r.Norm(); - - var matR = new Matrix(3, 3); - matR.Identity(); - - // if there is no rotation (theta == 0) return identity - if (theta == 0) - { - return matR; - } - - var rn = new Matrix(3, 1); - rn.Normalize(r); - - var matM = new Matrix(3, 3); - matM[0, 0] = 0; - matM[0, 1] = -rn[2]; - matM[0, 2] = rn[1]; - matM[1, 0] = rn[2]; - matM[1, 1] = 0; - matM[1, 2] = -rn[0]; - matM[2, 0] = -rn[1]; - matM[2, 1] = rn[0]; - matM[2, 2] = 0; - - var sinThetaM = new Matrix(3, 3); - sinThetaM.Scale(matM, Math.Sin(theta)); - matR.Add(sinThetaM); - - var matMM = new Matrix(3, 3); - matMM.Mult(matM, matM); - var cosThetaMM = new Matrix(3, 3); - cosThetaMM.Scale(matMM, 1 - Math.Cos(theta)); - matR.Add(cosThetaMM); - - return matR; - } - - // a port from http://www.euclideanspace.com/maths/geometry/rotations/conversions/matrixToAngle/index.htm - public static Matrix RotationVector(Matrix m) - { - double angle, x, y, z; // variables for result - - var v = new Matrix(3, 1); - - double epsilon = 0.01; // margin to allow for rounding errors - double epsilon2 = 0.1; // margin to distinguish between 0 and 180 degrees - - // optional check that input is pure rotation, 'isRotationMatrix' is defined at: - // http://www.euclideanspace.com/maths/algebra/matrix/orthogonal/rotation/ - // assert isRotationMatrix(m) : "not valid rotation matrix" ;// for debugging - if ((Math.Abs(m[0, 1] - m[1, 0]) < epsilon) - && (Math.Abs(m[0, 2] - m[2, 0]) < epsilon) - && (Math.Abs(m[1, 2] - m[2, 1]) < epsilon)) - { - // singularity found - // first check for identity matrix which must have +1 for all terms - // in leading diagonal and zero in other terms - if ((Math.Abs(m[0, 1] + m[1, 0]) < epsilon2) - && (Math.Abs(m[0, 2] + m[2, 0]) < epsilon2) - && (Math.Abs(m[1, 2] + m[2, 1]) < epsilon2) - && (Math.Abs(m[0, 0] + m[1, 1] + m[2, 2] - 3) < epsilon2)) - { - // this singularity is identity matrix so angle = 0 - // return new axisAngle(0, 1, 0, 0); // zero angle, arbitrary axis - v.Zero(); - return v; - } - - // otherwise this singularity is angle = 180 - angle = Math.PI; - double xx = (m[0, 0] + 1) / 2; - double yy = (m[1, 1] + 1) / 2; - double zz = (m[2, 2] + 1) / 2; - double xy = (m[0, 1] + m[1, 0]) / 4; - double xz = (m[0, 2] + m[2, 0]) / 4; - double yz = (m[1, 2] + m[2, 1]) / 4; - if ((xx > yy) && (xx > zz)) - { // m[0,0] is the largest diagonal term - if (xx < epsilon) - { - x = 0; - y = 0.7071; - z = 0.7071; - } - else - { - x = Math.Sqrt(xx); - y = xy / x; - z = xz / x; - } - } - else if (yy > zz) - { // m[1,1] is the largest diagonal term - if (yy < epsilon) - { - x = 0.7071; - y = 0; - z = 0.7071; - } - else - { - y = Math.Sqrt(yy); - x = xy / y; - z = yz / y; - } - } - else - { // m[2,2] is the largest diagonal term so base result on this - if (zz < epsilon) - { - x = 0.7071; - y = 0.7071; - z = 0; - } - else - { - z = Math.Sqrt(zz); - x = xz / z; - y = yz / z; - } - } - - // return axis angle - v[0] = x; - v[1] = y; - v[2] = z; - v.Scale(angle); - return v; - } - - // as we have reached here there are no singularities so we can handle normally - double s = Math.Sqrt(((m[2, 1] - m[1, 2]) * (m[2, 1] - m[1, 2])) - + ((m[0, 2] - m[2, 0]) * (m[0, 2] - m[2, 0])) - + ((m[1, 0] - m[0, 1]) * (m[1, 0] - m[0, 1]))); // used to normalise - if (Math.Abs(s) < 0.001) - { - s = 1; - } - - // prevent divide by zero, should not happen if matrix is orthogonal and should be - // caught by singularity test above, but I've left it in just in case - angle = Math.Acos((m[0, 0] + m[1, 1] + m[2, 2] - 1) / 2); - x = (m[2, 1] - m[1, 2]) / s; - y = (m[0, 2] - m[2, 0]) / s; - z = (m[1, 0] - m[0, 1]) / s; - - // return new axisAngle(angle, x, y, z); - v[0] = x; - v[1] = y; - v[2] = z; - v.Scale(angle); - return v; - } - } -#pragma warning restore SA1600 -} diff --git a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Mesh.cs b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Mesh.cs index a8fdc485a..c1bb79136 100644 --- a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Mesh.cs +++ b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Mesh.cs @@ -47,19 +47,19 @@ public class Mesh /// /// Create mesh from depth map. /// - /// Depth map image. + /// Depth map image. /// Color data image. /// Kinect calibration. /// Mesh. - public static Mesh MeshFromDepthMap(Shared depthMap, Shared colorData, IDepthDeviceCalibrationInfo calib) + public static Mesh MeshFromDepthMap(Shared depthImage, Shared colorData, IDepthDeviceCalibrationInfo calib) { Mesh mesh = new Mesh(); - int width = depthMap.Resource.Width; - int height = depthMap.Resource.Height; + int width = depthImage.Resource.Width; + int height = depthImage.Resource.Height; mesh.Vertices = new Vertex[width * height]; bool[] vertexValid = new bool[width * height]; mesh.Faces = new Face[2 * (width - 1) * (height - 1)]; - byte[] depthData = depthMap.Resource.ReadBytes(depthMap.Resource.Size); + byte[] depthData = depthImage.Resource.ReadBytes(depthImage.Resource.Size); byte[] pixelData = colorData.Resource.ReadBytes(colorData.Resource.Size); int count = 0; unsafe @@ -68,7 +68,7 @@ public static Mesh MeshFromDepthMap(Shared depthMap, Shared colorD { for (int j = 0; j < width; j++) { - ushort* src = (ushort*)((byte*)depthMap.Resource.ImageData.ToPointer() + (i * depthMap.Resource.Stride)) + j; + ushort* src = (ushort*)((byte*)depthImage.Resource.ImageData.ToPointer() + (i * depthImage.Resource.Stride)) + j; ushort depth = *src; Point2D pt = new Point2D(j, i); vertexValid[count] = (depth == 0) ? false : true; diff --git a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/MeshTests.cs b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/MeshTests.cs index 483498017..2ed99eb38 100644 --- a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/MeshTests.cs +++ b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/MeshTests.cs @@ -18,7 +18,7 @@ public class MeshTests : IDisposable { private KinectSensor sensor; private IDepthDeviceCalibrationInfo depthDeviceCalibrationInfo = null; - private Shared lastImage = null; + private Shared lastImage = null; private Shared lastColor = null; private bool disposed = false; diff --git a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/QuaternionTest.cs b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/QuaternionTest.cs deleted file mode 100644 index ff4f0f05c..000000000 --- a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/QuaternionTest.cs +++ /dev/null @@ -1,64 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Test.Psi.Kinect.Windows -{ - using System; - using MathNet.Spatial.Euclidean; - using MathNet.Spatial.Units; - using Microsoft.Kinect; - using Microsoft.VisualStudio.TestTools.UnitTesting; - - /// - /// Quaternion test. - /// - [TestClass] - [Ignore] - public class QuaternionTest : IDisposable - { - /// - public void Dispose() - { - this.Dispose(true); - GC.SuppressFinalize(this); - } - - /// - /// Very simple test to make sure that our quaternion math is correct. - /// This rotates 10 degrees around the Y axis via a quaternion and then - /// rotates the entire result by 90 degrees. After this the resulting quaternion - /// should be rotated 100 degrees around Y axis. - /// - [TestMethod] - [Timeout(60000)] - [Ignore] - public void TestQuaternion() - { - CoordinateSystem ncs = new CoordinateSystem(); - Vector3D v = new Vector3D(0.0, 1.0, 0.0); - ncs = ncs.RotateCoordSysAroundVector(v.Normalize(), Angle.FromDegrees(90.0 /* in LHS */)); - Vector4 q; - double axisX = 0.0; - double axisY = 1.0; - double axisZ = 0.0; - double sa = Math.Sin(-5.0 /*half angle in RHS*/ * 3.1415926 / 180.0); - double ca = Math.Cos(-5.0 /*half angle in RHS*/ * 3.1415926 / 180.0); - q.X = (float)(axisX * sa); - q.Y = (float)(axisY * sa); - q.Z = (float)(axisZ * sa); - q.W = (float)ca; - var rot = ncs.GetRotationSubMatrix(); - var qrot = Microsoft.Psi.Kinect.KinectExtensions.QuaternionToMatrix(q); - var qv = Microsoft.Psi.Kinect.KinectExtensions.MatrixToQuaternion(rot * qrot); - Vector4 axisAngle = Microsoft.Psi.Kinect.KinectExtensions.QuaternionAsAxisAngle(qv); - } - - /// - /// Dispose. - /// - /// Disposing. - protected virtual void Dispose(bool disposing) - { - } - } -} diff --git a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Test.Psi.Kinect.Windows.x64.csproj b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Test.Psi.Kinect.Windows.x64.csproj index ba32e8738..25d5c7be4 100644 --- a/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Test.Psi.Kinect.Windows.x64.csproj +++ b/Sources/Kinect/Test.Psi.Kinect.Windows.x64/Test.Psi.Kinect.Windows.x64.csproj @@ -1,20 +1,17 @@  net472 - false - false + ../../../Build/Test.Psi.ruleset Library - ../../../Build/Test.Psi.ruleset true true - ../../../Build/Test.Psi.ruleset true true @@ -261,8 +258,12 @@ - - + + all + runtime; build; native; contentfiles; analyzers + + + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Sources/Language/Microsoft.Psi.Language/Microsoft.Psi.Language.csproj b/Sources/Language/Microsoft.Psi.Language/Microsoft.Psi.Language.csproj index bdeca51a2..216b57fa9 100644 --- a/Sources/Language/Microsoft.Psi.Language/Microsoft.Psi.Language.csproj +++ b/Sources/Language/Microsoft.Psi.Language/Microsoft.Psi.Language.csproj @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Media/Microsoft.Psi.Media.Linux/LinuxVideoInterop.cs b/Sources/Media/Microsoft.Psi.Media.Linux/LinuxVideoInterop.cs index 4cdd92824..87d0b1ab5 100644 --- a/Sources/Media/Microsoft.Psi.Media.Linux/LinuxVideoInterop.cs +++ b/Sources/Media/Microsoft.Psi.Media.Linux/LinuxVideoInterop.cs @@ -12,7 +12,7 @@ namespace Microsoft.Psi.Media /// Structs, enums and static methods for interacting with Video4Linux drivers (V4L2). /// /// - /// This implimentation is based on this spec: https://www.linuxtv.org/downloads/legacy/video4linux/API/V4L2_API/spec-single/v4l2.html + /// This implementation is based on this spec: https://www.linuxtv.org/downloads/legacy/video4linux/API/V4L2_API/spec-single/v4l2.html /// The only dependencies are POSIX ioctl, mmap and munmap. /// internal static class LinuxVideoInterop @@ -342,11 +342,11 @@ internal enum Memory /// Query device capabilities (POSIX ioctl VIDIOC_QUERYCAP). /// /// Device name (e.g. "/dev/video0"). - /// Device capabilites. + /// Device capabilities. public static Capability QueryCapabilities(int fd) { var caps = default(Capability); - if (QueryCapabilities(fd, VIDIOC(IOCREAD, Marshal.SizeOf(caps), 0) /* VIDIOC_QUERYCAP */, ref caps) != 0) + if (NativeMethods.QueryCapabilities(fd, VIDIOC(IOCREAD, Marshal.SizeOf(caps), 0) /* VIDIOC_QUERYCAP */, ref caps) != 0) { throw new IOException("QueryCapabilities failed."); } @@ -386,7 +386,7 @@ public static IEnumerable EnumerateFormats(int fd) /// Success flag. public static bool EnumFormats(int fd, ref FormatDescription format) { - return EnumFormats(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 2) /* VIDIOC_ENUM_FMT */, ref format) >= 0; + return NativeMethods.EnumFormats(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 2) /* VIDIOC_ENUM_FMT */, ref format) >= 0; } /// @@ -396,7 +396,7 @@ public static bool EnumFormats(int fd, ref FormatDescription format) /// Video format struct to be populated. internal static void GetFormat(int fd, ref VideoFormat format) { - if (GetFormat(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 4) /* VIDIOC_G_FMT */, ref format) != 0) + if (NativeMethods.GetFormat(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 4) /* VIDIOC_G_FMT */, ref format) != 0) { throw new IOException("GetFormat failed."); } @@ -419,7 +419,7 @@ internal static void SetFormat(int fd, VideoFormat format) throw new ArgumentException("Formats other than video capture are not supported."); } - if (SetFormat(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 5) /* VIDIOC_S_FMT */, ref format) != 0) + if (NativeMethods.SetFormat(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(format), 5) /* VIDIOC_S_FMT */, ref format) != 0) { throw new IOException("SetFormat failed."); } @@ -438,7 +438,7 @@ internal static uint ReqBufs(int fd, uint count, Memory memory) req.Count = count; req.Memory = memory; req.Type = BufferType.VideoCapture; // note: only video capture supported - if (ReqBufs(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(req), 8) /* VIDIOC_REQBUFS */, ref req) != 0) + if (NativeMethods.ReqBufs(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(req), 8) /* VIDIOC_REQBUFS */, ref req) != 0) { throw new ArgumentException("ReqBufs failed."); } @@ -464,7 +464,7 @@ internal static Buffer QueryBuf(int fd, uint index) Memory = Memory.MemoryMapping, // note: memory mapped assumed }; - if (QueryBuf(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 9) /* VIDIOC_QUERYBUF */, ref buffer) != 0) + if (NativeMethods.QueryBuf(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 9) /* VIDIOC_QUERYBUF */, ref buffer) != 0) { throw new ArgumentException($"QueryBuf failed (index={index})."); } @@ -480,7 +480,16 @@ internal static Buffer QueryBuf(int fd, uint index) /// Pointer to mapped memory. internal static unsafe void* MemoryMap(int fd, Buffer buffer) { - var start = MemMap(IntPtr.Zero, buffer.Length, 0x1 /* PROT_READ */ | 0x2 /* PROT_WRITE */, 0x1 /* MAP_SHARED */, fd, buffer.Pointer); + void* start; + if (Environment.Is64BitOperatingSystem) + { + start = NativeMethods.MemMap64(IntPtr.Zero, buffer.Length, 0x1 /* PROT_READ */ | 0x2 /* PROT_WRITE */, 0x1 /* MAP_SHARED */, fd, buffer.Pointer); + } + else + { + start = NativeMethods.MemMap32(IntPtr.Zero, buffer.Length, 0x1 /* PROT_READ */ | 0x2 /* PROT_WRITE */, 0x1 /* MAP_SHARED */, fd, (uint)buffer.Pointer); + } + if (start == (void*)-1) { throw new ArgumentException("Memory map failed."); @@ -495,7 +504,17 @@ internal static Buffer QueryBuf(int fd, uint index) /// Buffer to unmap. internal static unsafe void MemoryUnmap(Buffer buffer) { - if (MemUnmap(buffer.Pointer, buffer.Length) != 0) + int result; + if (Environment.Is64BitOperatingSystem) + { + result = NativeMethods.MemUnmap64(buffer.Pointer, buffer.Length); + } + else + { + result = NativeMethods.MemUnmap32((uint)buffer.Pointer, buffer.Length); + } + + if (result != 0) { throw new ArgumentException("Memory unmap failed."); } @@ -508,7 +527,7 @@ internal static unsafe void MemoryUnmap(Buffer buffer) /// Buffer struct to enqueue. internal static void EnqueBuffer(int fd, Buffer buffer) { - if (EnqueBuffer(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 15) /* VIDIOC_QBUF */, ref buffer) != 0) + if (NativeMethods.EnqueBuffer(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 15) /* VIDIOC_QBUF */, ref buffer) != 0) { throw new ArgumentException("Enque buffer failed."); } @@ -523,7 +542,7 @@ internal static Buffer DequeBuffer(int fd) { var buffer = default(Buffer); buffer.Type = LinuxVideoInterop.BufferType.VideoCapture; - if (DequeBuffer(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 17) /* VIDIOC_DQBUF */, ref buffer) != 0) + if (NativeMethods.DequeBuffer(fd, VIDIOC(IOCREAD | IOCWRITE, Marshal.SizeOf(buffer), 17) /* VIDIOC_DQBUF */, ref buffer) != 0) { throw new ArgumentException("Deque buffer failed."); } @@ -538,7 +557,7 @@ internal static Buffer DequeBuffer(int fd) internal static void StreamOn(int fd) { var type = BufferType.VideoCapture; // note: only video capture supported - if (StreamOn(fd, VIDIOC(IOCWRITE, sizeof(uint), 18) /* VIDIOC_STREAMON */, ref type) != 0) + if (NativeMethods.StreamOn(fd, VIDIOC(IOCWRITE, sizeof(uint), 18) /* VIDIOC_STREAMON */, ref type) != 0) { throw new ArgumentException("StreamOn failed."); } @@ -551,7 +570,7 @@ internal static void StreamOn(int fd) internal static void StreamOff(int fd) { var type = BufferType.VideoCapture; // note: only video capture supported - if (StreamOff(fd, VIDIOC(IOCWRITE, Marshal.SizeOf(type), 19) /* VIDIOC_STREAMOFF */, ref type) != 0) + if (NativeMethods.StreamOff(fd, VIDIOC(IOCWRITE, Marshal.SizeOf(type), 19) /* VIDIOC_STREAMOFF */, ref type) != 0) { throw new ArgumentException("StreamOn failed."); } @@ -569,128 +588,6 @@ private static uint VIDIOC(uint readWrite, int size, int command) return readWrite | ((uint)size << 16) | V | (uint)command; } - /// - /// Query capabilites (POSIX ioctl VIDIOC_QUERYCAP). - /// - /// Device file descriptor. - /// Request type (VIDIOC_QUERYCAP). - /// Capabilities struct to be populated. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int QueryCapabilities(int fd, uint request, ref Capability caps); - - /// - /// Enumerate formats (POSIX ioctl VIDIOC_ENUM_FMT). - /// - /// Device file descriptor. - /// Request type (VIDIOC_ENUM_FMT). - /// Format descrition struct to be populated. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int EnumFormats(int fd, uint request, ref FormatDescription format); - - /// - /// Get format (POSIX ioctl VIDIOC_G_FMT). - /// - /// Device file descriptor. - /// Request type (VIDIOC_G_FMT). - /// Video format struct to be populated. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int GetFormat(int fd, uint request, ref VideoFormat format); - - /// - /// Set format (POSIX ioctl VIDIOC_S_FMT). - /// - /// Device file descriptor. - /// Request type (VIDIOC_S_FMT). - /// Video format. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int SetFormat(int fd, uint request, ref VideoFormat format); - - /// - /// Request buffers (POSIX ioctl VIDIOC_REQBUFS). - /// - /// Device file descriptor. - /// Request type (VIDIOC_REQBUFS). - /// Request buffers struct to be populated. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int ReqBufs(int fd, uint request, ref RequestBuffers req); - - /// - /// Query buffer (POSIX ioctl VIDIOC_QUERYBUF). - /// - /// Device file descriptor. - /// Request type (VIDIOC_QUERYBUF). - /// Buffer struct to be populated. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int QueryBuf(int fd, uint request, ref Buffer buf); - - /// - /// Memory map (POSIX mmap). - /// - /// Buffer start. - /// Buffer length. - /// Protection (PROT_EXEC/READ/WRITE/NONE). - /// Flags (MAP_SHARED/PRIVATE). - /// Device file descriptor. - /// Offset within buffer. - /// Pointer to mapped memory. - [DllImport("libc", EntryPoint="mmap", SetLastError=true)] - private static unsafe extern void* MemMap(IntPtr start, ulong length, int prot, int flags, int fd, ulong offset); - - /// - /// Memory unmap (POSIX munmap). - /// - /// Buffer start. - /// Buffer length. - /// Result flag. - [DllImport("libc", EntryPoint="munmap", SetLastError=true)] - private static unsafe extern int MemUnmap(ulong start, ulong length); - - /// - /// Enqueue buffer (POSIX ioctl VIDIOC_QBUF). - /// - /// Device file descriptor. - /// Request type (VIDIOC_QBUF). - /// Buffer struct to enqueue. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int EnqueBuffer(int fd, uint request, ref Buffer buffer); - - /// - /// Dequeue buffer (POSIX ioctl VIDIOC_DQBUF). - /// - /// Device file descriptor. - /// Request type (VIDIOC_DQBUF). - /// Buffer struct into which to dequeue. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int DequeBuffer(int fd, uint request, ref Buffer buffer); - - /// - /// Streaming on (POSIX ioctl VIDIOC_STREAMON). - /// - /// Device file descriptor. - /// Request type (VIDIOC_STREAMON). - /// Buffer type. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int StreamOn(int fd, uint request, ref BufferType buftype); - - /// - /// Streaming off (POSIX ioctl VIDIOC_STREAMOFF). - /// - /// Device file descriptor. - /// Request type (VIDIOC_STREAMOFF). - /// Buffer type. - /// Result flag. - [DllImport("libc", EntryPoint="ioctl", SetLastError=true)] - private static extern int StreamOff(int fd, uint request, ref BufferType buftype); - /// /// Capabilities struct (v4l2_capability). /// @@ -721,12 +618,12 @@ internal struct Capability public uint Version; /// - /// Driver capabilites. + /// Driver capabilities. /// public CapsFlags Caps; /// - /// Device capabilites. + /// Device capabilities. /// public CapsFlags DeviceCaps; @@ -1020,5 +917,152 @@ internal struct Buffer [MarshalAs(UnmanagedType.ByValArray, SizeConst = 2)] private uint[] reserved; } + + private static class NativeMethods + { + /// + /// Query capabilities (POSIX ioctl VIDIOC_QUERYCAP). + /// + /// Device file descriptor. + /// Request type (VIDIOC_QUERYCAP). + /// Capabilities struct to be populated. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int QueryCapabilities(int fd, uint request, ref Capability caps); + + /// + /// Enumerate formats (POSIX ioctl VIDIOC_ENUM_FMT). + /// + /// Device file descriptor. + /// Request type (VIDIOC_ENUM_FMT). + /// Format description struct to be populated. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int EnumFormats(int fd, uint request, ref FormatDescription format); + + /// + /// Get format (POSIX ioctl VIDIOC_G_FMT). + /// + /// Device file descriptor. + /// Request type (VIDIOC_G_FMT). + /// Video format struct to be populated. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int GetFormat(int fd, uint request, ref VideoFormat format); + + /// + /// Set format (POSIX ioctl VIDIOC_S_FMT). + /// + /// Device file descriptor. + /// Request type (VIDIOC_S_FMT). + /// Video format. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int SetFormat(int fd, uint request, ref VideoFormat format); + + /// + /// Request buffers (POSIX ioctl VIDIOC_REQBUFS). + /// + /// Device file descriptor. + /// Request type (VIDIOC_REQBUFS). + /// Request buffers struct to be populated. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int ReqBufs(int fd, uint request, ref RequestBuffers req); + + /// + /// Query buffer (POSIX ioctl VIDIOC_QUERYBUF). + /// + /// Device file descriptor. + /// Request type (VIDIOC_QUERYBUF). + /// Buffer struct to be populated. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int QueryBuf(int fd, uint request, ref Buffer buf); + + /// + /// Memory map (POSIX mmap). + /// + /// Buffer start. + /// Buffer length. + /// Protection (PROT_EXEC/READ/WRITE/NONE). + /// Flags (MAP_SHARED/PRIVATE). + /// Device file descriptor. + /// Offset within buffer. + /// Pointer to mapped memory. + [DllImport("libc", EntryPoint = "mmap", SetLastError = true)] + internal static unsafe extern void* MemMap32(IntPtr start, uint length, int prot, int flags, int fd, uint offset); + + /// + /// Memory map (POSIX mmap). + /// + /// Buffer start. + /// Buffer length. + /// Protection (PROT_EXEC/READ/WRITE/NONE). + /// Flags (MAP_SHARED/PRIVATE). + /// Device file descriptor. + /// Offset within buffer. + /// Pointer to mapped memory. + [DllImport("libc", EntryPoint = "mmap", SetLastError = true)] + internal static unsafe extern void* MemMap64(IntPtr start, ulong length, int prot, int flags, int fd, ulong offset); + + /// + /// Memory unmap (POSIX munmap). + /// + /// Buffer start. + /// Buffer length. + /// Result flag. + [DllImport("libc", EntryPoint = "munmap", SetLastError = true)] + internal static unsafe extern int MemUnmap32(uint start, uint length); + + /// + /// Memory unmap (POSIX munmap). + /// + /// Buffer start. + /// Buffer length. + /// Result flag. + [DllImport("libc", EntryPoint = "munmap", SetLastError = true)] + internal static unsafe extern int MemUnmap64(ulong start, ulong length); + + /// + /// Enqueue buffer (POSIX ioctl VIDIOC_QBUF). + /// + /// Device file descriptor. + /// Request type (VIDIOC_QBUF). + /// Buffer struct to enqueue. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int EnqueBuffer(int fd, uint request, ref Buffer buffer); + + /// + /// Dequeue buffer (POSIX ioctl VIDIOC_DQBUF). + /// + /// Device file descriptor. + /// Request type (VIDIOC_DQBUF). + /// Buffer struct into which to dequeue. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int DequeBuffer(int fd, uint request, ref Buffer buffer); + + /// + /// Streaming on (POSIX ioctl VIDIOC_STREAMON). + /// + /// Device file descriptor. + /// Request type (VIDIOC_STREAMON). + /// Buffer type. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int StreamOn(int fd, uint request, ref BufferType buftype); + + /// + /// Streaming off (POSIX ioctl VIDIOC_STREAMOFF). + /// + /// Device file descriptor. + /// Request type (VIDIOC_STREAMOFF). + /// Buffer type. + /// Result flag. + [DllImport("libc", EntryPoint = "ioctl", SetLastError = true)] + internal static extern int StreamOff(int fd, uint request, ref BufferType buftype); + } } } diff --git a/Sources/Media/Microsoft.Psi.Media.Linux/MediaCaptureInternal.cs b/Sources/Media/Microsoft.Psi.Media.Linux/MediaCaptureInternal.cs index 6475b850d..d80ef1dc7 100644 --- a/Sources/Media/Microsoft.Psi.Media.Linux/MediaCaptureInternal.cs +++ b/Sources/Media/Microsoft.Psi.Media.Linux/MediaCaptureInternal.cs @@ -63,7 +63,7 @@ public MediaCaptureInternal(string device) /// Open device. /// /// - /// This also queries the device capabilites and ensures that it supports video capture and streaming. + /// This also queries the device capabilities and ensures that it supports video capture and streaming. /// public void Open() { @@ -162,7 +162,7 @@ public unsafe void StreamBuffers() this.background = new Thread(new ThreadStart(this.ProcessFrames)) { IsBackground = true }; this.background.Start(); } - catch (Exception ex) + catch { for (var i = 0u; i < NumberOfDriverBuffers; i++) { @@ -173,7 +173,7 @@ public unsafe void StreamBuffers() } } - throw ex; + throw; } } @@ -254,12 +254,12 @@ internal PixelFormat(LinuxVideoInterop.FormatDescription internalFormat) public string Description => this.InternalFormat.Description; /// - /// Gets a value indicating whether whether pixel format is compressed. + /// Gets a value indicating whether pixel format is compressed. /// public bool IsCompressed => (this.InternalFormat.Flags & LinuxVideoInterop.FormatFlags.Compressed) == LinuxVideoInterop.FormatFlags.Compressed; /// - /// Gets a value indicating whether whether pixel format is emulated (non-native). + /// Gets a value indicating whether pixel format is emulated (non-native). /// public bool IsEmulated => (this.InternalFormat.Flags & LinuxVideoInterop.FormatFlags.Emulated) == LinuxVideoInterop.FormatFlags.Emulated; @@ -311,7 +311,7 @@ internal VideoFormat(LinuxVideoInterop.VideoFormat internalFormat) public uint Size => this.InternalFormat.Pixel.SizeImage; /// - /// Gets internal vidio format (used to pass back to driver). + /// Gets internal video format (used to pass back to driver). /// internal LinuxVideoInterop.VideoFormat InternalFormat => this.internalFormat; } diff --git a/Sources/Media/Microsoft.Psi.Media.Linux/Microsoft.Psi.Media.Linux.csproj b/Sources/Media/Microsoft.Psi.Media.Linux/Microsoft.Psi.Media.Linux.csproj index 5959cc412..1ac890bf0 100644 --- a/Sources/Media/Microsoft.Psi.Media.Linux/Microsoft.Psi.Media.Linux.csproj +++ b/Sources/Media/Microsoft.Psi.Media.Linux/Microsoft.Psi.Media.Linux.csproj @@ -39,6 +39,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Media/Microsoft.Psi.Media.Native.x64/Microsoft.Psi.Media.Native.x64.vcxproj b/Sources/Media/Microsoft.Psi.Media.Native.x64/Microsoft.Psi.Media.Native.x64.vcxproj index 5d3bfdd6a..bc61d71f3 100644 --- a/Sources/Media/Microsoft.Psi.Media.Native.x64/Microsoft.Psi.Media.Native.x64.vcxproj +++ b/Sources/Media/Microsoft.Psi.Media.Native.x64/Microsoft.Psi.Media.Native.x64.vcxproj @@ -32,14 +32,14 @@ DynamicLibrary true - v141 + v142 Unicode Spectre DynamicLibrary false - v141 + v142 true Unicode Spectre diff --git a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCapture.cs b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCapture.cs index 196b3e0c5..bdcf546f2 100644 --- a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCapture.cs +++ b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCapture.cs @@ -22,13 +22,13 @@ public class MediaCapture : IProducer>, ISourceComponent, IDisposa /// private readonly MediaCaptureConfiguration configuration; + private readonly IProducer audio; + /// /// The video capture device. /// private MediaCaptureDevice camera; - private IProducer audio; - /// /// Defines attributes of properties exposed by MediaCaptureDevice. /// @@ -46,7 +46,7 @@ public MediaCapture(Pipeline pipeline, string configurationFilename) this.configuration = configurationHelper.Configuration; if (this.configuration.CaptureAudio) { - this.audio = new Audio.AudioCapture(pipeline, new Audio.AudioCaptureConfiguration() { OutputFormat = Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm() }); + this.audio = new Audio.AudioCapture(pipeline, Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm()); } } @@ -55,13 +55,13 @@ public MediaCapture(Pipeline pipeline, string configurationFilename) /// /// Pipeline this component is a part of. /// Describes how to configure the media capture device. - public MediaCapture(Pipeline pipeline, MediaCaptureConfiguration configuration) + public MediaCapture(Pipeline pipeline, MediaCaptureConfiguration configuration = null) : this(pipeline) { - this.configuration = configuration; + this.configuration = configuration ?? new MediaCaptureConfiguration(); if (this.configuration.CaptureAudio) { - this.audio = new Audio.AudioCapture(pipeline, new Audio.AudioCaptureConfiguration() { OutputFormat = Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm() }); + this.audio = new Audio.AudioCapture(pipeline, Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm()); } } @@ -74,9 +74,8 @@ public MediaCapture(Pipeline pipeline, MediaCaptureConfiguration configuration) /// Frame rate. /// Should we create an audio capture device. /// Device ID. - /// Indicates whether video frames should be persisted. /// Indicates whether camera is shared amongst multiple applications. - public MediaCapture(Pipeline pipeline, int width, int height, double framerate = 15, bool captureAudio = false, string deviceId = null, bool persistVideoFrames = false, bool useInSharedMode = false) + public MediaCapture(Pipeline pipeline, int width, int height, double framerate = 15, bool captureAudio = false, string deviceId = null, bool useInSharedMode = false) : this(pipeline) { this.configuration = new MediaCaptureConfiguration() @@ -90,7 +89,7 @@ public MediaCapture(Pipeline pipeline, int width, int height, double framerate = }; if (this.configuration.CaptureAudio) { - this.audio = new Audio.AudioCapture(pipeline, new Audio.AudioCaptureConfiguration() { OutputFormat = Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm() }); + this.audio = new Audio.AudioCapture(pipeline, Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm()); } } @@ -122,7 +121,7 @@ public Emitter Audio /// /// Returns information about each property exposed by the media capture device. /// - /// MediaCaptureInfo object definiting ranges and availability of each property. + /// MediaCaptureInfo object defining ranges and availability of each property. public MediaCaptureInfo GetDeviceInfo() { return this.deviceInfo; @@ -134,18 +133,20 @@ public MediaCaptureInfo GetDeviceInfo() /// A new MediaCaptureConfiguration object with the device's current settings. public MediaCaptureConfiguration GetDeviceConfiguration() { - MediaCaptureConfiguration config = new MediaCaptureConfiguration(); - config.BacklightCompensation = this.GetValueBool(VideoProperty.BacklightCompensation, this.deviceInfo.BacklightCompensationInfo.Supported); - config.Brightness = this.GetValueInt(VideoProperty.Brightness, this.deviceInfo.BrightnessInfo.Supported); - config.ColorEnable = this.GetValueBool(VideoProperty.ColorEnable, this.deviceInfo.ColorEnableInfo.Supported); - config.Contrast = this.GetValueInt(VideoProperty.Contrast, this.deviceInfo.ContrastInfo.Supported); - config.Gain = this.GetValueInt(VideoProperty.Gain, this.deviceInfo.GainInfo.Supported); - config.Gamma = this.GetValueInt(VideoProperty.Gamma, this.deviceInfo.GammaInfo.Supported); - config.Hue = this.GetValueInt(VideoProperty.Hue, this.deviceInfo.HueInfo.Supported); - config.Saturation = this.GetValueInt(VideoProperty.Saturation, this.deviceInfo.SaturationInfo.Supported); - config.Sharpness = this.GetValueInt(VideoProperty.Sharpness, this.deviceInfo.SharpnessInfo.Supported); - config.WhiteBalance = this.GetValueInt(VideoProperty.WhiteBalance, this.deviceInfo.WhiteBalanceInfo.Supported); - config.Focus = this.GetValueInt(ManagedCameraControlProperty.Focus, this.deviceInfo.FocusInfo.Supported); + MediaCaptureConfiguration config = new MediaCaptureConfiguration + { + BacklightCompensation = this.GetValueBool(VideoProperty.BacklightCompensation, this.deviceInfo.BacklightCompensationInfo.Supported), + Brightness = this.GetValueInt(VideoProperty.Brightness, this.deviceInfo.BrightnessInfo.Supported), + ColorEnable = this.GetValueBool(VideoProperty.ColorEnable, this.deviceInfo.ColorEnableInfo.Supported), + Contrast = this.GetValueInt(VideoProperty.Contrast, this.deviceInfo.ContrastInfo.Supported), + Gain = this.GetValueInt(VideoProperty.Gain, this.deviceInfo.GainInfo.Supported), + Gamma = this.GetValueInt(VideoProperty.Gamma, this.deviceInfo.GammaInfo.Supported), + Hue = this.GetValueInt(VideoProperty.Hue, this.deviceInfo.HueInfo.Supported), + Saturation = this.GetValueInt(VideoProperty.Saturation, this.deviceInfo.SaturationInfo.Supported), + Sharpness = this.GetValueInt(VideoProperty.Sharpness, this.deviceInfo.SharpnessInfo.Supported), + WhiteBalance = this.GetValueInt(VideoProperty.WhiteBalance, this.deviceInfo.WhiteBalanceInfo.Supported), + Focus = this.GetValueInt(ManagedCameraControlProperty.Focus, this.deviceInfo.FocusInfo.Supported), + }; return config; } @@ -255,13 +256,11 @@ public void Start(Action notifyCompletionTime) this.camera.CaptureSample((data, length, timestamp) => { var time = DateTime.FromFileTimeUtc(timestamp); - using (var sharedImage = ImagePool.GetOrCreate(this.configuration.Width, this.configuration.Height, Microsoft.Psi.Imaging.PixelFormat.BGR_24bpp)) - { - sharedImage.Resource.CopyFrom(data); + using var sharedImage = ImagePool.GetOrCreate(this.configuration.Width, this.configuration.Height, Microsoft.Psi.Imaging.PixelFormat.BGR_24bpp); + sharedImage.Resource.CopyFrom(data); - var originatingTime = this.pipeline.GetCurrentTimeFromElapsedTicks(timestamp); - this.Out.Post(sharedImage, originatingTime); - } + var originatingTime = this.pipeline.GetCurrentTimeFromElapsedTicks(timestamp); + this.Out.Post(sharedImage, originatingTime); }); } else @@ -305,48 +304,6 @@ private void SetDeviceProperty(VideoProperty prop, MediaCaptureInfo.PropertyInfo } } - private MediaCaptureInfo.PropertyInfo GetInfo(ManagedCameraControlProperty prop) - { - MediaCaptureInfo.PropertyInfo info = new MediaCaptureInfo.PropertyInfo(); - int min = 0, max = 0, stepSize = 0, defValue = 0, flag = 0; - if (this.camera.GetRange(prop, ref min, ref max, ref stepSize, ref defValue, ref flag)) - { - info.MinValue = min; - info.MaxValue = max; - info.StepSize = stepSize; - info.DefaultValue = defValue; - info.AutoControlled = (flag == (int)VideoPropertyFlags.Auto) ? true : false; - info.Supported = true; - } - else - { - info.Supported = false; - } - - return info; - } - - private MediaCaptureInfo.PropertyInfo GetInfo(VideoProperty prop) - { - MediaCaptureInfo.PropertyInfo info = new MediaCaptureInfo.PropertyInfo(); - int min = 0, max = 0, stepSize = 0, defValue = 0, flag = 0; - if (this.camera.GetRange(prop, ref min, ref max, ref stepSize, ref defValue, ref flag)) - { - info.MinValue = min; - info.MaxValue = max; - info.StepSize = stepSize; - info.DefaultValue = defValue; - info.AutoControlled = (flag == (int)VideoPropertyFlags.Auto) ? true : false; - info.Supported = true; - } - else - { - info.Supported = false; - } - - return info; - } - private MediaCaptureConfiguration.PropertyValue GetValueInt(VideoProperty prop, bool supported) { int flags = 0; @@ -356,7 +313,7 @@ private MediaCaptureConfiguration.PropertyValue GetValueInt(VideoProperty p this.camera.GetProperty(prop, ref value, ref flags)) { propValue.Value = value; - propValue.Auto = (flags == (int)VideoPropertyFlags.Auto) ? true : false; + propValue.Auto = flags == (int)VideoPropertyFlags.Auto; } return propValue; @@ -371,7 +328,7 @@ private MediaCaptureConfiguration.PropertyValue GetValueInt(ManagedCameraCo this.camera.GetProperty(prop, ref value, ref flags)) { propValue.Value = value; - propValue.Auto = (flags == (int)VideoPropertyFlags.Auto) ? true : false; + propValue.Auto = flags == (int)VideoPropertyFlags.Auto; } return propValue; @@ -385,8 +342,8 @@ private MediaCaptureConfiguration.PropertyValue GetValueBool(VideoProperty if (supported && this.camera.GetProperty(prop, ref value, ref flags)) { - propValue.Value = (value == 1) ? true : false; - propValue.Auto = (flags == (int)VideoPropertyFlags.Auto) ? true : false; + propValue.Value = value == 1; + propValue.Auto = flags == (int)VideoPropertyFlags.Auto; } return propValue; diff --git a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCaptureConfiguration.cs b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCaptureConfiguration.cs index 262b10b41..e79be2180 100644 --- a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCaptureConfiguration.cs +++ b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaCaptureConfiguration.cs @@ -13,18 +13,12 @@ public class MediaCaptureConfiguration /// public static readonly MediaCaptureConfiguration Default = new MediaCaptureConfiguration() { - UseInSharedMode = false, - Width = 1280, - Height = 720, - Framerate = 15, - DeviceId = null, - CaptureAudio = false, }; /// /// Gets or sets a value indicating whether the capture device should include audio capture. /// - public bool CaptureAudio { get; set; } + public bool CaptureAudio { get; set; } = false; /// /// Gets or sets a value indicating whether Backlight Compensation is being applied. @@ -84,27 +78,27 @@ public class MediaCaptureConfiguration /// /// Gets or sets a value indicating whether the camera device is shared amongst multiple applications. /// - public bool UseInSharedMode { get; set; } + public bool UseInSharedMode { get; set; } = false; /// /// Gets or sets the camera resolution width. /// - public int Width { get; set; } + public int Width { get; set; } = 1280; /// /// Gets or sets the camera resolution height. /// - public int Height { get; set; } + public int Height { get; set; } = 720; /// /// Gets or sets the camera framerate. /// - public double Framerate { get; set; } + public double Framerate { get; set; } = 15; /// /// Gets or sets device id used to identify the camera. /// - public string DeviceId { get; set; } + public string DeviceId { get; set; } = null; /// /// Defines the type of a property on the media capture device. diff --git a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaSource.cs b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaSource.cs index 6703d50a5..3d7190280 100644 --- a/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaSource.cs +++ b/Sources/Media/Microsoft.Psi.Media.Windows.x64/MediaSource.cs @@ -18,7 +18,7 @@ namespace Microsoft.Psi.Media public class MediaSource : Generator, IDisposable { private bool disposed = false; - private string filename; + private Stream input; private short videoWidth; private short videoHeight; private SourceReader sourceReader; @@ -27,18 +27,47 @@ public class MediaSource : Generator, IDisposable private int audioStreamIndex = -1; private WaveFormat waveFormat; + /// + /// Keep track of the timestamp of the last image frame (computed from the value reported to us by media foundation). + /// + private DateTime lastPostedImageTime = DateTime.MinValue; + + /// + /// Keep track of the timestamp of the last audio buffer (computed from the value reported to us by media foundation). + /// + private DateTime lastPostedAudioTime = DateTime.MinValue; + + /// + /// Whether to drop out of order packets from media foundation. + /// + private bool dropOutOfOrderPackets = false; + /// /// Initializes a new instance of the class. /// /// Pipeline this component is a part of. /// Name of media file to play. - public MediaSource(Pipeline pipeline, string filename) + /// Optional flag specifying whether to drop out of order packets (defaults to false). + public MediaSource(Pipeline pipeline, string filename, bool dropOutOfOrderPackets = false) + : this(pipeline, File.OpenRead(filename), new FileInfo(filename).CreationTime, dropOutOfOrderPackets) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Pipeline this component is a part of. + /// Source stream of the media to consume. + /// Optional date/time that the media started. + /// Optional flag specifying whether to drop out of order packets (defaults to false). + public MediaSource(Pipeline pipeline, Stream input, DateTime? startTime = null, bool dropOutOfOrderPackets = false) : base(pipeline) { - FileInfo info = new FileInfo(filename); - pipeline.ProposeReplayTime(new TimeInterval(info.CreationTime, DateTime.MaxValue)); - this.start = info.CreationTime; - this.filename = filename; + var proposedReplayTime = startTime ?? DateTime.Now; + pipeline.ProposeReplayTime(new TimeInterval(proposedReplayTime, DateTime.MaxValue)); + this.start = proposedReplayTime; + this.input = input; + this.dropOutOfOrderPackets = dropOutOfOrderPackets; this.Image = pipeline.CreateEmitter>(this, nameof(this.Image)); this.Audio = pipeline.CreateEmitter(this, nameof(this.Audio)); this.InitializeMediaPipeline(); @@ -62,6 +91,7 @@ public void Dispose() if (!this.disposed) { this.sourceReader.Dispose(); + this.input.Dispose(); MediaManager.Shutdown(); this.disposed = true; } @@ -89,17 +119,45 @@ protected override DateTime GenerateNext(DateTime currentTime) if (streamIndex == this.imageStreamIndex) { - using (var sharedImage = ImagePool.GetOrCreate(this.videoWidth, this.videoHeight, Imaging.PixelFormat.BGR_24bpp)) + // Detect out of order originating times + if (originatingTime > this.lastPostedImageTime) { - sharedImage.Resource.CopyFrom(data); - this.Image.Post(sharedImage, originatingTime); + using (var sharedImage = ImagePool.GetOrCreate(this.videoWidth, this.videoHeight, Imaging.PixelFormat.BGR_24bpp)) + { + sharedImage.Resource.CopyFrom(data); + this.Image.Post(sharedImage, originatingTime); + this.lastPostedImageTime = originatingTime; + } + } + else if (!this.dropOutOfOrderPackets) + { + throw new InvalidOperationException( + $"The most recently captured image frame has a timestamp ({originatingTime.TimeOfDay}) which is before " + + $"that of the last posted image frame ({this.lastPostedImageTime.TimeOfDay}), as reported by the video stream. This could " + + $"be due to a timing glitch in the video stream. Set the 'dropOutOfOrderPackets' " + + $"parameter to true to handle this condition by dropping " + + $"packets with out of order timestamps."); } } else if (streamIndex == this.audioStreamIndex) { - AudioBuffer audioBuffer = new AudioBuffer(currentByteCount, this.waveFormat); - Marshal.Copy(data, audioBuffer.Data, 0, currentByteCount); - this.Audio.Post(audioBuffer, originatingTime); + // Detect out of order originating times + if (originatingTime > this.lastPostedAudioTime) + { + AudioBuffer audioBuffer = new AudioBuffer(currentByteCount, this.waveFormat); + Marshal.Copy(data, audioBuffer.Data, 0, currentByteCount); + this.Audio.Post(audioBuffer, originatingTime); + this.lastPostedAudioTime = originatingTime; + } + else if (!this.dropOutOfOrderPackets) + { + throw new InvalidOperationException( + $"The most recently captured audio buffer has a timestamp ({originatingTime.TimeOfDay}) which is before " + + $"that of the last posted audio buffer ({this.lastPostedAudioTime.TimeOfDay}), as reported by the audio stream. This could " + + $"be due to a timing glitch in the audio stream. Set the 'dropOutOfOrderPackets' " + + $"parameter to true to handle this condition by dropping " + + $"packets with out of order timestamps."); + } } buffer.Unlock(); @@ -129,7 +187,7 @@ private void InitializeMediaPipeline() MediaManager.Startup(false); MediaAttributes sourceReaderAttributes = new MediaAttributes(); sourceReaderAttributes.Set(SourceReaderAttributeKeys.EnableAdvancedVideoProcessing, true); - this.sourceReader = new SourceReader(this.filename, sourceReaderAttributes); + this.sourceReader = new SourceReader(this.input, sourceReaderAttributes); this.sourceReader.SetStreamSelection(SourceReaderIndex.AllStreams, false); int streamIndex = 0; diff --git a/Sources/Media/Microsoft.Psi.Media.Windows.x64/Microsoft.Psi.Media.Windows.x64.csproj b/Sources/Media/Microsoft.Psi.Media.Windows.x64/Microsoft.Psi.Media.Windows.x64.csproj index 3032e923d..4cce06dae 100644 --- a/Sources/Media/Microsoft.Psi.Media.Windows.x64/Microsoft.Psi.Media.Windows.x64.csproj +++ b/Sources/Media/Microsoft.Psi.Media.Windows.x64/Microsoft.Psi.Media.Windows.x64.csproj @@ -35,6 +35,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Media/Microsoft.Psi.Media.Windows.x64/Mpeg4Writer.cs b/Sources/Media/Microsoft.Psi.Media.Windows.x64/Mpeg4Writer.cs index a076ba504..8d2263683 100644 --- a/Sources/Media/Microsoft.Psi.Media.Windows.x64/Mpeg4Writer.cs +++ b/Sources/Media/Microsoft.Psi.Media.Windows.x64/Mpeg4Writer.cs @@ -94,7 +94,7 @@ public void Dispose() if (this.writer != null) { this.writer.Close(); - this.writer.Dispose(); + ((IDisposable)this.writer).Dispose(); // Cast to IDisposable to suppress false CA2213 warning this.writer = null; MP4Writer.Shutdown(); } diff --git a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.cpp b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.cpp index 2ff61f88b..4c8b2af41 100644 --- a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.cpp +++ b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.cpp @@ -15,6 +15,6 @@ using namespace System::Security::Permissions; [assembly:AssemblyCopyrightAttribute("Copyright (c) Microsoft Corporation. All rights reserved.")]; [assembly:ComVisible(false)]; [assembly:CLSCompliantAttribute(true)]; -[assembly:AssemblyVersionAttribute("0.11.82.2")]; -[assembly:AssemblyFileVersionAttribute("0.11.82.2")]; -[assembly:AssemblyInformationalVersionAttribute("0.11.82.2-beta")]; +[assembly:AssemblyVersionAttribute("0.12.53.2")]; +[assembly:AssemblyFileVersionAttribute("0.12.53.2")]; +[assembly:AssemblyInformationalVersionAttribute("0.12.53.2-beta")]; diff --git a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.rc b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.rc index e79501fb0..0895273fe 100644 Binary files a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.rc and b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/AssemblyInfo.rc differ diff --git a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.cpp b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.cpp index c43be1d5e..64f73471c 100644 --- a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.cpp +++ b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.cpp @@ -36,7 +36,7 @@ namespace Microsoft { } //********************************************************************** - // Sets up our MF writer for handling audio. This code always generates + // Sets up our Media Foundation writer for handling audio. This code always generates // AAC for the audio output and assumes the audio input is always PCM //********************************************************************** HRESULT MP4WriterUnmanagedData::SetupAudio(UINT32 bitsPerSample, UINT32 samplesPerSecond, UINT32 numChannels) @@ -222,10 +222,10 @@ namespace Microsoft { dstCol = dstRow; for (UINT32 x = 0; x < outputWidth; x++) { - dstCol[3] = srcCol[0]; - dstCol[2] = srcCol[1]; - dstCol[1] = srcCol[2]; - dstCol[0] = srcCol[3]; + dstCol[0] = srcCol[0]; + dstCol[1] = srcCol[1]; + dstCol[2] = srcCol[2]; + dstCol[3] = srcCol[3]; srcCol += 4; dstCol += 4; } diff --git a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.h b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.h index 53967dcdf..dbc190d52 100644 --- a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.h +++ b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/MP4Writer.h @@ -102,11 +102,7 @@ namespace Microsoft { ~MP4Writer() { - if (unmanagedData != nullptr) - { - delete unmanagedData; - unmanagedData = nullptr; - } + Close(); } HRESULT Open(String ^fn, MP4WriterConfiguration^ config); diff --git a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/Microsoft.Psi.Media_Interop.Windows.x64.vcxproj b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/Microsoft.Psi.Media_Interop.Windows.x64.vcxproj index 282ee2e4d..56139a740 100644 --- a/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/Microsoft.Psi.Media_Interop.Windows.x64.vcxproj +++ b/Sources/Media/Microsoft.Psi.Media_Interop.Windows.x64/Microsoft.Psi.Media_Interop.Windows.x64.vcxproj @@ -34,7 +34,7 @@ true true Unicode - v141 + v142 Spectre @@ -42,7 +42,7 @@ false true Unicode - v141 + v142 Spectre diff --git a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Microsoft.Psi.RealSense.Windows.x64.csproj b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Microsoft.Psi.RealSense.Windows.x64.csproj index 70b579232..951795615 100644 --- a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Microsoft.Psi.RealSense.Windows.x64.csproj +++ b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Microsoft.Psi.RealSense.Windows.x64.csproj @@ -67,6 +67,11 @@ + + 2.9.8 + runtime; build; native; contentfiles; analyzers + all + 1.1.118 runtime; build; native; contentfiles; analyzers; buildtransitive @@ -77,13 +82,13 @@ - + - + $(BuildDependsOn) CheckVariable - + \ No newline at end of file diff --git a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Properties/AssemblyInfo.cs b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Properties/AssemblyInfo.cs index 7a20b5b59..f1aebde16 100644 --- a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Properties/AssemblyInfo.cs +++ b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/Properties/AssemblyInfo.cs @@ -35,6 +35,6 @@ // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] -[assembly: AssemblyVersion("0.11.82.2")] -[assembly: AssemblyFileVersion("0.11.82.2")] -[assembly: AssemblyInformationalVersion("0.11.82.2-beta")] +[assembly: AssemblyVersion("0.12.53.2")] +[assembly: AssemblyFileVersion("0.12.53.2")] +[assembly: AssemblyInformationalVersion("0.12.53.2-beta")] diff --git a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/RealSenseSensor.cs b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/RealSenseSensor.cs index 3511011f4..be2f4d903 100644 --- a/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/RealSenseSensor.cs +++ b/Sources/RealSense/Microsoft.Psi.RealSense.Windows.x64/RealSenseSensor.cs @@ -27,7 +27,7 @@ public RealSenseSensor(Pipeline pipeline) { this.shutdown = false; this.ColorImage = pipeline.CreateEmitter>(this, "ColorImage"); - this.DepthImage = pipeline.CreateEmitter>(this, "DepthImage"); + this.DepthImage = pipeline.CreateEmitter>(this, "DepthImage"); this.pipeline = pipeline; } @@ -39,7 +39,7 @@ public RealSenseSensor(Pipeline pipeline) /// /// Gets the emitter that generates Depth images from the RealSense depth camera. /// - public Emitter> DepthImage { get; private set; } + public Emitter> DepthImage { get; private set; } /// /// Dispose method. @@ -48,7 +48,7 @@ public void Dispose() { if (this.device != null) { - this.device.Dispose(); + ((IDisposable)this.device).Dispose(); this.device = null; } } @@ -65,7 +65,7 @@ public void Start(Action notifyCompletionTime) } /// - public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) + public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) { if (this.thread != null) { @@ -81,7 +81,7 @@ public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) { this.device = null; } - + notifyCompleted(); } @@ -114,7 +114,7 @@ private void ThreadProc() throw new NotSupportedException("Expected 8bpp or 16bpp image."); } - var depthImage = ImagePool.GetOrCreate((int)this.device.GetDepthWidth(), (int)this.device.GetDepthHeight(), pixelFormat); + var depthImage = DepthImagePool.GetOrCreate((int)this.device.GetDepthWidth(), (int)this.device.GetDepthHeight()); uint depthImageSize = this.device.GetDepthHeight() * this.device.GetDepthStride(); while (!this.shutdown) { diff --git a/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/AssemblyInfo.cpp b/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/AssemblyInfo.cpp index 1cc197e68..a7955a821 100644 --- a/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/AssemblyInfo.cpp +++ b/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/AssemblyInfo.cpp @@ -31,9 +31,9 @@ using namespace System::Security::Permissions; // You can specify all the value or you can default the Revision and Build Numbers // by using the '*' as shown below: -[assembly:AssemblyVersionAttribute("0.11.82.2")]; -[assembly:AssemblyFileVersionAttribute("0.11.82.2")]; -[assembly:AssemblyInformationalVersionAttribute("0.11.82.2-beta")]; +[assembly:AssemblyVersionAttribute("0.12.53.2")]; +[assembly:AssemblyFileVersionAttribute("0.12.53.2")]; +[assembly:AssemblyInformationalVersionAttribute("0.12.53.2-beta")]; [assembly:ComVisible(false)]; diff --git a/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/Microsoft.Psi.RealSense_Interop.Windows.x64.vcxproj b/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/Microsoft.Psi.RealSense_Interop.Windows.x64.vcxproj index abeb422d5..df033293d 100644 --- a/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/Microsoft.Psi.RealSense_Interop.Windows.x64.vcxproj +++ b/Sources/RealSense/Microsoft.Psi.RealSense_Interop.Windows.x64/Microsoft.Psi.RealSense_Interop.Windows.x64.vcxproj @@ -23,7 +23,7 @@ DynamicLibrary true - v141 + v142 true Unicode Spectre @@ -31,7 +31,7 @@ DynamicLibrary false - v141 + v142 true Unicode Spectre @@ -126,13 +126,13 @@ - + - + $(BuildDependsOn) CheckVariable - + \ No newline at end of file diff --git a/Sources/Runtime/Microsoft.Psi.Interop/Microsoft.Psi.Interop.csproj b/Sources/Runtime/Microsoft.Psi.Interop/Microsoft.Psi.Interop.csproj index 22905777b..c004c4760 100644 --- a/Sources/Runtime/Microsoft.Psi.Interop/Microsoft.Psi.Interop.csproj +++ b/Sources/Runtime/Microsoft.Psi.Interop/Microsoft.Psi.Interop.csproj @@ -26,7 +26,11 @@ - + + + all + runtime; build; native; contentfiles; analyzers + @@ -36,7 +40,7 @@ - + diff --git a/Sources/Runtime/Microsoft.Psi.Windows/Microsoft.Psi.Windows.csproj b/Sources/Runtime/Microsoft.Psi.Windows/Microsoft.Psi.Windows.csproj index 75a0c99be..c1c9be833 100644 --- a/Sources/Runtime/Microsoft.Psi.Windows/Microsoft.Psi.Windows.csproj +++ b/Sources/Runtime/Microsoft.Psi.Windows/Microsoft.Psi.Windows.csproj @@ -28,6 +28,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Sources/Runtime/Microsoft.Psi.Windows/PerfCounterCollection.cs b/Sources/Runtime/Microsoft.Psi.Windows/PerfCounterCollection.cs index d882fc41b..061a67f96 100644 --- a/Sources/Runtime/Microsoft.Psi.Windows/PerfCounterCollection.cs +++ b/Sources/Runtime/Microsoft.Psi.Windows/PerfCounterCollection.cs @@ -7,7 +7,7 @@ namespace Microsoft.Psi using System.Diagnostics; /// - /// Performacne counter collection. + /// Performance counter collection. /// /// Performance counter key type. public class PerfCounterCollection : IPerfCounterCollection diff --git a/Sources/Runtime/Microsoft.Psi.Windows/PerfCounters.cs b/Sources/Runtime/Microsoft.Psi.Windows/PerfCounters.cs index 3d65c9f10..c5e077edd 100644 --- a/Sources/Runtime/Microsoft.Psi.Windows/PerfCounters.cs +++ b/Sources/Runtime/Microsoft.Psi.Windows/PerfCounters.cs @@ -26,7 +26,7 @@ public IPerfCounterCollection Enable(string category, string instance) } /// - /// Add performance counter defintions. + /// Add performance counter definitions. /// /// Category name. /// Performance counter definitions (key, name, help, type). diff --git a/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexDefinition.cs b/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexDefinition.cs index 89cc0c050..72e045d73 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexDefinition.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexDefinition.cs @@ -80,7 +80,7 @@ public IndexDefinition(int count, int elementStride) public IndexDefinition Slice(int start, int end) => this.Slice(new Range(start, end)); /// - /// Merges two index definitions into one dicontinuous index. The two are assumed to belong to the same dimension. + /// Merges two index definitions into one discontiguous index. The two are assumed to belong to the same dimension. /// /// The other definition. /// A combined definition. diff --git a/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexerNd.cs b/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexerNd.cs index fa4e0c5c5..54f940559 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexerNd.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Arrays/IndexerNd.cs @@ -112,7 +112,7 @@ public override IEnumerable Values { if (indices.Length != this.dimensions.Length) { - throw new ArgumentException("Invalid number of dimmensions. Expected " + this.dimensions.Length); + throw new ArgumentException("Invalid number of dimensions. Expected " + this.dimensions.Length); } int index = 0; diff --git a/Sources/Runtime/Microsoft.Psi/Common/Arrays/RangeIndexDefinition.cs b/Sources/Runtime/Microsoft.Psi/Common/Arrays/RangeIndexDefinition.cs index fefad281d..8b7961fa7 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Arrays/RangeIndexDefinition.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Arrays/RangeIndexDefinition.cs @@ -8,7 +8,7 @@ namespace Microsoft.Psi.Arrays using System.Linq; /// - /// Defines the possible values of an index as a continguous range. + /// Defines the possible values of an index as a contiguous range. /// internal class RangeIndexDefinition : IndexDefinition { diff --git a/Sources/Runtime/Microsoft.Psi/Common/BufferWriter.cs b/Sources/Runtime/Microsoft.Psi/Common/BufferWriter.cs index e2af1865c..b662589e7 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/BufferWriter.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/BufferWriter.cs @@ -86,7 +86,7 @@ public unsafe void Write(void* source, int lenghtInBytes) fixed (byte* buf = this.buffer) { - // more efficient then Array.Copy or cpblk IL instruction because it handles small sizes explicitly + // more efficient than Array.Copy or cpblk IL instruction because it handles small sizes explicitly // http://referencesource.microsoft.com/#mscorlib/system/buffer.cs,c2ca91c0d34a8f86 System.Buffer.MemoryCopy(source, buf + start, this.buffer.Length - start, lenghtInBytes); } diff --git a/Sources/Runtime/Microsoft.Psi/Common/Envelope.cs b/Sources/Runtime/Microsoft.Psi/Common/Envelope.cs index 5466a5033..3fb5d1442 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Envelope.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Envelope.cs @@ -17,7 +17,7 @@ public struct Envelope public int SourceId; /// - /// The sequence number of this message, unique within the stream idetified by . + /// The sequence number of this message, unique within the stream identified by . /// public int SequenceId; diff --git a/Sources/Runtime/Microsoft.Psi/Common/IStreamMetadata.cs b/Sources/Runtime/Microsoft.Psi/Common/IStreamMetadata.cs index 1aca3e937..ecbc16063 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/IStreamMetadata.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/IStreamMetadata.cs @@ -26,12 +26,17 @@ public interface IStreamMetadata string TypeName { get; } /// - /// Gets the name of the partation where the stream is stored. + /// Gets the name of the type of supplemental metadata for the stream the metadata represents. + /// + string SupplementalMetadataTypeName { get; } + + /// + /// Gets the name of the partition where the stream is stored. /// string PartitionName { get; } /// - /// Gets the path of the partation where the stream is stored. + /// Gets the path of the partition where the stream is stored. /// string PartitionPath { get; } diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/AdjacentValuesInterpolator.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/AdjacentValuesInterpolator.cs index 380b678fa..30f07ca82 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/AdjacentValuesInterpolator.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/AdjacentValuesInterpolator.cs @@ -16,7 +16,7 @@ namespace Microsoft.Psi.Common.Interpolators /// The interpolator results do not depend on the wall-clock time of the messages arriving /// on the secondary stream, i.e., they are based on originating times of messages. As a result, /// the interpolator might introduce an extra delay as it might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public class AdjacentValuesInterpolator : ReproducibleInterpolator { diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/FirstReproducibleInterpolator.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/FirstReproducibleInterpolator.cs index 784a6b309..19b8bbe7e 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/FirstReproducibleInterpolator.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/FirstReproducibleInterpolator.cs @@ -14,7 +14,7 @@ namespace Microsoft.Psi.Common.Interpolators /// The interpolator results do not depend on the wall-clock time of the messages arriving /// on the secondary stream, i.e., they are based on originating times of messages. As a result, /// the interpolator might introduce an extra delay as it might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public sealed class FirstReproducibleInterpolator : ReproducibleInterpolator { diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/LastReproducibleInterpolator.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/LastReproducibleInterpolator.cs index 980e20b83..0e883da87 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/LastReproducibleInterpolator.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/LastReproducibleInterpolator.cs @@ -14,7 +14,7 @@ namespace Microsoft.Psi.Common.Interpolators /// The interpolator results do not depend on the wall-clock time of the messages arriving /// on the secondary stream, i.e., they are based on originating times of messages. As a result, /// the interpolator might introduce an extra delay as it might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public sealed class LastReproducibleInterpolator : ReproducibleInterpolator { @@ -57,7 +57,7 @@ public override InterpolationResult Interpolate(DateTime interpolationTime, I } } - // Look for the last match that's stil within the window + // Look for the last match that's still within the window Message lastMatchingMessage = default; bool found = false; @@ -116,8 +116,8 @@ public override InterpolationResult Interpolate(DateTime interpolationTime, I } } - // If we arrive here, it means we did not find a last message in the lookup window, - // which also means there is no message in the loopkup window. + // If we arrive here, it means we did not find a last message in the lookback window, + // which also means there is no message in the lookback window. // If the stream was closed or last message is at or past the upper search bound, if (closedOriginatingTime.HasValue || lastMessageIsOutsideWindowEnd) diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/NearestReproducibleInterpolator.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/NearestReproducibleInterpolator.cs index 84b49a0b0..1a82c3112 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/NearestReproducibleInterpolator.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/NearestReproducibleInterpolator.cs @@ -14,7 +14,7 @@ namespace Microsoft.Psi.Common.Interpolators /// The interpolator results do not depend on the wall-clock time of the messages arriving /// on the secondary stream, i.e., they are based on originating times of messages. As a result, /// the interpolator might introduce an extra delay as it might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public sealed class NearestReproducibleInterpolator : ReproducibleInterpolator { diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{TIn,TResult}.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{TIn,TResult}.cs index 86ac0f664..b11171e9c 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{TIn,TResult}.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{TIn,TResult}.cs @@ -11,7 +11,7 @@ namespace Microsoft.Psi /// Reproducible interpolators produce results that do not depend on the wall-clock time of /// message arrival on a stream, i.e., they are based on originating times of messages. As a result, /// these interpolators might introduce extra delays as they might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public abstract class ReproducibleInterpolator : Interpolator { diff --git a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{T}.cs b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{T}.cs index df269e454..09ce661da 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{T}.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Interpolators/ReproducibleInterpolator{T}.cs @@ -12,7 +12,7 @@ namespace Microsoft.Psi /// Reproducible interpolators produce results that do not depend on the wall-clock time of /// message arrival on a stream, i.e., they are based on originating times of messages. As a result, /// these interpolators might introduce extra delays as they might have to wait for enough messages on the - /// secondary stream to proove that the interpolation result is correct, irrespective of any other messages + /// secondary stream to prove that the interpolation result is correct, irrespective of any other messages /// that might arrive later. public abstract class ReproducibleInterpolator : ReproducibleInterpolator { @@ -26,7 +26,7 @@ public abstract class ReproducibleInterpolator : ReproducibleInterpolator - /// Implicitly convert timespan to the equivalent of a reproducibla nearest match with that tolerance. + /// Implicitly convert timespan to the equivalent of a reproducible nearest match with that tolerance. /// /// Relative window tolerance within which to match messages. public static implicit operator ReproducibleInterpolator(TimeSpan tolerance) diff --git a/Sources/Runtime/Microsoft.Psi/Common/Intervals/IInterval.cs b/Sources/Runtime/Microsoft.Psi/Common/Intervals/IInterval.cs index 390b949be..bea1458b5 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Intervals/IInterval.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Intervals/IInterval.cs @@ -106,16 +106,16 @@ public interface IInterval /// /// Scale from left point by a span distance. /// - /// Span by whith to scale left. - /// Span by whith to scale right. + /// Span by which to scale left. + /// Span by which to scale right. /// Scaled interval. T Scale(TSpan left, TSpan right); /// /// Scale from left point by a factor. /// - /// Factor by whith to scale left. - /// Factor by whith to scale right. + /// Factor by which to scale left. + /// Factor by which to scale right. /// Scaled interval. T Scale(float left, float right); diff --git a/Sources/Runtime/Microsoft.Psi/Common/Metadata.cs b/Sources/Runtime/Microsoft.Psi/Common/Metadata.cs index b1673d275..178127a3e 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Metadata.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Metadata.cs @@ -22,7 +22,7 @@ public enum MetadataKind : ushort RuntimeInfo = 1, /// - /// Metadata usied in storing the schema definitions used when serializing and deserializing a type in a Psi Store. + /// Metadata using in storing the schema definitions used when serializing and deserializing a type in a Psi Store. /// TypeSchema = 2, } @@ -59,7 +59,7 @@ internal Metadata() public int Id { get; protected set; } /// - /// Gets or sets the name of the type of data conatined in the object the metadata represents. + /// Gets or sets the name of the type of data contained in the object the metadata represents. /// public string TypeName { get; protected set; } @@ -74,7 +74,7 @@ internal Metadata() public int Version { get; protected set; } /// - /// Gets or sets the metadata serializer verson number. + /// Gets or sets the metadata serializer version number. /// public int SerializerVersion { get; protected set; } diff --git a/Sources/Runtime/Microsoft.Psi/Common/Platform.cs b/Sources/Runtime/Microsoft.Psi/Common/Platform.cs index 7182f3a4b..48139e21a 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Platform.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Platform.cs @@ -92,22 +92,21 @@ internal sealed class Timer : ITimer public Timer(uint delay, Time.TimerDelegate handler, bool periodic) { uint ignore = 0; - this.id = TimeSetEvent(delay, 0, handler, ref ignore, (uint)(periodic ? 1 : 0)); + uint eventType = (uint)(periodic ? 1 : 0); + + // TIME_KILL_SYNCHRONOUS flag prevents timer event from occurring after timeKillEvent is called + eventType |= 0x100; + + this.id = NativeMethods.TimeSetEvent(delay, 0, handler, ref ignore, eventType); } public void Stop() { - if (TimeKillEvent(this.id) != 0) + if (NativeMethods.TimeKillEvent(this.id) != 0) { throw new ArgumentException("Invalid timer event ID."); } } - - [DllImport("winmm.dll", SetLastError = true, EntryPoint = "timeSetEvent")] - private static extern uint TimeSetEvent(uint delay, uint resolution, Time.TimerDelegate handler, ref uint userCtx, uint eventType); - - [DllImport("winmm.dll", SetLastError = true, EntryPoint = "timeKillEvent")] - private static extern uint TimeKillEvent(uint timerEventId); } internal sealed class HighResolutionTime : IHighResolutionTime @@ -115,7 +114,7 @@ internal sealed class HighResolutionTime : IHighResolutionTime public long TimeStamp() { long time; - if (!QueryPerformanceCounter(out time)) + if (!NativeMethods.QueryPerformanceCounter(out time)) { throw new NotImplementedException("QueryPerformanceCounter failed (supported on Windows XP and later)"); } @@ -126,7 +125,7 @@ public long TimeStamp() public long TimeFrequency() { long freq; - if (!QueryPerformanceFrequency(out freq)) + if (!NativeMethods.QueryPerformanceFrequency(out freq)) { throw new NotImplementedException("QueryPerformanceFrequency failed (supported on Windows XP and later)"); } @@ -137,7 +136,7 @@ public long TimeFrequency() public long SystemTime() { long time; - GetSystemTimePreciseAsFileTime(out time); + NativeMethods.GetSystemTimePreciseAsFileTime(out time); return time; } @@ -145,15 +144,6 @@ public ITimer TimerStart(uint delay, Time.TimerDelegate handler, bool periodic) { return new Timer(delay, handler, periodic); } - - [DllImport("kernel32.dll")] - private static extern void GetSystemTimePreciseAsFileTime(out long systemTimeAsFileTime); - - [DllImport("kernel32.dll")] - private static extern bool QueryPerformanceCounter(out long performanceCount); - - [DllImport("kernel32.dll")] - private static extern bool QueryPerformanceFrequency(out long frequency); } internal sealed class Threading : IThreading @@ -163,12 +153,31 @@ public void SetApartmentState(Thread thread, ApartmentState state) thread.SetApartmentState(state); } } + + private static class NativeMethods + { + [DllImport("winmm.dll", SetLastError = true, EntryPoint = "timeSetEvent")] + internal static extern uint TimeSetEvent(uint delay, uint resolution, Time.TimerDelegate handler, ref uint userCtx, uint eventType); + + [DllImport("winmm.dll", SetLastError = true, EntryPoint = "timeKillEvent")] + internal static extern uint TimeKillEvent(uint timerEventId); + + [DllImport("kernel32.dll")] + internal static extern void GetSystemTimePreciseAsFileTime(out long systemTimeAsFileTime); + + [DllImport("kernel32.dll")] + internal static extern bool QueryPerformanceCounter(out long performanceCount); + + [DllImport("kernel32.dll")] + internal static extern bool QueryPerformanceFrequency(out long frequency); + } } private static class Standard // Linux, Mac, ... { - internal sealed class Timer : ITimer + internal sealed class Timer : ITimer, IDisposable { + private readonly object timerDelegateLock = new object(); private System.Timers.Timer timer; public Timer(uint delay, Time.TimerDelegate handler, bool periodic) @@ -177,18 +186,34 @@ public Timer(uint delay, Time.TimerDelegate handler, bool periodic) this.timer.AutoReset = periodic; this.timer.Elapsed += (sender, args) => { - lock (this.timer) + lock (this.timerDelegateLock) { - handler(0, 0, UIntPtr.Zero, UIntPtr.Zero, UIntPtr.Zero); + // prevents handler from being called if timer has been stopped + if (this.timer != null) + { + handler(0, 0, UIntPtr.Zero, UIntPtr.Zero, UIntPtr.Zero); + } } }; this.timer.Start(); } + public void Dispose() + { + this.Stop(); + } + public void Stop() { - this.timer?.Stop(); - this.timer?.Dispose(); + if (this.timer != null) + { + lock (this.timerDelegateLock) + { + this.timer.Stop(); + this.timer.Dispose(); + this.timer = null; + } + } } private void Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e) diff --git a/Sources/Runtime/Microsoft.Psi/Common/PsiStreamMetadata.cs b/Sources/Runtime/Microsoft.Psi/Common/PsiStreamMetadata.cs index d644c5301..c7f40d008 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/PsiStreamMetadata.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/PsiStreamMetadata.cs @@ -6,6 +6,7 @@ namespace Microsoft.Psi using System; using System.Collections.Generic; using Microsoft.Psi.Common; + using Microsoft.Psi.Serialization; /// /// Specifies custom flags for Psi data streams. @@ -39,15 +40,17 @@ public enum StreamMetadataFlags : ushort /// public sealed class PsiStreamMetadata : Metadata, IStreamMetadata { + private const int CurrentVersion = 1; private const int TicksPerMicrosecond = 10; + private byte[] supplementalMetadataBytes = Array.Empty(); internal PsiStreamMetadata(string name, int id, string typeName) - : base(MetadataKind.StreamMetadata, name, id, typeName, 0, null, 0, 0) + : base(MetadataKind.StreamMetadata, name, id, typeName, CurrentVersion, null, 0, 0) { } internal PsiStreamMetadata(string name, int id, string typeName, int version, string serializerTypeName, int serializerVersion, ushort customFlags) - : base(MetadataKind.StreamMetadata, name, id, typeName, version, serializerTypeName, serializerVersion, customFlags) + : base(MetadataKind.StreamMetadata, name, id, typeName, version, serializerTypeName, serializerVersion, customFlags) { } @@ -141,6 +144,9 @@ internal set } } + /// + public string SupplementalMetadataTypeName { get; private set; } + /// /// Gets the average frequency of messages written to this stream. /// @@ -190,6 +196,58 @@ public void Update(TimeInterval messagesTimeInterval, TimeInterval messagesOrigi this.LastMessageOriginatingTime = messagesOriginatingTimeInterval.Right; } + /// + /// Gets supplemental stream metadata. + /// + /// Type of supplemental metadata. + /// Known serializers. + /// Supplemental metadata. + internal T GetSupplementalMetadata(KnownSerializers serializers) + { + if (string.IsNullOrEmpty(this.SupplementalMetadataTypeName)) + { + throw new InvalidOperationException("Stream does not contain supplemental metadata."); + } + + if (typeof(T) != Type.GetType(this.SupplementalMetadataTypeName)) + { + throw new InvalidCastException($"Supplemental metadata type mismatch ({this.SupplementalMetadataTypeName})."); + } + + var handler = serializers.GetHandler(); + var reader = new BufferReader(this.supplementalMetadataBytes); + var target = default(T); + handler.Deserialize(reader, ref target, new SerializationContext(serializers)); + return target; + } + + /// + /// Sets supplemental stream metadata. + /// + /// Type of supplemental metadata. + /// Supplemental metadata value. + /// Known serializers. + internal void SetSupplementalMetadata(T value, KnownSerializers serializers) + { + this.SupplementalMetadataTypeName = typeof(T).AssemblyQualifiedName; + var handler = serializers.GetHandler(); + var writer = new BufferWriter(this.supplementalMetadataBytes); + handler.Serialize(writer, value, new SerializationContext(serializers)); + this.supplementalMetadataBytes = writer.Buffer; + } + + /// + /// Update supplemental stream metadata from another stream metadata. + /// + /// Other stream metadata from which to copy supplemental metadata. + /// Updated stream metadata. + internal PsiStreamMetadata UpdateSupplementalMetadataFrom(PsiStreamMetadata other) + { + this.SupplementalMetadataTypeName = other.SupplementalMetadataTypeName; + this.supplementalMetadataBytes = other.supplementalMetadataBytes; + return this; + } + // custom deserializer with no dependency on the Serializer subsystem // order of fields is important for backwards compat and must be the same as the order in Serialize, don't change! internal new void Deserialize(BufferReader metadataBuffer) @@ -212,6 +270,14 @@ internal new void Deserialize(BufferReader metadataBuffer) this.RuntimeTypes.Add(metadataBuffer.ReadInt32(), metadataBuffer.ReadString()); } } + + if (this.Version >= 1) + { + this.SupplementalMetadataTypeName = metadataBuffer.ReadString(); + var len = metadataBuffer.ReadInt32(); + this.supplementalMetadataBytes = new byte[len]; + metadataBuffer.Read(this.supplementalMetadataBytes, len); + } } internal override void Serialize(BufferWriter metadataBuffer) @@ -235,6 +301,13 @@ internal override void Serialize(BufferWriter metadataBuffer) metadataBuffer.Write(pair.Value); } } + + metadataBuffer.Write(this.SupplementalMetadataTypeName); + metadataBuffer.Write(this.supplementalMetadataBytes.Length); + if (this.supplementalMetadataBytes.Length > 0) + { + metadataBuffer.Write(this.supplementalMetadataBytes); + } } private bool GetFlag(StreamMetadataFlags smflag) diff --git a/Sources/Runtime/Microsoft.Psi/Common/Shared.cs b/Sources/Runtime/Microsoft.Psi/Common/Shared.cs index cf2500ef9..8c2b2c178 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Shared.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Shared.cs @@ -153,7 +153,9 @@ public void Dispose() e.AddHistory("Instance created", this.constructorStackTrace); e.AddHistory("Instance disposed", this.disposeStackTrace); #endif +#pragma warning disable CA1065 // Do not raise exceptions in unexpected locations throw e; +#pragma warning restore CA1065 // Do not raise exceptions in unexpected locations } #if TRACKLEAKS else @@ -213,7 +215,7 @@ private class CustomSerializer : ISerializer> public TypeSchema Initialize(KnownSerializers serializers, TypeSchema targetSchema) { this.handler = serializers.GetHandler>(); - var type = this.GetType(); + var type = typeof(Shared); var name = TypeSchema.GetContractName(type, serializers.RuntimeVersion); var innerMember = new TypeMemberSchema("inner", typeof(SharedContainer).AssemblyQualifiedName, true); var schema = new TypeSchema(name, TypeSchema.GetId(name), type.AssemblyQualifiedName, TypeFlags.IsClass, new TypeMemberSchema[] { innerMember }, Version); diff --git a/Sources/Runtime/Microsoft.Psi/Common/SharedContainer.cs b/Sources/Runtime/Microsoft.Psi/Common/SharedContainer.cs index 374a1089c..e2bf60cc5 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/SharedContainer.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/SharedContainer.cs @@ -77,7 +77,7 @@ private class CustomSerializer : ISerializer> public TypeSchema Initialize(KnownSerializers serializers, TypeSchema targetSchema) { this.handler = serializers.GetHandler(); - var type = this.GetType(); + var type = typeof(SharedContainer); var name = TypeSchema.GetContractName(type, serializers.RuntimeVersion); var resourceMember = new TypeMemberSchema("resource", typeof(T).AssemblyQualifiedName, true); var schema = new TypeSchema(name, TypeSchema.GetId(name), type.AssemblyQualifiedName, TypeFlags.IsClass, new TypeMemberSchema[] { resourceMember }, Version); diff --git a/Sources/Runtime/Microsoft.Psi/Common/TickCalibration.cs b/Sources/Runtime/Microsoft.Psi/Common/TickCalibration.cs index caa74b2d9..54ed54a91 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/TickCalibration.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/TickCalibration.cs @@ -130,7 +130,7 @@ public long ConvertToFileTime(long ticks, bool recalibrate = true) /// Attempts to recalibrate elapsed ticks against the system time. The current elapsed ticks from /// the performance counter will be compared against the current system time and the calibration /// data will be modified only if it is determined that the times have drifted by more than the - /// maxmimum allowed amount since the last calibration. + /// maximum allowed amount since the last calibration. /// /// Forces the calibration data to be modified regardless of the observed drift. internal void Recalibrate(bool force = false) diff --git a/Sources/Runtime/Microsoft.Psi/Common/Time.cs b/Sources/Runtime/Microsoft.Psi/Common/Time.cs index b4581478d..0a5bda3f5 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/Time.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/Time.cs @@ -51,7 +51,7 @@ internal static DateTime GetTime(TimeSpan offset) /// /// Returns the system UTC time represented by the number of 100ns ticks from system boot. /// The tick counter is calibrated against system time to a precision that is determined - /// by the tickSyncPrecision argument of the constuctor + /// by the tickSyncPrecision argument of the constructor /// (1 microsecond by default). To account for OS system clock adjustments which may cause /// the tick counter to drift relative to the system clock, the calibration is repeated /// whenever the drift exceeds a predefined maximum (1 millisecond by default). diff --git a/Sources/Runtime/Microsoft.Psi/Common/TypeResolutionHelper.cs b/Sources/Runtime/Microsoft.Psi/Common/TypeResolutionHelper.cs index 18c933206..1f21c2a12 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/TypeResolutionHelper.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/TypeResolutionHelper.cs @@ -6,6 +6,7 @@ namespace Microsoft.Psi using System; using System.Linq; using System.Reflection; + using System.Text.RegularExpressions; /// /// Helper class for type resolution. @@ -23,9 +24,25 @@ public static Type GetVerifiedType(string typeName) return Type.GetType(typeName, AssemblyResolver, null); } + /// + /// Removes the assembly name from an assembly-qualified type name, returning the fully + /// qualified name of the type, including its namespace but not the assembly name. + /// + /// A string representing the assembly-qualified name of a type. + /// The fully qualified name of the type, including its namespace but not the assembly name. + internal static string RemoveAssemblyName(string assemblyQualifiedName) + { + string typeName = assemblyQualifiedName; + + // strip out all assembly names (including in nested type parameters) + typeName = Regex.Replace(typeName, @",\s[^,\[\]\*]+", string.Empty); + + return typeName; + } + private static Assembly AssemblyResolver(AssemblyName assemblyName) { - // Get the list of currently loaded asemblies + // Get the list of currently loaded assemblies Assembly[] loadedAssemblies = AppDomain.CurrentDomain.GetAssemblies(); // Attempt to match by full name first diff --git a/Sources/Runtime/Microsoft.Psi/Common/UnmanagedArray.cs b/Sources/Runtime/Microsoft.Psi/Common/UnmanagedArray.cs index b097b4d4d..976ec8865 100644 --- a/Sources/Runtime/Microsoft.Psi/Common/UnmanagedArray.cs +++ b/Sources/Runtime/Microsoft.Psi/Common/UnmanagedArray.cs @@ -86,7 +86,7 @@ public UnmanagedArray(int length, bool zeroMemory = true) /// /// Gets the number of elements in the array. /// - int ICollection.Count => this.length; + public int Count => this.length; /// /// Gets the size of the allocated memory, in bytes. @@ -96,7 +96,7 @@ public UnmanagedArray(int length, bool zeroMemory = true) /// /// Gets a value indicating whether the array can be modified or not. /// - bool ICollection.IsReadOnly => this.isReadOnly; + public bool IsReadOnly => this.isReadOnly; /// /// Gets or sets the value of the element at the specified index. @@ -110,7 +110,7 @@ public UnmanagedArray(int length, bool zeroMemory = true) { if (index < 0 || index > this.length) { - throw new IndexOutOfRangeException(); + throw new ArgumentException(); } return MemoryAccess.ReadValue(this.data + (index * ElementSize)); @@ -458,7 +458,7 @@ bool ICollection.Remove(T item) } /// - IEnumerator IEnumerable.GetEnumerator() + public IEnumerator GetEnumerator() { return new UnmanagedEnumerator(this); } diff --git a/Sources/Runtime/Microsoft.Psi/Components/Generator{T}.cs b/Sources/Runtime/Microsoft.Psi/Components/Generator{T}.cs index 1131621d3..211437eba 100644 --- a/Sources/Runtime/Microsoft.Psi/Components/Generator{T}.cs +++ b/Sources/Runtime/Microsoft.Psi/Components/Generator{T}.cs @@ -16,7 +16,7 @@ namespace Microsoft.Psi.Components /// The static functions provided by the wrap /// and are designed to make the common cases easier. /// - public class Generator : Generator, IProducer + public class Generator : Generator, IProducer, IDisposable { private readonly Enumerator enumerator; @@ -26,7 +26,7 @@ public class Generator : Generator, IProducer /// The pipeline to attach to. /// A lazy enumerator of data. /// The interval used to increment time on each generated message. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with the specified time. /// If true, mark this Generator instance as representing an infinite source (e.g., a live-running sensor). /// If false (default), it represents a finite source (e.g., Generating messages based on a finite file or IEnumerable). @@ -65,6 +65,12 @@ public Generator(Pipeline pipeline, IEnumerator<(T, DateTime)> enumerator, DateT /// public Emitter Out { get; } + /// + public void Dispose() + { + this.enumerator?.Dispose(); + } + /// /// Called to generate the next value. /// diff --git a/Sources/Runtime/Microsoft.Psi/Components/Processor.cs b/Sources/Runtime/Microsoft.Psi/Components/Processor.cs index 462ec3e0d..e7fbbbb4e 100644 --- a/Sources/Runtime/Microsoft.Psi/Components/Processor.cs +++ b/Sources/Runtime/Microsoft.Psi/Components/Processor.cs @@ -32,7 +32,7 @@ public Processor(Pipeline pipeline, Action> transfo } /// - /// Override this method to process the incomming message and potentially publish one or more output messages. + /// Override this method to process the incoming message and potentially publish one or more output messages. /// The input message payload is only valid for the duration of the call. /// If the data needs to be stored beyond the scope of this method, /// use the extension method to create a private copy. diff --git a/Sources/Runtime/Microsoft.Psi/Components/RelativeTimeWindow.cs b/Sources/Runtime/Microsoft.Psi/Components/RelativeTimeWindow.cs index a66ef58c7..6faea24ec 100644 --- a/Sources/Runtime/Microsoft.Psi/Components/RelativeTimeWindow.cs +++ b/Sources/Runtime/Microsoft.Psi/Components/RelativeTimeWindow.cs @@ -21,7 +21,6 @@ public class RelativeTimeWindow : ConsumerProducer /// Initializes a new instance of the class. @@ -29,60 +28,29 @@ public class RelativeTimeWindow : ConsumerProducerPipeline to which this component belongs. /// The relative time interval over which to gather messages. /// Select output message from collected window of input messages. - /// Whether to wait for seeing a complete window before computing the selector function and posting the first time. True by default. - public RelativeTimeWindow(Pipeline pipeline, RelativeTimeInterval relativeTimeInterval, Func>, TOutput> selector, bool waitForCompleteWindow = true) + public RelativeTimeWindow(Pipeline pipeline, RelativeTimeInterval relativeTimeInterval, Func>, TOutput> selector) : base(pipeline) { this.relativeTimeInterval = relativeTimeInterval; this.selector = selector; this.In.Unsubscribed += _ => this.OnUnsubscribed(); - - // If false, processing should begin immediately without waiting for full window length. - this.initialBuffer = waitForCompleteWindow; } /// protected override void Receive(TInput value, Envelope envelope) { - // clone and add the new message - var message = new Message(value, envelope).DeepClone(this.recycler); - - // time-based - if (this.initialBuffer) - { - var clone = this.buffer.DeepClone(); - this.buffer.Enqueue(message); - if (this.CheckRemoval()) - { - this.initialBuffer = false; - this.ProcessWindow(clone, false, this.Out); - this.ProcessRemoval(); - this.ProcessWindow(this.buffer, false, this.Out); - } - } - else - { - this.buffer.Enqueue(message); - this.ProcessRemoval(); - this.ProcessWindow(this.buffer, false, this.Out); - } + this.buffer.Enqueue(new Message(value, envelope).DeepClone(this.recycler)); + this.ProcessRemoval(); + this.ProcessWindow(this.buffer, false, this.Out); } private bool RemoveCondition(Message message) { - if (this.initialBuffer) - { - return message.OriginatingTime < (this.buffer.Last().OriginatingTime + this.relativeTimeInterval).Left; - } - else - { - return this.anchorMessageOriginatingTime > DateTime.MinValue && message.OriginatingTime < (this.anchorMessageOriginatingTime + this.relativeTimeInterval).Left; - } + return this.anchorMessageOriginatingTime > DateTime.MinValue && message.OriginatingTime < (this.anchorMessageOriginatingTime + this.relativeTimeInterval).Left; } private void ProcessWindow(IEnumerable> messageList, bool final, Emitter emitter) { - this.initialBuffer = false; var messages = messageList.ToArray(); var anchorMessageIndex = 0; if (this.anchorMessageSequenceId >= 0) diff --git a/Sources/Runtime/Microsoft.Psi/Components/Zip.cs b/Sources/Runtime/Microsoft.Psi/Components/Zip.cs index c23da190b..41ace4839 100644 --- a/Sources/Runtime/Microsoft.Psi/Components/Zip.cs +++ b/Sources/Runtime/Microsoft.Psi/Components/Zip.cs @@ -3,16 +3,17 @@ namespace Microsoft.Psi.Components { - using System; using System.Collections.Generic; using System.Linq; /// - /// Zip one or more streams (T) into a single stream (Message{T}) while ensuring delivery in originating time order (ordered within single tick by stream ID). + /// Zip one or more streams (T) into a single stream while ensuring delivery in originating time order. /// - /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// If multiple messages arrive with the same originating time, they are added in the output array in + /// the order of stream ids. /// The type of the messages. - public class Zip : IProducer> + public class Zip : IProducer { private readonly Pipeline pipeline; private readonly IList> inputs = new List>(); @@ -25,13 +26,13 @@ public class Zip : IProducer> public Zip(Pipeline pipeline) { this.pipeline = pipeline; - this.Out = pipeline.CreateEmitter>(this, nameof(this.Out)); + this.Out = pipeline.CreateEmitter(this, nameof(this.Out)); } /// /// Gets the output emitter. /// - public Emitter> Out { get; } + public Emitter Out { get; } /// /// Add input receiver. @@ -57,35 +58,33 @@ public Receiver AddInput(string name) } } - private void Receive(T clonedData, Envelope e, IRecyclingPool recycler) + private void Receive(T clonedData, Envelope envelope, IRecyclingPool recycler) { - this.buffer.Add((data: clonedData, envelope: e, recycler: recycler)); + this.buffer.Add((data: clonedData, envelope, recycler)); this.Publish(); } private void Publish() { - var frontier = this.inputs.Min(i => i.LastEnvelope.OriginatingTime); // earliest last originating time across inputs - var eligible = from m in this.buffer - where frontier > m.envelope.OriginatingTime // later messages seen on _all_ (non-closed) inputs - orderby m.envelope.OriginatingTime, m.envelope.SourceId // originating time order (by source ID if collision) - select m; - foreach (var (data, envelope, recycler) in eligible.ToArray()) + // find out the earliest last originating time across inputs + var frontier = this.inputs.Min(i => i.LastEnvelope.OriginatingTime); + + // get the groups of messages ready to be published + var eligible = this.buffer + .Where(m => m.envelope.OriginatingTime < frontier) + .OrderBy(m => m.envelope.OriginatingTime) + .ThenBy(m => m.envelope.SourceId) + .GroupBy(m => m.envelope.OriginatingTime); + + foreach (var group in eligible.ToArray()) { - var time = this.pipeline.GetCurrentTime(); - if (this.pipeline.FinalOriginatingTime > DateTime.MinValue && time >= this.pipeline.FinalOriginatingTime) + this.Out.Post(group.Select(t => t.data).ToArray(), group.Key); + + foreach (var (data, envelope, recycler) in group) { - // pipeline is closing and these messages would be dropped by the scheduler if posted with current time - time = this.Out.LastEnvelope.OriginatingTime.AddTicks(1); // squeeze final messages in before pipeline shutdown if possible - if (time > this.pipeline.FinalOriginatingTime) - { - throw new Exception("Zip has been forced to drop messages due to pipeline shutdown"); - } + this.buffer.Remove((data, envelope, recycler)); + recycler.Recycle(data); } - - this.Out.Post(Message.Create(data, envelope), time); - recycler.Recycle(data); - this.buffer.Remove((data, envelope, recycler)); } } } diff --git a/Sources/Runtime/Microsoft.Psi/Data/Exporter.cs b/Sources/Runtime/Microsoft.Psi/Data/Exporter.cs index 7a6f4029d..2501a3e2a 100644 --- a/Sources/Runtime/Microsoft.Psi/Data/Exporter.cs +++ b/Sources/Runtime/Microsoft.Psi/Data/Exporter.cs @@ -60,32 +60,6 @@ internal Exporter(Pipeline pipeline, string name, string path, bool createSubdir }); } - /// - /// Used to define which emitters should be written to the store using WriteEmitters. - /// - public enum WriteWhichEmitters - { - /// - /// Write all emitters. - /// - All, - - /// - /// Write specified emitters. - /// - Specified, - - /// - /// Write connected emitters. - /// - Connected, - - /// - /// Write attributed emitters. - /// - Attributed, - } - /// /// Gets the name of the store being written to. /// @@ -117,48 +91,38 @@ public override void Dispose() { this.writer.Dispose(); } + + this.throttle.Dispose(); } /// /// Writes the messages from the specified stream to the matching storage stream in this store. /// - /// The type of messages in the stream. + /// The type of messages in the stream. /// The source stream to write. /// The name of the storage stream. /// Indicates whether the stream contains large messages (typically >4k). If true, the messages will be written to the large message file. /// An optional delivery policy. - public void Write(Emitter source, string name, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) + public void Write(Emitter source, string name, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) { - // make sure we can serialize this type - var handler = this.serializers.GetHandler(); - - // add another input to the merger to hook up the serializer to - // and check for duplicate names in the process - var mergeInput = this.merger.Add(name); - - // name the stream if it's not already named - var connector = new MessageConnector(source.Pipeline, this, null); - - // defaults to lossless delivery policy unless otherwise specified - source.PipeTo(connector, deliveryPolicy ?? DeliveryPolicy.Unlimited); - source.Name = connector.Out.Name = name; - source.Closed += closeTime => this.writer.CloseStream(source.Id, closeTime); - - // tell the writer to write the serialized stream - var meta = this.writer.OpenStream(source.Id, name, largeMessages, handler.Name); - - // register this stream with the store catalog - this.pipeline.ConfigurationStore.Set(Store.StreamMetadataNamespace, name, meta); - - // hook up the serializer - var serializer = new SerializerComponent(this, this.serializers); + this.WriteToStorage(source, name, largeMessages, deliveryPolicy); + } - // The serializer and merger will act synchronously and throttle the connector for as long as - // the merger is busy writing data. This will cause messages to be queued or dropped at the input - // to the connector (per the user-supplied deliveryPolicy) until the merger is able to accept - // the next serialized data message. - serializer.PipeTo(mergeInput, allowWhileRunning: true, DeliveryPolicy.SynchronousOrThrottle); - connector.PipeTo(serializer, allowWhileRunning: true, DeliveryPolicy.SynchronousOrThrottle); + /// + /// Writes the messages from the specified stream to the matching storage stream in this store. + /// Additionally stores supplemental metadata value. + /// + /// The type of messages in the stream. + /// The type of supplemental stream metadata. + /// The source stream to write. + /// Supplemental metadata value. + /// The name of the storage stream. + /// Indicates whether the stream contains large messages (typically >4k). If true, the messages will be written to the large message file. + /// An optional delivery policy. + public void Write(Emitter source, TSupplementalMetadata supplementalMetadataValue, string name, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) + { + var meta = this.WriteToStorage(source, name, largeMessages, deliveryPolicy); + meta.SetSupplementalMetadata(supplementalMetadataValue, this.serializers); } /// @@ -203,88 +167,16 @@ public void WriteEnvelopes(Emitter source, string name, DeliveryPolicy } /// - /// Given a pipeline source object (such as KinectSensor) this method will - /// write all emitters that match the requested properties to the current store. - /// This is useful for writing streams from a sensor to a store for later playback - /// via Importer.OpenAs(). + /// Writes the messages from the specified stream to the matching storage stream in this store. /// - /// Type of source to get emitters from (e.g. KinectSensor). - /// The source to write data from. - /// Used to indicate which emitters should be written. - /// List of stream names if streamToEmit==WriteWhichEmitters.Specified. - public void WriteEmitters(T sensor, WriteWhichEmitters streamsToEmit = WriteWhichEmitters.Connected, string[] streamNames = null) + /// The type of messages in the stream. + /// The source stream to write. + /// The name of the storage stream. + /// Source stream metadata. + /// An optional delivery policy. + internal void Write(Emitter source, string name, PsiStreamMetadata metadata, DeliveryPolicy deliveryPolicy = null) { - var objType = typeof(T); - var objProps = objType.GetProperties(); - foreach (var f in objProps) - { - var objPropType = f.PropertyType; - if (objPropType.GetInterface(nameof(IEmitter)) != null) - { - bool emit = false; - switch (streamsToEmit) - { - case WriteWhichEmitters.All: - emit = true; - break; - case WriteWhichEmitters.Specified: - if (streamNames != null) - { - foreach (var name in streamNames) - { - if (name == f.Name) - { - emit = true; - break; - } - } - } - - break; - case WriteWhichEmitters.Connected: - var theEmitter = f.GetValue(sensor); - if (theEmitter != null) - { - var hasSubscribers = objPropType.GetProperty("HasSubscribers").GetValue(theEmitter); - if ((bool)hasSubscribers) - { - emit = true; - } - } - - break; - case WriteWhichEmitters.Attributed: - if (Attribute.IsDefined(f, typeof(WriteStreamAttribute))) - { - emit = true; - } - - break; - } - - if (emit) - { - var propInfo = typeof(T).GetProperty(f.Name); - var writeMethods = typeof(Exporter).GetMethods(); - System.Reflection.MethodInfo writeMethod = null; - foreach (var mi in writeMethods) - { - if (mi.Name == nameof(this.Write) && mi.IsGenericMethod == true) - { - writeMethod = mi; - break; - } - } - - if (writeMethod != null) - { - var writeStream = writeMethod.MakeGenericMethod(objPropType.GetGenericArguments()); - object[] args = { propInfo.GetValue(sensor), f.Name, true, DeliveryPolicy.LatestMessage }; - writeStream.Invoke(this, args); - } - } - } - } + this.WriteToStorage(source, name, metadata.IsIndexed, deliveryPolicy).UpdateSupplementalMetadataFrom(metadata); } internal void Write(Emitter> source, PsiStreamMetadata meta, DeliveryPolicy> deliveryPolicy = null) @@ -303,13 +195,48 @@ internal void Write(Emitter> source, PsiStreamMetadata met } /// - /// Class that defines an attribute used to indicate which Emitters should be - /// written when WriteEmitters is called. - /// Clients may place the [WriteStream] attribute on each emitter property - /// they want written. + /// Writes the messages from the specified stream to the matching storage stream in this store. /// - public class WriteStreamAttribute : Attribute + /// The type of messages in the stream. + /// The source stream to write. + /// The name of the storage stream. + /// Indicates whether the stream contains large messages (typically >4k). If true, the messages will be written to the large message file. + /// An optional delivery policy. + /// Stream metadata. + private PsiStreamMetadata WriteToStorage(Emitter source, string name, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) { + // make sure we can serialize this type + var handler = this.serializers.GetHandler(); + + // add another input to the merger to hook up the serializer to + // and check for duplicate names in the process + var mergeInput = this.merger.Add(name); + + // name the stream if it's not already named + var connector = new MessageConnector(source.Pipeline, this, null); + + // defaults to lossless delivery policy unless otherwise specified + source.PipeTo(connector, deliveryPolicy ?? DeliveryPolicy.Unlimited); + source.Name = connector.Out.Name = name; + source.Closed += closeTime => this.writer.CloseStream(source.Id, closeTime); + + // tell the writer to write the serialized stream + var meta = this.writer.OpenStream(source.Id, name, largeMessages, handler.Name); + + // register this stream with the store catalog + this.pipeline.ConfigurationStore.Set(Store.StreamMetadataNamespace, name, meta); + + // hook up the serializer + var serializer = new SerializerComponent(this, this.serializers); + + // The serializer and merger will act synchronously and throttle the connector for as long as + // the merger is busy writing data. This will cause messages to be queued or dropped at the input + // to the connector (per the user-supplied deliveryPolicy) until the merger is able to accept + // the next serialized data message. + serializer.PipeTo(mergeInput, allowWhileRunning: true, DeliveryPolicy.SynchronousOrThrottle); + connector.PipeTo(serializer, allowWhileRunning: true, DeliveryPolicy.SynchronousOrThrottle); + + return meta; } } } diff --git a/Sources/Runtime/Microsoft.Psi/Data/Importer.cs b/Sources/Runtime/Microsoft.Psi/Data/Importer.cs index 0c552b50f..197256a35 100644 --- a/Sources/Runtime/Microsoft.Psi/Data/Importer.cs +++ b/Sources/Runtime/Microsoft.Psi/Data/Importer.cs @@ -24,7 +24,6 @@ public sealed class Importer : Subpipeline, IDisposable private readonly Splitter, int> splitter; private readonly Dictionary streams = new Dictionary(); private readonly Pipeline pipeline; - private KnownSerializers serializers; /// /// Initializes a new instance of the class. @@ -51,7 +50,7 @@ public Importer(Pipeline pipeline, string name, string path) /// Gets the set of types that this Importer can deserialize. /// Types can be added or re-mapped using the method. /// - public KnownSerializers Serializers => this.serializers; + public KnownSerializers Serializers { get; private set; } /// /// Gets the metadata of all the storage streams in this store. @@ -88,6 +87,18 @@ public override void Dispose() /// The metadata associated with the storage stream. public PsiStreamMetadata GetMetadata(string streamName) => this.reader.GetMetadata(streamName); + /// + /// Returns the supplemental metadata for a specified storage stream. + /// + /// Type of supplemental metadata. + /// The name of the storage stream. + /// The metadata associated with the storage stream. + public T GetSupplementalMetadata(string streamName) + { + var meta = this.reader.GetMetadata(streamName); + return meta.GetSupplementalMetadata(this.Serializers); + } + /// /// Indicates whether the store contains the specified storage stream. /// @@ -153,7 +164,7 @@ public T OpenAs() /// A stream that publishes the data read from the store. public IProducer OpenStream(string streamName, T reusableInstance = default(T)) { - return this.OpenStream(streamName, new DeserializerComponent(this, this.serializers, reusableInstance)); + return this.OpenStream(streamName, new DeserializerComponent(this, this.Serializers, reusableInstance)); } /// @@ -165,7 +176,7 @@ public IProducer OpenStream(string streamName, T reusableInstance = defaul /// A stream of dynamic that publishes the data read from the store. public IProducer OpenDynamicStream(string streamName) { - return this.OpenStream(streamName, new DynamicDeserializerComponent(this, this.reader.OpenStream(streamName).TypeName, this.serializers.Schemas), false); + return this.OpenStream(streamName, new DynamicDeserializerComponent(this, this.reader.OpenStream(streamName).TypeName, this.Serializers.Schemas), false); } /// @@ -177,6 +188,12 @@ public IProducer OpenDynamicStream(string streamName) /// A stream of raw messages that publishes the data read from the store. internal Emitter> OpenRawStream(PsiStreamMetadata meta) { + if (meta.OriginatingLifetime != null && !meta.OriginatingLifetime.IsEmpty && meta.OriginatingLifetime.IsFinite) + { + // propose a replay time that covers the stream lifetime + this.ProposeReplayTime(meta.OriginatingLifetime); + } + this.reader.OpenStream(meta); // this checks for duplicates but bypasses type checks return this.splitter.Add(meta.Id); } @@ -194,15 +211,15 @@ private IProducer OpenStream(string streamName, ConsumerProducer(); + var handler = this.Serializers.GetHandler(); if (handler.Name != streamType) { - if (this.serializers.Schemas.TryGetValue(streamType, out var streamTypeSchema) && - this.serializers.Schemas.TryGetValue(requestedType, out var requestedTypeSchema)) + if (this.Serializers.Schemas.TryGetValue(streamType, out var streamTypeSchema) && + this.Serializers.Schemas.TryGetValue(requestedType, out var requestedTypeSchema)) { // validate compatibility - will throw if types are incompatible streamTypeSchema.ValidateCompatibleWith(requestedTypeSchema); @@ -252,12 +269,12 @@ private IProducer OpenStream(string streamName, ConsumerProducerThe version of the runtime that produced the store. private void LoadMetadata(IEnumerable metadata, RuntimeInfo runtimeVersion) { - if (this.serializers == null) + if (this.Serializers == null) { - this.serializers = new KnownSerializers(runtimeVersion); + this.Serializers = new KnownSerializers(runtimeVersion); } - this.serializers.RegisterMetadata(metadata); + this.Serializers.RegisterMetadata(metadata); } } } diff --git a/Sources/Runtime/Microsoft.Psi/Data/Store.cs b/Sources/Runtime/Microsoft.Psi/Data/Store.cs index 7f4865a63..1a7f5739f 100644 --- a/Sources/Runtime/Microsoft.Psi/Data/Store.cs +++ b/Sources/Runtime/Microsoft.Psi/Data/Store.cs @@ -5,12 +5,11 @@ namespace Microsoft.Psi { using System; using System.Collections.Generic; - using System.Data; using System.IO; using System.Linq; + using System.Reflection; using System.Threading; using Microsoft.Psi.Common; - using Microsoft.Psi.Components; using Microsoft.Psi.Data; using Microsoft.Psi.Persistence; using Microsoft.Psi.Serialization; @@ -35,7 +34,7 @@ public static class Store /// /// The Exporter maintains a collection of serializers it knows about, which it uses to serialize /// the data it writes to the store. By default, the Exporter derives the correct serializers - /// from the type argument passed to . In other words, + /// from the TMessage type argument passed to . In other words, /// for the most part simply knowing the stream type is sufficient to determine all the types needed to /// serialize the messages in the stream. /// Use the parameter to override the default behavior and provide a custom set of serializers. @@ -85,19 +84,37 @@ public static Exporter Create(Pipeline pipeline, string name, string rootPath, b /// /// Writes the specified stream to a multi-stream store. /// - /// The type of messages in the stream. + /// The type of messages in the stream. /// The source stream to write. /// The name of the persisted stream. /// The store writer, created by e.g. . /// Indicates whether the stream contains large messages (typically >4k). If true, the messages will be written to the large message file. /// An optional delivery policy. /// The input stream. - public static IProducer Write(this IProducer source, string name, Exporter writer, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) + public static IProducer Write(this IProducer source, string name, Exporter writer, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) { writer.Write(source.Out, name, largeMessages, deliveryPolicy); return source; } + /// + /// Writes the specified stream to a multi-stream store. + /// + /// The type of messages in the stream. + /// The type of supplemental stream metadata. + /// The source stream to write. + /// Supplemental metadata value. + /// The name of the persisted stream. + /// The store writer, created by e.g. . + /// Indicates whether the stream contains large messages (typically >4k). If true, the messages will be written to the large message file. + /// An optional delivery policy. + /// The input stream. + public static IProducer Write(this IProducer source, TSupplementalMetadata supplementalMetadataValue, string name, Exporter writer, bool largeMessages = false, DeliveryPolicy deliveryPolicy = null) + { + writer.Write(source.Out, supplementalMetadataValue, name, largeMessages, deliveryPolicy); + return source; + } + /// /// Writes the envelopes for the specified stream to a multi-stream store. /// @@ -121,7 +138,7 @@ public static IProducer WriteEnvelopes(this IProducer source, str /// Returns true if the store exists. public static bool Exists(string name, string path) { - return (path == null) ? InfiniteFileReader.IsActive(StoreCommon.GetCatalogFileName(name), path) : StoreCommon.TryGetPathToLatestVersion(name, path, out string fullPath); + return (path == null) ? InfiniteFileReader.IsActive(StoreCommon.GetCatalogFileName(name), path) : StoreCommon.TryGetPathToLatestVersion(name, path, out _); } /// @@ -143,50 +160,18 @@ public static bool IsClosed(string name, string path) } /// - /// Repairs an invalid store. + /// Repairs an invalid store in place. /// /// The name of the store to check. /// The path of the store to check. - /// Indicates whether the original store should be deleted. - public static void Repair(string name, string path, bool deleteOldStore = true) + /// Indicates whether the original store should be deleted. + public static void Repair(string name, string path, bool deleteOriginalStore = true) { - string storePath = StoreCommon.GetPathToLatestVersion(name, path); - string tempFolderPath = Path.Combine(path, $"Repair-{Guid.NewGuid()}"); - - // call Crop over the entire store interval to regenerate and repair the streams in the store - Store.Crop((name, storePath), (name, tempFolderPath), TimeInterval.Infinite); - - // create a _BeforeRepair folder in which to save the original store files - var beforeRepairPath = Path.Combine(storePath, $"BeforeRepair-{Guid.NewGuid()}"); - Directory.CreateDirectory(beforeRepairPath); - - // Move the original store files to the BeforeRepair folder. Do this even if the deleteOldStore - // flag is true, as deleting the original store files immediately may occasionally fail. This can - // happen because the InfiniteFileReader disposes of its MemoryMappedView in a background - // thread, which may still be in progress. If deleteOldStore is true, we will delete the - // BeforeRepair folder at the very end (by which time any open MemoryMappedViews will likely - // have finished disposing). - foreach (var file in Directory.EnumerateFiles(storePath)) - { - var fileInfo = new FileInfo(file); - File.Move(file, Path.Combine(beforeRepairPath, fileInfo.Name)); - } - - // move the repaired store files to the original folder - foreach (var file in Directory.EnumerateFiles(Path.Combine(tempFolderPath, $"{name}.0000"))) - { - var fileInfo = new FileInfo(file); - File.Move(file, Path.Combine(storePath, fileInfo.Name)); - } - - // cleanup temporary folder - Directory.Delete(tempFolderPath, true); - - if (deleteOldStore) - { - // delete the old store files - Directory.Delete(beforeRepairPath, true); - } + PerformStoreOperationInPlace( + (name, path), + nameof(Repair), + (name, path, temp) => Store.Crop((name, path), (name, temp), TimeInterval.Infinite), + deleteOriginalStore); } /// @@ -204,15 +189,15 @@ public static void Repair(string name, string path, bool deleteOldStore = true) /// An optional callback to which human-friendly information will be logged. public static void Crop((string Name, string Path) input, (string Name, string Path) output, TimeSpan start, RelativeTimeInterval length, bool createSubdirectory = true, IProgress progress = null, Action loggingCallback = null) { - Crop(input, output, store => new TimeInterval(store.ActiveTimeInterval.Left + start, length), createSubdirectory, progress, loggingCallback); + Copy(input, output, store => new TimeInterval(store.OriginatingTimeInterval.Left + start, length), null, createSubdirectory, progress, loggingCallback); } /// - /// Crops a store between the extents of a specified interval, generating a new store. + /// Crops a store between the extents of a specified originating time interval, generating a new store. /// /// The name and path of the store to crop. /// The name and path of the cropped store. - /// The time interval to which to crop the store. + /// The originating time interval to which to crop the store. /// /// Indicates whether to create a numbered subdirectory for each cropped store /// generated by multiple calls to this method. @@ -221,7 +206,88 @@ public static void Crop((string Name, string Path) input, (string Name, string P /// An optional callback to which human-friendly information will be logged. public static void Crop((string Name, string Path) input, (string Name, string Path) output, TimeInterval cropInterval, bool createSubdirectory = true, IProgress progress = null, Action loggingCallback = null) { - Crop(input, output, _ => cropInterval, createSubdirectory, progress, loggingCallback); + Copy(input, output, _ => cropInterval, null, createSubdirectory, progress, loggingCallback); + } + + /// + /// Crops a store in place between the extents of a specified interval. + /// + /// The name and path of the store to crop. + /// Start of crop interval relative to beginning of store. + /// Length of crop interval. + /// Indicates whether the original store should be deleted. + /// An optional progress reporter for progress updates. + /// An optional callback to which human-friendly information will be logged. + public static void CropInPlace((string Name, string Path) input, TimeSpan start, RelativeTimeInterval length, bool deleteOriginalStore = true, IProgress progress = null, Action loggingCallback = null) + { + PerformStoreOperationInPlace( + input, + nameof(Crop), + (name, path, temp) => Copy((name, path), (name, temp), store => new TimeInterval(store.OriginatingTimeInterval.Left + start, length), null, false, progress, loggingCallback), + deleteOriginalStore); + } + + /// + /// Crops a store in place between the extents of a specified originating time interval. + /// + /// The name and path of the store to crop. + /// The originating time interval to which to crop the store. + /// Indicates whether the original store should be deleted. + /// An optional progress reporter for progress updates. + /// An optional callback to which human-friendly information will be logged. + public static void CropInPlace((string Name, string Path) input, TimeInterval cropInterval, bool deleteOriginalStore = true, IProgress progress = null, Action loggingCallback = null) + { + PerformStoreOperationInPlace( + input, + nameof(Crop), + (name, path, temp) => Copy((name, path), (name, temp), _ => cropInterval, null, false, progress, loggingCallback), + deleteOriginalStore); + } + + /// + /// Copies a store, or a subset of it. + /// + /// The name and path of the store to crop. + /// The name and path of the cropped store. + /// An optional function that defines an originating time interval to copy. By default, the extents of the entire store. + /// An optional predicate that specifies which streams to include. By default, include all streams. By default, all streams are copied. + /// + /// Indicates whether to create a numbered subdirectory for each cropped store + /// generated by multiple calls to this method. + /// + /// An optional progress reporter for progress updates. + /// An optional callback to which human-friendly information will be logged. + public static void Copy( + (string Name, string Path) input, + (string Name, string Path) output, + Func cropIntervalFunction = null, + Predicate includeStreamPredicate = null, + bool createSubdirectory = true, + IProgress progress = null, + Action loggingCallback = null) + { + using var pipeline = Pipeline.Create(); + Importer inputStore = Store.Open(pipeline, input.Name, input.Path); + Exporter outputStore = Store.Create(pipeline, output.Name, output.Path, createSubdirectory, inputStore.Serializers); + + // setup the defaults + cropIntervalFunction ??= s => s.OriginatingTimeInterval; + includeStreamPredicate ??= _ => true; + + // copy all streams from inputStore to outputStore + foreach (var streamInfo in inputStore.AvailableStreams) + { + if (includeStreamPredicate(streamInfo)) + { + inputStore.CopyStream(streamInfo.Name, outputStore); + } + } + + // run the pipeline to copy over the specified timeInterval + loggingCallback?.Invoke("Copying store ..."); + pipeline.RunAsync(cropIntervalFunction(inputStore), false, progress); + pipeline.WaitAll(); + loggingCallback?.Invoke("Done."); } /// @@ -234,90 +300,86 @@ public static void Crop((string Name, string Path) input, (string Name, string P /// An optional callback to which human-friendly information will be logged. public static void Concatenate(IEnumerable<(string Name, string Path)> storeFiles, (string Name, string Path) output, IProgress progress = null, Action loggingCallback = null) { - using (var pipeline = Pipeline.Create(nameof(Concatenate), DeliveryPolicy.Unlimited)) + using var pipeline = Pipeline.Create(nameof(Concatenate), DeliveryPolicy.Unlimited); + var outputStore = Store.Create(pipeline, output.Name, output.Path); + var outputReceivers = new Dictionary>>(); // per-stream receivers writing to store + var inputStores = storeFiles.Select(file => (file, Store.Open(pipeline, file.Name, Path.GetFullPath(file.Path)))); + var streamMetas = inputStores.SelectMany(s => s.Item2.AvailableStreams).GroupBy(s => s.Name); // streams across input stores, grouped by name + var totalMessageCount = 0; // total messages in all stores + var totalWrittenCount = 0; // total currently written (to track progress) + var totalInFlight = 0; + foreach (var group in streamMetas) { - var outputStore = Store.Create(pipeline, output.Name, output.Path); - var outputReceivers = new Dictionary>>(); // per-stream receivers writing to store - var inputStores = storeFiles.Select(file => (file, Store.Open(pipeline, file.Name, Path.GetFullPath(file.Path)))); - var streamMetas = inputStores.SelectMany(s => s.Item2.AvailableStreams).GroupBy(s => s.Name); // streams across input stores, grouped by name - var totalMessageCount = 0; // total messages in all stores - var totalWrittenCount = 0; // total currently written (to track progress) - var totalInFlight = 0; - foreach (var group in streamMetas) - { - var name = group.Key; - loggingCallback?.Invoke($"Stream: {name}"); - var emitter = pipeline.CreateEmitter>(pipeline, name); - var meta = group.First(); // using meta from first store in which stream appears - outputStore.Write(emitter, meta); - outputReceivers.Add(name, pipeline.CreateReceiver>( - pipeline, - m => - { - emitter.Post(Message.Create(m.Data, m.OriginatingTime, m.Time, meta.Id, emitter.LastEnvelope.SequenceId + 1), m.OriginatingTime); - Interlocked.Decrement(ref totalInFlight); - Interlocked.Increment(ref totalWrittenCount); - if (totalWrittenCount % 10000 == 0) - { - progress?.Report((double)totalWrittenCount / totalMessageCount); - } - }, - name)); - - // validate types match across stores and stream lifetimes don't overlap - foreach (var stream in group) + var name = group.Key; + loggingCallback?.Invoke($"Stream: {name}"); + var emitter = pipeline.CreateEmitter>(pipeline, name); + var meta = group.First(); // using meta from first store in which stream appears + outputStore.Write(emitter, meta); + outputReceivers.Add(name, pipeline.CreateReceiver>( + pipeline, + m => { - totalMessageCount += stream.MessageCount; - loggingCallback?.Invoke($" Partition: {stream.PartitionName} {stream.Id} ({stream.TypeName.Split(',')[0]}) {stream.OriginatingLifetime.Left}-{stream.OriginatingLifetime.Right}"); - if (group.GroupBy(pair => pair.TypeName).Count() != 1) + emitter.Post(Message.Create(m.Data, m.OriginatingTime, m.Time, meta.Id, emitter.LastEnvelope.SequenceId + 1), m.OriginatingTime); + Interlocked.Decrement(ref totalInFlight); + Interlocked.Increment(ref totalWrittenCount); + if (totalWrittenCount % 10000 == 0) { - throw new ArgumentException("Type Mismatch"); + progress?.Report((double)totalWrittenCount / totalMessageCount); } + }, + name)); + + // validate types match across stores and stream lifetimes don't overlap + foreach (var stream in group) + { + totalMessageCount += stream.MessageCount; + loggingCallback?.Invoke($" Partition: {stream.PartitionName} {stream.Id} ({stream.TypeName.Split(',')[0]}) {stream.OriginatingLifetime.Left}-{stream.OriginatingLifetime.Right}"); + if (group.GroupBy(pair => pair.TypeName).Count() != 1) + { + throw new ArgumentException("Type Mismatch"); + } - foreach (var crosscheck in group) + foreach (var crosscheck in group) + { + if (crosscheck != stream && crosscheck.OriginatingLifetime.IntersectsWith(stream.OriginatingLifetime)) { - if (crosscheck != stream && crosscheck.OriginatingLifetime.IntersectsWith(stream.OriginatingLifetime)) - { - throw new ArgumentException("Originating Lifetime Overlap"); - } + throw new ArgumentException("Originating Lifetime Overlap"); } } } + } - // for each store (in order), create pipeline to read to end; piping to receivers in outer pipeline - var orderedStores = inputStores.OrderBy(i => i.Item2.OriginatingTimeInterval.Left).Select(i => (i.file.Name, i.file.Path)); - Generators.Sequence(pipeline, orderedStores, TimeSpan.FromTicks(1)).Do(file => + // for each store (in order), create pipeline to read to end; piping to receivers in outer pipeline + var orderedStores = inputStores.OrderBy(i => i.Item2.OriginatingTimeInterval.Left).Select(i => (i.file.Name, i.file.Path)); + Generators.Sequence(pipeline, orderedStores, TimeSpan.FromTicks(1)).Do(file => + { + loggingCallback?.Invoke($"Processing: {file.Name} {file.Path}"); + using var p = Pipeline.Create(file.Name, DeliveryPolicy.Unlimited); + var store = Store.Open(p, file.Name, Path.GetFullPath(file.Path)); + foreach (var stream in store.AvailableStreams) { - loggingCallback?.Invoke($"Processing: {file.Name} {file.Path}"); - using (var p = Pipeline.Create(file.Name, DeliveryPolicy.Unlimited)) - { - var store = Store.Open(p, file.Name, Path.GetFullPath(file.Path)); - foreach (var stream in store.AvailableStreams) - { - var connector = store.CreateOutputConnectorTo>(pipeline, stream.Name); - store.OpenRawStream(stream).Do(_ => Interlocked.Increment(ref totalInFlight)).PipeTo(connector); - connector.Out.PipeTo(outputReceivers[stream.Name], true); // while outer pipeline running - } + var connector = store.CreateOutputConnectorTo>(pipeline, stream.Name); + store.OpenRawStream(stream).Do(_ => Interlocked.Increment(ref totalInFlight)).PipeTo(connector); + connector.Out.PipeTo(outputReceivers[stream.Name], true); // while outer pipeline running + } - p.Run(); + p.Run(); - while (totalInFlight > 0) - { - Thread.Sleep(100); - } + while (totalInFlight > 0) + { + Thread.Sleep(100); + } - // unsubscribe to reuse receivers with next store - foreach (var r in outputReceivers.Values) - { - r.OnUnsubscribe(); - } - } - }); + // unsubscribe to reuse receivers with next store + foreach (var r in outputReceivers.Values) + { + r.OnUnsubscribe(); + } + }); - loggingCallback?.Invoke("Concatenating..."); - pipeline.Run(); - loggingCallback?.Invoke("Done."); - } + loggingCallback?.Invoke("Concatenating..."); + pipeline.Run(); + loggingCallback?.Invoke("Done."); } /// @@ -351,35 +413,166 @@ public static bool TryGetMetadata(Pipeline pipeline, string streamName, out PsiS } /// - /// Crops a store between the extents of a specified interval, generating a new store. + /// Edit a store, or a subset of it. /// /// The name and path of the store to crop. /// The name and path of the cropped store. - /// Function creating crop interval. - /// - /// Indicates whether to create a numbered subdirectory for each cropped store - /// generated by multiple calls to this method. - /// + /// Dictionary of per-stream sequence of edits to be applied. Whether to update/insert or delete, an optional message to upsert and originating times. + /// An optional predicate that specifies which streams to include. By default, include all streams. By default, all streams are copied. + /// Indicates whether to create a numbered subdirectory for each cropped store generated by multiple calls to this method. /// An optional progress reporter for progress updates. /// An optional callback to which human-friendly information will be logged. - private static void Crop((string Name, string Path) input, (string Name, string Path) output, Func cropIntervalFn, bool createSubdirectory = true, IProgress progress = null, Action loggingCallback = null) + internal static void Edit( + (string Name, string Path) input, + (string Name, string Path) output, + IDictionary> streamEdits, + Predicate includeStreamPredicate = null, + bool createSubdirectory = true, + IProgress progress = null, + Action loggingCallback = null) { - using (var pipeline = Pipeline.Create("CropStore")) + includeStreamPredicate ??= _ => true; + using var pipeline = Pipeline.Create(); + Importer inputStore = Store.Open(pipeline, input.Name, input.Path); + Exporter outputStore = Store.Create(pipeline, output.Name, output.Path, createSubdirectory, inputStore.Serializers); + + // edit/copy all streams from inputStore to outputStore + foreach (var streamInfo in inputStore.AvailableStreams) { - Importer inputStore = Store.Open(pipeline, input.Name, input.Path); - Exporter outputStore = Store.Create(pipeline, output.Name, output.Path, createSubdirectory, inputStore.Serializers); + if (includeStreamPredicate(streamInfo)) + { + if (streamEdits.TryGetValue(streamInfo.Name, out var edits)) + { + var method = typeof(Store).GetMethod(nameof(Store.EditStreamWithDynamicUpserts), BindingFlags.NonPublic | BindingFlags.Static); + var streamType = Type.GetType(streamInfo.TypeName); + var generic = method.MakeGenericMethod(streamType); + generic.Invoke(inputStore, new object[] { inputStore, streamInfo.Name, edits, outputStore, null }); + } + else + { + inputStore.CopyStream(streamInfo.Name, outputStore); + } + } + } - // copy all streams from inputStore to outputStore - foreach (var streamInfo in inputStore.AvailableStreams) + loggingCallback?.Invoke("Editing store ..."); + + var minOriginatingTime = inputStore.OriginatingTimeInterval.Left; + var maxOriginatingTime = inputStore.OriginatingTimeInterval.Right; + foreach (var edit in streamEdits.Values) + { + foreach (var message in edit) { - inputStore.CopyStream(streamInfo.Name, outputStore); + var time = message.originatingTime; + minOriginatingTime = time < minOriginatingTime ? time : minOriginatingTime; + maxOriginatingTime = time > maxOriginatingTime ? time : maxOriginatingTime; } + } + + pipeline.RunAsync(new TimeInterval(minOriginatingTime, maxOriginatingTime), false, progress); + pipeline.WaitAll(); + loggingCallback?.Invoke("Done."); + } - // run the pipeline to copy over the specified cropInterval - loggingCallback?.Invoke("Cropping..."); - pipeline.RunAsync(cropIntervalFn(inputStore), false, progress); - pipeline.WaitAll(); - loggingCallback?.Invoke("Done."); + /// + /// Edit a store in place, or a subset of it. + /// + /// The name and path of the store to crop. + /// Dictionary of per-stream sequence of edits to be applied. Whether to update/insert or delete, an optional message to upsert and originating times. + /// An optional predicate that specifies which streams to include. By default, include all streams. By default, all streams are copied. + /// Indicates whether the original store should be deleted. + /// An optional progress reporter for progress updates. + /// An optional callback to which human-friendly information will be logged. + internal static void EditInPlace( + (string Name, string Path) input, + IDictionary> streamEdits, + Predicate includeStreamPredicate = null, + bool deleteOriginalStore = true, + IProgress progress = null, + Action loggingCallback = null) + { + PerformStoreOperationInPlace( + input, + nameof(Edit), + (name, path, temp) => Edit((name, path), (name, temp), streamEdits, includeStreamPredicate, false, progress, loggingCallback), + deleteOriginalStore); + } + + /// + /// Edit messages in the specified storage stream and write to an exporter; applying updates/inserts and deletes. + /// + /// Importer from which to get stream being edited. + /// The name of the storage stream to edit. + /// A sequence of edits to be applied. Whether to update/insert or delete, an optional message to upsert and originating times. + /// The store into which to output. + /// An optional delivery policy. + private static void EditStream(this Importer importer, string streamName, IEnumerable<(bool upsert, T message, DateTime originatingTime)> edits, Exporter writer, DeliveryPolicy deliveryPolicy = null) + { + var stream = importer.OpenStream(streamName); + var edited = stream.EditStream(edits, deliveryPolicy); + var meta = importer.GetMetadata(streamName); + writer.Write(edited.Out, streamName, meta, deliveryPolicy); + } + + /// + /// Edit messages in the specified storage stream and write to an exporter; applying updates/inserts and deletes. + /// + /// Importer from which to get stream being edited. + /// The name of the storage stream to edit. + /// A sequence of edits to be applied. Whether to update/insert or delete, an optional message to upsert and originating times. + /// The store into which to output. + /// An optional delivery policy. + private static void EditStreamWithDynamicUpserts(this Importer importer, string streamName, IEnumerable<(bool upsert, dynamic message, DateTime originatingTime)> edits, Exporter writer, DeliveryPolicy deliveryPolicy = null) + { + var typedEdits = edits.Select(e => (e.upsert, (T)(e.message ?? default(T)), e.originatingTime)); + importer.EditStream(streamName, typedEdits, writer); + } + + /// + /// Perform store operation in place. + /// + /// The name and path of the store on which to perform operation. + /// Name of operation to perform. + /// Operation function to perform. + /// Indicates whether the original store should be deleted. + private static void PerformStoreOperationInPlace((string Name, string Path) input, string operationName, Action operationAction, bool deleteOriginalStore) + { + string storePath = StoreCommon.GetPathToLatestVersion(input.Name, input.Path); + string tempFolderPath = Path.Combine(input.Path, $"{operationName}-{Guid.NewGuid()}"); + + // invoke operation over the store; expected to generate a resulting store in the temp folder + operationAction(input.Name, storePath, tempFolderPath); + + // create a Before* folder in which to save the original store files + var beforePath = Path.Combine(storePath, $"Before{operationName}-{Guid.NewGuid()}"); + Directory.CreateDirectory(beforePath); + + // Move the original store files to the Before* folder. Do this even if the deleteOriginalStore + // flag is true, as deleting the original store files immediately may occasionally fail. This can + // happen because the InfiniteFileReader disposes of its MemoryMappedView in a background + // thread, which may still be in progress. If deleteOriginalStore is true, we will delete the + // Before* folder at the very end (by which time any open MemoryMappedViews will likely + // have finished disposing). + foreach (var file in Directory.EnumerateFiles(storePath)) + { + var fileInfo = new FileInfo(file); + File.Move(file, Path.Combine(beforePath, fileInfo.Name)); + } + + // move the new store files to the original folder + foreach (var file in Directory.EnumerateFiles(Path.Combine(tempFolderPath, $"{input.Name}.0000"))) + { + var fileInfo = new FileInfo(file); + File.Move(file, Path.Combine(storePath, fileInfo.Name)); + } + + // cleanup temporary folder + Directory.Delete(tempFolderPath, true); + + if (deleteOriginalStore) + { + // delete the old store files + Directory.Delete(beforePath, true); } } } diff --git a/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnostics.cs b/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnostics.cs index 95f27edae..edee0ca8c 100644 --- a/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnostics.cs +++ b/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnostics.cs @@ -12,7 +12,7 @@ namespace Microsoft.Psi.Diagnostics /// Represents diagnostic information about a pipeline. /// /// - /// This is a sumarized snapshot of the graph with aggregated message statistics which is posted to the + /// This is a summarized snapshot of the graph with aggregated message statistics which is posted to the /// diagnostics stream. It has a much smaller memory footprint compared with PipelineDiagnosticsInternal. /// public class PipelineDiagnostics diff --git a/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnosticsInternal.cs b/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnosticsInternal.cs index bd567056a..0f4ded72c 100644 --- a/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnosticsInternal.cs +++ b/Sources/Runtime/Microsoft.Psi/Diagnostics/PipelineDiagnosticsInternal.cs @@ -12,7 +12,7 @@ namespace Microsoft.Psi.Diagnostics /// /// /// This is used while gathering live diagnostics information. It is optimized for lookups with Dictionaries and - /// maintains latency, processing time, message size histories. This information is sumarized before being posted + /// maintains latency, processing time, message size histories. This information is summarized before being posted /// as PipelineDiagnostics. /// internal class PipelineDiagnosticsInternal diff --git a/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs b/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs index a1527b9f1..271ae8584 100644 --- a/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs +++ b/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs @@ -73,6 +73,7 @@ public class Pipeline : IDisposable private IProgress progressReporter; private Time.TimerDelegate progressDelegate; private Platform.ITimer progressTimer; + private bool pipelineRunEventHandled = false; /// /// Initializes a new instance of the class. @@ -94,9 +95,6 @@ public class Pipeline : IDisposable { this.scheduler = new Scheduler(this.ErrorHandler, threadCount, allowSchedulingOnExternalThreads, name); this.schedulerContext = new SchedulerContext(); - - // stop the scheduler when the main pipeline completes - this.PipelineCompleted += (sender, args) => this.scheduler.Stop(args.AbandonedPendingWorkitems); } /// @@ -277,7 +275,7 @@ private set /// /// /// This is an rather than a as we need the - /// event to trigger one and only one action when signalled. + /// event to trigger one and only one action when signaled. /// protected AutoResetEvent NoRemainingCompletableComponents { get; } = new AutoResetEvent(false); @@ -700,6 +698,10 @@ internal void Dispose(bool abandonPendingWorkItems) this.DisposeComponents(); this.components = null; this.DiagnosticsCollector?.PipelineDisposed(this); + this.completed.Dispose(); + this.scheduler.Dispose(); + this.schedulerContext.Dispose(); + this.activationContext.Dispose(); } internal void AddComponent(PipelineElement pe) @@ -783,17 +785,7 @@ internal void CompleteComponent(PipelineElement component, DateTime finalOrigina } /// - /// Signal pipeline completion. - /// - /// Abandons the pending work items. - internal void Complete(bool abandonPendingWorkitems) - { - this.PipelineCompleted?.Invoke(this, new PipelineCompletedEventArgs(this.Clock.GetCurrentTime(), abandonPendingWorkitems, this.errors)); - this.state = State.Completed; - } - - /// - /// Apply action to this pipeline and to all descendent Subpipelines. + /// Apply action to this pipeline and to all descendant Subpipelines. /// /// Action to apply. internal void ForThisPipelineAndAllDescendentSubpipelines(Action action) @@ -822,13 +814,12 @@ internal PipelineElement GetOrCreateNode(object component) var name = component.GetType().Name; var fullName = $"{id}.{name}"; node = new PipelineElement(id, fullName, component); - this.AddComponent(node); - if (this.IsRunning && node.IsSource && !(component is Subpipeline)) { throw new InvalidOperationException($"Source component added when pipeline already running. Consider using Subpipeline."); } + this.AddComponent(node); this.DiagnosticsCollector?.PipelineElementCreate(this, node, component); } @@ -886,8 +877,24 @@ internal virtual void Stop(DateTime finalOriginatingTime, bool abandonPendingWor this.ReportProgress(); // ensure that final progress is reported } + // stop the scheduler if this is the main pipeline + if (!(this is Subpipeline)) + { + this.scheduler.Stop(abandonPendingWorkitems); + } + + // pipeline has completed + this.state = State.Completed; + + // raise the pipeline completed event + this.OnPipelineCompleted( + new PipelineCompletedEventArgs( + this.FinalOriginatingTime, + abandonPendingWorkitems, + this.errors)); + + // signal all threads waiting on pipeline completion this.completed.Set(); - this.Complete(abandonPendingWorkitems); } /// @@ -923,9 +930,7 @@ protected virtual IDisposable RunAsync(ReplayDescriptor descriptor, Clock clock, this.DiagnosticsCollector?.PipelineStart(this); // raise the event prior to starting the components - this.PipelineRun?.Invoke(this, new PipelineRunEventArgs(this.StartTime)); - - this.state = State.Running; + this.OnPipelineRun(new PipelineRunEventArgs(this.StartTime)); // keep track of completable source components foreach (var component in this.components) @@ -951,6 +956,9 @@ protected virtual IDisposable RunAsync(ReplayDescriptor descriptor, Clock clock, // wait for component activation to finish this.scheduler.PauseForQuiescence(this.activationContext); + // all components started - pipeline is now running + this.state = State.Running; + // now start scheduling work on the main scheduler context this.scheduler.StartScheduling(this.schedulerContext); @@ -1091,9 +1099,9 @@ private static bool OnlyCyclicInputs(PipelineElement node, PipelineElement origi /// Mapping of emitter IDs to corresponding nodes. /// Known nodes (with inputs) representing Connectors. /// Whether to consider nodes that are members of a pure cycle to be finalizable. - /// Whether to consider only direct cycles (node is it's own parent). + /// Whether to consider only self-cycles (node is it's own parent). /// An indication of eligibility for finalization. - private static bool IsNodeFinalizable(PipelineElement node, Dictionary emitterNodes, Dictionary inputConnectors, bool includeCycles, bool onlyDirectCycles) + private static bool IsNodeFinalizable(PipelineElement node, Dictionary emitterNodes, Dictionary inputConnectors, bool includeCycles, bool onlySelfCycles) { if (node.IsFinalized) { @@ -1115,7 +1123,7 @@ private static bool IsNodeFinalizable(PipelineElement node, Dictionary + /// Raises the event. + /// + /// A that contains the event data. + private void OnPipelineRun(PipelineRunEventArgs e) + { + // ensure that event will only be raised once + if (!this.pipelineRunEventHandled) + { + // raise the PipelineRun event for all subpipelines + this.ForThisPipelineAndAllDescendentSubpipelines(p => + { + p.PipelineRun?.Invoke(p, e); + p.pipelineRunEventHandled = true; + }); + } + } + + /// + /// Raises the event. + /// + /// A that contains the event data. + private void OnPipelineCompleted(PipelineCompletedEventArgs e) + { + this.PipelineCompleted?.Invoke(this, e); + } + /// /// Deactivates all active components. /// @@ -1178,8 +1213,35 @@ private int DeactivateComponents() } /// - /// Finalize child components. + /// Finalizes all components within the pipeline and all subpipelines. The graph of the pipeline is iteratively + /// inspected for nodes (representing components) that are finalizable. On each pass, nodes with no active + /// inputs (i.e. whose receivers are all unsubscribed) are finalized immediately and their emitters are closed. + /// The act of finalizing a node and closing its emitters may in turn cause downstream nodes to also become + /// finalizable if all of their receivers become unsubscribed. This process is repeated until no finalizable + /// nodes are found. Remaining active nodes are then inspected for their participation in cycles. + /// + /// We identify three kinds of nodes, depending on their participation in various types of cycles: + /// - A node in a self-cycle whose active inputs are all directly connected to its outputs. + /// - A node participating in only simple (or pure) cycles where every one of its active inputs (and those of + /// its predecessor nodes) cycle back to itself. + /// - A node which has at least one active input on a directed path that is not a simple cycle back to itself. + /// + /// Once there are no immediately finalizable nodes found, any nodes containing self cycles are finalized next. + /// This may cause direct successor nodes to become finalizable once they have no active inputs. When there are + /// again no more finalizable nodes, the graph is inspected for nodes in pure cycles. A node in the cycle is + /// chosen arbitrarily and finalized to break the cycle. This process is iterated over until all that remains + /// are nodes which have inputs that are not exclusively simple cycles (e.g. a cyclic node with a predecessor + /// that is also on a different cycle), or nodes with such cycles upstream. The node with the most number of + /// outputs (used as a heuristic to finalize terminal nodes last) is finalized, then the remaining nodes are + /// evaluated for finalization using the same order of criteria as before (nodes with no active inputs, nodes + /// with only self-cycles, nodes in only simple cycles, etc.) until all nodes have been finalized. /// + /// + /// Prior to calling this method, all source components should first be deactivated such that they are no + /// longer producing new source messages. The act of finalizing a node may cause it to post new messages to + /// downstream components. It is therefore important to allow the pipeline to drain once there are no longer + /// any nodes without active inputs remaining (these are always safe to finalize immediately). + /// private void FinalizeComponents() { IList nodes = GatherActiveNodes(this).ToList(); // all non-finalized node within pipeline and subpipelines @@ -1192,61 +1254,87 @@ private void FinalizeComponents() { foreach (var output in node.Outputs) { + // used for looking up source nodes from active receiver inputs when testing for cycles emitterNodes.Add(output.Value.Id, node); } if (node.Inputs.Count > 0 && IsConnector(node)) { + // Used for looking up the input-side of a pipeline-bridging connector (these + // have an input node and an output node on each side of the bridge). inputConnectors.Add(node.StateObject, node); } } - // finalize eligible nodes with no active receivers - var finalizable = nodes.Where(n => IsNodeFinalizable(n, emitterNodes, inputConnectors, false, false)); - if (finalizable.Count() == 0) + // initially we exclude nodes which are in a cycle + bool includeCycles = false; + bool onlySelfCycles = false; + + // This is a LINQ expression which queries the list of all active nodes for nodes which are eligible to + // be finalized immediately (i.e. without waiting for messages currently in the pipeline to drain). + // Initially, only nodes which have no active receivers are considered finalizable. If there are no + // such nodes found, messages in the pipeline are allowed to drain. We then progressively loosen the + // requirements by including nodes with self-cycles, followed by nodes which are part of pure cycles + // (nodes which have *only* cyclic inputs). Once a node has been finalized, more nodes may then become + // finalizable as their receivers unsubscribe from the emitters of finalized node. The predicate + // function IsNodeFinalizable() in the LINQ expression is evaluated for each node in the finalization + // loop to determine its eligibility for finalization. If true, then the node may be finalized safely. + var finalizable = nodes.Where(n => IsNodeFinalizable(n, emitterNodes, inputConnectors, includeCycles, onlySelfCycles)); + + // if we cannot find any nodes to finalize + if (!finalizable.Any()) { // try letting messages drain, then look for finalizable nodes again (there may have been closing messages in-flight) this.PauseForQuiescence(); - finalizable = nodes.Where(n => IsNodeFinalizable(n, emitterNodes, inputConnectors, false, false)); - } - - if (finalizable.Count() == 0) - { - // try eliminating direct cycles first (nodes receiving only from themselves - these are completely safe) - finalizable = nodes.Where(n => IsNodeFinalizable(n, emitterNodes, inputConnectors, true, true)); - } - if (finalizable.Count() == 0) - { - // try eliminating indirect cycles (nodes indirectly from themselves - these finalize in arbitrary order!) - finalizable = nodes.Where(n => IsNodeFinalizable(n, emitterNodes, inputConnectors, true, false)); -#if DEBUG - if (finalizable.Count() > 0) + // if we still cannot find any nodes to finalize + if (!finalizable.Any()) { - Debug.WriteLine("FINALIZING INDIRECT CYCLES (UNSAFE ARBITRARY FINALIZATION ORDER)"); - foreach (var node in finalizable) + // include nodes containing self-cycles (nodes receiving only from themselves - these are completely safe to finalize) + includeCycles = true; + onlySelfCycles = true; + + // if there are no nodes containing self-cycles + if (!finalizable.Any()) { - Debug.WriteLine($" FINALIZING {node.Name} {node.StateObject} {node.StateObject.GetType()}"); - } - } + // include nodes in pure cycles (nodes receiving indirectly from themselves - these finalize in arbitrary order!) + onlySelfCycles = false; +#if DEBUG + if (finalizable.Any()) + { + Debug.WriteLine("FINALIZING SIMPLE CYCLES (UNSAFE ARBITRARY FINALIZATION ORDER)"); + foreach (var node in finalizable) + { + Debug.WriteLine($" FINALIZING {node.Name} {node.StateObject} {node.StateObject.GetType()}"); + } + } #endif - } - if (finalizable.Count() == 0) - { - // finally, eliminate all remaining (cycles and nodes with only cycles upstream - these finalize in semi-arbitrary order!) - // finalize remaining nodes in order of number of active outputs (most to least; e.g. terminal nodes last) as a heuristic - finalizable = nodes.OrderBy(n => -n.Outputs.Where(o => o.Value.HasSubscribers).Count()); + // no finalizable nodes found (including self and simple cycles) + if (!finalizable.Any()) + { + // All remaining nodes are either nodes which are part of a non-pure cycle (i.e. not all of its inputs + // and those of its predecessors cycle back to itself), or nodes with such cycles upstream. Pick the + // node with the most number of active inputs to finalize first, then search the graph again for + // finalizable nodes with no active inputs. + var node = nodes.OrderBy(n => -n.Outputs.Where(o => o.Value.HasSubscribers).Count()).FirstOrDefault(); + if (node != null) + { #if DEBUG - if (finalizable.Count() > 0) - { - Debug.WriteLine("ONLY SEPARATED CYCLES REMAINING (UNSAFE ARBITRARY FINALIZATION ORDER)"); - foreach (var node in finalizable) - { - Debug.WriteLine($" FINALIZING {node.Name} {node.StateObject} {node.StateObject.GetType()}"); + Debug.WriteLine("ONLY NON-SIMPLE CYCLES REMAINING (UNSAFE ARBITRARY FINALIZATION ORDER)"); + Debug.WriteLine($" FINALIZING {node.Name} {node.StateObject} {node.StateObject.GetType()}"); +#endif + + // finalize the first node (i.e. with the most number of active inputs) + node.Final(this.FinalOriginatingTime); + } + + // revert to searching for finalizable nodes with no active inputs + includeCycles = false; + onlySelfCycles = false; + } } } -#endif } foreach (var node in finalizable) @@ -1286,7 +1374,7 @@ private void NotifyPipelineFinalizing(DateTime finalOriginatingTime) this.FinalOriginatingTime = finalOriginatingTime; this.schedulerContext.FinalizeTime = finalOriginatingTime; - // propagate the final originating time to all descendent subpipelines and their respective scheduler contexts + // propagate the final originating time to all descendant subpipelines and their respective scheduler contexts foreach (var sub in this.components.Where(c => c.StateObject is Subpipeline && !c.IsFinalized).Select(c => c.StateObject as Subpipeline)) { // if subpipeline is already stopping then don't override its final originating time @@ -1342,8 +1430,12 @@ private double EstimateProgress() } else if (component.StateObject is Subpipeline sub) { - // recursively estimate progress for subpipelines - componentProgress = sub.EstimateProgress(); + // ensures dynamically-added subpipelines are actually running + if (sub.IsRunning) + { + // recursively estimate progress for subpipelines + componentProgress = sub.EstimateProgress(); + } } else if (component.Outputs.Count > 0 || component.Inputs.Count > 0) { diff --git a/Sources/Runtime/Microsoft.Psi/Executive/Subpipeline.cs b/Sources/Runtime/Microsoft.Psi/Executive/Subpipeline.cs index 412d9aa55..2ec6addd4 100644 --- a/Sources/Runtime/Microsoft.Psi/Executive/Subpipeline.cs +++ b/Sources/Runtime/Microsoft.Psi/Executive/Subpipeline.cs @@ -65,7 +65,7 @@ public static Subpipeline Create(Pipeline parent, string name = null, DeliveryPo /// /// This is called by the parent subpipeline, if any. /// Delegate to call to notify of completion time. - void ISourceComponent.Start(Action notifyCompletionTime) + public void Start(Action notifyCompletionTime) { this.notifyCompletionTime = notifyCompletionTime; this.InitializeCompletionTimes(); @@ -75,7 +75,7 @@ void ISourceComponent.Start(Action notifyCompletionTime) } /// - void ISourceComponent.Stop(DateTime finalOriginatingTime, Action notifyCompleted) + public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) { this.notifyCompleted = notifyCompleted; diff --git a/Sources/Runtime/Microsoft.Psi/Microsoft.Psi.csproj b/Sources/Runtime/Microsoft.Psi/Microsoft.Psi.csproj index 00ec08961..30d8b07b1 100644 --- a/Sources/Runtime/Microsoft.Psi/Microsoft.Psi.csproj +++ b/Sources/Runtime/Microsoft.Psi/Microsoft.Psi.csproj @@ -1,10 +1,10 @@ - + netstandard2.0 - true - Microsoft.Psi.Runtime - Provides the core APIs and components for Platform for Situated Intelligence. + true + Microsoft.Psi.Runtime + Provides the core APIs and components for Platform for Situated Intelligence. @@ -46,9 +46,14 @@ + + + all + runtime; build; native; contentfiles; analyzers + @@ -78,8 +83,12 @@ - + + + + + diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Connectors.cs b/Sources/Runtime/Microsoft.Psi/Operators/Connectors.cs index 25349b13d..034dc2159 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Connectors.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Connectors.cs @@ -3,7 +3,6 @@ namespace Microsoft.Psi { - using System; using Microsoft.Psi.Components; /// @@ -12,7 +11,7 @@ namespace Microsoft.Psi public static partial class Operators { /// - /// Connnects a stream producer to a stream consumer. As a result, all messages in the stream will be routed to the consumer for processing. + /// Connects a stream producer to a stream consumer. As a result, all messages in the stream will be routed to the consumer for processing. /// /// The type of messages in the stream. /// The type of consumer. @@ -40,7 +39,29 @@ public static Connector CreateConnector(this Pipeline p, string name) } /// - /// Connnects a stream producer to a stream consumer. As a result, all messages in the stream will be routed to the consumer for processing. + /// Creates a stream in a specified target pipeline, based on a given input stream (that may belong in a different pipeline). + /// + /// The type of the messages on the input stream. + /// The input stream. + /// Pipeline to which to bridge. + /// An optional name for the connector (defaults to BridgeConnector). + /// An optional delivery policy. + /// The bridged stream. + public static IProducer BridgeTo(this IProducer input, Pipeline targetPipeline, string name = null, DeliveryPolicy deliveryPolicy = null) + { + if (input.Out.Pipeline == targetPipeline) + { + return input; + } + else + { + var connector = new Connector(input.Out.Pipeline, targetPipeline, name ?? "BridgeConnector"); + return input.PipeTo(connector, deliveryPolicy); + } + } + + /// + /// Connects a stream producer to a stream consumer. As a result, all messages in the stream will be routed to the consumer for processing. /// /// /// This is an internal-only method which provides the option to allow connections between producers and consumers in running pipelines. diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Enumerable.cs b/Sources/Runtime/Microsoft.Psi/Operators/Enumerable.cs index 186ac5a86..b4b3d5ea7 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Enumerable.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Enumerable.cs @@ -35,7 +35,7 @@ public static IEnumerable ToEnumerable(this IProducer source, Func /// Type of stream messages. - public class StreamEnumerable : IEnumerable, IEnumerable + public class StreamEnumerable : IEnumerable, IEnumerable, IDisposable { private readonly StreamEnumerator enumerator; @@ -61,6 +61,12 @@ public StreamEnumerable(IProducer source, Func predicate = null, Del processor.In.Unsubscribed += _ => this.enumerator.Update.Set(); } + /// + public void Dispose() + { + this.enumerator.Dispose(); + } + /// public IEnumerator GetEnumerator() { @@ -98,6 +104,7 @@ public StreamEnumerator(Func predicate) public void Dispose() { + this.enqueued.Dispose(); } public bool MoveNext() diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Generators.cs b/Sources/Runtime/Microsoft.Psi/Operators/Generators.cs index ffa011664..0e0edb0c9 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Generators.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Generators.cs @@ -22,7 +22,7 @@ public static class Generators /// The function that generates a new value based on the previous value. /// The number of messages to publish. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// Indicates whether the stream should be kept open after all messages in the sequence have been posted. @@ -41,7 +41,7 @@ public static IProducer Sequence(Pipeline pipeline, T initialValue, FuncThe initial value. /// The function that generates a new value based on the previous value. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// A stream of values of type T. @@ -58,7 +58,7 @@ public static IProducer Sequence(Pipeline pipeline, T initialValue, FuncThe pipeline that will run this generator. /// The sequence to publish. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// Indicates whether the stream should be kept open after all messages in the sequence have been posted. @@ -121,7 +121,7 @@ public static IProducer Return(Pipeline pipeline, T value) /// The value to publish. /// The number of messages to publish. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// Indicates whether the stream should be kept open after the specified number of messages have been posted. @@ -139,7 +139,7 @@ public static IProducer Repeat(Pipeline pipeline, T value, int count, Time /// The pipeline that will run this generator. /// The value to publish. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// A stream of values of type T. @@ -156,7 +156,7 @@ public static IProducer Repeat(Pipeline pipeline, T value, TimeSpan interv /// The starting value. /// The number of messages to publish. /// The desired time interval between consecutive messages. Defaults to 1 tick. - /// If non-null, this parameter specifies a time to align the generator messages with. If the paramater + /// If non-null, this parameter specifies a time to align the generator messages with. If the parameter /// is non-null, the messages will have originating times that align with the specified time. /// A stream of consecutive integers. /// Indicates whether the stream should be kept open after the specified number of messages have been posted. diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Interpolators.cs b/Sources/Runtime/Microsoft.Psi/Operators/Interpolators.cs index 58ba7a4d4..6140c1cbf 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Interpolators.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Interpolators.cs @@ -19,7 +19,7 @@ public static partial class Operators /// Source stream. /// Interval at which to apply the interpolator. /// Interpolator to use for generating results. - /// If non-null, this parameter specifies a time to align the sampling messages with. If the paramater + /// If non-null, this parameter specifies a time to align the sampling messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// An optional delivery policy. @@ -69,7 +69,7 @@ public static partial class Operators /// Source stream. /// Interval at which to apply the interpolator. /// The tolerance within which to search for the nearest message. - /// If non-null, this parameter specifies a time to align the sampling messages with. If the paramater + /// If non-null, this parameter specifies a time to align the sampling messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// An optional delivery policy. @@ -95,8 +95,10 @@ public static partial class Operators /// Type of source (and output) messages. /// Source stream. /// Interval at which to apply the interpolator. - /// The relative time interval within which to search for the nearest message. - /// If non-null, this parameter specifies a time to align the sampling messages with. If the paramater + /// The relative time interval within which to search for the nearest message. + /// If the parameter is not specified the relative time interval is + /// used,resulting in sampling the nearest point to the clock signal on the source stream. + /// If non-null, this parameter specifies a time to align the sampling messages with. If the parameter /// is non-null, the messages will have originating times that align with (i.e., are an integral number of intervals away from) the /// specified alignment time. /// An optional delivery policy. @@ -104,10 +106,11 @@ public static partial class Operators public static IProducer Sample( this IProducer source, TimeSpan samplingInterval, - RelativeTimeInterval relativeTimeInterval, + RelativeTimeInterval relativeTimeInterval = null, DateTime? alignmentDateTime = null, DeliveryPolicy deliveryPolicy = null) { + relativeTimeInterval ??= RelativeTimeInterval.Infinite; return source.Interpolate( samplingInterval, Reproducible.Nearest(relativeTimeInterval), @@ -145,17 +148,20 @@ public static partial class Operators /// Type of messages on the clock stream. /// Source stream. /// Clock stream that dictates the interpolation points. - /// The relative time interval within which to search for the nearest message. + /// The relative time interval within which to search for the nearest message. + /// If the parameter is not specified the relative time interval is + /// used,resulting in sampling the nearest point to the clock signal on the source stream. /// An optional delivery policy for the source stream. /// An optional delivery policy for the clock stream. /// Sampled stream. public static IProducer Sample( this IProducer source, IProducer clock, - RelativeTimeInterval relativeTimeInterval, + RelativeTimeInterval relativeTimeInterval = null, DeliveryPolicy sourceDeliveryPolicy = null, DeliveryPolicy clockDeliveryPolicy = null) { + relativeTimeInterval ??= RelativeTimeInterval.Infinite; return source.Interpolate(clock, Reproducible.Nearest(relativeTimeInterval), sourceDeliveryPolicy, clockDeliveryPolicy); } } diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Joins.cs b/Sources/Runtime/Microsoft.Psi/Operators/Joins.cs index f3df29a83..5c2179c02 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Joins.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Joins.cs @@ -1438,7 +1438,7 @@ public static partial class Operators } /// - /// Joins a primary stream of integers with an enumeration of seconary streams based on a specified reproducible interpolator. + /// Joins a primary stream of integers with an enumeration of secondary streams based on a specified reproducible interpolator. /// /// Type of input messages. /// Type of the interpolation result. diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Processors.cs b/Sources/Runtime/Microsoft.Psi/Operators/Processors.cs index 97442d66a..11e821178 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Processors.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Processors.cs @@ -4,6 +4,8 @@ namespace Microsoft.Psi { using System; + using System.Collections.Generic; + using System.Linq; using Microsoft.Psi.Components; /// @@ -166,5 +168,29 @@ public static IProducer Do(this IProducer source, Action action, Del { return Do(source, (d, e) => action(d), deliveryPolicy); } + + /// + /// Edit messages in a stream; applying updates/inserts and deletes. + /// + /// The input message type. + /// The source stream to edit. + /// A sequence of edits to be applied. Whether to update/insert or delete, an optional message to upsert and originating times. + /// An optional delivery policy. + /// A stream of the same type as the source stream with edits applied. + internal static IProducer EditStream(this IProducer source, IEnumerable<(bool upsert, T message, DateTime originatingTime)> edits, DeliveryPolicy deliveryPolicy = null) + { + var originalStream = source.Select(m => (original: true, upsert: true, m), deliveryPolicy); + var orderedEdits = edits.OrderBy(e => e.originatingTime).Select(e => ((original: false, e.upsert, e.message), e.originatingTime)); + var editsStream = Generators.Sequence(source.Out.Pipeline, orderedEdits); + return originalStream.Zip(editsStream, DeliveryPolicy.Unlimited).Process<(bool original, bool upsert, T message)[], T>( + ((bool original, bool upsert, T message)[] messages, Envelope envelope, Emitter emitter) => + { + var m = messages[0].original && messages.Length > 1 ? messages[1] : messages[0]; + if (m.upsert) + { + emitter.Post(m.message, envelope.OriginatingTime); + } + }); + } } } \ No newline at end of file diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Statistical.cs b/Sources/Runtime/Microsoft.Psi/Operators/Statistical.cs index b88fefc96..ff62a410f 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Statistical.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Statistical.cs @@ -711,7 +711,7 @@ public static IProducer Std(this IProducer source, DeliveryPol var oldmean = mean; mean = mean + ((value - mean) / count); q = q + ((value - oldmean) * (value - mean)); - return count == 1 ? 0m : (decimal)Math.Sqrt((double)q / (count - 1)); // reture 0m for first value + return count == 1 ? 0m : (decimal)Math.Sqrt((double)q / (count - 1)); // return 0m for first value }, deliveryPolicy); } diff --git a/Sources/Runtime/Microsoft.Psi/Operators/VectorProcessors.cs b/Sources/Runtime/Microsoft.Psi/Operators/VectorProcessors.cs index 47848a7cd..5659ba3df 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/VectorProcessors.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/VectorProcessors.cs @@ -427,6 +427,49 @@ public static partial class Operators return source.PipeTo(p, deliveryPolicy); } + /// + /// Transforms a stream of messages by splitting it into a set of substreams (indexed by a key), + /// applying a sub-pipeline to each of these streams, and assembling the results into a corresponding + /// output stream. + /// + /// The type of input messages. + /// Type of the substream key. + /// Type of the substream messages. + /// Type of the subpipeline output for each substream. + /// Source stream. + /// A function that splits the input by generating a dictionary of key-value pairs for each given input message. + /// Stream transform to be applied to each substream. + /// When true, a result is produced even if a message is dropped in processing one of the input elements. In this case the corresponding output element is set to default. + /// Default value to use when messages are dropped in processing one of the input elements. + /// An optional delivery policy. + /// Predicate function determining whether and when (originating time) to terminate branches (defaults to when key no longer present), given the current key, dictionary of values and the originating time of the last message containing the key. + /// Name for the parallel composite component (defaults to ParallelSparse). + /// Pipeline-level default delivery policy to be used by the parallel composite component (defaults to if unspecified). + /// Stream of output dictionaries. + public static IProducer> Parallel( + this IProducer source, + Func> splitter, + Func, IProducer> streamTransform, + bool outputDefaultIfDropped = false, + TBranchOut defaultValue = default, + DeliveryPolicy deliveryPolicy = null, + Func, DateTime, (bool, DateTime)> branchTerminationPolicy = null, + string name = null, + DeliveryPolicy defaultParallelDeliveryPolicy = null) + { + var p = new ParallelSparseSelect>( + source.Out.Pipeline, + splitter, + (k, s) => streamTransform(s), + _ => _, + outputDefaultIfDropped, + defaultValue, + branchTerminationPolicy, + name, + defaultParallelDeliveryPolicy); + return source.PipeTo(p, deliveryPolicy); + } + /// /// Transforms a stream of dictionary messages by creating a stream for each key in the dictionary, /// applying a sub-pipeline to each of these streams, and assembling the results into a corresponding output @@ -436,7 +479,7 @@ public static partial class Operators /// Type of input dictionary values. /// Type of output dictionary values. /// Source stream. - /// Function mapping from an input element stream to an output output element stream. + /// Function mapping from an input element stream to an output element stream. /// When true, a result is produced even if a message is dropped in processing one of the input elements. In this case the corresponding output element is set to default. /// Default value to use when messages are dropped in processing one of the input elements. /// An optional delivery policy. diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Windows.cs b/Sources/Runtime/Microsoft.Psi/Operators/Windows.cs index a552ead12..8b524953b 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Windows.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Windows.cs @@ -88,23 +88,6 @@ public static IProducer Window(this IProducer sourc return PipeTo(source, window, deliveryPolicy); } - /// - /// Process windows of messages by relative time interval. - /// - /// Type of source messages. - /// Type of output messages. - /// Source stream of messages. - /// The relative time interval over which to gather messages. - /// Selector function. - /// Whether to wait for the full window before output. - /// An optional delivery policy. - /// Output stream. - public static IProducer Window(this IProducer source, RelativeTimeInterval relativeTimeInterval, Func>, TOutput> selector, bool waitForCompleteWindow, DeliveryPolicy deliveryPolicy = null) - { - var window = new RelativeTimeWindow(source.Out.Pipeline, relativeTimeInterval, selector, waitForCompleteWindow); - return PipeTo(source, window, deliveryPolicy); - } - /// /// Process windows of messages by relative time interval. /// diff --git a/Sources/Runtime/Microsoft.Psi/Operators/Zips.cs b/Sources/Runtime/Microsoft.Psi/Operators/Zips.cs index 84d6462e0..21592184e 100644 --- a/Sources/Runtime/Microsoft.Psi/Operators/Zips.cs +++ b/Sources/Runtime/Microsoft.Psi/Operators/Zips.cs @@ -14,18 +14,20 @@ namespace Microsoft.Psi public static partial class Operators { /// - /// Zip one or more streams (T) into a single stream (Message{T}) while ensuring delivery in originating time order (ordered within single tick by stream ID). + /// Zip one or more streams (T) into a single stream while ensuring delivery in originating time order. /// - /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// If multiple messages arrive with the same originating time, they are added in the output array in + /// the order of stream ids. /// Type of messages. - /// Collection of homogeneous inputs. + /// Collection of input streams to zip. /// An optional delivery policy. /// Stream of zipped messages. - public static IProducer> Zip(IEnumerable> inputs, DeliveryPolicy deliveryPolicy = null) + public static IProducer Zip(IEnumerable> inputs, DeliveryPolicy deliveryPolicy = null) { if (inputs.Count() == 0) { - throw new ArgumentException("Zip requires one or more inputs."); + throw new ArgumentException($"{nameof(Zip)} requires one or more inputs."); } var zip = new Zip(inputs.First().Out.Pipeline); @@ -38,15 +40,17 @@ public static IProducer> Zip(IEnumerable> inputs, Del } /// - /// Zip two streams (T) into a single stream (Message{T}) while ensuring delivery in originating time order (ordered within single tick by stream ID). + /// Zip two streams (T) into a single stream while ensuring delivery in originating time order. /// - /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// Messages are produced in originating-time order; potentially delayed in wall-clock time. + /// If multiple messages arrive with the same originating time, they are added in the output array in + /// the order of stream ids. /// Type of messages. /// First input stream. /// Second input stream with same message type. /// An optional delivery policy. /// Stream of zipped messages. - public static IProducer> Zip(this IProducer input1, IProducer input2, DeliveryPolicy deliveryPolicy = null) + public static IProducer Zip(this IProducer input1, IProducer input2, DeliveryPolicy deliveryPolicy = null) { return Zip(new List>() { input1, input2 }, deliveryPolicy); } diff --git a/Sources/Runtime/Microsoft.Psi/Persistence/IndexEntry.cs b/Sources/Runtime/Microsoft.Psi/Persistence/IndexEntry.cs index 1d1124979..e63eea4ac 100644 --- a/Sources/Runtime/Microsoft.Psi/Persistence/IndexEntry.cs +++ b/Sources/Runtime/Microsoft.Psi/Persistence/IndexEntry.cs @@ -11,7 +11,7 @@ namespace Microsoft.Psi.Persistence /// /// This structure is used in two places: the index file and the large data file. /// To facilitate seeking, each data file is accompanied by an index file containing records of this type. - /// Each record indicates the largest time and orginating time values seen up to the specified position. + /// Each record indicates the largest time and originating time values seen up to the specified position. /// The position is a composite value, consisting of the extent and the relative position within the extent. /// These records allow seeking close to (but guaranteed before) a given time. /// Reading from the position provided by the index entry guarantees that all the messages with the @@ -19,7 +19,7 @@ namespace Microsoft.Psi.Persistence /// /// To enable efficient reading of streams, the Store breaks streams in two categories: small and large. /// When writing large messages, an index entry is written into the main data file, - /// pointing ot a location in the large data file where the actual message resides. + /// pointing to a location in the large data file where the actual message resides. /// public struct IndexEntry { diff --git a/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileReader.cs b/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileReader.cs index d971d928e..f9ddcb8f0 100644 --- a/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileReader.cs +++ b/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileReader.cs @@ -79,6 +79,9 @@ public void Dispose() { this.writePulse.Dispose(); this.CloseCurrent(); + + // may have already been disposed in CloseCurrent + this.view?.Dispose(); } // Seeks to the next block (assumes the position points to a block entry) @@ -216,10 +219,8 @@ private void LoadNextExtent() { try { - using (var file = File.Open(fullName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) - { - this.mappedFile = MemoryMappedFile.CreateFromFile(file, null, 0, MemoryMappedFileAccess.Read, HandleInheritability.Inheritable, false); - } + var file = File.Open(fullName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); + this.mappedFile = MemoryMappedFile.CreateFromFile(file, null, 0, MemoryMappedFileAccess.Read, HandleInheritability.Inheritable, false); } catch (IOException) { diff --git a/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs b/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs index ee2529486..c4a596949 100644 --- a/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs +++ b/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileWriter.cs @@ -140,6 +140,9 @@ public void Dispose() this.globalWritePulse = null; this.activeWriterMutex.Dispose(); this.activeWriterMutex = null; + + // may have already been disposed in CloseCurrent + this.view?.Dispose(); } public void Write(BufferWriter bufferWriter) diff --git a/Sources/Runtime/Microsoft.Psi/Persistence/StoreReader.cs b/Sources/Runtime/Microsoft.Psi/Persistence/StoreReader.cs index 1ebe2e728..3b5b81cf1 100644 --- a/Sources/Runtime/Microsoft.Psi/Persistence/StoreReader.cs +++ b/Sources/Runtime/Microsoft.Psi/Persistence/StoreReader.cs @@ -187,7 +187,7 @@ public void CloseStream(int id) } /// - /// Closes allthe storage streams. + /// Closes all the storage streams. /// public void CloseAllStreams() { diff --git a/Sources/Runtime/Microsoft.Psi/Persistence/StoreWriter.cs b/Sources/Runtime/Microsoft.Psi/Persistence/StoreWriter.cs index e72127631..195ebe11a 100644 --- a/Sources/Runtime/Microsoft.Psi/Persistence/StoreWriter.cs +++ b/Sources/Runtime/Microsoft.Psi/Persistence/StoreWriter.cs @@ -122,20 +122,20 @@ public void Dispose() /// /// Creates a logical storage stream to write messages to. - /// The storage stream characteristics are extracted form the prvoided metadata descriptor. + /// The storage stream characteristics are extracted from the provided metadata descriptor. /// /// The metadata describing the stream to open. /// The complete metadata for the storage stream just created. public PsiStreamMetadata OpenStream(PsiStreamMetadata meta) { - return this.OpenStream(meta.Id, meta.Name, meta.IsIndexed, meta.TypeName); + return this.OpenStream(meta.Id, meta.Name, meta.IsIndexed, meta.TypeName).UpdateSupplementalMetadataFrom(meta); } /// /// Creates a logical storage stream to write messages to. /// /// The id of the stream, unique for this store. All messages with this stream id will be written to this storage stream. - /// The name of the stream. This name can be later used to open the sorage stream for reading. + /// The name of the stream. This name can be later used to open the storage stream for reading. /// Indicates whether the stream is indexed or not. Indexed streams have a small index entry in the main data file and the actual message body in a large data file. /// A name identifying the type of the messages in this stream. This is usually a fully-qualified type name or a data contract name, but can be anything that the caller wants. /// The complete metadata for the storage stream just created. diff --git a/Sources/Runtime/Microsoft.Psi/Remoting/RemoteExporter.cs b/Sources/Runtime/Microsoft.Psi/Remoting/RemoteExporter.cs index 1a0cbe525..cd3f890ff 100644 --- a/Sources/Runtime/Microsoft.Psi/Remoting/RemoteExporter.cs +++ b/Sources/Runtime/Microsoft.Psi/Remoting/RemoteExporter.cs @@ -142,6 +142,8 @@ public void Dispose() this.connections = null; this.metaClientThread = null; this.dataClientThread = null; + + this.dataTransport.Dispose(); } private void AddConnection(Connection connection) @@ -302,10 +304,10 @@ public void Connect() this.stream.Write(writer.Buffer, 0, len); this.storeReader = new StoreReader(this.storeName, this.storePath, this.MetaUpdateHandler, true); } - catch (Exception ex) + catch (Exception) { this.Disconnect(); - throw ex; + throw; } } diff --git a/Sources/Runtime/Microsoft.Psi/Remoting/RemoteImporter.cs b/Sources/Runtime/Microsoft.Psi/Remoting/RemoteImporter.cs index 748ed091c..ceb66e5df 100644 --- a/Sources/Runtime/Microsoft.Psi/Remoting/RemoteImporter.cs +++ b/Sources/Runtime/Microsoft.Psi/Remoting/RemoteImporter.cs @@ -152,6 +152,8 @@ public void Dispose() this.storeWriter.Dispose(); this.storeWriter = null; + + this.connected.Dispose(); } private void StartMetaClient() diff --git a/Sources/Runtime/Microsoft.Psi/Remoting/Transport/TcpTransport.cs b/Sources/Runtime/Microsoft.Psi/Remoting/Transport/TcpTransport.cs index 3c70f3e41..14bd2875a 100644 --- a/Sources/Runtime/Microsoft.Psi/Remoting/Transport/TcpTransport.cs +++ b/Sources/Runtime/Microsoft.Psi/Remoting/Transport/TcpTransport.cs @@ -116,6 +116,7 @@ public void WriteMessage(Envelope envelope, byte[] message) public void Dispose() { + this.stream.Close(); this.client.Dispose(); this.client = null; } diff --git a/Sources/Runtime/Microsoft.Psi/Scheduling/PriorityQueue.cs b/Sources/Runtime/Microsoft.Psi/Scheduling/PriorityQueue.cs index 1c8c41e56..12beefe8c 100644 --- a/Sources/Runtime/Microsoft.Psi/Scheduling/PriorityQueue.cs +++ b/Sources/Runtime/Microsoft.Psi/Scheduling/PriorityQueue.cs @@ -10,14 +10,14 @@ namespace Microsoft.Psi.Scheduling /// A generic ordered queue that sorts items based on the specified Comparer. /// /// Type of item in the list. - public abstract class PriorityQueue + public abstract class PriorityQueue : IDisposable { // the head of the ordered work item list is always empty private readonly PriorityQueueNode head = new PriorityQueueNode(0); private readonly PriorityQueueNode emptyHead = new PriorityQueueNode(0); private readonly Comparison comparer; + private readonly ManualResetEvent empty = new ManualResetEvent(true); private IPerfCounterCollection counters; - private ManualResetEvent empty = new ManualResetEvent(true); private int count; private int nextId; @@ -38,6 +38,12 @@ public PriorityQueue(string name, Comparison comparer) internal WaitHandle Empty => this.empty; + /// + public void Dispose() + { + this.empty.Dispose(); + } + /// /// Try peeking at first item; returning indication of success. /// diff --git a/Sources/Runtime/Microsoft.Psi/Scheduling/Scheduler.cs b/Sources/Runtime/Microsoft.Psi/Scheduling/Scheduler.cs index 6e3efc574..b0ab60c51 100644 --- a/Sources/Runtime/Microsoft.Psi/Scheduling/Scheduler.cs +++ b/Sources/Runtime/Microsoft.Psi/Scheduling/Scheduler.cs @@ -9,7 +9,7 @@ namespace Microsoft.Psi.Scheduling /// /// Maintains a queue of workitems and schedules worker threads to empty them. /// - public sealed class Scheduler + public sealed class Scheduler : IDisposable { private readonly string name; private readonly SimpleSemaphore threadSemaphore; @@ -17,6 +17,9 @@ public sealed class Scheduler private readonly bool allowSchedulingOnExternalThreads; private readonly ManualResetEvent stopped = new ManualResetEvent(true); private readonly AutoResetEvent futureAdded = new AutoResetEvent(false); + private readonly ThreadLocal nextWorkitem = new ThreadLocal(); + private readonly ThreadLocal isSchedulerThread = new ThreadLocal(() => false); + private readonly ThreadLocal currentWorkitemTime = new ThreadLocal(() => DateTime.MaxValue); // the queue of pending workitems, ordered by start time private readonly WorkItemQueue globalWorkitems; @@ -24,9 +27,6 @@ public sealed class Scheduler private Thread futuresThread; private IPerfCounterCollection counters; private bool forcedShutdownRequested; - private ThreadLocal nextWorkitem = new ThreadLocal(); - private ThreadLocal isSchedulerThread = new ThreadLocal(() => false); - private ThreadLocal currentWorkitemTime = new ThreadLocal(() => DateTime.MaxValue); private Clock clock; private bool delayFutureWorkitemsUntilDue; private bool started = false; @@ -77,6 +77,19 @@ internal bool IsStarted } } + /// + public void Dispose() + { + this.threadSemaphore.Dispose(); + this.stopped.Dispose(); + this.futureAdded.Dispose(); + this.globalWorkitems.Dispose(); + this.futureWorkitems.Dispose(); + this.nextWorkitem.Dispose(); + this.isSchedulerThread.Dispose(); + this.currentWorkitemTime.Dispose(); + } + /// /// Attempts to execute the delegate immediately, on the calling thread, without scheduling. /// diff --git a/Sources/Runtime/Microsoft.Psi/Scheduling/SchedulerContext.cs b/Sources/Runtime/Microsoft.Psi/Scheduling/SchedulerContext.cs index 3a8919960..a9d1eb504 100644 --- a/Sources/Runtime/Microsoft.Psi/Scheduling/SchedulerContext.cs +++ b/Sources/Runtime/Microsoft.Psi/Scheduling/SchedulerContext.cs @@ -11,7 +11,7 @@ namespace Microsoft.Psi.Scheduling /// Maintains a count of the number of work items currently in-flight and an event /// to signal when there are no remaining work items in the context. /// - public sealed class SchedulerContext + public sealed class SchedulerContext : IDisposable { private readonly object syncLock = new object(); private readonly ManualResetEvent empty = new ManualResetEvent(true); @@ -34,6 +34,12 @@ public sealed class SchedulerContext internal bool Started { get; private set; } = false; + /// + public void Dispose() + { + this.empty.Dispose(); + } + /// /// Starts scheduling work on the context. /// diff --git a/Sources/Runtime/Microsoft.Psi/Scheduling/SimpleSemaphore.cs b/Sources/Runtime/Microsoft.Psi/Scheduling/SimpleSemaphore.cs index 367fa613c..4c7215dd2 100644 --- a/Sources/Runtime/Microsoft.Psi/Scheduling/SimpleSemaphore.cs +++ b/Sources/Runtime/Microsoft.Psi/Scheduling/SimpleSemaphore.cs @@ -3,17 +3,18 @@ namespace Microsoft.Psi.Scheduling { + using System; using System.Threading; /// /// Implements a semaphore class that limits the number of threads entering a resource and provides an event when all threads finished. /// - public class SimpleSemaphore + public class SimpleSemaphore : IDisposable { + private readonly ManualResetEvent empty; + private readonly ManualResetEvent available; private readonly int maxThreadCount; private int count; - private ManualResetEvent empty; - private ManualResetEvent available; /// /// Initializes a new instance of the class. @@ -41,6 +42,13 @@ public SimpleSemaphore(int maxThreadCount) /// public int MaxThreadCount => this.maxThreadCount; + /// + public void Dispose() + { + this.empty.Dispose(); + this.available.Dispose(); + } + /// /// Try to enter the semaphore. /// diff --git a/Sources/Runtime/Microsoft.Psi/Serialization/KnownSerializers.cs b/Sources/Runtime/Microsoft.Psi/Serialization/KnownSerializers.cs index c6b380f9b..4d06540de 100644 --- a/Sources/Runtime/Microsoft.Psi/Serialization/KnownSerializers.cs +++ b/Sources/Runtime/Microsoft.Psi/Serialization/KnownSerializers.cs @@ -423,7 +423,14 @@ private SerializationHandler AddHandler() name = TypeSchema.GetContractName(type, this.runtimeVersion); } - int id = this.schemas.TryGetValue(name, out schema) ? schema.Id : TypeSchema.GetId(name); + if (!this.schemas.TryGetValue(name, out schema)) + { + // try to match to an existing schema without assembly/version info + string typeName = TypeResolutionHelper.RemoveAssemblyName(type.AssemblyQualifiedName); + schema = this.schemas.Values.FirstOrDefault(s => TypeResolutionHelper.RemoveAssemblyName(s.TypeName) == typeName); + } + + int id = schema?.Id ?? TypeSchema.GetId(name); serializer = this.CreateSerializer(); handler = SerializationHandler.Create(serializer, name, id); diff --git a/Sources/Runtime/Microsoft.Psi/Serialization/RefHandler.cs b/Sources/Runtime/Microsoft.Psi/Serialization/RefHandler.cs index 0769fe7b2..41575138d 100644 --- a/Sources/Runtime/Microsoft.Psi/Serialization/RefHandler.cs +++ b/Sources/Runtime/Microsoft.Psi/Serialization/RefHandler.cs @@ -100,7 +100,7 @@ public override void Deserialize(BufferReader reader, ref T target, Serializatio if (target == null) { // a custom serializer was implemented incorrectly - throw new InvalidDataException("The serializer detected an unresolved circular reference to an instance of type {typeof(T)}. The custom serializer for this type needs to implement the ISerializerEx interface."); + throw new InvalidDataException($"The serializer detected an unresolved circular reference to an instance of type {typeof(T)}. The custom serializer for this type needs to implement the ISerializerEx interface."); } return; diff --git a/Sources/Runtime/Microsoft.Psi/Serialization/StructHandler.cs b/Sources/Runtime/Microsoft.Psi/Serialization/StructHandler.cs index 24fe52951..3a15d6ca5 100644 --- a/Sources/Runtime/Microsoft.Psi/Serialization/StructHandler.cs +++ b/Sources/Runtime/Microsoft.Psi/Serialization/StructHandler.cs @@ -27,7 +27,7 @@ public StructHandler(ISerializer innerSerializer, string contractName, int id if (typeof(T).IsByRef) { - throw new InvalidOperationException("Cannot use a valut type handler with a class serializer"); + throw new InvalidOperationException("Cannot use a value type handler with a class serializer"); } } diff --git a/Sources/Runtime/Microsoft.Psi/Streams/Emitter{T}.cs b/Sources/Runtime/Microsoft.Psi/Streams/Emitter{T}.cs index ef6d9e99f..e4152002e 100644 --- a/Sources/Runtime/Microsoft.Psi/Streams/Emitter{T}.cs +++ b/Sources/Runtime/Microsoft.Psi/Streams/Emitter{T}.cs @@ -18,7 +18,7 @@ namespace Microsoft.Psi /// /// The type of messages in the stream. [Serializer(typeof(Emitter<>.NonSerializer))] - public class Emitter : IEmitter, IProducer + public sealed class Emitter : IEmitter, IProducer { private readonly object owner; private readonly Pipeline pipeline; diff --git a/Sources/Runtime/Microsoft.Psi/Streams/Receiver{T}.cs b/Sources/Runtime/Microsoft.Psi/Streams/Receiver{T}.cs index 23dade3ae..597b505cf 100644 --- a/Sources/Runtime/Microsoft.Psi/Streams/Receiver{T}.cs +++ b/Sources/Runtime/Microsoft.Psi/Streams/Receiver{T}.cs @@ -24,7 +24,7 @@ namespace Microsoft.Psi /// /// The type of messages that can be received. [Serializer(typeof(Receiver<>.NonSerializer))] - public class Receiver : IReceiver, IConsumer + public sealed class Receiver : IReceiver, IConsumer { private readonly Action> onReceived; private readonly PipelineElement element; diff --git a/Sources/Runtime/Test.Psi.Windows/Test.Psi.Windows.csproj b/Sources/Runtime/Test.Psi.Windows/Test.Psi.Windows.csproj index cc45db11b..2a02190c1 100644 --- a/Sources/Runtime/Test.Psi.Windows/Test.Psi.Windows.csproj +++ b/Sources/Runtime/Test.Psi.Windows/Test.Psi.Windows.csproj @@ -1,20 +1,17 @@  net472 - false - false + ../../../Build/Test.Psi.ruleset Exe Test.Psi.ConsoleMain - ../../../Build/Test.Psi.ruleset true AnyCPU - ../../../Build/Test.Psi.ruleset true AnyCPU @@ -34,6 +31,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Sources/Runtime/Test.Psi/APITester.cs b/Sources/Runtime/Test.Psi/APITester.cs index efba5d76d..babb2d9aa 100644 --- a/Sources/Runtime/Test.Psi/APITester.cs +++ b/Sources/Runtime/Test.Psi/APITester.cs @@ -10,7 +10,7 @@ namespace Test.Psi using Microsoft.VisualStudio.TestTools.UnitTesting; /// - /// Scenarios and usage around simple pipelines, join and repeat + /// Scenarios and usage around simple pipelines, join and repeat. /// [TestClass] public class APITester diff --git a/Sources/Runtime/Test.Psi/CustomSerializationTester.cs b/Sources/Runtime/Test.Psi/CustomSerializationTester.cs index 30c0b16d9..68bde3cb7 100644 --- a/Sources/Runtime/Test.Psi/CustomSerializationTester.cs +++ b/Sources/Runtime/Test.Psi/CustomSerializationTester.cs @@ -61,7 +61,7 @@ public class TypeWithPolymorphicField // serializer that skips one property public class TestCustomSerializer : ISerializer { - public int Version => throw new NotImplementedException(); + public int Version => throw new NotSupportedException(); public TypeSchema Initialize(KnownSerializers serializers, TypeSchema targetSchema) { diff --git a/Sources/Runtime/Test.Psi/DeliveryPolicyTest.cs b/Sources/Runtime/Test.Psi/DeliveryPolicyTest.cs index d2bda5d22..b13d5d926 100644 --- a/Sources/Runtime/Test.Psi/DeliveryPolicyTest.cs +++ b/Sources/Runtime/Test.Psi/DeliveryPolicyTest.cs @@ -33,7 +33,7 @@ public void ThrottledTimer() p.WaitAll(100); } - // Timer continues to post so messages will be dropped at receiver B until C stops throttling it + // Timer continues to post so messages will be dropped at receiver B until C stops throttling it Assert.IsTrue(countA > 0); Assert.IsTrue(countA > countB); Assert.AreEqual(countB, countC); @@ -77,12 +77,13 @@ public void LatestDelivery() using (var p = Pipeline.Create()) { var numGen = Generators.Range(p, 10, 3, TimeSpan.FromMilliseconds(100)); - numGen.Do(m => - { - Thread.Sleep(500); - loopCount++; - lastMsg = m; - }, DeliveryPolicy.LatestMessage); + numGen.Do( + m => + { + Thread.Sleep(500); + loopCount++; + lastMsg = m; + }, DeliveryPolicy.LatestMessage); p.RunAsync(); p.WaitAll(700); @@ -252,18 +253,25 @@ public void DeliveryPolicyWithGuarantees() p.Run(); } - // with latest we only get the first message since the others are dropped, and the - // last message. - CollectionAssert.AreEqual(latest, new List() { 0, 9 }); + // with latest we may drop some messages except the last message. + CollectionAssert.IsSubsetOf(new List() { 9 }, latest); + CollectionAssert.AllItemsAreUnique(latest); // ensure uniqueness + CollectionAssert.AreEqual(latest.OrderBy(x => x).ToList(), latest); // ensure order // with guarantees we get all the even messages. - CollectionAssert.AreEqual(latestWithGuarantees, new List() { 0, 2, 4, 6, 8 }); + CollectionAssert.IsSubsetOf(new List() { 0, 2, 4, 6, 8 }, latestWithGuarantees); + CollectionAssert.AllItemsAreUnique(latestWithGuarantees); + CollectionAssert.AreEqual(latestWithGuarantees.OrderBy(x => x).ToList(), latestWithGuarantees); // with guarantees we get all the even and multiple of 3 messages. - CollectionAssert.AreEqual(latestWithGuaranteesChained, new List() { 0, 2, 3, 4, 6, 8, 9 }); + CollectionAssert.IsSubsetOf(new List() { 0, 2, 3, 4, 6, 8, 9 }, latestWithGuaranteesChained); + CollectionAssert.AllItemsAreUnique(latestWithGuaranteesChained); + CollectionAssert.AreEqual(latestWithGuaranteesChained.OrderBy(x => x).ToList(), latestWithGuaranteesChained); // with guarantees, even when queue size is 2, we get all the even messages. - CollectionAssert.AreEqual(queueSizeTwoWithGuarantees, new List() { 0, 2, 4, 6, 8 }); + CollectionAssert.IsSubsetOf(new List() { 0, 2, 4, 6, 8 }, queueSizeTwoWithGuarantees); + CollectionAssert.AllItemsAreUnique(queueSizeTwoWithGuarantees); + CollectionAssert.AreEqual(queueSizeTwoWithGuarantees.OrderBy(x => x).ToList(), queueSizeTwoWithGuarantees); } } } \ No newline at end of file diff --git a/Sources/Runtime/Test.Psi/DynamicDeserializationTests.cs b/Sources/Runtime/Test.Psi/DynamicDeserializationTests.cs index 8b29b5243..1b3c4a2cb 100644 --- a/Sources/Runtime/Test.Psi/DynamicDeserializationTests.cs +++ b/Sources/Runtime/Test.Psi/DynamicDeserializationTests.cs @@ -7,8 +7,8 @@ namespace Test.Psi using System.Dynamic; using System.IO; using System.Linq; - using Microsoft.VisualStudio.TestTools.UnitTesting; using Microsoft.Psi; + using Microsoft.VisualStudio.TestTools.UnitTesting; using Test.Psi.Common; [TestClass] @@ -33,14 +33,14 @@ public void Cleanup() public void PrimitiveToDynamicTest() { // simple primitives - Assert.AreEqual(123, InstanceToDynamic(123)); - Assert.AreEqual(Math.E, InstanceToDynamic(Math.E)); - Assert.AreEqual(true, InstanceToDynamic(true)); - Assert.AreEqual('x', InstanceToDynamic('x')); - Assert.AreEqual("Hello", InstanceToDynamic("Hello")); + Assert.AreEqual(123, this.InstanceToDynamic(123)); + Assert.AreEqual(Math.E, this.InstanceToDynamic(Math.E)); + Assert.AreEqual(true, this.InstanceToDynamic(true)); + Assert.AreEqual('x', this.InstanceToDynamic('x')); + Assert.AreEqual("Hello", this.InstanceToDynamic("Hello")); // collections of primitives - CollectionAssert.AreEqual(new[] { 1, 2, 3 }, InstanceToDynamic(new[] { 1, 2, 3 })); + CollectionAssert.AreEqual(new[] { 1, 2, 3 }, this.InstanceToDynamic(new[] { 1, 2, 3 })); } public class TestObject @@ -85,8 +85,8 @@ public void AssertTestObject(dynamic dyn) public void ObjectToDynamicTest() { var obj = new TestObject(); - var dyn = InstanceToDynamic(obj); - AssertTestObject(dyn); + var dyn = this.InstanceToDynamic(obj); + this.AssertTestObject(dyn); } [TestMethod] @@ -96,11 +96,11 @@ public void CollectionWithDuplicatesToDynamicTest() // serialization detects duplicates and flags, deserializer caches and returns existing instances var obj = new TestObject(); var arr = new[] { obj, obj }; - var dyn = InstanceToDynamic(arr); + var dyn = this.InstanceToDynamic(arr); Assert.IsTrue(dyn is object[]); Assert.AreEqual(2, dyn.Length); - AssertTestObject(dyn[0]); - AssertTestObject(dyn[1]); + this.AssertTestObject(dyn[0]); + this.AssertTestObject(dyn[1]); Assert.AreSame(dyn[0], dyn[1]); // literally the same object! dyn[0].Foo = "Bar"; Assert.AreEqual("Bar", dyn[1].Foo); // spooky action at a distance (same object!) @@ -111,14 +111,14 @@ public void CollectionWithDuplicatesToDynamicTest() public void ComplexObjectToDynamicTest() { var obj = new ComplexTestObject(); - var dyn = InstanceToDynamic(obj); + var dyn = this.InstanceToDynamic(obj); Assert.IsTrue(dyn is ExpandoObject); - AssertTestObject(dyn.NestedA); - AssertTestObject(dyn.NestedB); - AssertTestObject(dyn.Self.NestedA); - AssertTestObject(dyn.Self.Self.NestedA); - AssertTestObject(dyn.Self.Self.Self.NestedA); - AssertTestObject(dyn.Self.Self.Self.Self.NestedA); // I could go on... + this.AssertTestObject(dyn.NestedA); + this.AssertTestObject(dyn.NestedB); + this.AssertTestObject(dyn.Self.NestedA); + this.AssertTestObject(dyn.Self.Self.NestedA); + this.AssertTestObject(dyn.Self.Self.Self.NestedA); + this.AssertTestObject(dyn.Self.Self.Self.Self.NestedA); // I could go on... Assert.AreSame(dyn.NestedA, dyn.NestedB); // literally the same object! Assert.AreSame(dyn, dyn.Self); // also, literally the same object! } @@ -131,8 +131,8 @@ public class ComplexTestObject public ComplexTestObject() { - Self = this; // cyclic! - NestedA = NestedB = new TestObject(); // duplicate! + this.Self = this; // cyclic! + this.NestedA = this.NestedB = new TestObject(); // duplicate! } } diff --git a/Sources/Runtime/Test.Psi/EmitterTester.cs b/Sources/Runtime/Test.Psi/EmitterTester.cs index 21418390f..93fdef642 100644 --- a/Sources/Runtime/Test.Psi/EmitterTester.cs +++ b/Sources/Runtime/Test.Psi/EmitterTester.cs @@ -15,7 +15,7 @@ namespace Test.Psi public class EmitterTester { // make out local version of immediate, to make sure it's synchronous even in debug builds - private static readonly DeliveryPolicy immediate = new DeliveryPolicy(1, int.MaxValue, null, true, true); + private static readonly DeliveryPolicy Immediate = new DeliveryPolicy(1, int.MaxValue, null, true, true); [TestMethod] [Timeout(60000)] @@ -57,7 +57,7 @@ public void ReceiveClassByRef() }, "receiver"); - Generators.Return(p, c).PipeTo(receiver, immediate); + Generators.Return(p, c).PipeTo(receiver, Immediate); p.Run(); } @@ -109,7 +109,7 @@ public void ReceiveStructByRef() }, "receiver"); - Generators.Return(p, s).PipeTo(receiver, immediate); + Generators.Return(p, s).PipeTo(receiver, Immediate); p.Run(); } @@ -159,8 +159,8 @@ public void ValidateMessages() } catch (AggregateException errors) { - // expecting an ArgumentOutOfRangeException wrapped in an AggregateException - Assert.AreEqual(1, errors.InnerExceptions.Count); + // expecting at least one ArgumentOutOfRangeException wrapped in an AggregateException + Assert.IsTrue(errors.InnerExceptions.Count > 0); Assert.IsInstanceOfType(errors.InnerException, typeof(ArgumentOutOfRangeException)); CollectionAssert.AreEqual(new[] { 0, 1, 2, 3, 4, 5 }, results); } diff --git a/Sources/Runtime/Test.Psi/FunctionalTests.cs b/Sources/Runtime/Test.Psi/FunctionalTests.cs index 51f25868c..30565bbca 100644 --- a/Sources/Runtime/Test.Psi/FunctionalTests.cs +++ b/Sources/Runtime/Test.Psi/FunctionalTests.cs @@ -123,7 +123,7 @@ public void ImmutablePipeline() validateSync: false); } - // uses types that need cloning but ccan be reclaimed + // uses types that need cloning but can be reclaimed [TestMethod] [Timeout(60000)] public void RefTypePipeline() @@ -347,7 +347,7 @@ public void RunDataFlowPipeline(Func create, Func initiali Console.WriteLine(resultCount); if (sources.Length == 0) { - throw new Exception("This was here just to keeo source alive in release mode, why did it hit?"); + throw new Exception("This was here just to keep source alive in release mode, why did it hit?"); } } diff --git a/Sources/Runtime/Test.Psi/GeneratorsTests.cs b/Sources/Runtime/Test.Psi/GeneratorsTests.cs index 17491b03a..d6789ce32 100644 --- a/Sources/Runtime/Test.Psi/GeneratorsTests.cs +++ b/Sources/Runtime/Test.Psi/GeneratorsTests.cs @@ -3,13 +3,13 @@ namespace Test.Psi { - using Microsoft.Psi; - using Microsoft.Psi.Components; - using Microsoft.VisualStudio.TestTools.UnitTesting; using System; using System.Collections.Generic; using System.Linq; using System.Threading; + using Microsoft.Psi; + using Microsoft.Psi.Components; + using Microsoft.VisualStudio.TestTools.UnitTesting; /// /// Runs a series of tests for stream generators. @@ -52,7 +52,6 @@ public void RepeatWithClockEnforcement() Assert.IsTrue(lastMessageTime >= originatingTimes.Last()); } - [TestMethod] [Timeout(60000)] public void RepeatNoClockEnforcement() diff --git a/Sources/Runtime/Test.Psi/InteropTests.cs b/Sources/Runtime/Test.Psi/InteropTests.cs index 11c56985c..cdec8beef 100644 --- a/Sources/Runtime/Test.Psi/InteropTests.cs +++ b/Sources/Runtime/Test.Psi/InteropTests.cs @@ -4,23 +4,24 @@ namespace Test.Psi { using System; - using System.Reactive; - using System.Reactive.Linq; using System.IO; using System.Linq; + using System.Reactive; + using System.Reactive.Linq; using System.Text; using System.Threading; - using Microsoft.VisualStudio.TestTools.UnitTesting; using Microsoft.Psi; - using Microsoft.Psi.Interop.Serialization; using Microsoft.Psi.Interop.Format; + using Microsoft.Psi.Interop.Serialization; using Microsoft.Psi.Interop.Transport; + using Microsoft.VisualStudio.TestTools.UnitTesting; using Test.Psi.Common; [TestClass] public class InteropTests { private string path = Path.Combine(Environment.CurrentDirectory, nameof(PersistenceTest)); + private DateTime originatingTime; [TestInitialize] public void Setup() @@ -35,30 +36,28 @@ public void Cleanup() TestRunner.SafeDirectoryDelete(this.path, true); } - private DateTime originatingTime; - private void AssertStringSerialization(dynamic value, string expected, IFormatSerializer serializer, IFormatDeserializer deserializer, bool roundTrip = true) { - var serialized = serializer.SerializeMessage(value, originatingTime); + var serialized = serializer.SerializeMessage(value, this.originatingTime); Assert.AreEqual(expected, Encoding.UTF8.GetString(serialized.Item1, serialized.Item2, serialized.Item3)); if (roundTrip) { var deserialized = deserializer.DeserializeMessage(serialized.Item1, serialized.Item2, serialized.Item3); - Assert.AreEqual(originatingTime, deserialized.Item2); + Assert.AreEqual(this.originatingTime, deserialized.Item2); - var roundtrip = serializer.SerializeMessage(deserialized.Item1, originatingTime); + var roundtrip = serializer.SerializeMessage(deserialized.Item1, this.originatingTime); Assert.AreEqual(expected, Encoding.UTF8.GetString(roundtrip.Item1, roundtrip.Item2, roundtrip.Item3)); } } private void AssertBinarySerialization(dynamic value, IFormatSerializer serializer, IFormatDeserializer deserializer) { - var serialized = serializer.SerializeMessage(value, originatingTime); + var serialized = serializer.SerializeMessage(value, this.originatingTime); var deserialized = deserializer.DeserializeMessage(serialized.Item1, serialized.Item2, serialized.Item3); - Assert.AreEqual(originatingTime, deserialized.Item2); + Assert.AreEqual(this.originatingTime, deserialized.Item2); - var roundtrip = serializer.SerializeMessage(deserialized.Item1, originatingTime); + var roundtrip = serializer.SerializeMessage(deserialized.Item1, this.originatingTime); Enumerable.SequenceEqual(serialized.Item1, roundtrip.Item1); Assert.AreEqual(serialized.Item2, roundtrip.Item2); Assert.AreEqual(serialized.Item3, roundtrip.Item3); @@ -69,12 +68,12 @@ private void AssertBinarySerialization(dynamic value, IFormatSerializer serializ public void JsonFormatSerializerTest() { var json = JsonFormat.Instance; - AssertStringSerialization(123, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":123}", json, json); - AssertStringSerialization(true, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":true}", json, json); - AssertStringSerialization(2.71828, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":2.71828}", json, json); - AssertStringSerialization("Howdy", @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":""Howdy""}", json, json); - AssertStringSerialization((object)null, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":null}", json, json); - AssertStringSerialization(new [] { 1, 2, 3 }, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":[1,2,3]}", json, json); + this.AssertStringSerialization(123, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":123}", json, json); + this.AssertStringSerialization(true, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":true}", json, json); + this.AssertStringSerialization(2.71828, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":2.71828}", json, json); + this.AssertStringSerialization("Howdy", @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":""Howdy""}", json, json); + this.AssertStringSerialization((object)null, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":null}", json, json); + this.AssertStringSerialization(new[] { 1, 2, 3 }, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":[1,2,3]}", json, json); var structured = new { @@ -85,17 +84,17 @@ public void JsonFormatSerializerTest() X = 213, Y = 107, Width = 42, - Height = 61 - } + Height = 61, + }, }; - AssertStringSerialization(structured, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":{""ID"":123,""Confidence"":0.92,""Face"":{""X"":213,""Y"":107,""Width"":42,""Height"":61}}}", json, json); + this.AssertStringSerialization(structured, @"{""originatingTime"":""1971-11-03T00:00:00.1234567Z"",""message"":{""ID"":123,""Confidence"":0.92,""Face"":{""X"":213,""Y"":107,""Width"":42,""Height"":61}}}", json, json); // also verify "manually" - var serialized = json.SerializeMessage(structured, originatingTime); + var serialized = json.SerializeMessage(structured, this.originatingTime); var deserialized = json.DeserializeMessage(serialized.Item1, serialized.Item2, serialized.Item3); var message = deserialized.Item1; var timestamp = deserialized.Item2; - Assert.AreEqual(originatingTime, timestamp); + Assert.AreEqual(this.originatingTime, timestamp); Assert.AreEqual(123, message.ID); Assert.AreEqual(0.92, message.Confidence); Assert.AreEqual(213, message.Face.X); @@ -109,13 +108,13 @@ public void JsonFormatSerializerTest() public void CsvFormatSerializerTest() { var csv = CsvFormat.Instance; - AssertStringSerialization(123, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,123\r\n", csv, csv); - AssertStringSerialization(true, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,True\r\n", csv, csv); - AssertStringSerialization(2.71828, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,2.71828\r\n", csv, csv); - AssertStringSerialization("Howdy", "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,Howdy\r\n", csv, csv); + this.AssertStringSerialization(123, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,123\r\n", csv, csv); + this.AssertStringSerialization(true, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,True\r\n", csv, csv); + this.AssertStringSerialization(2.71828, "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,2.71828\r\n", csv, csv); + this.AssertStringSerialization("Howdy", "_OriginatingTime_,_Value_\r\n1971-11-03T00:00:00.1234567Z,Howdy\r\n", csv, csv); // special case - AssertStringSerialization(new double[] { 1, 2, 3 }, "_OriginatingTime_,_Column0_,_Column1_,_Column2_\r\n1971-11-03T00:00:00.1234567Z,1,2,3\r\n", csv, csv); + this.AssertStringSerialization(new double[] { 1, 2, 3 }, "_OriginatingTime_,_Column0_,_Column1_,_Column2_\r\n1971-11-03T00:00:00.1234567Z,1,2,3\r\n", csv, csv); var structured = new { @@ -127,11 +126,12 @@ public void CsvFormatSerializerTest() Y = 107, Width = 42, Height = 61, - Points = new [] { 123, 456 } - } + Points = new[] { 123, 456 }, + }, }; + // notice Face is traversed but flattened - no hierarchy allowed - AssertStringSerialization(structured, "_OriginatingTime_,ID,Confidence,X,Y,Width,Height\r\n1971-11-03T00:00:00.1234567Z,123,0.92,213,107,42,61\r\n", csv, csv); + this.AssertStringSerialization(structured, "_OriginatingTime_,ID,Confidence,X,Y,Width,Height\r\n1971-11-03T00:00:00.1234567Z,123,0.92,213,107,42,61\r\n", csv, csv); var structuredAmbiguous = new { @@ -143,12 +143,12 @@ public void CsvFormatSerializerTest() X = 213, Y = 107, Width = 42, - Height = 61 - } + Height = 61, + }, }; // notice Face is traversed but flattened - no hierarchy allowed - AssertStringSerialization(structuredAmbiguous, "_OriginatingTime_,ID,Confidence,Confidence,X,Y,Width,Height\r\n1971-11-03T00:00:00.1234567Z,123,0.92,0.89,213,107,42,61\r\n", csv, csv, false); + this.AssertStringSerialization(structuredAmbiguous, "_OriginatingTime_,ID,Confidence,Confidence,X,Y,Width,Height\r\n1971-11-03T00:00:00.1234567Z,123,0.92,0.89,213,107,42,61\r\n", csv, csv, false); var flat = new { @@ -157,9 +157,9 @@ public void CsvFormatSerializerTest() FaceX = 213, FaceY = 107, FaceWidth = 42, - FaceHeight = 61 + FaceHeight = 61, }; - AssertStringSerialization(flat, "_OriginatingTime_,ID,Confidence,FaceX,FaceY,FaceWidth,FaceHeight\r\n1971-11-03T00:00:00.1234567Z,123,0.92,213,107,42,61\r\n", csv, csv); + this.AssertStringSerialization(flat, "_OriginatingTime_,ID,Confidence,FaceX,FaceY,FaceWidth,FaceHeight\r\n1971-11-03T00:00:00.1234567Z,123,0.92,213,107,42,61\r\n", csv, csv); } [TestMethod] @@ -167,11 +167,11 @@ public void CsvFormatSerializerTest() public void MessagePackFormatSerializerTest() { var msg = MessagePackFormat.Instance; - AssertBinarySerialization(123, msg, msg); - AssertBinarySerialization(true, msg, msg); - AssertBinarySerialization(2.71828, msg, msg); - AssertBinarySerialization("Howdy", msg, msg); - AssertBinarySerialization(new [] { 1, 2, 3 }, msg, msg); + this.AssertBinarySerialization(123, msg, msg); + this.AssertBinarySerialization(true, msg, msg); + this.AssertBinarySerialization(2.71828, msg, msg); + this.AssertBinarySerialization("Howdy", msg, msg); + this.AssertBinarySerialization(new[] { 1, 2, 3 }, msg, msg); var structured = new { @@ -182,16 +182,16 @@ public void MessagePackFormatSerializerTest() X = 213, Y = 107, Width = 42, - Height = 61 - } + Height = 61, + }, }; // can't round-trip ExpandoObjects, so verifying "manually" - var serialized = msg.SerializeMessage(structured, originatingTime); + var serialized = msg.SerializeMessage(structured, this.originatingTime); var deserialized = msg.DeserializeMessage(serialized.Item1, serialized.Item2, serialized.Item3); var message = deserialized.Item1; var timestamp = deserialized.Item2; - Assert.AreEqual(originatingTime, timestamp); + Assert.AreEqual(this.originatingTime, timestamp); Assert.AreEqual(123, message.ID); Assert.AreEqual(0.92, message.Confidence); Assert.AreEqual(213, message.Face.X); @@ -244,7 +244,7 @@ private void NetMQTransportTest() { Console.WriteLine("Starting client..."); var client = new NetMQSource(p, topic, address, JsonFormat.Instance); - client.Do(x => complete = (x == 9)).Do(x => Console.WriteLine($"MSG: {x}")); + client.Do(x => complete = x == 9).Do(x => Console.WriteLine($"MSG: {x}")); results = client.ToObservable().ToListObservable(); p.RunAsync(); @@ -287,10 +287,10 @@ private void NetMQTransportMultiTopicTest() { Console.WriteLine("Starting client..."); var client0 = new NetMQSource(p, topic0, address, JsonFormat.Instance); - client0.Do(x => complete0 = (x == 9)).Do(x => Console.WriteLine($"MSG0: {x}")); + client0.Do(x => complete0 = x == 9).Do(x => Console.WriteLine($"MSG0: {x}")); results0 = client0.ToObservable().ToListObservable(); var client1 = new NetMQSource(p, topic1, address, JsonFormat.Instance); - client1.Do(x => complete1 = (x == 9)).Do(x => Console.WriteLine($"MSG1: {x}")); + client1.Do(x => complete1 = x == 9).Do(x => Console.WriteLine($"MSG1: {x}")); results1 = client1.ToObservable().ToListObservable(); p.RunAsync(); diff --git a/Sources/Runtime/Test.Psi/InterpolateTests.cs b/Sources/Runtime/Test.Psi/InterpolateTests.cs index 115c396b8..cfccfd851 100644 --- a/Sources/Runtime/Test.Psi/InterpolateTests.cs +++ b/Sources/Runtime/Test.Psi/InterpolateTests.cs @@ -9,7 +9,7 @@ namespace Test.Psi using Microsoft.VisualStudio.TestTools.UnitTesting; [TestClass] - public class InterpolateTest + public class InterpolateTests { [TestMethod] [Timeout(60000)] diff --git a/Sources/Runtime/Test.Psi/InterpolatorTests.cs b/Sources/Runtime/Test.Psi/InterpolatorTests.cs index a92298b49..29d7a28f7 100644 --- a/Sources/Runtime/Test.Psi/InterpolatorTests.cs +++ b/Sources/Runtime/Test.Psi/InterpolatorTests.cs @@ -4,9 +4,6 @@ namespace Test.Psi { using System; - using System.ComponentModel.DataAnnotations; - using System.Reflection; - using System.Text; using Microsoft.Psi; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -22,7 +19,7 @@ public void AvailableNearest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -68,7 +65,7 @@ public void AvailableNearest_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -114,7 +111,7 @@ public void AvailableNearest_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -160,7 +157,7 @@ public void AvailableNearest_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -215,7 +212,7 @@ public void AvailableNearest_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should not match the right end of the interval @@ -273,7 +270,7 @@ public void AvailableFirst_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -319,7 +316,7 @@ public void AvailableFirst_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -365,7 +362,7 @@ public void AvailableFirst_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -420,7 +417,7 @@ public void AvailableFirst_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should not match the right end of the interval because it's open. @@ -449,7 +446,7 @@ public void AvailableLast() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -495,7 +492,7 @@ public void AvailableLast_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -541,7 +538,7 @@ public void AvailableLast_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -587,7 +584,7 @@ public void AvailableLast_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -646,7 +643,7 @@ public void AvailableLast_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should return does not exist b/c message at 30 falls after the open window @@ -675,7 +672,7 @@ public void ReproducibleNearest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -721,7 +718,7 @@ public void ReproducibleNearest_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -767,7 +764,7 @@ public void ReproducibleNearest_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -813,7 +810,7 @@ public void ReproducibleNearest_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -868,7 +865,7 @@ public void ReproducibleNearest_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should not match the right end of the interval @@ -926,7 +923,7 @@ public void ReproducibleFirst_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -972,7 +969,7 @@ public void ReproducibleFirst_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -1018,7 +1015,7 @@ public void ReproducibleFirst_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -1073,7 +1070,7 @@ public void ReproducibleFirst_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should not match the right end of the interval because it's open. @@ -1102,7 +1099,7 @@ public void ReproducibleLast() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -1148,7 +1145,7 @@ public void ReproducibleLast_LeftBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -1194,7 +1191,7 @@ public void ReproducibleLast_RightBounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -1240,7 +1237,7 @@ public void ReproducibleLast_Bounded() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message (outside upper bound) @@ -1299,7 +1296,7 @@ public void ReproducibleLast_OpenIntervalTest() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate should return does not exist b/c message at 30 falls after the open window @@ -1328,7 +1325,7 @@ public void Linear() { new Message(1, new DateTime(10), new DateTime(11), 0, 0), new Message(2, new DateTime(20), new DateTime(21), 0, 1), - new Message(3, new DateTime(30), new DateTime(31), 0, 2) + new Message(3, new DateTime(30), new DateTime(31), 0, 2), }; // Interpolate at point later than last message @@ -1385,7 +1382,6 @@ public void Linear() Assert.AreEqual(InterpolationResult.InsufficientData(), result); } - private InterpolationResult MakeResult(Message msg) { return InterpolationResult.Create(msg.Data, msg.OriginatingTime); diff --git a/Sources/Runtime/Test.Psi/JoinTests.cs b/Sources/Runtime/Test.Psi/JoinTests.cs index 2a7d8e340..23afedfc7 100644 --- a/Sources/Runtime/Test.Psi/JoinTests.cs +++ b/Sources/Runtime/Test.Psi/JoinTests.cs @@ -50,19 +50,22 @@ public void DynamicJoinClosingSecondaryOrDefault() // value N/A 1 2 3 4 5 6 N/A N/A N/A // gamma-result [1 2 - 4 - -] // out 1 2 0 4 0 0 - var input = Generators.Sequence(p, new List>() - { - new Dictionary(), - new Dictionary() { { 1, 1 } }, - new Dictionary() { { 1, 2 } }, - new Dictionary() { { 1, 3 } }, - new Dictionary() { { 1, 4 } }, - new Dictionary() { { 1, 5 } }, - new Dictionary() { { 1, 6 } }, - new Dictionary(), - new Dictionary(), - new Dictionary(), - }, TimeSpan.FromTicks(1)); + var input = Generators.Sequence( + p, + new List>() + { + new Dictionary(), + new Dictionary() { { 1, 1 } }, + new Dictionary() { { 1, 2 } }, + new Dictionary() { { 1, 3 } }, + new Dictionary() { { 1, 4 } }, + new Dictionary() { { 1, 5 } }, + new Dictionary() { { 1, 6 } }, + new Dictionary(), + new Dictionary(), + new Dictionary(), + }, + TimeSpan.FromTicks(1)); var resultsParallelOrDefault = new List(); input.Parallel(s => s.Where(x => x != 3 && x <= 4), outputDefaultIfDropped: true).Do(d => @@ -257,7 +260,7 @@ public void TupleCollapsingJoin() ValueTuple.Create("A6", "B6", "C6", "D6", "E6", "F6", "G6"), ValueTuple.Create("A7", "B7", "C7", "D7", "E7", "F7", "G7"), ValueTuple.Create("A8", "B8", "C8", "D8", "E8", "F8", "G8"), - ValueTuple.Create("A9", "B9", "C9", "D9", "E9", "F9", "G9") + ValueTuple.Create("A9", "B9", "C9", "D9", "E9", "F9", "G9"), }, results)); } @@ -301,7 +304,7 @@ public void TupleCollapsingReversedJoin() ValueTuple.Create("A6", "B6", "C6", "D6", "E6", "F6", "G6"), ValueTuple.Create("A7", "B7", "C7", "D7", "E7", "F7", "G7"), ValueTuple.Create("A8", "B8", "C8", "D8", "E8", "F8", "G8"), - ValueTuple.Create("A9", "B9", "C9", "D9", "E9", "F9", "G9") + ValueTuple.Create("A9", "B9", "C9", "D9", "E9", "F9", "G9"), }, results)); } diff --git a/Sources/Runtime/Test.Psi/OpProcessorTests.cs b/Sources/Runtime/Test.Psi/OpProcessorTests.cs index 336668ef7..9baff39de 100644 --- a/Sources/Runtime/Test.Psi/OpProcessorTests.cs +++ b/Sources/Runtime/Test.Psi/OpProcessorTests.cs @@ -6,6 +6,7 @@ namespace Test.Psi using System; using System.Collections.Generic; using System.Linq; + using System.Reactive.Linq; using Microsoft.Psi; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -33,5 +34,33 @@ public void SelectClosure() CollectionAssert.AreEqual(new double[] { 100, 75, 50 }, results); } + + [TestMethod] + [Timeout(60000)] + public void StreamEditing() + { + using (var pipeline = Pipeline.Create()) + { + var start = new DateTime(1971, 11, 03); + var range = Generators.Sequence(pipeline, new[] + { + ('B', start.AddSeconds(1)), + ('C', start.AddSeconds(2)), + ('D', start.AddSeconds(3)), + ('E', start.AddSeconds(4)), + }); + var edited = range.EditStream(new[] + { + (true, 'F', start.AddSeconds(5)), // insert F after E + (false, default(char), start.AddSeconds(2)), // delete C + (true, 'A', start), // insert A before B + (true, 'X', start.AddSeconds(3)), // update D to X + }).ToObservable().ToListObservable(); + pipeline.Run(); + + var editedResults = edited.AsEnumerable().ToArray(); + Assert.IsTrue(Enumerable.SequenceEqual(new[] { 'A', 'B', 'X', 'E', 'F' }, editedResults)); + } + } } } diff --git a/Sources/Runtime/Test.Psi/OperatorTests.cs b/Sources/Runtime/Test.Psi/OperatorTests.cs index 3c9e943d6..28d9fef1a 100644 --- a/Sources/Runtime/Test.Psi/OperatorTests.cs +++ b/Sources/Runtime/Test.Psi/OperatorTests.cs @@ -56,7 +56,7 @@ public void DelayOperator() [Timeout(60000)] public void RepeatTest() { - using (var pipeline = Pipeline.Create(nameof(RepeatTest))) + using (var pipeline = Pipeline.Create(nameof(this.RepeatTest))) { var startTime = DateTime.UtcNow; @@ -288,7 +288,6 @@ public void Average() } } - [TestMethod] [Timeout(60000)] public void AverageWithCondition() @@ -369,27 +368,27 @@ public void AverageOverHistory() pipeline.Run(); Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 1.5, 2.5, 3.5, 4.5 }, intAverageHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2.5, 3.5, 4.5 }, intAverageHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2, 3, 4 }, intAverageHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, 3, 3.5, 4, 5 }, nullableIntAverageHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, null, null, 3, 3.5, 4, 4.5 }, nullableIntAverageHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 1.5, 2.5, 3.5, 4.5 }, longAverageHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2.5, 3.5, 4.5 }, longAverageHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2, 3, 4 }, longAverageHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, 3, 3.5, 4, 5 }, nullableLongAverageHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, null, null, 3, 3.5, 4, 4.5 }, nullableLongAverageHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 1.5f, 2.5f, 3.5f, 4.5f }, floatAverageHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 0.5f, 1, 1.5f, 2.5f, 3.5f, 4.5f }, floatAverageHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 0.5f, 1, 1.5f, 2f, 3f, 4f }, floatAverageHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { null, 3, 3.5f, 4, 5 }, nullableFloatAverageHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { null, null, null, 3, 3.5f, 4, 4.5f }, nullableFloatAverageHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 1.5, 2.5, 3.5, 4.5 }, doubleAverageHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2.5, 3.5, 4.5 }, doubleAverageHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0.5, 1, 1.5, 2, 3, 4 }, doubleAverageHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, 3, 3.5, 4, 5 }, nullableDoubleAverageHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, null, null, 3, 3.5, 4, 4.5 }, nullableDoubleAverageHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 1.5m, 2.5m, 3.5m, 4.5m }, decimalAverageHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 0.5m, 1, 1.5m, 2.5m, 3.5m, 4.5m }, decimalAverageHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 0.5m, 1, 1.5m, 2m, 3m, 4m }, decimalAverageHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { null, 3, 3.5m, 4, 5 }, nullableDecimalAverageHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { null, null, null, 3, 3.5m, 4, 4.5m }, nullableDecimalAverageHistByTime.AsEnumerable())); } @@ -407,12 +406,11 @@ public void CountOverHistory() pipeline.Run(); - Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 1, 2, 3, 4, 4, 4, 4 }, countHistoryByTime.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 1, 2, 3, 4, 4, 4, 4 }, longCountHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 1, 2, 3, 4, 5, 5, 5 }, countHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 1, 2, 3, 4, 5, 5, 5 }, longCountHistoryByTime.AsEnumerable())); } } - [TestMethod] [Timeout(60000)] public void SumOverHistory() @@ -462,27 +460,27 @@ public void SumOverHistory() pipeline.Run(); Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 6, 10, 14, 18 }, intSumHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 1, 3, 6, 10, 14, 18 }, intSumHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 1, 3, 6, 10, 15, 20 }, intSumHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new int?[] { 0, 3, 7, 12, 15 }, nullableIntSumHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new int?[] { 0, 0, 0, 3, 7, 12, 18 }, nullableIntSumHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 6, 10, 14, 18 }, longSumHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 0, 1, 3, 6, 10, 14, 18 }, longSumHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 0, 1, 3, 6, 10, 15, 20 }, longSumHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long?[] { 0, 3, 7, 12, 15 }, nullableLongSumHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long?[] { 0, 0, 0, 3, 7, 12, 18 }, nullableLongSumHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 6, 10, 14, 18 }, floatSumHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 1, 3, 6, 10, 14, 18 }, floatSumHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 1, 3, 6, 10, 15, 20 }, floatSumHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { 0, 3, 7, 12, 15 }, nullableFloatSumHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { 0, 0, 0, 3, 7, 12, 18 }, nullableFloatSumHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 6, 10, 14, 18 }, doubleSumHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 1, 3, 6, 10, 14, 18 }, doubleSumHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 1, 3, 6, 10, 15, 20 }, doubleSumHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { 0, 3, 7, 12, 15 }, nullableDoubleSumHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { 0, 0, 0, 3, 7, 12, 18 }, nullableDoubleSumHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 6, 10, 14, 18 }, decimalSumHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 1, 3, 6, 10, 14, 18 }, decimalSumHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 1, 3, 6, 10, 15, 20 }, decimalSumHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { 0, 3, 7, 12, 15 }, nullableDecimalSumHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { 0, 0, 0, 3, 7, 12, 18 }, nullableDecimalSumHistByTime.AsEnumerable())); } @@ -537,27 +535,27 @@ public void MinOverHistory() pipeline.Run(); Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 1, 2, 3 }, intMinHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 0, 0, 0, 1, 2, 3 }, intMinHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 0, 0, 0, 0, 1, 2 }, intMinHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new int?[] { null, 3, 3, 3, 4 }, nullableIntMinHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new int?[] { null, null, null, 3, 3, 3, 3 }, nullableIntMinHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 0, 1, 2, 3 }, longMinHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 0, 0, 0, 0, 1, 2, 3 }, longMinHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new long[] { 0, 0, 0, 0, 0, 1, 2 }, longMinHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long?[] { null, 3, 3, 3, 4 }, nullableLongMinHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new long?[] { null, null, null, 3, 3, 3, 3 }, nullableLongMinHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 1, 2, 3 }, floatMinHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 0, 0, 0, 1, 2, 3 }, floatMinHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new float[] { 0, 0, 0, 0, 0, 1, 2 }, floatMinHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { null, 3, 3, 3, 4 }, nullableFloatMinHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new float?[] { null, null, null, 3, 3, 3, 3 }, nullableFloatMinHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 1, 2, 3 }, doubleMinHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0, 0, 0, 1, 2, 3 }, doubleMinHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new double[] { 0, 0, 0, 0, 0, 1, 2 }, doubleMinHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, 3, 3, 3, 4 }, nullableDoubleMinHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new double?[] { null, null, null, 3, 3, 3, 3 }, nullableDoubleMinHistByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 1, 2, 3 }, decimalMinHistoryBySize.AsEnumerable())); - Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 0, 0, 0, 1, 2, 3 }, decimalMinHistoryByTime.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new decimal[] { 0, 0, 0, 0, 0, 1, 2 }, decimalMinHistoryByTime.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { null, 3, 3, 3, 4 }, nullableDecimalMinHistBySize.AsEnumerable())); Assert.IsTrue(Enumerable.SequenceEqual(new decimal?[] { null, null, null, 3, 3, 3, 3 }, nullableDecimalMinHistByTime.AsEnumerable())); } @@ -726,7 +724,6 @@ public void MinMaxWithCondition() } } - [TestMethod] [Timeout(60000)] public void MinMaxWithConditionAndComparer() @@ -747,11 +744,12 @@ public void MinMaxWithConditionAndComparer() [TestMethod] [Timeout(60000)] - public void BufferBySize() + public void BufferBySizeInclusive() { using (var pipeline = Pipeline.Create()) { - var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromTicks(1)).Window(0, 2).ToObservable().ToListObservable(); + var inclusiveIntInterval = new IntInterval(0, true, 2, true); + var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromTicks(1)).Window(inclusiveIntInterval).ToObservable().ToListObservable(); var timestamps = Generators.Range(pipeline, 0, 5, TimeSpan.FromTicks(1)).Select((_, e) => e.OriginatingTime).Window(0, 2).Select((m, e) => Tuple.Create(m.ToArray(), e.OriginatingTime)).ToObservable().ToListObservable(); pipeline.Run(); @@ -771,6 +769,34 @@ public void BufferBySize() } } + [TestMethod] + [Timeout(60000)] + public void BufferBySizeExclusive() + { + using (var pipeline = Pipeline.Create()) + { + var inclusiveIntInterval = new IntInterval(0, true, 2, false); + var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromTicks(1)).Window(inclusiveIntInterval).ToObservable().ToListObservable(); + var timestamps = Generators.Range(pipeline, 0, 5, TimeSpan.FromTicks(1)).Select((_, e) => e.OriginatingTime).Window(0, 2).Select((m, e) => Tuple.Create(m.ToArray(), e.OriginatingTime)).ToObservable().ToListObservable(); + pipeline.Run(); + + var bufferResults = buffers.AsEnumerable().ToArray(); + Assert.AreEqual(4, bufferResults.Length); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 1 }, bufferResults[0])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 1, 2 }, bufferResults[1])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 2, 3 }, bufferResults[2])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 3, 4 }, bufferResults[3])); + + var timestampResults = timestamps.AsEnumerable().ToArray(); + Assert.AreEqual(3, timestampResults.Length); + foreach (var buf in timestamps) + { + // buffer timestamp matches _first_ message in buffer + Assert.AreEqual(buf.Item1.First(), buf.Item2); + } + } + } + [TestMethod] [Timeout(60000)] public void FutureWindowBySizeWithSelector() @@ -833,11 +859,12 @@ public void HistoryBySizeWithSelector() [TestMethod] [Timeout(60000)] - public void HistoryByTime() + public void HistoryByTimeInclusive() { using (var pipeline = Pipeline.Create()) { - var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromMilliseconds(1)).Window(RelativeTimeInterval.Past(TimeSpan.FromMilliseconds(2))).ToObservable().ToListObservable(); + var inclusiveRelativeTime = RelativeTimeInterval.Past(TimeSpan.FromMilliseconds(2)); + var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromMilliseconds(1)).Window(inclusiveRelativeTime).ToObservable().ToListObservable(); var timestamps = Generators.Range(pipeline, 0, 5, TimeSpan.FromMilliseconds(1)).Select((_, e) => e.OriginatingTime).Window(RelativeTimeInterval.Past(TimeSpan.FromMilliseconds(2))).Select((m, e) => Tuple.Create(m.ToArray(), e.OriginatingTime)).ToObservable().ToListObservable(); pipeline.Run(); @@ -859,6 +886,35 @@ public void HistoryByTime() } } + [TestMethod] + [Timeout(60000)] + public void HistoryByTimeExclusive() + { + using (var pipeline = Pipeline.Create()) + { + var exclusiveRelativeTime = RelativeTimeInterval.Past(TimeSpan.FromMilliseconds(2), false); + var buffers = Generators.Range(pipeline, 0, 5, TimeSpan.FromMilliseconds(1)).Window(exclusiveRelativeTime).ToObservable().ToListObservable(); + var timestamps = Generators.Range(pipeline, 0, 5, TimeSpan.FromMilliseconds(1)).Select((_, e) => e.OriginatingTime).Window(RelativeTimeInterval.Past(TimeSpan.FromMilliseconds(2))).Select((m, e) => Tuple.Create(m.ToArray(), e.OriginatingTime)).ToObservable().ToListObservable(); + pipeline.Run(); + + var results = buffers.AsEnumerable().ToArray(); + Assert.AreEqual(5, results.Length); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0 }, results[0])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 0, 1 }, results[1])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 1, 2 }, results[2])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 2, 3 }, results[3])); + Assert.IsTrue(Enumerable.SequenceEqual(new int[] { 3, 4 }, results[4])); + + var timestampResults = timestamps.AsEnumerable().ToArray(); + Assert.AreEqual(5, timestampResults.Length); + foreach (var buf in timestamps) + { + // buffer timestamp matches _last_ message in buffer + Assert.AreEqual(buf.Item1.Last(), buf.Item2); + } + } + } + [TestMethod] [Timeout(60000)] public void NonOriginSpanningPastWindowByTime() @@ -877,8 +933,8 @@ public void NonOriginSpanningPastWindowByTime() Assert.AreEqual(0, results[0]); // empty Assert.AreEqual(0, results[1]); // 0 Assert.AreEqual(1, results[2]); // 0 + 1 - Assert.AreEqual(3, results[3]); // 1 + 2 - Assert.AreEqual(5, results[4]); // 2 + 3 + Assert.AreEqual(3, results[3]); // 0 + 1 + 2 + Assert.AreEqual(6, results[4]); // 1 + 2 + 3 } } @@ -1053,32 +1109,48 @@ DateTime ToTime(int i) } // test growing (right) window - pairs are (obsolete, start, end) windows with expected windowed data - Test(new[] { - ((2, 2, 4), new[] { 2, 3, 4 }), // 2..4 growing to right - ((2, 2, 5), new[] { 2, 3, 4, 5 }), // 2..5 - ((2, 2, 6), new[] { 2, 3, 4, 5, 6 }), // 2..6 - }, true, true); + Test( + new[] + { + ((2, 2, 4), new[] { 2, 3, 4 }), // 2..4 growing to right + ((2, 2, 5), new[] { 2, 3, 4, 5 }), // 2..5 + ((2, 2, 6), new[] { 2, 3, 4, 5, 6 }), // 2..6 + }, + true, + true); // test left inclusivity - Test(new[] { - ((2, 2, 4), new[] { 3, 4 }), - ((2, 2, 5), new[] { 3, 4, 5 }), - ((2, 2, 6), new[] { 3, 4, 5, 6 }), - }, false /* not including left */, true); + Test( + new[] + { + ((2, 2, 4), new[] { 3, 4 }), + ((2, 2, 5), new[] { 3, 4, 5 }), + ((2, 2, 6), new[] { 3, 4, 5, 6 }), + }, + false /* not including left */, + true); // test right inclusivity - Test(new[] { - ((2, 2, 4), new[] { 2, 3 }), - ((2, 2, 5), new[] { 2, 3, 4 }), - ((2, 2, 6), new[] { 2, 3, 4, 5}), - }, true, false /* not including right */); + Test( + new[] + { + ((2, 2, 4), new[] { 2, 3 }), + ((2, 2, 5), new[] { 2, 3, 4 }), + ((2, 2, 6), new[] { 2, 3, 4, 5 }), + }, + true, + false /* not including right */); // test left & right inclusivity - Test(new[] { - ((2, 2, 4), new[] { 3 }), - ((2, 2, 5), new[] { 3, 4 }), - ((2, 2, 6), new[] { 3, 4, 5}), - }, false /* not including left */, false /* not including right */); + Test( + new[] + { + ((2, 2, 4), new[] { 3 }), + ((2, 2, 5), new[] { 3, 4 }), + ((2, 2, 6), new[] { 3, 4, 5 }), + }, + false /* not including left */, + false /* not including right */); // test left-most window Test(new[] { ((0, 0, 2), new[] { 0, 1, 2 }), }, true, true); @@ -1090,26 +1162,38 @@ DateTime ToTime(int i) Test(new[] { ((7, 7, 100), new[] { 7, 8, 9 }), }, true, true); // test sliding window - Test(new[] { - ((2, 2, 4), new[] { 2, 3, 4 }), // 2..4 sliding to right - ((3, 3, 5), new[] { 3, 4, 5 }), // 3..6 - ((4, 4, 6), new[] { 4, 5, 6 }), // 4..7 - }, true, true); + Test( + new[] + { + ((2, 2, 4), new[] { 2, 3, 4 }), // 2..4 sliding to right + ((3, 3, 5), new[] { 3, 4, 5 }), // 3..6 + ((4, 4, 6), new[] { 4, 5, 6 }), // 4..7 + }, + true, + true); // test growing (left) window - Test(new[] { - ((1, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left - ((1, 2, 5), new[] { 2, 3, 4, 5 }), // 2..4 - ((1, 1, 5), new[] { 1, 2, 3, 4, 5}), // 1..5 - }, true, true); + Test( + new[] + { + ((1, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left + ((1, 2, 5), new[] { 2, 3, 4, 5 }), // 2..4 + ((1, 1, 5), new[] { 1, 2, 3, 4, 5 }), // 1..5 + }, + true, + true); // invalid if obsolete time moves backward! try { - Test(new[] { - ((3, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left - ((2, 3, 5), new[] { 3, 4, 5 }), // 3..4 boom! (2 earlier than previous [3] obsolete) - }, true, true); + Test( + new[] + { + ((3, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left + ((2, 3, 5), new[] { 3, 4, 5 }), // 3..4 boom! (2 earlier than previous [3] obsolete) + }, + true, + true); Assert.Fail("Expected exception due to obsolete time backtracking"); } catch (Exception ex) @@ -1120,10 +1204,14 @@ DateTime ToTime(int i) // invalid if window requests are before previous obsolete time try { - Test(new[] { - ((3, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left - ((3, 2, 5), new[] { 1, 3, 4, 5 }), // 2..5 boom! (2 has alreadly been obsoleted) - }, true, true); + Test( + new[] + { + ((3, 3, 5), new[] { 3, 4, 5 }), // 3..5 growing to left + ((3, 2, 5), new[] { 1, 3, 4, 5 }), // 2..5 boom! (2 has already been obsoleted) + }, + true, + true); Assert.Fail("Expected exception due to window request into obsoleted inputs"); } catch (Exception ex) @@ -1136,9 +1224,9 @@ DateTime ToTime(int i) [Timeout(60000)] public void StdOverIEnumerable() { - Assert.IsTrue(Math.Abs(new [] { 727.7m, 1086.5m, 1091.0m, 1361.3m, 1490.5m, 1956.1m }.Std() - 420.96248961952256m) < 0.0000000001m); // decimal - Assert.IsTrue(Math.Abs(new [] { 727.7, 1086.5, 1091.0, 1361.3, 1490.5, 1956.1 }.Std() - 420.96248961952256) < double.Epsilon); // double - Assert.IsTrue(Math.Abs(new [] { 727.7f, 1086.5f, 1091.0f, 1361.3f, 1490.5f, 1956.1f }.Std() - 420.96248961952256f) < float.Epsilon); // float + Assert.IsTrue(Math.Abs(new[] { 727.7m, 1086.5m, 1091.0m, 1361.3m, 1490.5m, 1956.1m }.Std() - 420.96248961952256m) < 0.0000000001m); // decimal + Assert.IsTrue(Math.Abs(new[] { 727.7, 1086.5, 1091.0, 1361.3, 1490.5, 1956.1 }.Std() - 420.96248961952256) < double.Epsilon); // double + Assert.IsTrue(Math.Abs(new[] { 727.7f, 1086.5f, 1091.0f, 1361.3f, 1490.5f, 1956.1f }.Std() - 420.96248961952256f) < float.Epsilon); // float Assert.AreEqual(0, new double[] { }.Std()); } @@ -1162,7 +1250,7 @@ public void StdOverWindows() [Timeout(60000)] public void ZipAndMerge() { - var zipped = new List(); + var zipped = new List(); var merged = new List(); using (var p = Pipeline.Create()) @@ -1179,22 +1267,52 @@ public void ZipAndMerge() var sourceB = Generators.Range(p, 0, 10, TimeSpan.FromMilliseconds(15)).Select(i => $"B{i}"); var sourceC = Generators.Range(p, 0, 10, TimeSpan.FromMilliseconds(20)).Select(i => $"C{i}").Delay(TimeSpan.FromMilliseconds(100)); - Operators.Merge(new [] { sourceA, sourceB, sourceC }).Do(x => merged.Add(x.Data)); // non-deterministic order - Operators.Zip(new [] { sourceA, sourceB, sourceC }).Do(x => zipped.Add(x.Data)); // ordered by originating time, then stream ID within single tick + Operators.Merge(new[] { sourceA, sourceB, sourceC }).Do(x => merged.Add(x.Data.DeepClone())); // non-deterministic order + Operators.Zip(new[] { sourceA, sourceB, sourceC }).Do(x => zipped.Add(x.DeepClone())); // ordered by originating time, then stream ID within single tick p.Run(); } // Zipped and ordered by originating time (with stream ID tie-breaker) - Assert.IsTrue(Enumerable.SequenceEqual(new[] { "A0", "B0", "C0", "A1", "B1", "A2", "C1", "A3", "B2", "A4", - "C2", "B3", "A5", "A6", "B4", "C3", "A7", "B5", "A8", "C4", - "A9", "B6", "C5", "B7", "B8", "C6", "B9", "C7", "C8", "C9" }, zipped)); - + var zippedShouldBe = new[] + { + new string[] { "A0", "B0", "C0" }, + new string[] { "A1" }, + new string[] { "B1" }, + new string[] { "A2", "C1" }, + new string[] { "A3", "B2" }, + new string[] { "A4", "C2" }, + new string[] { "B3" }, + new string[] { "A5" }, + new string[] { "A6", "B4", "C3" }, + new string[] { "A7" }, + new string[] { "B5" }, + new string[] { "A8", "C4" }, + new string[] { "A9", "B6" }, + new string[] { "C5" }, + new string[] { "B7" }, + new string[] { "B8", "C6" }, + new string[] { "B9" }, + new string[] { "C7" }, + new string[] { "C8" }, + new string[] { "C9" }, + }; + + Assert.AreEqual(zipped.Count, zippedShouldBe.Length); + for (int i = 0; i < zipped.Count; i++) + { + Assert.IsTrue(Enumerable.SequenceEqual(zipped[i], zippedShouldBe[i])); + } // Since merging is non-deterministic, all we test here is that all messages arrive - Assert.IsTrue(Enumerable.SequenceEqual(new[] { "A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7", "A8", "A9", - "B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7", "B8", "B9", - "C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7", "C8", "C9" }, merged.OrderBy(_ => _))); + Assert.IsTrue(Enumerable.SequenceEqual( + new[] + { + "A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7", "A8", "A9", + "B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7", "B8", "B9", + "C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7", "C8", "C9", + }, + merged.OrderBy(_ => _))); } [TestMethod] diff --git a/Sources/Runtime/Test.Psi/PersistenceTest.cs b/Sources/Runtime/Test.Psi/PersistenceTest.cs index 43c4e05a7..f1ce87d65 100644 --- a/Sources/Runtime/Test.Psi/PersistenceTest.cs +++ b/Sources/Runtime/Test.Psi/PersistenceTest.cs @@ -7,6 +7,7 @@ namespace Test.Psi using System.Collections.Generic; using System.IO; using System.Linq; + using System.Reactive.Linq; using System.Runtime.Serialization; using System.Threading; using Microsoft.Psi; @@ -167,6 +168,7 @@ public void MultipleWriteAttempts() [Timeout(60000)] public void PersistSingleStream() { + // generate a sequence from 1 .. 100, simulating also a latency and write to store var count = 100; var name = nameof(this.PersistSingleStream); using (var p = Pipeline.Create("write")) @@ -228,6 +230,116 @@ public void PersistMultipleStreams() Assert.AreEqual(factor * count * (count + 1) / 2, sum2); } + [TestMethod] + [Timeout(60000)] + public void CopyStore() + { + // generate a sequence from 1 .. 100 + var count = 100; + var name = nameof(this.CopyStore); + var supplemental = Math.E; // supplemental metadata + using (var p = Pipeline.Create("write")) + { + var writeStore = Store.Create(p, name, this.path); + var seq = Generators.Sequence(p, 1, i => i + 1, count, TimeSpan.FromTicks(1)) + .Select( + x => + { + Thread.Sleep(1); + return x; + }, + DeliveryPolicy.Unlimited); + seq.Write(supplemental, "seq", writeStore); + p.Run(); + } + + var copyName = $"{name}Copy"; + Store.Copy((name, this.path), (copyName, this.path)); + + // now replay the contents and verify we get the right sum + double sum = 0; + using (var p2 = Pipeline.Create("read")) + { + var readStore = Store.Open(p2, copyName, this.path); + var seq2 = readStore.OpenStream("seq"); + Assert.AreEqual(supplemental, readStore.GetSupplementalMetadata("seq"), double.Epsilon); + var verifier = seq2.Do(s => sum += s); + p2.Run(); + } + + Assert.AreEqual(count * (count + 1) / 2, sum); + } + + [TestMethod] + [Timeout(60000)] + public void EditStore() + { + var name = nameof(this.EditStore); + var supplemental = Math.E; // supplemental metadata + var now = DateTime.UtcNow; + DateTime[] originatingTimes = new DateTime[] + { + now, + now.AddMilliseconds(1), + now.AddMilliseconds(2), + now.AddMilliseconds(3), + now.AddMilliseconds(4), + now.AddMilliseconds(5), + }; + using (var p = Pipeline.Create("write")) + { + var writeStore = Store.Create(p, name, this.path); + var stream0 = Generators.Sequence(p, new[] + { + (7, originatingTimes[0]), + (42, originatingTimes[1]), + (1, originatingTimes[2]), + (123, originatingTimes[3]), + (1971, originatingTimes[4]), + (0, originatingTimes[5]), + }); + stream0.Write("stream0", writeStore); + var stream1 = Generators.Sequence(p, new[] + { + ('B', originatingTimes[1]), + ('C', originatingTimes[2]), + ('E', originatingTimes[3]), + ('F', originatingTimes[4]), + }); + var times = stream1.Select((_, e) => e.OriginatingTime).ToObservable().ToListObservable(); + stream1.Write(supplemental, "stream1", writeStore); + p.Run(); + originatingTimes = times.AsEnumerable().ToArray(); + } + + var edits = new Dictionary>(); + edits.Add( + "stream1", + new (bool, dynamic, DateTime)[] + { + //// TODO: why does Zip never get unsubscribed?! (true, 'G', originatingTimes[3].AddTicks(1)), // insert G after F (past end of original stream) + (true, 'A', originatingTimes[0].AddTicks(-1)), // insert A before B (before start of original stream) + (true, 'D', originatingTimes[1].AddTicks(1)), // insert D between C and E + (false, null, originatingTimes[1]), // delete C + (true, 'X', originatingTimes[2]), // update E to X + }); + + var editName = $"{name}Edited"; + Store.Edit((name, this.path), (editName, this.path), edits); + + // now replay the contents and verify we get the right sum + using (var p2 = Pipeline.Create("read")) + { + var readStore = Store.Open(p2, editName, this.path); + var stream0 = readStore.OpenStream("stream0").ToObservable().ToListObservable(); + var stream1 = readStore.OpenStream("stream1").ToObservable().ToListObservable(); + Assert.AreEqual(supplemental, readStore.GetSupplementalMetadata("stream1"), double.Epsilon); + p2.Run(); + Assert.IsTrue(Enumerable.SequenceEqual(new[] { 7, 42, 1, 123, 1971, 0 }, stream0.AsEnumerable())); + Assert.IsTrue(Enumerable.SequenceEqual(new[] { 'A', 'B', 'D', 'X', 'F' }, stream1.AsEnumerable())); + } + } + [TestMethod] [Timeout(60000)] public void Seek() @@ -277,14 +389,14 @@ public void Seek() [Timeout(60000)] public void ReadWhilePersistingToDisk() { - this.ReadWhilePersisting(nameof(ReadWhilePersistingToDisk), this.path); + this.ReadWhilePersisting(nameof(this.ReadWhilePersistingToDisk), this.path); } [TestMethod] [Timeout(60000)] public void ReadWhilePersistingInMemory() { - this.ReadWhilePersisting(nameof(ReadWhilePersistingInMemory), null); + this.ReadWhilePersisting(nameof(this.ReadWhilePersistingInMemory), null); } // with a null path, the file is only in memory (system file). With a non-null path, the file is also written to disk @@ -685,12 +797,12 @@ public void RealTimePlayback() [TestMethod] [Timeout(60000)] - public void CopyStore() + public void CopyStream() { var count = 100; var before = new Envelope[count + 1]; var after = new Envelope[count + 1]; - var name = nameof(this.CopyStore); + var name = nameof(this.CopyStream); using (var p = Pipeline.Create("write")) { @@ -789,7 +901,7 @@ public void CropStore() } // verify the results in the interval after the cropped range - for (int i = (count - 4); i <= count; i++) + for (int i = count - 4; i <= count; i++) { Assert.AreEqual(0, after[i].SequenceId); Assert.AreEqual(0, after[i].OriginatingTime.Ticks); @@ -822,66 +934,54 @@ public void RepairInvalidStore() // pipeline terminated normally so store should be valid Assert.IsTrue(Store.IsClosed(name, this.path)); - // now generate an invalid store - var p2 = Pipeline.Create("write2"); + // Now create a new pipeline in which we will simulate an invalid store by taking a + // snapshot of the store files midway through the execution (use synchronous policy + // to ensure that the messages actually hit the exporter before the snapshot is taken). + var p2 = Pipeline.Create("write2", DeliveryPolicy.SynchronousOrThrottle); var invalidStore = Store.Create(p2, name, this.path); string tempFolder = Path.Combine(this.path, Guid.NewGuid().ToString()); - - try + var seq2 = Generators.Sequence(p2, 1, i => i + 1, count, TimeSpan.FromTicks(1)); + seq2.Do((m, e) => { - var seq2 = Generators.Sequence(p2, 1, i => i + 1, count, TimeSpan.FromTicks(1)); - seq2.Do((m, e) => + // Halfway through, simulate abrupt termination of the pipeline by copying the store + // files while the pipeline is running, resulting in a store in an invalid state. + if (e.OriginatingTime.Ticks == count / 2) { - if (e.OriginatingTime.Ticks >= count / 2) - { - // Simulate abrupt termination of the pipeline by copying the store files - // while the pipeline is running, resulting in a store in an invalid state. + // at this point the store should still be open + Assert.IsFalse(Store.IsClosed(name, this.path)); - // at this point the store should still be open - Assert.IsFalse(Store.IsClosed(name, this.path)); + // We need to temporarily save the invalid store before disposing the pipeline, + // since the store will be rendered valid when the pipeline is terminated. + Directory.CreateDirectory(tempFolder); - // We need to temporarily save the invalid store before disposing the pipeline, - // since the store will be rendered valid when the pipeline is terminated. - Directory.CreateDirectory(tempFolder); + // copy the store files to the temp folder - we will restore them later + foreach (var file in Directory.EnumerateFiles(invalidStore.Path)) + { + var fileInfo = new FileInfo(file); + File.Copy(file, Path.Combine(tempFolder, fileInfo.Name)); + } + } + }).Write("seq", invalidStore); + seq2.Select(i => i.ToString()).Write("seqString", invalidStore); - // copy the store files to the temp folder - we will restore them later - foreach (var file in Directory.EnumerateFiles(invalidStore.Path)) - { - var fileInfo = new FileInfo(file); - File.Copy(file, Path.Combine(tempFolder, fileInfo.Name)); - } + // run the pipeline with exception handling enabled + p2.Run(new ReplayDescriptor(new DateTime(1), false)); + p2.Dispose(); - // throw an exception and terminate the pipeline - throw new Exception(); - } - }).Write("seq", invalidStore); - seq2.Select(i => i.ToString()).Write("seqString", invalidStore); + // after disposing the pipeline, the store becomes valid + Assert.IsTrue(Store.IsClosed(name, this.path)); - // run the pipeline with exception handling enabled - p2.Run(new ReplayDescriptor(new DateTime(1), false)); - } - catch + // delete the (now valid) store files + foreach (var file in Directory.EnumerateFiles(invalidStore.Path)) { + TestRunner.SafeFileDelete(file); } - finally - { - p2.Dispose(); - - // after disposing the pipeline, the store becomes valid - Assert.IsTrue(Store.IsClosed(name, this.path)); - // delete the (now valid) store files - foreach (var file in Directory.EnumerateFiles(invalidStore.Path)) - { - TestRunner.SafeFileDelete(file); - } - - // restore the invalid store files from the temp folder - foreach (var file in Directory.EnumerateFiles(tempFolder)) - { - var fileInfo = new FileInfo(file); - File.Move(file, Path.Combine(invalidStore.Path, fileInfo.Name)); - } + // restore the invalid store files from the temp folder + foreach (var file in Directory.EnumerateFiles(tempFolder)) + { + var fileInfo = new FileInfo(file); + File.Move(file, Path.Combine(invalidStore.Path, fileInfo.Name)); } // the generated store should be invalid prior to repairing @@ -904,14 +1004,19 @@ public void RepairInvalidStore() p3.Run(ReplayDescriptor.ReplayAll); } - // verify the results in the repaired store prior to the exception - for (int i = 0; i < count / 2; i++) + // Since we cannot guarantee when the messages written to the "invalid store" actually got + // flushed from the memory-mapped file, just verify that we got some of the initial data. + for (int i = 0; i < 3; i++) { Assert.AreEqual(valid[i].SequenceId, invalid[i].SequenceId); Assert.AreEqual(valid[i].OriginatingTime, invalid[i].OriginatingTime); } - // verify there are no results after the exception + // log how much of the store actually got written for diagnostic purposes only + int lastIndex = invalid.LastOrDefault(e => e.SequenceId != 0).SequenceId; + Console.WriteLine($"Last sequence id found: {lastIndex}"); + + // verify there are no results after the point at which the store was rendered invalid for (int j = count / 2; j <= count; j++) { Assert.AreEqual(0, invalid[j].SequenceId); @@ -928,7 +1033,7 @@ public void StoreWriteReadSpeedTest() { // here we write large messages to a store. This will quickly overflow the extents, causing // MemoryMappedViews to be disposed. This is a potentially blocking call which we now do on a - // separate thread. Prior to this fix, it would block writing/reading the store with a human-noticable + // separate thread. Prior to this fix, it would block writing/reading the store with a human-noticeable // delay of several seconds. var payload = new byte[1024 * 1024 * 10]; @@ -1225,6 +1330,36 @@ public void StoreProgressTest() Assert.AreEqual(1.0, lastValue); } + [TestMethod] + [Timeout(60000)] + public void PersistingStreamSupplementalMetadata() + { + var name = nameof(this.PersistingStreamSupplementalMetadata); + + // create store with supplemental meta + using (var p = Pipeline.Create("write")) + { + var store = Store.Create(p, name, this.path); + var stream0 = Generators.Range(p, 0, 10, TimeSpan.FromTicks(1)); + var stream1 = Generators.Range(p, 0, 10, TimeSpan.FromTicks(1)); + stream0.Write("NoMeta", store, true); + stream1.Write(("Favorite irrational number", Math.E), "WithMeta", store); + } + + // read it back with an importer + using (var q = Pipeline.Create("read")) + { + var store = Store.Open(q, name, this.path); + Assert.IsNull(store.GetMetadata("NoMeta").SupplementalMetadataTypeName); + Assert.AreEqual(typeof(ValueTuple).AssemblyQualifiedName, store.GetMetadata("WithMeta").SupplementalMetadataTypeName); + var supplemental0 = store.GetSupplementalMetadata<(string, double)>("WithMeta"); + Assert.AreEqual("Favorite irrational number", supplemental0.Item1); + Assert.AreEqual(Math.E, supplemental0.Item2); + Assert.ThrowsException(() => store.GetSupplementalMetadata("NoMeta")); + Assert.ThrowsException(() => store.GetSupplementalMetadata("WithMeta")); + } + } + [DataContract(Name = "TestDataContract")] private class DataContractTypeV1 { diff --git a/Sources/Runtime/Test.Psi/PipelineTest.cs b/Sources/Runtime/Test.Psi/PipelineTest.cs index 4682ad406..870b21806 100644 --- a/Sources/Runtime/Test.Psi/PipelineTest.cs +++ b/Sources/Runtime/Test.Psi/PipelineTest.cs @@ -6,13 +6,13 @@ namespace Test.Psi using System; using System.Collections.Generic; using System.Linq; - using System.Threading; using System.Reactive; using System.Reactive.Linq; + using System.Threading; using Microsoft.Psi; using Microsoft.Psi.Components; - using Microsoft.VisualStudio.TestTools.UnitTesting; using Microsoft.Psi.Diagnostics; + using Microsoft.VisualStudio.TestTools.UnitTesting; [TestClass] public class PipelineTest @@ -88,10 +88,6 @@ public void Subpipelines() public class TestReactiveCompositeComponent : Subpipeline { - public Receiver In { private set; get; } - - public Emitter Out { private set; get; } - public TestReactiveCompositeComponent(Pipeline parent) : base(parent, "TestReactiveCompositeComponent") { @@ -101,6 +97,10 @@ public TestReactiveCompositeComponent(Pipeline parent) this.Out = output.Out; input.Select(i => i * 2).PipeTo(output); } + + public Receiver In { get; private set; } + + public Emitter Out { get; private set; } } [TestMethod] @@ -122,8 +122,6 @@ public void SubpipelineAsReactiveComponent() public class TestFiniteSourceCompositeComponent : Subpipeline { - public Emitter Out { private set; get; } - public TestFiniteSourceCompositeComponent(Pipeline parent) : base(parent, "TestFiniteSourceCompositeComponent") { @@ -131,6 +129,8 @@ public TestFiniteSourceCompositeComponent(Pipeline parent) this.Out = output.Out; Generators.Range(this, 0, 10, TimeSpan.FromTicks(1)).Out.PipeTo(output); } + + public Emitter Out { get; private set; } } [TestMethod] @@ -150,8 +150,6 @@ public void SubpipelineAsFiniteSourceComponent() public class TestInfiniteSourceCompositeComponent : Subpipeline { - public Emitter Out { private set; get; } - public TestInfiniteSourceCompositeComponent(Pipeline parent) : base(parent, "TestInfiniteSourceCompositeComponent") { @@ -160,6 +158,8 @@ public TestInfiniteSourceCompositeComponent(Pipeline parent) var timer = Timers.Timer(this, TimeSpan.FromMilliseconds(10)); timer.Aggregate(0, (i, _) => i + 1).PipeTo(output); } + + public Emitter Out { get; private set; } } [TestMethod] @@ -174,7 +174,7 @@ public void SubpipelineAsInfiniteSourceComponent() var infinite = new TestInfiniteSourceCompositeComponent(p); Assert.AreEqual(p, infinite.Out.Pipeline); // composite component shouldn't expose the fact that subpipeline is involved results = infinite.Out.ToObservable().ToListObservable(); - p.PipelineCompleted += ((_, __) => completed = true); + p.PipelineCompleted += (_, __) => completed = true; p.RunAsync(); Thread.Sleep(200); Assert.IsFalse(completed); // note that infinite source composite-component subpipeline never completes (parent pipeline must be disposed explicitly) @@ -321,6 +321,7 @@ public void ComponentInitStartOrderingWhenExceedingSchedulerThreadPool() pipeline.Run(); } } + private class FiniteToInfiniteTestComponent : ISourceComponent { private Action notifyCompletionTime; @@ -464,8 +465,6 @@ public class GeneratorWithLatency : Generator private readonly TimeSpan interval; private readonly TimeSpan latency; - public Emitter Out { get; } - public GeneratorWithLatency(Pipeline pipeline, TimeSpan interval, TimeSpan latency) : base(pipeline, isInfiniteSource: true) { @@ -474,6 +473,8 @@ public GeneratorWithLatency(Pipeline pipeline, TimeSpan interval, TimeSpan laten this.Out = pipeline.CreateEmitter(this, nameof(this.Out)); } + public Emitter Out { get; } + protected override DateTime GenerateNext(DateTime currentTime) { // introduce a delay (in wall-clock time) to artificially slow down the generator @@ -506,7 +507,8 @@ public void PipelineShutdownWithLatency() var results = seq.AsEnumerable().ToArray(); CollectionAssert.AreEqual( - new[] { + new[] + { 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, @@ -517,7 +519,7 @@ public void PipelineShutdownWithLatency() 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, - 10 + 10, }, results); } } @@ -525,10 +527,6 @@ public void PipelineShutdownWithLatency() // A composite component which contains a GeneratorWithLatency used to densify an input stream public class TestSourceComponentWithGenerator : Subpipeline { - public Receiver In { get; } - - public Emitter Out { get; } - public TestSourceComponentWithGenerator(Pipeline parent, TimeSpan interval, TimeSpan latency) : base(parent, "sub") { @@ -544,6 +542,10 @@ public TestSourceComponentWithGenerator(Pipeline parent, TimeSpan interval, Time var densified = clock.Out.Join(input.Out, RelativeTimeInterval.Past()).Select(x => x.Item2); densified.PipeTo(output.In); } + + public Receiver In { get; } + + public Emitter Out { get; } } [TestMethod] @@ -714,10 +716,199 @@ public void FinalizationTestLongCycle() Assert.IsTrue(log.Contains("CEmitterZ Closed")); Assert.IsTrue(log.Contains("CEmitterGen Closed")); + // Emitters should have closed in C, A, B order + Assert.IsTrue(log.IndexOf("CEmitterAny Closed") < log.IndexOf("AEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("CEmitterX Closed") < log.IndexOf("AEmitterX Closed")); + Assert.IsTrue(log.IndexOf("CEmitterY Closed") < log.IndexOf("AEmitterY Closed")); + Assert.IsTrue(log.IndexOf("CEmitterZ Closed") < log.IndexOf("AEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("CEmitterGen Closed") < log.IndexOf("AEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("AEmitterAny Closed") < log.IndexOf("BEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("AEmitterX Closed") < log.IndexOf("BEmitterX Closed")); + Assert.IsTrue(log.IndexOf("AEmitterY Closed") < log.IndexOf("BEmitterY Closed")); + Assert.IsTrue(log.IndexOf("AEmitterZ Closed") < log.IndexOf("BEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("AEmitterGen Closed") < log.IndexOf("BEmitterGen Closed")); + + // subscribed receivers should have been unsubscribed + Assert.IsTrue(log.Contains("AReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("BReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("CReceiverX Unsubscribed")); + + // non-subscribed receivers should have done *nothing* + Assert.IsFalse(log.Contains("AReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("AReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("BReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("BReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("CReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("CReceiverZ Unsubscribed")); + } + + [TestMethod] + [Timeout(60000)] + public void FinalizationTestDisjointCycles() + { + var log = new List(); + using (var p = Pipeline.Create()) + { + /* + * =---= =---= =---= =---= + * | A |--->| C | | B |--->| D | + * =---= =---= =---= =---= + * ^ | ^ | + * | | | | + * +--------+ +--------+ + */ + var d = new FinalizationTestComponent(p, "D", log); + var c = new FinalizationTestComponent(p, "C", log); + var b = new FinalizationTestComponent(p, "B", log); + var a = new FinalizationTestComponent(p, "A", log); + a.Generator.PipeTo(c.ReceiverX); + b.Generator.PipeTo(d.ReceiverX); + c.RelayFromX.PipeTo(a.ReceiverX); + d.RelayFromX.PipeTo(b.ReceiverX); + p.Run(); + } + + // all emitters should have closed + Assert.IsTrue(log.Contains("AEmitterAny Closed")); + Assert.IsTrue(log.Contains("AEmitterX Closed")); + Assert.IsTrue(log.Contains("AEmitterY Closed")); + Assert.IsTrue(log.Contains("AEmitterZ Closed")); + Assert.IsTrue(log.Contains("AEmitterGen Closed")); + + Assert.IsTrue(log.Contains("BEmitterAny Closed")); + Assert.IsTrue(log.Contains("BEmitterX Closed")); + Assert.IsTrue(log.Contains("BEmitterY Closed")); + Assert.IsTrue(log.Contains("BEmitterZ Closed")); + Assert.IsTrue(log.Contains("BEmitterGen Closed")); + + Assert.IsTrue(log.Contains("CEmitterAny Closed")); + Assert.IsTrue(log.Contains("CEmitterX Closed")); + Assert.IsTrue(log.Contains("CEmitterY Closed")); + Assert.IsTrue(log.Contains("CEmitterZ Closed")); + Assert.IsTrue(log.Contains("CEmitterGen Closed")); + + Assert.IsTrue(log.Contains("DEmitterAny Closed")); + Assert.IsTrue(log.Contains("DEmitterX Closed")); + Assert.IsTrue(log.Contains("DEmitterY Closed")); + Assert.IsTrue(log.Contains("DEmitterZ Closed")); + Assert.IsTrue(log.Contains("DEmitterGen Closed")); + + // Emitters should have closed in D, C, then B or A order + Assert.IsTrue(log.IndexOf("DEmitterAny Closed") < log.IndexOf("CEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("DEmitterX Closed") < log.IndexOf("CEmitterX Closed")); + Assert.IsTrue(log.IndexOf("DEmitterY Closed") < log.IndexOf("CEmitterY Closed")); + Assert.IsTrue(log.IndexOf("DEmitterZ Closed") < log.IndexOf("CEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("DEmitterGen Closed") < log.IndexOf("CEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("CEmitterAny Closed") < log.IndexOf("BEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("CEmitterX Closed") < log.IndexOf("BEmitterX Closed")); + Assert.IsTrue(log.IndexOf("CEmitterY Closed") < log.IndexOf("BEmitterY Closed")); + Assert.IsTrue(log.IndexOf("CEmitterZ Closed") < log.IndexOf("BEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("CEmitterGen Closed") < log.IndexOf("BEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("CEmitterAny Closed") < log.IndexOf("AEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("CEmitterX Closed") < log.IndexOf("AEmitterX Closed")); + Assert.IsTrue(log.IndexOf("CEmitterY Closed") < log.IndexOf("AEmitterY Closed")); + Assert.IsTrue(log.IndexOf("CEmitterZ Closed") < log.IndexOf("AEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("CEmitterGen Closed") < log.IndexOf("AEmitterGen Closed")); + + // subscribed receivers should have been unsubscribed + Assert.IsTrue(log.Contains("AReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("BReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("CReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("DReceiverX Unsubscribed")); + + // non-subscribed receivers should have done *nothing* + Assert.IsFalse(log.Contains("AReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("AReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("BReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("BReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("CReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("CReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("DReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("DReceiverZ Unsubscribed")); + } + + [TestMethod] + [Timeout(60000)] + public void FinalizationTestDisjointUpstreamSelfCycles() + { + var log = new List(); + using (var p = Pipeline.Create()) + { + /* + * =---= =---= =---= =---= + * +-->| A |--->| B | +-->| C |--->| D | + * | =---= =---= | =---= =---= + * | | | | + * +-----+ +-----+ + */ + var d = new FinalizationTestComponent(p, "D", log); + var c = new FinalizationTestComponent(p, "C", log); // finalized 1st although constructed 2nd + var b = new FinalizationTestComponent(p, "B", log); + var a = new FinalizationTestComponent(p, "A", log); + a.Generator.PipeTo(a.ReceiverX); + a.RelayFromX.PipeTo(b.ReceiverX); + c.Generator.PipeTo(c.ReceiverX); + c.RelayFromX.PipeTo(d.ReceiverX); + p.Run(); + } + + // all emitters should have closed + Assert.IsTrue(log.Contains("AEmitterAny Closed")); + Assert.IsTrue(log.Contains("AEmitterX Closed")); + Assert.IsTrue(log.Contains("AEmitterY Closed")); + Assert.IsTrue(log.Contains("AEmitterZ Closed")); + Assert.IsTrue(log.Contains("AEmitterGen Closed")); + + Assert.IsTrue(log.Contains("BEmitterAny Closed")); + Assert.IsTrue(log.Contains("BEmitterX Closed")); + Assert.IsTrue(log.Contains("BEmitterY Closed")); + Assert.IsTrue(log.Contains("BEmitterZ Closed")); + Assert.IsTrue(log.Contains("BEmitterGen Closed")); + + Assert.IsTrue(log.Contains("CEmitterAny Closed")); + Assert.IsTrue(log.Contains("CEmitterX Closed")); + Assert.IsTrue(log.Contains("CEmitterY Closed")); + Assert.IsTrue(log.Contains("CEmitterZ Closed")); + Assert.IsTrue(log.Contains("CEmitterGen Closed")); + + Assert.IsTrue(log.Contains("DEmitterAny Closed")); + Assert.IsTrue(log.Contains("DEmitterX Closed")); + Assert.IsTrue(log.Contains("DEmitterY Closed")); + Assert.IsTrue(log.Contains("DEmitterZ Closed")); + Assert.IsTrue(log.Contains("DEmitterGen Closed")); + + // Emitters should have closed in C, A, then D or B order + Assert.IsTrue(log.IndexOf("CEmitterAny Closed") < log.IndexOf("AEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("CEmitterX Closed") < log.IndexOf("AEmitterX Closed")); + Assert.IsTrue(log.IndexOf("CEmitterY Closed") < log.IndexOf("AEmitterY Closed")); + Assert.IsTrue(log.IndexOf("CEmitterZ Closed") < log.IndexOf("AEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("CEmitterGen Closed") < log.IndexOf("AEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("AEmitterAny Closed") < log.IndexOf("DEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("AEmitterX Closed") < log.IndexOf("DEmitterX Closed")); + Assert.IsTrue(log.IndexOf("AEmitterY Closed") < log.IndexOf("DEmitterY Closed")); + Assert.IsTrue(log.IndexOf("AEmitterZ Closed") < log.IndexOf("DEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("AEmitterGen Closed") < log.IndexOf("DEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("AEmitterAny Closed") < log.IndexOf("BEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("AEmitterX Closed") < log.IndexOf("BEmitterX Closed")); + Assert.IsTrue(log.IndexOf("AEmitterY Closed") < log.IndexOf("BEmitterY Closed")); + Assert.IsTrue(log.IndexOf("AEmitterZ Closed") < log.IndexOf("BEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("AEmitterGen Closed") < log.IndexOf("BEmitterGen Closed")); + // subscribed receivers should have been unsubscribed Assert.IsTrue(log.Contains("AReceiverX Unsubscribed")); Assert.IsTrue(log.Contains("BReceiverX Unsubscribed")); Assert.IsTrue(log.Contains("CReceiverX Unsubscribed")); + Assert.IsTrue(log.Contains("DReceiverX Unsubscribed")); // non-subscribed receivers should have done *nothing* Assert.IsFalse(log.Contains("AReceiverY Unsubscribed")); @@ -728,6 +919,9 @@ public void FinalizationTestLongCycle() Assert.IsFalse(log.Contains("CReceiverY Unsubscribed")); Assert.IsFalse(log.Contains("CReceiverZ Unsubscribed")); + + Assert.IsFalse(log.Contains("DReceiverY Unsubscribed")); + Assert.IsFalse(log.Contains("DReceiverZ Unsubscribed")); } [TestMethod] @@ -869,7 +1063,7 @@ public void FinalizationTestFeedbackLoop() [Timeout(60000)] public void FinalizationTestWithSubpipeline() { - // this exercised traversal of Connector cross-pipeline bridges + // this exercises traversal of Connector cross-pipeline bridges // internally, a node (PipelineElement) is created on each side with a shared state object (the Connector component) // finalization traverses these boundaries @@ -1496,6 +1690,7 @@ public void FinalizationTestWithSeparatedCycles() d.RelayFromX.PipeTo(c.ReceiverX); b.Generator.PipeTo(c.ReceiverY); c.Generator.PipeTo(b.ReceiverY); + b.RelayFromY.PipeTo(c.ReceiverZ); p.Run(); } @@ -1524,12 +1719,32 @@ public void FinalizationTestWithSeparatedCycles() Assert.IsTrue(log.Contains("DEmitterZ Closed")); Assert.IsTrue(log.Contains("DEmitterGen Closed")); + // Should finalize B first since it has the most active outputs, then A, then either C or D (in reality construction order: D then C) + Assert.IsTrue(log.IndexOf("BEmitterAny Closed") < log.IndexOf("AEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("BEmitterX Closed") < log.IndexOf("AEmitterX Closed")); + Assert.IsTrue(log.IndexOf("BEmitterY Closed") < log.IndexOf("AEmitterY Closed")); + Assert.IsTrue(log.IndexOf("BEmitterZ Closed") < log.IndexOf("AEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("BEmitterGen Closed") < log.IndexOf("AEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("AEmitterAny Closed") < log.IndexOf("CEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("AEmitterX Closed") < log.IndexOf("CEmitterX Closed")); + Assert.IsTrue(log.IndexOf("AEmitterY Closed") < log.IndexOf("CEmitterY Closed")); + Assert.IsTrue(log.IndexOf("AEmitterZ Closed") < log.IndexOf("CEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("AEmitterGen Closed") < log.IndexOf("CEmitterGen Closed")); + + Assert.IsTrue(log.IndexOf("AEmitterAny Closed") < log.IndexOf("DEmitterAny Closed")); + Assert.IsTrue(log.IndexOf("AEmitterX Closed") < log.IndexOf("DEmitterX Closed")); + Assert.IsTrue(log.IndexOf("AEmitterY Closed") < log.IndexOf("DEmitterY Closed")); + Assert.IsTrue(log.IndexOf("AEmitterZ Closed") < log.IndexOf("DEmitterZ Closed")); + Assert.IsTrue(log.IndexOf("AEmitterGen Closed") < log.IndexOf("DEmitterGen Closed")); + // subscribed receivers should have been unsubscribed Assert.IsTrue(log.Contains("AReceiverX Unsubscribed")); Assert.IsTrue(log.Contains("BReceiverX Unsubscribed")); Assert.IsTrue(log.Contains("BReceiverY Unsubscribed")); Assert.IsTrue(log.Contains("CReceiverX Unsubscribed")); Assert.IsTrue(log.Contains("CReceiverY Unsubscribed")); + Assert.IsTrue(log.Contains("CReceiverZ Unsubscribed")); Assert.IsTrue(log.Contains("DReceiverX Unsubscribed")); // non-subscribed receivers should have done *nothing* @@ -1538,8 +1753,6 @@ public void FinalizationTestWithSeparatedCycles() Assert.IsFalse(log.Contains("BReceiverZ Unsubscribed")); - Assert.IsFalse(log.Contains("CReceiverZ Unsubscribed")); - Assert.IsFalse(log.Contains("DReceiverY Unsubscribed")); Assert.IsFalse(log.Contains("DReceiverZ Unsubscribed")); } @@ -1561,7 +1774,12 @@ public void FinalizationTestUnsubscribedHandler() var emitter = p.CreateEmitter(collector, "Emitter"); // on unsubscribe, post collected messages (with an artificial latency to slow them down) - receiver.Unsubscribed += _ => collector.ForEach(m => { Thread.Sleep(33); emitter.Post(m.data, m.env.OriginatingTime); }); + receiver.Unsubscribed += _ => collector.ForEach( + m => + { + Thread.Sleep(33); + emitter.Post(m.data, m.env.OriginatingTime); + }); // log posted messages emitter.Do((d, e) => log.Add($"{e.OriginatingTime.TimeOfDay}:{d}")); @@ -1759,10 +1977,13 @@ public void DiagnosticsTest() b.RelayFromX.PipeTo(cOut.In); cOut.Out.PipeTo(d.ReceiverX); - p.Diagnostics.Do(diag => graph = graph ?? diag); + p.Diagnostics.Do(diag => graph = diag.DeepClone()); p.RunAsync(); - while (graph == null) Thread.Sleep(10); + while (graph == null) + { + Thread.Sleep(10); + } } Assert.AreEqual(2, graph.GetPipelineCount()); // total graphs @@ -1777,11 +1998,11 @@ public void DiagnosticsTest() // example complex query: average latency at emitter across reactive components in leaf subpipelines var complex = graph.GetAllPipelineDiagnostics() - .Where(p => p.SubpipelineDiagnostics.Length == 0) // leaf subpipelines - .GetAllPipelineElements() - .Where(e => e.Kind == PipelineElementKind.Reactive) // reactive components - .GetAllReceiverDiagnostics() - .Select(r => r.MessageLatencyAtEmitter); // average latency at emitter into each component's receivers + .Where(p => p.SubpipelineDiagnostics.Length == 0) // leaf subpipelines + .GetAllPipelineElements() + .Where(e => e.Kind == PipelineElementKind.Reactive) // reactive components + .GetAllReceiverDiagnostics() + .Select(r => r.MessageLatencyAtEmitter); // average latency at emitter into each component's receivers Assert.AreEqual("default", graph.Name); Assert.IsTrue(graph.IsPipelineRunning); @@ -1899,6 +2120,10 @@ public void PipelineProgressTestSubpipeline() // run and wait for pipeline to complete pipeline.RunAsync(replay, new Progress(x => progress.Add(x))); + + // test adding a dynamic subpipeline after main pipeline has started + var subpipeline2 = Subpipeline.Create(pipeline, "subpipeline2"); + pipeline.WaitAll(); } @@ -1920,6 +2145,75 @@ public void PipelineProgressTestSubpipeline() Assert.AreEqual(1.0, lastValue); } + [TestMethod] + [Timeout(60000)] + public void SubpipelineWiringOnPipelineRun() + { + var results = new List(); + + using (var pipeline = Pipeline.Create()) + { + var inputs = Generators.Range(pipeline, 0, 10, TimeSpan.FromTicks(1)); + var subSquare = Subpipeline.Create(pipeline); + var connSquare = subSquare.CreateInputConnectorFrom(pipeline, "square"); + var square = new Processor(subSquare, (x, e, emitter) => emitter.Post(x * x, e.OriginatingTime)); + var subAddOne = Subpipeline.Create(pipeline); + + // wiring between parent pipeline and first child subpipeline + subSquare.PipelineRun += (_, __) => + { + connSquare.PipeTo(square); + inputs.PipeTo(connSquare); + }; + + // second child subpipeline creates grandchild subpipeline + subAddOne.PipelineRun += (_, __) => + { + var subSubAddOne = Subpipeline.Create(subAddOne); + + // wiring from first child subpipeline to grandchild subpipeline, the back to parent pipeline + subSubAddOne.PipelineRun += (s, e) => + { + var connAddOne = subSubAddOne.CreateInputConnectorFrom(subSquare, "addOne"); + var addOne = new Processor(subSubAddOne, (x, env, emitter) => emitter.Post(x + 1, env.OriginatingTime)); + var connResult = subSubAddOne.CreateOutputConnectorTo(pipeline, "result"); + square.PipeTo(connAddOne); + connAddOne.PipeTo(addOne); + addOne.PipeTo(connResult); + + // capture result stream + connResult.Do(x => results.Add(x)); + }; + }; + + pipeline.Run(); + } + + // verify result stream y = x^2 + 1 + CollectionAssert.AreEqual(Enumerable.Range(0, 10).Select(x => (x * x) + 1).ToArray(), results); + } + + [TestMethod] + public void OnPipelineCompleted() + { + var output = new List(); + using (var p = Pipeline.Create()) + { + Generators.Return(p, 0); + p.PipelineCompleted += (_, __) => + { + // slow handler to test that it completes execution before Pipeline.Run() returns + Thread.Sleep(100); + output.Add("Completed"); + }; + + p.Run(); + } + + output.Add("Disposed"); + CollectionAssert.AreEqual(new[] { "Completed", "Disposed" }, output); + } + private class FinalizationTestComponent : ISourceComponent { private readonly Pipeline pipeline; @@ -1954,6 +2248,22 @@ public FinalizationTestComponent(Pipeline pipeline, string name, List lo this.ReceiverZ.Unsubscribed += _ => this.Log($"{this.name}ReceiverZ Unsubscribed"); } + public Receiver ReceiverX { get; private set; } // relays to EmitterX and W + + public Receiver ReceiverY { get; private set; } // relays to EmitterY and W + + public Receiver ReceiverZ { get; private set; } // relays to EmitterY and W + + public Emitter RelayFromAny { get; private set; } // relays from ReceiverX or Y + + public Emitter RelayFromX { get; private set; } // relays from ReceiverX + + public Emitter RelayFromY { get; private set; } // relays from ReceiverY + + public Emitter RelayFromZ { get; private set; } // relays from ReceiverY + + public Emitter Generator { get; private set; } // emits at 10ms intervals + public void Start(Action notifyCompletionTime) { this.notifyCompletionTime = notifyCompletionTime; @@ -1971,7 +2281,7 @@ public void Start(Action notifyCompletionTime) public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) { - if (timer != null) + if (this.timer != null) { this.timer.Elapsed -= this.Elapsed; } @@ -2010,21 +2320,21 @@ private void EmitFromEach(int m, DateTime time) private void ReceiveX(int m, Envelope e) { this.Log($"{this.name}ReceiveX {m}"); - EmitFromEach(m, e.OriginatingTime); + this.EmitFromEach(m, e.OriginatingTime); this.RelayFromX.Post(m, e.OriginatingTime); } private void ReceiveY(int m, Envelope e) { this.Log($"{this.name}ReceiveY {m}"); - EmitFromEach(m, e.OriginatingTime); + this.EmitFromEach(m, e.OriginatingTime); this.RelayFromY.Post(m, e.OriginatingTime); } private void ReceiveZ(int m, Envelope e) { this.Log($"{this.name}ReceiveZ {m}"); - EmitFromEach(m, e.OriginatingTime); + this.EmitFromEach(m, e.OriginatingTime); this.RelayFromZ.Post(m, e.OriginatingTime); } @@ -2035,22 +2345,6 @@ private void Log(string entry) this.log.Add(entry); } } - - public Receiver ReceiverX { get; private set; } // relays to EmitterX and W - - public Receiver ReceiverY { get; private set; } // relays to EmitterY and W - - public Receiver ReceiverZ { get; private set; } // relays to EmitterY and W - - public Emitter RelayFromAny { get; private set; } // relays from ReceiverX or Y - - public Emitter RelayFromX { get; private set; } // relays from ReceiverX - - public Emitter RelayFromY { get; private set; } // relays from ReceiverY - - public Emitter RelayFromZ { get; private set; } // relays from ReceiverY - - public Emitter Generator { get; private set; } // emits at 10ms intervals } } } diff --git a/Sources/Runtime/Test.Psi/RemotingTests.cs b/Sources/Runtime/Test.Psi/RemotingTests.cs index 37311ea55..6fa46cc17 100644 --- a/Sources/Runtime/Test.Psi/RemotingTests.cs +++ b/Sources/Runtime/Test.Psi/RemotingTests.cs @@ -99,7 +99,7 @@ private void ReceiveAndValidate(Process server) doneEvt.WaitOne(20000); #if !ShellExecute server.StandardInput.WriteLine(); - server.StandardInput.WriteLine(); // the test executon framework is also waiting for a line + server.StandardInput.WriteLine(); // the test execution framework is also waiting for a line #endif server.WaitForExit(); Console.WriteLine("Server completed."); @@ -115,7 +115,7 @@ private void ReceiveAndValidate(Process server) /// /// Starts the Test.Psi.exe process with the specified entry point (needs to be a public method) /// - /// The name of a public mehtod. Doesn't need to have the [TestMethod] annotation. + /// The name of a public method. Doesn't need to have the [TestMethod] annotation. /// A process. Caller should ensure the process terminates (e.g via process.Kill) private Process StartServer(string entryPoint) { diff --git a/Sources/Runtime/Test.Psi/SchedulerTester.cs b/Sources/Runtime/Test.Psi/SchedulerTester.cs index 7c286ffce..244fa91c3 100644 --- a/Sources/Runtime/Test.Psi/SchedulerTester.cs +++ b/Sources/Runtime/Test.Psi/SchedulerTester.cs @@ -238,7 +238,7 @@ public void FutureSchedulingWithClockEnforcement() Assert.IsFalse(results.Any(l => l < 0)); } - // validate that items scheduled in the future get delivered imediately when the clock is not enforced + // validate that items scheduled in the future get delivered immediately when the clock is not enforced [TestMethod] [Timeout(60000)] public void FutureSchedulingWithoutClockEnforcement() diff --git a/Sources/Runtime/Test.Psi/SerializationTester.cs b/Sources/Runtime/Test.Psi/SerializationTester.cs index 51bda7b4f..e1a240d33 100644 --- a/Sources/Runtime/Test.Psi/SerializationTester.cs +++ b/Sources/Runtime/Test.Psi/SerializationTester.cs @@ -20,7 +20,7 @@ public class SerializationTester internal enum FooEnum : uint { One = 0x1, - Two = 0x2 + Two = 0x2, } [TestMethod] diff --git a/Sources/Runtime/Test.Psi/SharedTester.cs b/Sources/Runtime/Test.Psi/SharedTester.cs index 4077c817c..9a70e58d7 100644 --- a/Sources/Runtime/Test.Psi/SharedTester.cs +++ b/Sources/Runtime/Test.Psi/SharedTester.cs @@ -9,8 +9,8 @@ namespace Test.Psi using System.Threading; using Microsoft.Psi; using Microsoft.Psi.Common; - using Microsoft.Psi.Serialization; using Microsoft.Psi.Imaging; + using Microsoft.Psi.Serialization; using Microsoft.VisualStudio.TestTools.UnitTesting; [TestClass] @@ -56,7 +56,7 @@ public void RefCountedTest() var sharedPool = new SharedPool(() => UnmanagedBuffer.Allocate(100), 1); var shared = sharedPool.GetOrCreate(); // refcount = 1 - // a private copy shoudl point to the same resource + // a private copy should point to the same resource var otherShared = shared.DeepClone(); // refcount = 1 + 1 Assert.AreNotEqual(shared, otherShared); Assert.AreEqual(shared.Inner, otherShared.Inner); @@ -343,7 +343,7 @@ public void SharedImagePoolCollisionTest() Assert.AreEqual(7, bmp75.Width); Assert.AreEqual(5, bmp75.Height); - var shared57 = ImagePool.GetOrCreate(bmp57); + var shared57 = ImagePool.GetOrCreateFromBitmap(bmp57); Assert.AreEqual(5, shared57.Resource.Width); Assert.AreEqual(7, shared57.Resource.Height); @@ -352,7 +352,7 @@ public void SharedImagePoolCollisionTest() // stride and total size of the recycled image could be incorrect. shared57.Dispose(); // release to be recycled - var shared75 = ImagePool.GetOrCreate(bmp75); // should *not* get the recycled image + var shared75 = ImagePool.GetOrCreateFromBitmap(bmp75); // should *not* get the recycled image Assert.AreEqual(7, shared75.Resource.Width); Assert.AreEqual(5, shared75.Resource.Height); } diff --git a/Sources/Runtime/Test.Psi/StatisticalTests.cs b/Sources/Runtime/Test.Psi/StatisticalTests.cs index 326b2e79b..110db2d87 100644 --- a/Sources/Runtime/Test.Psi/StatisticalTests.cs +++ b/Sources/Runtime/Test.Psi/StatisticalTests.cs @@ -15,7 +15,8 @@ public class StatisticalTests [Timeout(60000)] public void StatisticsMinDouble() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new double[] { }, // empty sequence new double[] { } // expected output @@ -43,15 +44,15 @@ public void StatisticsMinDouble() ( new[] { double.PositiveInfinity, 1.0, double.NegativeInfinity, -1.0, double.NaN }, // sequence with +/- infinity new[] { double.PositiveInfinity, 1.0, double.NegativeInfinity, double.NegativeInfinity, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMinDoubleArray() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new double[][] // sequence of enumerations { @@ -60,11 +61,10 @@ public void StatisticsMinDoubleArray() new[] { 1.0, double.NaN, -1.0 }, new[] { double.NaN, 2.0, -1.0 }, new[] { double.NegativeInfinity, 2.0, -1.0 }, - new[] { double.PositiveInfinity, 2.0, -1.0 } + new[] { double.PositiveInfinity, 2.0, -1.0 }, }, new[] { 1.0, 1.0, double.NaN, double.NaN, double.NegativeInfinity, -1.0 } // expected output - ) - ); + )); } [TestMethod] @@ -75,12 +75,12 @@ public void StatisticsMinEmptyDoubleArray() try { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new double[][] { new double[] { } }, // sequence containing an empty enumeration new double[] { } // no expected output - should throw - ) - ); + )); } catch (Exception e) { @@ -96,7 +96,8 @@ public void StatisticsMinEmptyDoubleArray() [Timeout(60000)] public void StatisticsMinNullableDoubleArray() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new double?[][] // sequence of enumerations { @@ -107,18 +108,18 @@ public void StatisticsMinNullableDoubleArray() new double?[] { null, 1.0, null, double.NaN, null, -1.0, null }, new double?[] { null, double.NaN, null, 2.0, null, -1.0, null }, new double?[] { null, double.NegativeInfinity, null, 2.0, null, -1.0, null }, - new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.0, null } + new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.0, null }, }, new double?[] { null, null, 1.0, 1.0, double.NaN, double.NaN, double.NegativeInfinity, -1.0 } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMinFloat() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new float[] { }, // empty sequence new float[] { } // expected output @@ -146,15 +147,15 @@ public void StatisticsMinFloat() ( new[] { float.PositiveInfinity, 1.0f, float.NegativeInfinity, -1.0f, float.NaN }, // sequence with +/- infinity new[] { float.PositiveInfinity, 1.0f, float.NegativeInfinity, float.NegativeInfinity, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMinFloatArray() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new float[][] // sequence of enumerations { @@ -163,11 +164,10 @@ public void StatisticsMinFloatArray() new[] { 1.0f, float.NaN, -1.0f }, new[] { float.NaN, 2.0f, -1.0f }, new[] { float.NegativeInfinity, 2.0f, -1.0f }, - new[] { float.PositiveInfinity, 2.0f, -1.0f } + new[] { float.PositiveInfinity, 2.0f, -1.0f }, }, new[] { 1.0f, 1.0f, float.NaN, float.NaN, float.NegativeInfinity, -1.0f } // expected output - ) - ); + )); } [TestMethod] @@ -178,12 +178,12 @@ public void StatisticsMinEmptyFloatArray() try { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new float[][] { new float[] { } }, // sequence containing an empty enumeration new float[] { } // no expected output - should throw - ) - ); + )); } catch (Exception e) { @@ -199,7 +199,8 @@ public void StatisticsMinEmptyFloatArray() [Timeout(60000)] public void StatisticsMinNullableFloatArray() { - this.RunTest(Operators.Min, + this.RunTest( + Operators.Min, ( new float?[][] // sequence of enumerations { @@ -210,18 +211,18 @@ public void StatisticsMinNullableFloatArray() new float?[] { null, 1.0f, null, float.NaN, null, -1.0f, null }, new float?[] { null, float.NaN, null, 2.0f, null, -1.0f, null }, new float?[] { null, float.NegativeInfinity, null, 2.0f, null, -1.0f, null }, - new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.0f, null } + new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.0f, null }, }, new float?[] { null, null, 1.0f, 1.0f, float.NaN, float.NaN, float.NegativeInfinity, -1.0f } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMaxDouble() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new double[] { }, // empty sequence new double[] { } // expected output @@ -249,15 +250,15 @@ public void StatisticsMaxDouble() ( new[] { double.NegativeInfinity, -1.0, double.PositiveInfinity, 1.0, double.NaN }, // sequence with +/- infinity new[] { double.NegativeInfinity, -1.0, double.PositiveInfinity, double.PositiveInfinity, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMaxDoubleArray() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new double[][] // sequence of enumerations { @@ -266,11 +267,10 @@ public void StatisticsMaxDoubleArray() new[] { 1.0, double.NaN, -1.0 }, new[] { double.NaN, 2.0, -1.0 }, new[] { double.NegativeInfinity, 2.0, -1.0 }, - new[] { double.PositiveInfinity, 2.0, -1.0 } + new[] { double.PositiveInfinity, 2.0, -1.0 }, }, new[] { 1.0, 2.0, double.NaN, double.NaN, 2.0, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] @@ -281,12 +281,12 @@ public void StatisticsMaxEmptyDoubleArray() try { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new double[][] { new double[] { } }, // sequence containing an empty enumeration new double[] { } // no expected output - should throw - ) - ); + )); } catch (Exception e) { @@ -302,7 +302,8 @@ public void StatisticsMaxEmptyDoubleArray() [Timeout(60000)] public void StatisticsMaxNullableDoubleArray() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new double?[][] // sequence of enumerations { @@ -313,18 +314,18 @@ public void StatisticsMaxNullableDoubleArray() new double?[] { null, 1.0, null, double.NaN, null, -1.0, null }, new double?[] { null, double.NaN, null, 2.0, null, -1.0, null }, new double?[] { null, double.NegativeInfinity, null, 2.0, null, -1.0, null }, - new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.0, null } + new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.0, null }, }, new double?[] { null, null, 1.0, 2.0, double.NaN, double.NaN, 2.0, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMaxFloat() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new float[] { }, // empty sequence new float[] { } // expected output @@ -352,15 +353,15 @@ public void StatisticsMaxFloat() ( new[] { float.NegativeInfinity, -1.0f, float.PositiveInfinity, 1.0f, float.NaN }, // sequence with +/- infinity new[] { float.NegativeInfinity, -1.0f, float.PositiveInfinity, float.PositiveInfinity, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsMaxFloatArray() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new float[][] // sequence of enumerations { @@ -369,11 +370,10 @@ public void StatisticsMaxFloatArray() new[] { 1.0f, float.NaN, -1.0f }, new[] { float.NaN, 2.0f, -1.0f }, new[] { float.NegativeInfinity, 2.0f, -1.0f }, - new[] { float.PositiveInfinity, 2.0f, -1.0f } + new[] { float.PositiveInfinity, 2.0f, -1.0f }, }, new[] { 1.0f, 2.0f, float.NaN, float.NaN, 2.0f, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] @@ -384,12 +384,12 @@ public void StatisticsMaxEmptyFloatArray() try { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new float[][] { new float[] { } }, // sequence containing an empty enumeration new float[] { } // no expected output - should throw - ) - ); + )); } catch (Exception e) { @@ -405,7 +405,8 @@ public void StatisticsMaxEmptyFloatArray() [Timeout(60000)] public void StatisticsMaxNullableFloatArray() { - this.RunTest(Operators.Max, + this.RunTest( + Operators.Max, ( new float?[][] // sequence of enumerations { @@ -416,18 +417,18 @@ public void StatisticsMaxNullableFloatArray() new float?[] { null, 1.0f, null, float.NaN, null, -1.0f, null }, new float?[] { null, float.NaN, null, 2.0f, null, -1.0f, null }, new float?[] { null, float.NegativeInfinity, null, 2.0f, null, -1.0f, null }, - new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.0f, null } + new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.0f, null }, }, new float?[] { null, null, 1.0f, 2.0f, float.NaN, float.NaN, 2.0f, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumDouble() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new double[] { }, // empty sequence new double[] { } // expected output @@ -455,15 +456,15 @@ public void StatisticsSumDouble() ( new[] { double.NegativeInfinity, -1.0, double.PositiveInfinity, 1.0, double.NaN }, // sequence with +/- infinity new[] { double.NegativeInfinity, double.NegativeInfinity, double.NaN, double.NaN, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumDoubleArray() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new double[][] // sequence of enumerations { @@ -472,18 +473,18 @@ public void StatisticsSumDoubleArray() new[] { 1.0, double.NaN, -1.5, 3.0 }, new[] { double.NaN, 2.0, -1.5, 3.0 }, new[] { double.NegativeInfinity, 2.0, -1.5, 3.0 }, - new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 } + new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 }, }, new[] { 0, 4.5, double.NaN, double.NaN, double.NegativeInfinity, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumNullableDoubleArray() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new double?[][] // sequence of enumerations { @@ -493,18 +494,18 @@ public void StatisticsSumNullableDoubleArray() new double?[] { null, 1.0, null, double.NaN, null, -1.5, null, 3.0, null }, new double?[] { null, double.NaN, null, 2.0, null, -1.5, null, 3.0, null }, new double?[] { null, double.NegativeInfinity, null, 2.0, null, -1.5, null, 3.0, null }, - new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.5, null, 3.0, null } + new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.5, null, 3.0, null }, }, new double?[] { 0, 0, 4.5, double.NaN, double.NaN, double.NegativeInfinity, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumFloat() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new float[] { }, // empty sequence new float[] { } // expected output @@ -532,15 +533,15 @@ public void StatisticsSumFloat() ( new[] { float.NegativeInfinity, -1.0f, float.PositiveInfinity, 1.0f, float.NaN }, // sequence with +/- infinity new[] { float.NegativeInfinity, float.NegativeInfinity, float.NaN, float.NaN, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumFloatArray() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new float[][] // sequence of enumerations { @@ -549,18 +550,18 @@ public void StatisticsSumFloatArray() new[] { 1.0f, float.NaN, -1.5f, 3.0f }, new[] { float.NaN, 2.0f, -1.5f, 3.0f }, new[] { float.NegativeInfinity, 2.0f, -1.5f, 3.0f }, - new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f } + new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f }, }, new[] { 0f, 4.5f, float.NaN, float.NaN, float.NegativeInfinity, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsSumNullableFloatArray() { - this.RunTest(Operators.Sum, + this.RunTest( + Operators.Sum, ( new float?[][] // sequence of enumerations { @@ -570,18 +571,18 @@ public void StatisticsSumNullableFloatArray() new float?[] { null, 1.0f, null, float.NaN, null, -1.5f, null, 3.0f, null }, new float?[] { null, float.NaN, null, 2.0f, null, -1.5f, null, 3.0f, null }, new float?[] { null, float.NegativeInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null }, - new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null } + new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null }, }, new float?[] { 0f, 0f, 4.5f, float.NaN, float.NaN, float.NegativeInfinity, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageDouble() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new double[] { }, // empty sequence new double[] { } // expected output @@ -609,15 +610,15 @@ public void StatisticsAverageDouble() ( new[] { double.NegativeInfinity, -1.0, double.PositiveInfinity, 1.0, double.NaN }, // sequence with +/- infinity new[] { double.NegativeInfinity, double.NegativeInfinity, double.NaN, double.NaN, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageDoubleArray() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new double[][] // sequence of enumerations { @@ -626,18 +627,18 @@ public void StatisticsAverageDoubleArray() new[] { 1.0, double.NaN, -1.5, 3.0 }, new[] { double.NaN, 2.0, -1.5, 3.0 }, new[] { double.NegativeInfinity, 2.0, -1.5, 3.0 }, - new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 } + new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 }, }, new[] { 1.0, 1.125, double.NaN, double.NaN, double.NegativeInfinity, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageNullableDoubleArray() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new double?[][] // sequence of enumerations { @@ -648,18 +649,18 @@ public void StatisticsAverageNullableDoubleArray() new double?[] { null, 1.0, null, double.NaN, null, -1.5, null, 3.0, null }, new double?[] { null, double.NaN, null, 2.0, null, -1.5, null, 3.0, null }, new double?[] { null, double.NegativeInfinity, null, 2.0, null, -1.5, null, 3.0, null }, - new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.5, null, 3.0, null } + new double?[] { null, double.PositiveInfinity, null, 2.0, null, -1.5, null, 3.0, null }, }, new double?[] { null, null, 1.0, 1.125, double.NaN, double.NaN, double.NegativeInfinity, double.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageFloat() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new float[] { }, // empty sequence new float[] { } // expected output @@ -687,15 +688,15 @@ public void StatisticsAverageFloat() ( new[] { float.NegativeInfinity, -1.0f, float.PositiveInfinity, 1.0f, float.NaN }, // sequence with +/- infinity new[] { float.NegativeInfinity, float.NegativeInfinity, float.NaN, float.NaN, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageFloatArray() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new float[][] // sequence of enumerations { @@ -704,18 +705,18 @@ public void StatisticsAverageFloatArray() new[] { 1.0f, float.NaN, -1.5f, 3.0f }, new[] { float.NaN, 2.0f, -1.5f, 3.0f }, new[] { float.NegativeInfinity, 2.0f, -1.5f, 3.0f }, - new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f } + new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f }, }, new[] { 1.0f, 1.125f, float.NaN, float.NaN, float.NegativeInfinity, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsAverageNullableFloatArray() { - this.RunTest(Operators.Average, + this.RunTest( + Operators.Average, ( new float?[][] // sequence of enumerations { @@ -726,18 +727,18 @@ public void StatisticsAverageNullableFloatArray() new float?[] { null, 1.0f, null, float.NaN, null, -1.5f, null, 3.0f, null }, new float?[] { null, float.NaN, null, 2.0f, null, -1.5f, null, 3.0f, null }, new float?[] { null, float.NegativeInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null }, - new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null } + new float?[] { null, float.PositiveInfinity, null, 2.0f, null, -1.5f, null, 3.0f, null }, }, new float?[] { null, null, 1.0f, 1.125f, float.NaN, float.NaN, float.NegativeInfinity, float.PositiveInfinity } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdDouble() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new double[] { }, // empty sequence new double[] { } // expected output @@ -765,15 +766,15 @@ public void StatisticsStdDouble() ( new[] { double.NegativeInfinity, -1.0, double.PositiveInfinity, 1.0, double.NaN }, // sequence with +/- infinity new[] { double.NaN, double.NaN, double.NaN, double.NaN, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdDoubleArray() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new double[][] // sequence of enumerations { @@ -784,18 +785,18 @@ public void StatisticsStdDoubleArray() new[] { 1.0, double.NaN, -1.5, 3.0 }, new[] { double.NaN, 2.0, -1.5, 3.0 }, new[] { double.NegativeInfinity, 2.0, -1.5, 3.0 }, - new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 } + new[] { double.PositiveInfinity, 2.0, -1.5, 3.0 }, }, new[] { 0, 0, 0.70710678118654757, 1.9311050377094112, double.NaN, double.NaN, double.NaN, double.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdFloat() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new float[] { }, // empty sequence new float[] { } // expected output @@ -823,15 +824,15 @@ public void StatisticsStdFloat() ( new[] { float.NegativeInfinity, -1.0f, float.PositiveInfinity, 1.0f, float.NaN }, // sequence with +/- infinity new[] { float.NaN, float.NaN, float.NaN, float.NaN, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdFloatArray() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new float[][] // sequence of enumerations { @@ -842,18 +843,18 @@ public void StatisticsStdFloatArray() new[] { 1.0f, float.NaN, -1.5f, 3.0f }, new[] { float.NaN, 2.0f, -1.5f, 3.0f }, new[] { float.NegativeInfinity, 2.0f, -1.5f, 3.0f }, - new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f } + new[] { float.PositiveInfinity, 2.0f, -1.5f, 3.0f }, }, new[] { 0f, 0f, 0.707106769f, 1.931105f, float.NaN, float.NaN, float.NaN, float.NaN } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdDecimal() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new decimal[] { }, // empty sequence new decimal[] { } // expected output @@ -861,15 +862,15 @@ public void StatisticsStdDecimal() ( new[] { -1.0m, -2.0m, -3.0m, 0.0m, 1.0m, 2.0m }, // real numbers only new[] { 0.0m, 0.707106781186548m, 1m, 1.29099444873581m, 1.58113883008419m, 1.87082869338697m } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdDecimalArray() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new decimal[][] // sequence of enumerations { @@ -879,15 +880,15 @@ public void StatisticsStdDecimalArray() new[] { 1.0m, 2.0m, -1.5m, 3.0m }, }, new[] { 0m, 0m, 0.707106781186548m, 1.93110503770941m } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdInt() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new int[] { }, // empty sequence new double[] { } // expected output @@ -895,15 +896,15 @@ public void StatisticsStdInt() ( new[] { -1, -2, -3, 0, 1, 2 }, new[] { double.NaN, 0.70710678118654757, 1, 1.2909944487358056, 1.5811388300841898, 1.8708286933869707 } // expected output - ) - ); + )); } [TestMethod] [Timeout(60000)] public void StatisticsStdIntArray() { - this.RunTest(Operators.Std, + this.RunTest( + Operators.Std, ( new int[][] // sequence of enumerations { @@ -913,8 +914,7 @@ public void StatisticsStdIntArray() new[] { 1, 2, -1, 3 }, }, new[] { 0, 0, 0.70710678118654757, 1.707825127659933 } // expected output - ) - ); + )); } /// diff --git a/Sources/Runtime/Test.Psi/Test.Psi.csproj b/Sources/Runtime/Test.Psi/Test.Psi.csproj index 31856d657..c802d7da6 100644 --- a/Sources/Runtime/Test.Psi/Test.Psi.csproj +++ b/Sources/Runtime/Test.Psi/Test.Psi.csproj @@ -2,7 +2,7 @@ Exe - netcoreapp2.0 + netcoreapp3.1 @@ -22,17 +22,11 @@ - true - true - Test.Psi - Microsoft Corporation - Microsoft Corporation - - Test.Psi - Copyright (c) Microsoft Corporation. All rights reserved. - false - Test.Psi.ConsoleMain - false + true + true + Test.Psi + Test.Psi + Test.Psi.ConsoleMain @@ -44,9 +38,17 @@ - - - + + all + runtime; build; native; contentfiles; analyzers + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + diff --git a/Sources/Runtime/Test.Psi/TimeTester.cs b/Sources/Runtime/Test.Psi/TimeTester.cs index b46a95b4f..ff66ef2a8 100644 --- a/Sources/Runtime/Test.Psi/TimeTester.cs +++ b/Sources/Runtime/Test.Psi/TimeTester.cs @@ -18,7 +18,7 @@ public void Time_GetTimeFromElapsedTicks() // The clock sync precision in ticks long syncPrecision = 10; - // Get QPC frequency and compute a sync preceision in QPC cycles + // Get QPC frequency and compute a sync precision in QPC cycles long qpcFrequency = Platform.Specific.TimeFrequency(); double qpcToHns = 10000000.0 / qpcFrequency; long qpcSyncPrecision = (long)(syncPrecision / qpcToHns); @@ -209,7 +209,7 @@ public void Time_TickCalibrationCapacity() long ft = cal.ConvertToFileTime(ticks); Assert.AreEqual(ft, cal.ConvertToFileTime(ticks)); - // Add more calibration data until the capacityis reached. After initialization, + // Add more calibration data until the capacity is reached. After initialization, // converter will already have one calibration entry, so these add to it. Note // the different adjustment factor (2:1) that will allow us to distinguish this // from the previous calibration data. diff --git a/Sources/Runtime/Test.Psi/TypeResolutionTests.cs b/Sources/Runtime/Test.Psi/TypeResolutionTests.cs new file mode 100644 index 000000000..63527b033 --- /dev/null +++ b/Sources/Runtime/Test.Psi/TypeResolutionTests.cs @@ -0,0 +1,77 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Test.Psi +{ + using System; + using System.Collections.Generic; + using System.Threading; + using Microsoft.Psi; + using Microsoft.VisualStudio.TestTools.UnitTesting; + + [TestClass] + public class TypeResolutionTests + { + [TestMethod] + [Timeout(60000)] + public void TypeNameTest() + { + // primitive type + string typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(int).AssemblyQualifiedName); + Assert.AreEqual(typeof(int).FullName, typeName); + Assert.AreEqual(typeof(int), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(int[]).AssemblyQualifiedName); + Assert.AreEqual(typeof(int[]).FullName, typeName); + Assert.AreEqual(typeof(int[]), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(int[,]).AssemblyQualifiedName); + Assert.AreEqual(typeof(int[,]).FullName, typeName); + Assert.AreEqual(typeof(int[,]), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(string).AssemblyQualifiedName); + Assert.AreEqual(typeof(string).FullName, typeName); + Assert.AreEqual(typeof(string), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(object).AssemblyQualifiedName); + Assert.AreEqual(typeof(object).FullName, typeName); + Assert.AreEqual(typeof(object), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(byte*).AssemblyQualifiedName); + Assert.AreEqual(typeof(byte*).FullName, typeName); + Assert.AreEqual(typeof(byte*), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(List<>).AssemblyQualifiedName); + Assert.AreEqual(typeof(List<>).FullName, typeName); + Assert.AreEqual(typeof(List<>), Type.GetType(typeName)); + + // Note - Type.FullName does not remove the AQN from the inner type parameters of generic + // types, so we won't test the result for equality with typeof(List).FullName + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(List).AssemblyQualifiedName); + Assert.AreEqual(typeof(List), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(IEnumerable).AssemblyQualifiedName); + Assert.AreEqual(typeof(IEnumerable), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(IDictionary>).AssemblyQualifiedName); + Assert.AreEqual(typeof(IDictionary>), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(Func).AssemblyQualifiedName); + Assert.AreEqual(typeof(Func), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName(typeof(NestedClass>).AssemblyQualifiedName); + Assert.AreEqual(typeof(NestedClass>), Type.GetType(typeName)); + + typeName = TypeResolutionHelper.RemoveAssemblyName("Namespace.TypeName, AssemblyName WithSpaces-v1.0.0.0, Version=1.0.0.0"); + Assert.AreEqual("Namespace.TypeName", typeName); + + typeName = TypeResolutionHelper.RemoveAssemblyName("Namespace.TypeName`2[[Nested.TypeName1, AssemblyName WithSpaces-v1.0.0.0, Version=1.0.0.0], [Nested.TypeName2[], AssemblyName, Culture=neutral]], AssemblyName, PublicKeyToken=null"); + Assert.AreEqual("Namespace.TypeName`2[[Nested.TypeName1], [Nested.TypeName2[]]]", typeName); + } + + // empty class for type name testing + private class NestedClass + { + } + } +} diff --git a/Sources/Runtime/Test.Psi/VectorTests.cs b/Sources/Runtime/Test.Psi/VectorTests.cs index 0356272bd..789659bb2 100644 --- a/Sources/Runtime/Test.Psi/VectorTests.cs +++ b/Sources/Runtime/Test.Psi/VectorTests.cs @@ -4,10 +4,10 @@ namespace Test.Psi { using System; - using System.Threading; using System.Collections.Generic; using System.Linq; using System.Text; + using System.Threading; using Microsoft.Psi; using Microsoft.Psi.Components; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -103,7 +103,6 @@ public void VariableLengthVectorStatelessParallel() CollectionAssert.AreEqual(new int[] { 0, 10, 30, 60, 100 }, results.ToArray()); } - [TestMethod] [Timeout(60000)] public void SparseVectorStatelessParallel() @@ -131,7 +130,8 @@ public void SparseVectorStatelessParallel() [Timeout(60000)] public void SparseVectorWithGapParallel() { - var sequence = new List>() { + var sequence = new List>() + { new Dictionary { { 1, 100 } }, new Dictionary { { 1, 100 } }, new Dictionary { { 1, 100 } }, @@ -181,7 +181,8 @@ public void SparseVectorWithGapParallel() [Timeout(60000)] public void SparseVectorBranchTerminationPolicy() { - var sequence = new List>() { + var sequence = new List>() + { new Dictionary { { 1, 100 } }, new Dictionary { { 1, 100 } }, new Dictionary { { 1, 100 } }, @@ -311,7 +312,7 @@ public void SparseVectorBufferCompletionTest() new Dictionary { { 2, 26 } }, new Dictionary { { 2, 27 } }, new Dictionary { { 2, 28 } }, - new Dictionary { { 2, 29 } } + new Dictionary { { 2, 29 } }, }; var results = new List(); @@ -340,8 +341,10 @@ public void SparseVectorBufferCompletionTest() { sb.Append($"{v},"); } + sb.Remove(sb.Length - 1, 1); } + results.Add(sb.ToString().Substring(1)); }); p.Run(); @@ -484,7 +487,7 @@ public void SparseVectorWithGammaCreatingHolesAndOutputDefaultIfDropped() p.RunAsync(); - void step(int expected) + void Step(int expected) { stepper.Step(); while (results.Count != expected) @@ -497,34 +500,34 @@ void step(int expected) Assert.AreEqual(0, results.Count); - step(1); // A C + Step(1); // A C CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 3, 'C' } }, results[0]); - step(2); // A C E + Step(2); // A C E CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 3, 'C' }, { 5, 'E' } }, results[1]); - step(2); // - B - D E (holes) no new results + Step(2); // - B - D E (holes) no new results - step(3); // A B C D - (hole) stream 5/E closed + Step(3); // A B C D - (hole) stream 5/E closed CollectionAssert.AreEquivalent(new Dictionary { { 1, '\0' }, { 2, 'B' }, { 3, '\0' }, { 4, 'D' }, { 5, 'E' } }, results[2]); // note default '\0' values - step(4); // A - C - + Step(4); // A - C - CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' }, { 3, 'C' }, { 4, 'D' }, { 5, '\0' } }, results[3]); // note default '\0' value - step(6); // A B C D E stream 5/E "reopens" two outputs + Step(6); // A B C D E stream 5/E "reopens" two outputs CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, '\0' }, { 3, 'C' }, { 4, '\0' } }, results[4]); // note default '\0' values CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' }, { 3, 'C' }, { 4, 'D' }, { 5, 'E' } }, results[5]); - step(7); // A B C D E + Step(7); // A B C D E CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' }, { 3, 'C' }, { 4, 'D' }, { 5, 'E' } }, results[6]); - step(8); // A B C streams 4/D and 5/E closed + Step(8); // A B C streams 4/D and 5/E closed CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' }, { 3, 'C' } }, results[7]); - step(9); // A B stream 3/C closed + Step(9); // A B stream 3/C closed CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' } }, results[8]); - step(10); // A B streams 1/A and 2/B now closed + Step(10); // A B streams 1/A and 2/B now closed CollectionAssert.AreEquivalent(new Dictionary { { 1, 'A' }, { 2, 'B' } }, results[9]); Assert.IsFalse(stepper.Step()); // we're done @@ -549,6 +552,8 @@ public ManualSteppingGenerator(Pipeline pipeline, IEnumerable sequence) this.Out = pipeline.CreateEmitter(this, "Out"); } + public Emitter Out { get; private set; } + public bool Step() { if (this.index < this.sequence.Count) @@ -571,8 +576,6 @@ public void Stop(DateTime finalOriginatingTime, Action notifyCompleted) { notifyCompleted(); } - - public Emitter Out { get; private set; } } } } diff --git a/Sources/Speech/Microsoft.Psi.Speech.Windows/Microsoft.Psi.Speech.Windows.csproj b/Sources/Speech/Microsoft.Psi.Speech.Windows/Microsoft.Psi.Speech.Windows.csproj index f09b902f1..31e42b2e4 100644 --- a/Sources/Speech/Microsoft.Psi.Speech.Windows/Microsoft.Psi.Speech.Windows.csproj +++ b/Sources/Speech/Microsoft.Psi.Speech.Windows/Microsoft.Psi.Speech.Windows.csproj @@ -24,6 +24,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechIntentDetector.cs b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechIntentDetector.cs index 1663a00a6..7febd178f 100644 --- a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechIntentDetector.cs +++ b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechIntentDetector.cs @@ -99,6 +99,7 @@ public void Dispose() { // Unregister handlers so they won't fire while disposing. this.speechRecognitionEngine.LoadGrammarCompleted -= this.OnLoadGrammarCompleted; + this.speechRecognitionEngine.Dispose(); } } diff --git a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizer.cs b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizer.cs index ad02c90f5..ec64bd3a8 100644 --- a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizer.cs +++ b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizer.cs @@ -194,7 +194,7 @@ private enum EmitterGroup public Emitter AudioLevelUpdated { get; } /// - /// Gets the output stream of emulate recognize completed completed events. + /// Gets the output stream of emulate recognize completed events. /// public Emitter EmulateRecognizeCompleted { get; } @@ -242,6 +242,7 @@ public void Dispose() // Free any other managed objects here. this.recognizeComplete.Dispose(); this.recognizeComplete = null; + this.inputAudioStream.Dispose(); } /// diff --git a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizerConfiguration.cs b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizerConfiguration.cs index a40a89489..3bdf1267b 100644 --- a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizerConfiguration.cs +++ b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechRecognizerConfiguration.cs @@ -20,12 +20,6 @@ public sealed class SystemSpeechRecognizerConfiguration /// public SystemSpeechRecognizerConfiguration() { - this.Language = "en-us"; - this.Grammars = null; - this.BufferLengthInMs = 1000; - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.InputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// @@ -36,7 +30,7 @@ public SystemSpeechRecognizerConfiguration() /// (U.S. English). Other supported locales include "en-gb", "de-de", "es-es", "fr-fr", "ja-jp", /// "zh-cn" and "zh-tw". /// - public string Language { get; set; } + public string Language { get; set; } = "en-us"; /// /// Gets or sets the list of grammar files. @@ -49,7 +43,7 @@ public SystemSpeechRecognizerConfiguration() /// context-free grammar used for free text dictation. /// [XmlArrayItem("Grammar")] - public GrammarInfo[] Grammars { get; set; } + public GrammarInfo[] Grammars { get; set; } = null; /// /// Gets or sets the length of the recognizer's input stream buffer in milliseconds. @@ -61,31 +55,31 @@ public SystemSpeechRecognizerConfiguration() /// on the length of audio to buffer in milliseconds and the audio input format. By default, a 1000 ms /// buffer is used. It is safe to leave this value unchanged. /// - public int BufferLengthInMs { get; set; } + public int BufferLengthInMs { get; set; } = 1000; /// /// Gets or sets the number of milliseconds during which the internal speech detection /// engine accepts input containing only silence before making a state transition. /// - public int InitialSilenceTimeoutMs { get; set; } + public int InitialSilenceTimeoutMs { get; set; } = 0; /// /// Gets or sets the number of milliseconds during which the internal speech detection /// engine accepts input containing only background noise before making a state transition. /// - public int BabbleTimeoutMs { get; set; } + public int BabbleTimeoutMs { get; set; } = 0; /// /// Gets or sets the number of milliseconds of silence that the internal speech detection /// engine will accept at the end of unambiguous input before making a state transition. /// - public int EndSilenceTimeoutMs { get; set; } + public int EndSilenceTimeoutMs { get; set; } = 150; /// /// Gets or sets the number of milliseconds of silence that the internal speech detection /// engine will accept at the end of ambiguous input before making a state transition. /// - public int EndSilenceTimeoutAmbiguousMs { get; set; } + public int EndSilenceTimeoutAmbiguousMs { get; set; } = 500; /// /// Gets or sets the expected input format of the audio stream. @@ -96,6 +90,6 @@ public SystemSpeechRecognizerConfiguration() /// static methods to create the appropriate /// object. If not specified, a default value of 16000 Hz is assumed. /// - public WaveFormat InputFormat { get; set; } + public WaveFormat InputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechSynthesizerConfiguration.cs b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechSynthesizerConfiguration.cs index 9bbe4802c..87a5b8609 100644 --- a/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechSynthesizerConfiguration.cs +++ b/Sources/Speech/Microsoft.Psi.Speech.Windows/SystemSpeechSynthesizerConfiguration.cs @@ -19,60 +19,50 @@ public sealed class SystemSpeechSynthesizerConfiguration /// public SystemSpeechSynthesizerConfiguration() { - this.Voice = "Microsoft Zira Desktop"; - this.PersistAudio = false; - this.UseDefaultAudioPlaybackDevice = false; - this.BufferLengthInMs = 1000; - this.ProsodyRate = 1.0; - this.ProsodyPitch = "default"; - this.ProsodyVolume = "default"; - - // Defaults to 16 kHz, 16-bit, 1-channel PCM samples - this.OutputFormat = WaveFormat.Create16kHz1Channel16BitPcm(); } /// /// Gets or sets the text-to-speech voice to use. /// - public string Voice { get; set; } + public string Voice { get; set; } = "Microsoft Zira Desktop"; /// /// Gets or sets a value indicating whether the output audio stream is persisted. /// - public bool PersistAudio { get; set; } + public bool PersistAudio { get; set; } = false; /// /// Gets or sets a value indicating whether synthesized speech audio /// output should be redirected to the default audio device instead of /// the output stream. Useful for debugging purposes. /// - public bool UseDefaultAudioPlaybackDevice { get; set; } + public bool UseDefaultAudioPlaybackDevice { get; set; } = false; /// /// Gets or sets the length of the synthesizer's output audio buffer in milliseconds. /// - public int BufferLengthInMs { get; set; } + public int BufferLengthInMs { get; set; } = 1000; /// /// Gets or sets the prosody rate for the speech. /// - public double ProsodyRate { get; set; } + public double ProsodyRate { get; set; } = 1.0; /// /// Gets or sets the prosody pitch for the speech. Possible values: x-low, low, medium, high, x-high, or default /// Todo: make this an enum. /// - public string ProsodyPitch { get; set; } + public string ProsodyPitch { get; set; } = "default"; /// /// Gets or sets the prosody volume for the speech. Possible values: silent, x-soft, soft, medium, loud, x-loud, or default /// Todo: make this an enum. /// - public string ProsodyVolume { get; set; } + public string ProsodyVolume { get; set; } = "default"; /// /// Gets or sets the output format of the audio stream. /// - public WaveFormat OutputFormat { get; set; } + public WaveFormat OutputFormat { get; set; } = WaveFormat.Create16kHz1Channel16BitPcm(); } } diff --git a/Sources/Speech/Microsoft.Psi.Speech/GrammarInfo.cs b/Sources/Speech/Microsoft.Psi.Speech/GrammarInfo.cs index 94e4679dc..b059b718a 100644 --- a/Sources/Speech/Microsoft.Psi.Speech/GrammarInfo.cs +++ b/Sources/Speech/Microsoft.Psi.Speech/GrammarInfo.cs @@ -9,7 +9,7 @@ namespace Microsoft.Psi.Speech /// Represents information about a grammar. /// /// - /// This information may be used to define a set of files containing grammar definitions to be comsumed by a speech recognition component. + /// This information may be used to define a set of files containing grammar definitions to be consumed by a speech recognition component. /// public class GrammarInfo { diff --git a/Sources/Speech/Microsoft.Psi.Speech/Microsoft.Psi.Speech.csproj b/Sources/Speech/Microsoft.Psi.Speech/Microsoft.Psi.Speech.csproj index ccf6ae028..8b96f796c 100644 --- a/Sources/Speech/Microsoft.Psi.Speech/Microsoft.Psi.Speech.csproj +++ b/Sources/Speech/Microsoft.Psi.Speech/Microsoft.Psi.Speech.csproj @@ -33,6 +33,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Speech/Microsoft.Psi.Speech/SimpleVoiceActivityDetector.cs b/Sources/Speech/Microsoft.Psi.Speech/SimpleVoiceActivityDetector.cs index 7f2993007..a00fcf354 100644 --- a/Sources/Speech/Microsoft.Psi.Speech/SimpleVoiceActivityDetector.cs +++ b/Sources/Speech/Microsoft.Psi.Speech/SimpleVoiceActivityDetector.cs @@ -61,7 +61,7 @@ public SimpleVoiceActivityDetector(Pipeline pipeline, SimpleVoiceActivityDetecto var voiceActivityDetected = logEnergyThreshold.Window(0, voiceActivityDetectionFrames - 1).Average(); var silenceDetected = logEnergyThreshold.Window(-(silenceDetectionFrames - 1), 0).Average(); - // Use Aggregate opertator to update the state (isSpeaking) based on the current state. + // Use Aggregate operator to update the state (isSpeaking) based on the current state. var vad = voiceActivityDetected.Join(silenceDetected).Aggregate( false, (isSpeaking, v) => isSpeaking ? v.Item2 != 0 : v.Item1 == 1.0); diff --git a/Sources/Speech/Test.Psi.Speech.Windows/SystemSpeechRecognizerTests.cs b/Sources/Speech/Test.Psi.Speech.Windows/SystemSpeechRecognizerTests.cs index 45b9eb1ef..ab0c6acc0 100644 --- a/Sources/Speech/Test.Psi.Speech.Windows/SystemSpeechRecognizerTests.cs +++ b/Sources/Speech/Test.Psi.Speech.Windows/SystemSpeechRecognizerTests.cs @@ -80,7 +80,7 @@ private void RecognizeSpeechFromWaveFile(string filename, string expectedText, s return; } - // Read the WaveFormat from the file header so we can set the recognizer configuaration. + // Read the WaveFormat from the file header so we can set the recognizer configuration. WaveFormat format = WaveFileHelper.ReadWaveFileHeader(filename); // Initialize components and wire up pipeline. @@ -98,7 +98,7 @@ private void RecognizeSpeechFromWaveFile(string filename, string expectedText, s } // Add results from outputs. Note that we need to call DeepClone on each result as we - // do not want them to be resused by the runtime. + // do not want them to be reused by the runtime. var results = new List(); recognizer.Out.Do(r => results.Add(r.DeepClone())); recognizer.PartialRecognitionResults.Do(r => results.Add(r.DeepClone())); diff --git a/Sources/Speech/Test.Psi.Speech.Windows/Test.Psi.Speech.Windows.csproj b/Sources/Speech/Test.Psi.Speech.Windows/Test.Psi.Speech.Windows.csproj index 40249ce73..23ae5f434 100644 --- a/Sources/Speech/Test.Psi.Speech.Windows/Test.Psi.Speech.Windows.csproj +++ b/Sources/Speech/Test.Psi.Speech.Windows/Test.Psi.Speech.Windows.csproj @@ -1,28 +1,27 @@ - - - net472 - - Exe - Test.Psi.Speech.Windows.ConsoleMain - false - false - + + + net472 + + Exe + Test.Psi.Speech.Windows.ConsoleMain + ../../../Build/Test.Psi.ruleset + true AnyCPU - + true AnyCPU - + - + - + @@ -30,29 +29,37 @@ - - - - PreserveNewest - - - PreserveNewest - - + + + + PreserveNewest + + + PreserveNewest + + - + - - - + + all + runtime; build; native; contentfiles; analyzers + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + - - - - - - - + + + + + + + \ No newline at end of file diff --git a/Sources/Speech/Test.Psi.Speech/Test.Psi.Speech.csproj b/Sources/Speech/Test.Psi.Speech/Test.Psi.Speech.csproj index 172deb932..a07cf1889 100644 --- a/Sources/Speech/Test.Psi.Speech/Test.Psi.Speech.csproj +++ b/Sources/Speech/Test.Psi.Speech/Test.Psi.Speech.csproj @@ -1,7 +1,7 @@  - netcoreapp2.0 + netcoreapp3.1 false @@ -35,10 +35,14 @@ - + + all + runtime; build; native; contentfiles; analyzers + + - - + + diff --git a/Sources/Toolkits/FiniteStateMachine/Microsoft.Psi.FiniteStateMachine/Microsoft.Psi.FiniteStateMachine.csproj b/Sources/Toolkits/FiniteStateMachine/Microsoft.Psi.FiniteStateMachine/Microsoft.Psi.FiniteStateMachine.csproj index 5d9134222..c9da3efd8 100644 --- a/Sources/Toolkits/FiniteStateMachine/Microsoft.Psi.FiniteStateMachine/Microsoft.Psi.FiniteStateMachine.csproj +++ b/Sources/Toolkits/FiniteStateMachine/Microsoft.Psi.FiniteStateMachine/Microsoft.Psi.FiniteStateMachine.csproj @@ -28,6 +28,10 @@ + + all + runtime; build; native; contentfiles; analyzers + diff --git a/Sources/Tools/PsiStoreTool/Program.cs b/Sources/Tools/PsiStoreTool/Program.cs index 342ce61dd..9dd5772bb 100644 --- a/Sources/Tools/PsiStoreTool/Program.cs +++ b/Sources/Tools/PsiStoreTool/Program.cs @@ -38,10 +38,11 @@ private static int Main(string[] args) Console.WriteLine($"Platform for Situated Intelligence Store Tool"); try { - return Parser.Default.ParseArguments(args) + return Parser.Default.ParseArguments(args) .MapResult( - (Verbs.ListStreams opts) => Utility.ListStreams(opts.Store, opts.Path), + (Verbs.ListStreams opts) => Utility.ListStreams(opts.Store, opts.Path, opts.ShowSize), (Verbs.Info opts) => Utility.DisplayStreamInfo(opts.Stream, opts.Store, opts.Path), + (Verbs.RemoveStream opts) => Utility.RemoveStream(opts.Stream, opts.Store, opts.Path), (Verbs.Messages opts) => Utility.DisplayStreamMessages(opts.Stream, opts.Store, opts.Path, opts.Number), (Verbs.Save opts) => Utility.SaveStreamMessages(opts.Stream, opts.Store, opts.Path, opts.File, opts.Format), (Verbs.Send opts) => Utility.SendStreamMessages(opts.Stream, opts.Store, opts.Path, opts.Topic, opts.Address, opts.Format), diff --git a/Sources/Tools/PsiStoreTool/PsiStoreTool.csproj b/Sources/Tools/PsiStoreTool/PsiStoreTool.csproj index 6facd7936..ccd3e11c3 100644 --- a/Sources/Tools/PsiStoreTool/PsiStoreTool.csproj +++ b/Sources/Tools/PsiStoreTool/PsiStoreTool.csproj @@ -2,19 +2,19 @@ Exe - netcoreapp2.0 + netcoreapp3.1 ../../../Build/Microsoft.Psi.ruleset false - bin\Debug\netcoreapp2.0\PsiStoreTool.xml + bin\Debug\netcoreapp3.1\PsiStoreTool.xml true - bin\Release\netcoreapp2.0\PsiStoreTool.xml + bin\Release\netcoreapp3.1\PsiStoreTool.xml true @@ -29,6 +29,10 @@ + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers diff --git a/Sources/Tools/PsiStoreTool/Readme.md b/Sources/Tools/PsiStoreTool/Readme.md index a09128b52..96e4b49db 100644 --- a/Sources/Tools/PsiStoreTool/Readme.md +++ b/Sources/Tools/PsiStoreTool/Readme.md @@ -55,7 +55,7 @@ AnotherStream (MyNamespace.MyType) Count: 3394 ``` -This displays the name and .NET type of each stream. +This displays the name and .NET type of each stream. Adding '-s true' option enables listing the size of each stream (the information displayed includes both the average message size and the total size of all messages in the store). To get info about a particular stream: diff --git a/Sources/Tools/PsiStoreTool/Utility.cs b/Sources/Tools/PsiStoreTool/Utility.cs index eba429dfc..5dc7eac61 100644 --- a/Sources/Tools/PsiStoreTool/Utility.cs +++ b/Sources/Tools/PsiStoreTool/Utility.cs @@ -38,23 +38,37 @@ internal static class Utility /// /// Store name. /// Store path. + /// Indicates whether to show the stream size information. /// Success flag. - internal static int ListStreams(string store, string path) + internal static int ListStreams(string store, string path, bool showSize = false) { - Console.WriteLine($"Available Streams (store={store}, path={path})"); + var stringBuilder = new StringBuilder(); + using (var pipeline = Pipeline.Create()) { var data = Store.Open(pipeline, store, Path.GetFullPath(path)); - var count = 0; - foreach (var stream in data.AvailableStreams) + + stringBuilder.AppendLine($"{data.AvailableStreams.Count()} Available Streams (store={store}, path={path})"); + if (showSize) { - Console.WriteLine($"{stream.Name} ({stream.TypeName.Split(',')[0]})"); - count++; + stringBuilder.AppendLine("[Avg. Message Size / Total Size]; * marks indexed streams"); + foreach (var stream in data.AvailableStreams.OrderByDescending(s => (double)s.MessageCount * s.AverageMessageSize)) + { + var isIndexed = stream.IsIndexed ? "* " : " "; + stringBuilder.AppendLine($"{isIndexed}[{(double)stream.AverageMessageSize / 1024:0.00}Kb / {(stream.AverageMessageSize * (double)stream.MessageCount) / (1024 * 1024):0.00}Mb] {stream.Name} ({stream.TypeName.Split(',')[0]})"); + } + } + else + { + foreach (var stream in data.AvailableStreams) + { + stringBuilder.AppendLine($"{stream.Name} ({stream.TypeName.Split(',')[0]})"); + } } - - Console.WriteLine($"Count: {count}"); } + Console.WriteLine(stringBuilder.ToString()); + return 0; } @@ -75,6 +89,7 @@ internal static int DisplayStreamInfo(string stream, string store, string path) Console.WriteLine($"ID: {meta.Id}"); Console.WriteLine($"Name: {meta.Name}"); Console.WriteLine($"TypeName: {meta.TypeName}"); + Console.WriteLine($"SupplementalMetadataTypeName: {meta.SupplementalMetadataTypeName}"); Console.WriteLine($"MessageCount: {meta.MessageCount}"); Console.WriteLine($"AverageFrequency: {meta.AverageFrequency}"); Console.WriteLine($"AverageLatency: {meta.AverageLatency}"); @@ -98,6 +113,52 @@ internal static int DisplayStreamInfo(string stream, string store, string path) return 0; } + /// + /// Removes a stream from the store. + /// + /// Stream name. + /// Store name. + /// Store path. + /// Success flag. + internal static int RemoveStream(string stream, string store, string path) + { + string tempFolderPath = Path.Combine(path, $"Copy-{Guid.NewGuid()}"); + + // copy all streams to the new path, excluding the specified stream by name + Store.Copy((store, path), (store, tempFolderPath), null, s => s.Name != stream, false); + + // create a SafeCopy folder in which to save the original store files + var safeCopyPath = Path.Combine(path, $"Original-{Guid.NewGuid()}"); + Directory.CreateDirectory(safeCopyPath); + + // Move the original store files to the BeforeRepair folder. Do this even if the deleteOldStore + // flag is true, as deleting the original store files immediately may occasionally fail. This can + // happen because the InfiniteFileReader disposes of its MemoryMappedView in a background + // thread, which may still be in progress. If deleteOldStore is true, we will delete the + // BeforeRepair folder at the very end (by which time any open MemoryMappedViews will likely + // have finished disposing). + foreach (var file in Directory.EnumerateFiles(path)) + { + var fileInfo = new FileInfo(file); + File.Move(file, Path.Combine(safeCopyPath, fileInfo.Name)); + } + + // move the repaired store files to the original folder + foreach (var file in Directory.EnumerateFiles(Path.Combine(tempFolderPath))) + { + var fileInfo = new FileInfo(file); + File.Move(file, Path.Combine(path, fileInfo.Name)); + } + + // cleanup temporary folder + Directory.Delete(tempFolderPath, true); + + // delete the old store files + Directory.Delete(safeCopyPath, true); + + return 0; + } + /// /// Print (first n) messages from stream. /// @@ -187,7 +248,7 @@ internal static int ConcatenateStores(string stores, string path, string output) /// Success flag. internal static int CropStore(string store, string path, string output, string start, string length) { - Console.WriteLine($"Concatenating stores (stores={store}, path={path}, output={output}, start={start}, length={length})"); + Console.WriteLine($"Cropping store (store={store}, path={path}, output={output}, start={start}, length={length})"); var startTime = TimeSpan.Zero; // start at beginning if unspecified if (!string.IsNullOrWhiteSpace(start) && !TimeSpan.TryParse(start, CultureInfo.InvariantCulture, out startTime)) @@ -236,7 +297,7 @@ internal static int ListTasks(IEnumerable assemblies) /// Store path. /// Task name. /// Optional assemblies containing task. - /// Addidional configuration arguments. + /// Additional configuration arguments. /// Success flag. internal static int ExecuteTask(string stream, string store, string path, string name, IEnumerable assemblies, IEnumerable args) { @@ -275,7 +336,7 @@ internal static int ExecuteTask(string stream, string store, string path, string { if (importer == null) { - throw new ArgumentException("Error: Task requires a store, but no store argument suplied (-s)."); + throw new ArgumentException("Error: Task requires a store, but no store argument supplied (-s)."); } parameters[i] = importer; @@ -362,7 +423,7 @@ internal static int ExecuteTask(string stream, string store, string path, string { if (importer == null) { - throw new ArgumentException("Error: Task requires a stream within a store, but no store argument suplied (-s)."); + throw new ArgumentException("Error: Task requires a stream within a store, but no store argument supplied (-s)."); } importer.OpenDynamicStream(stream).Do((m, e) => diff --git a/Sources/Tools/PsiStoreTool/Verbs.cs b/Sources/Tools/PsiStoreTool/Verbs.cs index e08f0b5c1..ebe77f413 100644 --- a/Sources/Tools/PsiStoreTool/Verbs.cs +++ b/Sources/Tools/PsiStoreTool/Verbs.cs @@ -59,6 +59,11 @@ internal abstract class BaseTransportStreamCommand : BaseStreamCommand [Verb("list", HelpText = "List streams within a Psi data store.")] internal class ListStreams : BaseStoreCommand { + /// + /// Gets or sets a value indicating whether to show stream size information. + /// + [Option('s', "showsize", Required = false, HelpText = "Shows stream size information.", Default = false)] + public bool ShowSize { get; set; } } /// @@ -69,6 +74,14 @@ internal class Info : BaseStreamCommand { } + /// + /// Remove stream verb. + /// + [Verb("removestream", HelpText = "Removes a stream from a store.")] + internal class RemoveStream : BaseStreamCommand + { + } + /// /// Display messages verb. /// diff --git a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindow.xaml b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindow.xaml index 48aad86d0..d6a9c5196 100644 --- a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindow.xaml +++ b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindow.xaml @@ -15,7 +15,7 @@ xmlns:viewmodels="clr-namespace:Microsoft.Psi.Visualization.ViewModels;assembly=Microsoft.Psi.Visualization.Common.Windows" xmlns:xctk="clr-namespace:Xceed.Wpf.Toolkit.PropertyGrid;assembly=Xceed.Wpf.Toolkit" xmlns:i="clr-namespace:System.Windows.Interactivity;assembly=System.Windows.Interactivity" - Title="Platform for Situated Intelligence Studio" + Title="{Binding TitleText}" WindowState="{Binding AppSettings.WindowState, Mode=TwoWay}" Top="{Binding AppSettings.WindowPositionTop, Mode=TwoWay}" Left="{Binding AppSettings.WindowPositionLeft, Mode=TwoWay}" @@ -26,10 +26,10 @@ - + + + + diff --git a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindowViewModel.cs b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindowViewModel.cs index bcec8b766..f9a947a96 100644 --- a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindowViewModel.cs +++ b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/MainWindowViewModel.cs @@ -17,6 +17,7 @@ namespace Microsoft.Psi.PsiStudio using Microsoft.Psi.Visualization; using Microsoft.Psi.Visualization.Base; using Microsoft.Psi.Visualization.Data; + using Microsoft.Psi.Visualization.Helpers; using Microsoft.Psi.Visualization.Navigation; using Microsoft.Psi.Visualization.ViewModels; using Microsoft.Psi.Visualization.VisualizationObjects; @@ -29,6 +30,8 @@ namespace Microsoft.Psi.PsiStudio /// public class MainWindowViewModel : ObservableObject { + private readonly TimeSpan nudgeTimeSpan = TimeSpan.FromSeconds(1 / 30.0); + private readonly TimeSpan jumpTimeSpan = TimeSpan.FromSeconds(1 / 6.0); private readonly string newLayoutName = ""; private List availableLayouts = new List(); private LayoutInfo currentLayout = null; @@ -56,6 +59,10 @@ public class MainWindowViewModel : ObservableObject private RelayCommand playPauseCommand; private RelayCommand toggleCursorFollowsMouseComand; + private RelayCommand nudgeRightCommand; + private RelayCommand nudgeLeftCommand; + private RelayCommand jumpRightCommand; + private RelayCommand jumpLeftCommand; private RelayCommand openStoreCommand; private RelayCommand openDatasetCommand; private RelayCommand saveDatasetCommand; @@ -100,6 +107,7 @@ public MainWindowViewModel() Application.Current.MainWindow.ContentRendered += this.MainWindow_Activated; // Listen for property change events from the visualization context (specifically when the visualization container changes) + VisualizationContext.Instance.PropertyChanging += this.VisualizationContext_PropertyChanging; VisualizationContext.Instance.PropertyChanged += this.VisualizationContext_PropertyChanged; // Load the available layouts @@ -108,8 +116,8 @@ public MainWindowViewModel() // Set the current layout if it's in the available layouts, otherwise make "new layout" the current layout LayoutInfo lastLayout = this.AvailableLayouts.FirstOrDefault(l => l.Name == this.AppSettings.CurrentLayoutName); this.currentLayout = lastLayout ?? this.AvailableLayouts[0]; - } - + } + /// /// Gets the name of this application for use when constructing paths etc. /// @@ -144,6 +152,31 @@ public object SelectedPropertiesObject set => this.Set(nameof(this.SelectedPropertiesObject), ref this.selectedPropertiesObject, value); } + /// + /// Gets the text to display in the application's titlebar. + /// + public string TitleText + { + get + { + StringBuilder text = new StringBuilder("Platform for Situated Intelligence Studio"); + if (VisualizationContext.Instance.DatasetViewModel != null) + { + text.Append(" - "); + text.Append(VisualizationContext.Instance.DatasetViewModel.Name); + + if (VisualizationContext.Instance.VisualizationContainer.SnapToVisualizationObject != null) + { + text.Append(" [cursor snaps to "); + text.Append(VisualizationContext.Instance.VisualizationContainer.SnapToVisualizationObject.Name); + text.Append(" stream]"); + } + } + + return text.ToString(); + } + } + /// /// Gets the play/pause command. /// @@ -183,6 +216,86 @@ public RelayCommand ToggleCursorFollowsMouseComand } } + /// + /// Gets the nudge cursor right command. + /// + [Browsable(false)] + [IgnoreDataMember] + public RelayCommand NudgeRightCommand + { + get + { + if (this.nudgeRightCommand == null) + { + this.nudgeRightCommand = new RelayCommand( + () => this.MoveCursorBy(this.nudgeTimeSpan, SnappingBehavior.Next), + () => VisualizationContext.Instance.IsDatasetLoaded() && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live); + } + + return this.nudgeRightCommand; + } + } + + /// + /// Gets the nudge cursor left command. + /// + [Browsable(false)] + [IgnoreDataMember] + public RelayCommand NudgeLeftCommand + { + get + { + if (this.nudgeLeftCommand == null) + { + this.nudgeLeftCommand = new RelayCommand( + () => this.MoveCursorBy(-this.nudgeTimeSpan, SnappingBehavior.Previous), + () => VisualizationContext.Instance.IsDatasetLoaded() && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live); + } + + return this.nudgeLeftCommand; + } + } + + /// + /// Gets the jump cursor right command. + /// + [Browsable(false)] + [IgnoreDataMember] + public RelayCommand JumpRightCommand + { + get + { + if (this.jumpRightCommand == null) + { + this.jumpRightCommand = new RelayCommand( + () => this.MoveCursorBy(this.jumpTimeSpan, SnappingBehavior.Next), + () => VisualizationContext.Instance.IsDatasetLoaded() && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live); + } + + return this.jumpRightCommand; + } + } + + /// + /// Gets the jump cursor left command. + /// + [Browsable(false)] + [IgnoreDataMember] + public RelayCommand JumpLeftCommand + { + get + { + if (this.jumpLeftCommand == null) + { + this.jumpLeftCommand = new RelayCommand( + () => this.MoveCursorBy(-this.jumpTimeSpan, SnappingBehavior.Previous), + () => VisualizationContext.Instance.IsDatasetLoaded() && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live); + } + + return this.jumpLeftCommand; + } + } + /// /// Gets the open store command. /// @@ -654,7 +767,7 @@ public RelayCommand SynchronizeTreesCommand } /// - /// Gets the selected visualzation changed command. + /// Gets the selected visualization changed command. /// [Browsable(false)] [IgnoreDataMember] @@ -881,10 +994,13 @@ public LayoutInfo CurrentLayout /// public void OnClosing() { - // Put the current state of the timeing buttons into the settings object + // Put the current state of the timing buttons into the settings object this.AppSettings.ShowAbsoluteTiming = this.VisualizationContainer.Navigator.ShowAbsoluteTiming; this.AppSettings.ShowTimingRelativeToSessionStart = this.VisualizationContainer.Navigator.ShowTimingRelativeToSessionStart; this.AppSettings.ShowTimingRelativeToSelectionStart = this.VisualizationContainer.Navigator.ShowTimingRelativeToSelectionStart; + + // Save the settings + this.AppSettings.Save(); } /*/// @@ -937,6 +1053,21 @@ public void AddAnnotation(Window owner) } }*/ + private void MoveCursorBy(TimeSpan timeSpan, SnappingBehavior snappingBehavior) + { + var visContainer = this.VisualizationContainer; + var nav = visContainer.Navigator; + var time = nav.Cursor + timeSpan; + if (visContainer.SnapToVisualizationObject is IStreamVisualizationObject vo) + { + nav.MoveTo(vo.GetSnappedTime(time, snappingBehavior) ?? time); + } + else + { + nav.MoveTo(time); + } + } + private void OpenCurrentLayout() { // Attempt to open the current layout @@ -1010,7 +1141,7 @@ private void UpdateLayoutList() Directory.CreateDirectory(this.LayoutsDirectory); } - // Find all the layout files and add them to the the list of available layouts + // Find all the layout files and add them to the list of available layouts FileInfo[] files = directoryInfo.GetFiles("*.plo"); foreach (FileInfo fileInfo in files) { @@ -1129,13 +1260,69 @@ private void SynchronizeDatasetsTreeToVisualizationsTree() } } + private void VisualizationContext_PropertyChanging(object sender, PropertyChangingEventArgs e) + { + if (e.PropertyName == nameof(VisualizationContext.VisualizationContainer)) + { + // Unhook property changed events from old visualization container + if (VisualizationContext.Instance.VisualizationContainer != null) + { + VisualizationContext.Instance.VisualizationContainer.PropertyChanged -= this.VisualizationContainer_PropertyChanged; + } + + this.RaisePropertyChanging(nameof(this.VisualizationContainer)); + } + else if (e.PropertyName == nameof(VisualizationContext.DatasetViewModel)) + { + // Unhook property changed events from old dataset view model + if (VisualizationContext.Instance.DatasetViewModel != null) + { + VisualizationContext.Instance.DatasetViewModel.PropertyChanged -= this.DatasetViewModel_PropertyChanged; + } + + this.RaisePropertyChanged(nameof(this.TitleText)); + } + } + private void VisualizationContext_PropertyChanged(object sender, PropertyChangedEventArgs e) { if (e.PropertyName == nameof(VisualizationContext.VisualizationContainer)) { + // Hook property changed events to new visualization container + if (VisualizationContext.Instance.VisualizationContainer != null) + { + VisualizationContext.Instance.VisualizationContainer.PropertyChanged += this.VisualizationContainer_PropertyChanged; + } + this.RaisePropertyChanged(nameof(this.VisualizationContainer)); } - } + else if (e.PropertyName == nameof(VisualizationContext.DatasetViewModel)) + { + // Hook property changed events to new dataset view model + if (VisualizationContext.Instance.DatasetViewModel != null) + { + VisualizationContext.Instance.DatasetViewModel.PropertyChanged += this.DatasetViewModel_PropertyChanged; + } + + this.RaisePropertyChanged(nameof(this.TitleText)); + } + } + + private void DatasetViewModel_PropertyChanged(object sender, PropertyChangedEventArgs e) + { + if (e.PropertyName == nameof(DatasetViewModel.Name)) + { + this.RaisePropertyChanged(nameof(this.TitleText)); + } + } + + private void VisualizationContainer_PropertyChanged(object sender, PropertyChangedEventArgs e) + { + if (e.PropertyName == nameof(VisualizationContext.Instance.VisualizationContainer.SnapToVisualizationObject)) + { + this.RaisePropertyChanged(nameof(this.TitleText)); + } + } /*/// /// Display the settings dialog. diff --git a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/Microsoft.Psi.PsiStudio.csproj b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/Microsoft.Psi.PsiStudio.csproj index 54edc7143..77b9032cc 100644 --- a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/Microsoft.Psi.PsiStudio.csproj +++ b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/Microsoft.Psi.PsiStudio.csproj @@ -62,9 +62,10 @@ - - - + + all + runtime; build; native; contentfiles; analyzers + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/PsiStudioSettings.cs b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/PsiStudioSettings.cs index da13f0199..44e812116 100644 --- a/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/PsiStudioSettings.cs +++ b/Sources/Tools/PsiStudio/Microsoft.Psi.PsiStudio/PsiStudioSettings.cs @@ -7,8 +7,10 @@ namespace Microsoft.Psi.PsiStudio using System.Collections.Generic; using System.IO; using System.Reflection; + using System.Windows; using System.Xml; using System.Xml.Serialization; + using Microsoft.Psi.Visualization.Windows; /// /// Persisted application settings. @@ -38,23 +40,6 @@ public PsiStudioSettings() this.AdditionalAssemblies = null; } - /// - /// Finalizes an instance of the class. - /// - ~PsiStudioSettings() - { - if (string.IsNullOrWhiteSpace(this.settingsFilename)) - { - throw new InvalidOperationException("Coould not save the settings to file because PsiStudioSettings.Load() was not previously called to set the filepath."); - } - - using (var writer = XmlWriter.Create(this.settingsFilename, new XmlWriterSettings() { Indent = true })) - { - var xmlSerializer = new XmlSerializer(typeof(PsiStudioSettings)); - xmlSerializer.Serialize(writer, this); - } - } - /// /// Gets or sets the main window left position. /// @@ -152,6 +137,51 @@ public static PsiStudioSettings Load(string settingsFilename) return settings; } + /// + /// Saves the settings to the settings file. + /// + public void Save() + { + if (string.IsNullOrWhiteSpace(this.settingsFilename)) + { + throw new InvalidOperationException("Could not save the settings to file because PsiStudioSettings.Load() was not previously called to set the filepath."); + } + + // To avoid obliterating the existing settings file if something goes wrong, write + // the settings to a temporary file and then copy the file over the existing settings + // file once we know we were successful. + string tempFilename = this.CreateTempFilename(); + + try + { + using (var writer = XmlWriter.Create(tempFilename, new XmlWriterSettings() { Indent = true })) + { + var xmlSerializer = new XmlSerializer(typeof(PsiStudioSettings)); + xmlSerializer.Serialize(writer, this); + } + + // Delete the existing settings file + File.Delete(this.settingsFilename); + + // Move the temp file to be the new settings file + File.Move(tempFilename, this.settingsFilename); + } + catch (Exception ex) + { + new MessageBoxWindow( + Application.Current.MainWindow, + "Save Settings Error", + $"An error occurred while attempting to save the application settings{Environment.NewLine}{Environment.NewLine}{ex.Message}", + "Close", + null).ShowDialog(); + } + } + + private string CreateTempFilename() + { + return this.settingsFilename.Substring(0, this.settingsFilename.IndexOf('.')) + ".tmp"; + } + /// /// Loads settings from the xml settings file. /// diff --git a/Sources/Tools/PsiStudio/Test.Psi.PsiStudio/Test.Psi.PsiStudio.csproj b/Sources/Tools/PsiStudio/Test.Psi.PsiStudio/Test.Psi.PsiStudio.csproj index bb43073b5..ddb860122 100644 --- a/Sources/Tools/PsiStudio/Test.Psi.PsiStudio/Test.Psi.PsiStudio.csproj +++ b/Sources/Tools/PsiStudio/Test.Psi.PsiStudio/Test.Psi.PsiStudio.csproj @@ -4,8 +4,7 @@ Exe Test.Psi.PsiStudio.ConsoleMain - false - false + ../../../../Build/Test.Psi.ruleset true @@ -26,6 +25,16 @@ + + + all + runtime; build; native; contentfiles; analyzers + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + diff --git a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/GraphViewer.cs b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/GraphViewer.cs index dfe53e0f6..d63da5ce4 100644 --- a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/GraphViewer.cs +++ b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/GraphViewer.cs @@ -39,7 +39,7 @@ namespace Microsoft.Msagl.WpfGraphControl /// /// Graph viewer. /// - public class GraphViewer : IViewer + public class GraphViewer : IViewer, IDisposable { private const double DesiredPathThicknessInInches = 0.008; @@ -513,6 +513,12 @@ public static Size MeasureText(string text, FontFamily family, double size, Visu return new Size(formattedText.Width, formattedText.Height); } + /// + public void Dispose() + { + this.backgroundWorker?.Dispose(); + } + /// /// Update graph being viewed in place (assuming structure hasn't changed, otherwise triggers full re-layout). /// diff --git a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/Microsoft.Msagl.WpfGraphControl.csproj b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/Microsoft.Msagl.WpfGraphControl.csproj index 80a3da1af..e48bcae7f 100644 --- a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/Microsoft.Msagl.WpfGraphControl.csproj +++ b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/Microsoft.Msagl.WpfGraphControl.csproj @@ -21,6 +21,10 @@ + + all + runtime; build; native; contentfiles; analyzers + 1.1.3 diff --git a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/NativeMethods.cs b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/NativeMethods.cs index cea8dadd3..2733c6520 100644 --- a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/NativeMethods.cs +++ b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/NativeMethods.cs @@ -7,7 +7,7 @@ namespace Microsoft.Msagl.WpfGraphControl using System.Runtime.InteropServices; /// - /// Navive methods. + /// Native methods. /// internal static class NativeMethods { diff --git a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/VEdge.cs b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/VEdge.cs index 9819b298e..6c4e35ae6 100644 --- a/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/VEdge.cs +++ b/Sources/Visualization/Microsoft.Msagl.WpfGraphControl/VEdge.cs @@ -148,7 +148,7 @@ public bool MarkedForDragging public IViewerNode Target { get; private set; } /// - /// Gets or sets the radious of the polyline corner. + /// Gets or sets the radius of the polyline corner. /// public double RadiusOfPolylineCorner { get; set; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/DepthImageToImageAdapter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/DepthImageToImageAdapter.cs new file mode 100644 index 000000000..8f49076af --- /dev/null +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/DepthImageToImageAdapter.cs @@ -0,0 +1,36 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.Adapters +{ + using Microsoft.Psi.Imaging; + using Microsoft.Psi.Visualization.Data; + + /// + /// Represents an adapter that converts a depth image to an image. + /// + [StreamAdapter] + public class DepthImageToImageAdapter : StreamAdapter, Shared> + { + /// + /// Initializes a new instance of the class. + /// + public DepthImageToImageAdapter() + : base(Adapter) + { + } + + private static Shared Adapter(Shared sharedDepthImage, Envelope envelope) + { + Shared sharedImage = null; + + if ((sharedDepthImage != null) && (sharedDepthImage.Resource != null)) + { + sharedImage = ImagePool.GetOrCreate(sharedDepthImage.Resource.Width, sharedDepthImage.Resource.Height, PixelFormat.Gray_16bpp); + sharedImage.Resource.CopyFrom(sharedDepthImage.Resource); + } + + return sharedImage; + } + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToDepthImageAdapter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToDepthImageAdapter.cs new file mode 100644 index 000000000..2b62c3ee4 --- /dev/null +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToDepthImageAdapter.cs @@ -0,0 +1,37 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.Adapters +{ + using Microsoft.Psi.Imaging; + using Microsoft.Psi.Visualization.Data; + + /// + /// Represents an adapter that converts an encoded depth image to a depth image. + /// + [StreamAdapter] + public class EncodedDepthImageToDepthImageAdapter : StreamAdapter, Shared> + { + /// + /// Initializes a new instance of the class. + /// + public EncodedDepthImageToDepthImageAdapter() + : base(Adapter) + { + } + + private static Shared Adapter(Shared sharedEncodedDepthImage, Envelope envelope) + { + Shared sharedDepthImage = null; + + if ((sharedEncodedDepthImage != null) && (sharedEncodedDepthImage.Resource != null)) + { + sharedDepthImage = DepthImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height); + var decoder = new DepthImageFromStreamDecoder(); + decoder.DecodeFromStream(sharedEncodedDepthImage.Resource.ToStream(), sharedDepthImage.Resource); + } + + return sharedDepthImage; + } + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToImageAdapter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToImageAdapter.cs new file mode 100644 index 000000000..d99b0f7ba --- /dev/null +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedDepthImageToImageAdapter.cs @@ -0,0 +1,39 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.Adapters +{ + using Microsoft.Psi.Imaging; + using Microsoft.Psi.Visualization.Data; + + /// + /// Represents an adapter that converts an encoded depth image to an image. + /// + [StreamAdapter] + public class EncodedDepthImageToImageAdapter : StreamAdapter, Shared> + { + /// + /// Initializes a new instance of the class. + /// + public EncodedDepthImageToImageAdapter() + : base(Adapter) + { + } + + private static Shared Adapter(Shared sharedEncodedDepthImage, Envelope envelope) + { + Shared sharedImage = null; + + if ((sharedEncodedDepthImage != null) && (sharedEncodedDepthImage.Resource != null)) + { + var sharedDepthImage = DepthImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height); + sharedImage = ImagePool.GetOrCreate(sharedEncodedDepthImage.Resource.Width, sharedEncodedDepthImage.Resource.Height, PixelFormat.Gray_16bpp); + var decoder = new DepthImageFromStreamDecoder(); + decoder.DecodeFromStream(sharedEncodedDepthImage.Resource.ToStream(), sharedDepthImage.Resource); + sharedDepthImage.Resource.CopyTo(sharedImage.Resource); + } + + return sharedImage; + } + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedImageToImageAdapter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedImageToImageAdapter.cs index 01dd676f7..1b4eb399f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedImageToImageAdapter.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/EncodedImageToImageAdapter.cs @@ -3,6 +3,7 @@ namespace Microsoft.Psi.Visualization.Adapters { + using System; using Microsoft.Psi.Imaging; using Microsoft.Psi.Visualization.Data; @@ -20,14 +21,28 @@ public EncodedImageToImageAdapter() { } - private static Shared Adapter(Shared encodedImage, Envelope env) + private static Shared Adapter(Shared sharedEncodedImage, Envelope envelope) { Shared sharedImage = null; - if ((encodedImage != null) && (encodedImage.Resource != null)) + if ((sharedEncodedImage != null) && (sharedEncodedImage.Resource != null)) { - sharedImage = ImagePool.GetOrCreate(encodedImage.Resource.Width, encodedImage.Resource.Height, ImageDecoder.GetPixelFormat(encodedImage.Resource)); - ImageDecoder.DecodeTo(encodedImage.Resource, sharedImage.Resource); + // The code below maintains back-compatibility with encoded images which did not store the pixel format + // on the instance, but only in the stream. If the pixel format is unknown, we call upon the decoder to + // retrieve the pixel format. This might be less performant, but enables decoding in the right format + // even from older versions of encoded images. + var decoder = new ImageFromStreamDecoder(); + var pixelFormat = sharedEncodedImage.Resource.PixelFormat == PixelFormat.Undefined ? + decoder.GetPixelFormat(sharedEncodedImage.Resource.ToStream()) : sharedEncodedImage.Resource.PixelFormat; + + // If the decoder does not return a valid pixel format, we throw an exception. + if (pixelFormat == PixelFormat.Undefined) + { + throw new ArgumentException("The encoded image does not contain a supported pixel format."); + } + + sharedImage = ImagePool.GetOrCreate(sharedEncodedImage.Resource.Width, sharedEncodedImage.Resource.Height, pixelFormat); + decoder.DecodeFromStream(sharedEncodedImage.Resource.ToStream(), sharedImage.Resource); } return sharedImage; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/ObjectAdapter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/ObjectAdapter.cs index 868fe6128..7be05ffcb 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/ObjectAdapter.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/ObjectAdapter.cs @@ -3,7 +3,6 @@ namespace Microsoft.Psi.Visualization.Adapters { - using System.Runtime.Serialization; using Microsoft.Psi.Visualization.Data; /// @@ -22,7 +21,9 @@ public ObjectAdapter() private static object Adapter(T value, Envelope env) { - return value; + // If the data is shared, clone it because the caller will + // dereference the source soon after this method returns. + return SourceIsSharedType ? value.DeepClone() : value; } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/PassthroughAdapter{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/PassthroughAdapter{T}.cs index 0f485d83b..b77e814c2 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/PassthroughAdapter{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Adapters/PassthroughAdapter{T}.cs @@ -3,7 +3,6 @@ namespace Microsoft.Psi.Visualization.Adapters { - using System; using Microsoft.Psi.Visualization.Data; /// @@ -22,12 +21,9 @@ public PassthroughAdapter() private static T Adapter(T data, Envelope env) { - if (data is IDisposable disposableData) - { - return data.DeepClone(); - } - - return data; + // If the data is shared, clone it because the caller will + // dereference the source soon after this method returns. + return SourceIsSharedType ? data.DeepClone() : data; } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataCollection{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataCollection{T}.cs index 31118ab57..1a7ad786a 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataCollection{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataCollection{T}.cs @@ -17,9 +17,9 @@ public class ObservableDataCollection : ObservableCollection observableSource; /// - /// Sets the obvervable source. + /// Sets the observable source. /// - /// Souce collection to observe. + /// Source collection to observe. public void SetSource(IList source) { if (this.observableSource != null) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataItem{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataItem{T}.cs index b3269c037..64260c68b 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataItem{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableDataItem{T}.cs @@ -16,7 +16,7 @@ public class ObservableDataItem : ObservableObject /// /// Initializes a new instance of the class. /// - /// The initiali data value. + /// The initial data value. public ObservableDataItem(T data) { this.data = data; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableKeyedCache{TKey,TItem}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableKeyedCache{TKey,TItem}.cs index 963343972..52285e954 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableKeyedCache{TKey,TItem}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableKeyedCache{TKey,TItem}.cs @@ -38,7 +38,7 @@ public class ObservableKeyedCache : ObservableSortedCollection getKeyForItem; /// - /// Number of collection changed events recieved before pruning the cache. + /// Number of collection changed events received before pruning the cache. /// private uint pruneThreshold = ObservableSortedCollection.DefaultCapacity; @@ -48,7 +48,7 @@ public class ObservableKeyedCache : ObservableSortedCollection - /// Number of collection changed events recieved, since last pruning of the cache. + /// Number of collection changed events received, since last pruning of the cache. /// private uint collectionChangedCount = 0; @@ -60,7 +60,7 @@ public class ObservableKeyedCache : ObservableSortedCollection /// Initializes a new instance of the class that uses the default comparers. /// - /// Funtion that returns a key given an item. + /// Function that returns a key given an item. /// is null. public ObservableKeyedCache(Func getKeyForItem) : this(null, null, getKeyForItem) @@ -403,7 +403,7 @@ internal ObservableKeyedView(ObservableKeyedCache cache, ViewMode m /// /// Occurs when an item is added, removed, changed, moved, or the entire list is refreshed. - /// This event does not fire range based events for compatability to wpf controls. It turns adds into resets. + /// This event does not fire range based events for compatibility to wpf controls. It turns adds into resets. /// public virtual event NotifyCollectionChangedEventHandler CollectionChanged; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableSortedCollection{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableSortedCollection{T}.cs index 4edcb12c9..40607f1a6 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableSortedCollection{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Collections/ObservableSortedCollection{T}.cs @@ -28,7 +28,7 @@ public class ObservableSortedCollection : System.Collections.Generic.IList private const string IndexerName = "Item[]"; /// - /// Underlying sorted array of itmes. + /// Underlying sorted array of items. /// private SortedArray items; @@ -79,7 +79,7 @@ public ObservableSortedCollection(int capacity, IComparer comparer, IEquality /// /// Occurs when an item is added, removed, changed, moved, or the entire list is refreshed. - /// This event does not fire range based events for compatability to wpf controls. It turns adds into resets. + /// This event does not fire range based events for compatibility to wpf controls. It turns adds into resets. /// public virtual event NotifyCollectionChangedEventHandler CollectionChanged; @@ -127,7 +127,7 @@ public ObservableSortedCollection(int capacity, IComparer comparer, IEquality return this.items[index]; } - set { throw new NotSupportedException("ObservableSortedCollection does not support assignemt by index."); } + set { throw new NotSupportedException("ObservableSortedCollection does not support assignment by index."); } } /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/DisplayImage.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/DisplayImage.cs index 8f0012f69..bbb46ab44 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/DisplayImage.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/DisplayImage.cs @@ -62,6 +62,7 @@ public void UpdateImage(Shared image) lock (this.imageLock) { this.psiImage?.Dispose(); + if (image == null || image.Resource == null) { this.psiImage = null; @@ -74,6 +75,33 @@ public void UpdateImage(Shared image) this.UpdateBitmap(); } + /// + /// Update the underlying image with the specified image. + /// + /// New encoded image. + public void UpdateImage(Shared encodedImage) + { + lock (this.imageLock) + { + if (encodedImage == null || encodedImage.Resource == null) + { + this.psiImage?.Dispose(); + this.psiImage = null; + return; + } + + if (this.psiImage == null) + { + this.psiImage = Shared.Create(new Imaging.Image(encodedImage.Resource.Width, encodedImage.Resource.Height, Imaging.PixelFormat.BGR_24bpp)); + } + + var decoder = new ImageFromStreamDecoder(); + decoder.DecodeFromStream(encodedImage.Resource.ToStream(), this.psiImage.Resource); + } + + this.UpdateBitmap(); + } + /// /// Crop image to specified dimensions and return newly cropped image. Does not alter current image. /// @@ -84,7 +112,8 @@ public void UpdateImage(Shared image) /// The newly cropped image. public DisplayImage Crop(int left, int top, int width, int height) { - Shared croppedCopy = this.psiImage.Resource.Crop(left, top, width, height); + Shared croppedCopy = this.psiImage.SharedPool.GetOrCreate(); + this.psiImage.Resource.Crop(croppedCopy.Resource, left, top, width, height); var displayImage = new DisplayImage(); displayImage.UpdateImage(croppedCopy); return displayImage; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/SnappingBehavior.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/SnappingBehavior.cs new file mode 100644 index 000000000..eb8e857f2 --- /dev/null +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Common/SnappingBehavior.cs @@ -0,0 +1,26 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.Helpers +{ + /// + /// Defines various timeline snapping behaviors. + /// + public enum SnappingBehavior + { + /// + /// Snap to nearest message. + /// + Nearest, + + /// + /// Snap to nearest previous message. + /// + Previous, + + /// + /// Snap to nearest next message. + /// + Next, + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/PlacementConverter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/PlacementConverter.cs index df0dc9cf6..6f9dc73ff 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/PlacementConverter.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/PlacementConverter.cs @@ -8,7 +8,7 @@ namespace Microsoft.Psi.Visualization.Converters using System.Windows.Data; /// - /// Provides a way to apply custom logic to a binding. Specifically, converting from numeric value to a scaled offest. + /// Provides a way to apply custom logic to a binding. Specifically, converting from numeric value to a scaled offset. /// public class PlacementConverter : IValueConverter { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/TimeSpanConverter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/TimeSpanConverter.cs index 4e611d975..2910018e8 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/TimeSpanConverter.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Converters/TimeSpanConverter.cs @@ -24,7 +24,7 @@ public object Convert(object[] values, Type targetType, object parameter, Cultur return timespan.ToString("hh\\:mm\\:ss\\.ffff"); } - return "Invaid TimeSpan"; + return "Invalid TimeSpan"; } /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/AdaptingInstantDataProvider{TSrc,TDest}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/AdaptingInstantDataProvider{TSrc,TDest}.cs index fb55deb60..942945555 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/AdaptingInstantDataProvider{TSrc,TDest}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/AdaptingInstantDataProvider{TSrc,TDest}.cs @@ -6,7 +6,6 @@ namespace Microsoft.Psi.Visualization.Data using System; using System.Collections.Generic; using System.Linq; - using System.Threading.Tasks; using Microsoft.Psi.Persistence; /// @@ -19,7 +18,7 @@ namespace Microsoft.Psi.Visualization.Data public class AdaptingInstantDataProvider : IAdaptingInstantDataProvider { /// - /// Flag indicating whether type paramamter TSrc is Shared{} or not. + /// Flag indicating whether type parameter TDest is Shared{} or not. /// private readonly bool adaptedDataIsSharedType = typeof(TDest).IsGenericType && typeof(TDest).GetGenericTypeDefinition() == typeof(Shared<>); @@ -84,39 +83,20 @@ public InstantDataTarget UnregisterInstantDataTarget(Guid registrationToken) /// public void PushData(TSrc sourceData, IndexEntry indexEntry) { - // Adapt the data to the type required by target. The data - // adapter will release the reference to the source data automatically. + // Adapt the data to the type required by target. TDest adaptedData = this.streamAdapter.AdaptData(sourceData); - // Create a non-volatile copy of the list of targets - List targetList; - lock (this.targets) - { - targetList = this.targets.Values.ToList(); - } - - // Call each of the targets with the new data. If the adapted data is shared, - // then do a deep clone of each item before calling the callback and release the - // reference to the adapted data once we're done. - bool createClone = this.adaptedDataIsSharedType && adaptedData != null; - - foreach (InstantDataTarget callbackTarget in targetList) + // Call each of the targets with the adapted data, cloning it if it's shared. + foreach (InstantDataTarget instantDataTarget in this.targets.Values.ToList()) { - this.RunPushTask(callbackTarget, createClone ? adaptedData.DeepClone() : adaptedData, indexEntry); + instantDataTarget.Callback.Invoke(adaptedData, indexEntry); } - if (createClone) + // We're done with the adapted data, so decrement its reference count if it's shared + if (this.adaptedDataIsSharedType && adaptedData != null) { (adaptedData as IDisposable).Dispose(); } } - - private void RunPushTask(InstantDataTarget target, TDest data, IndexEntry indexEntry) - { - Task.Run(() => - { - target.Callback.Invoke(data, indexEntry); - }); - } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataManager.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataManager.cs index b52502152..2ed9c2719 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataManager.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataManager.cs @@ -9,10 +9,12 @@ namespace Microsoft.Psi.Visualization.Data using System.Linq; using System.Reflection; using System.Threading.Tasks; + using System.Windows; using System.Windows.Threading; using Microsoft.Psi.Persistence; using Microsoft.Psi.Visualization.Adapters; using Microsoft.Psi.Visualization.Collections; + using Microsoft.Psi.Visualization.Windows; /// /// Provides cached-controlled read access to data stores used by the visualization runtime. @@ -82,7 +84,7 @@ public void Dispose() /// Information about the stream source and the required stream adapter. /// The epsilon window to use when reading data at a given time. /// The method to call when new data is available. - /// The initial time range over which data is expectd. + /// The initial time range over which data is expected. /// A registration token that must be used by the target to unregister from updates or to modify the read epsilon. public Guid RegisterInstantDataTarget(StreamBinding streamBinding, RelativeTimeInterval cursorEpsilon, Action callback, TimeInterval viewRange) { @@ -153,7 +155,7 @@ public void ReadInstantData(DateTime cursorTime) } /// - /// Notifies the data manager that the possible range of data that mey be read has changed. + /// Notifies the data manager that the possible range of data that may be read has changed. /// /// The new view range of the navigator. public void OnInstantViewRangeChanged(TimeInterval viewRange) @@ -165,10 +167,10 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) } /// - /// Creates a view of the messages identified by the matching start and end times and asychronously fills it in. + /// Creates a view of the messages identified by the matching start and end times and asynchronously fills it in. /// /// The type of the message to read. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Start time of messages to read. /// End time of messages to read. /// Observable view of data. @@ -184,10 +186,10 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) } /// - /// Creates a view of the messages identified by the matching tail count and asychronously fills it in. + /// Creates a view of the messages identified by the matching tail count and asynchronously fills it in. /// /// The type of the message to read. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Number of messages to included in tail. /// Observable view of data. public ObservableKeyedCache>.ObservableKeyedView ReadStream(StreamBinding streamBinding, uint tailCount) @@ -202,10 +204,10 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) } /// - /// Creates a view of the messages identified by the matching tail range and asychronously fills it in. + /// Creates a view of the messages identified by the matching tail range and asynchronously fills it in. /// /// The type of the message to read. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Function to determine range included in tail. /// Observable view of data. public ObservableKeyedCache>.ObservableKeyedView ReadStream(StreamBinding streamBinding, Func tailRange) @@ -223,7 +225,7 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) /// Gets a view over the specified time range of the cached summary data. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// The start time of the view range. /// The end time of the view range. /// The time interval each summary value should cover. @@ -238,7 +240,7 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) /// Gets a view over the specified time range of the cached summary data. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// The time interval each summary value should cover. /// Number of items to include in view. /// A view over the cached summary data that covers the specified time range. @@ -252,7 +254,7 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) /// Gets a view over the specified time range of the cached summary data. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// The time interval each summary value should cover. /// Tail duration function. Computes the view range start time given an end time. Applies to live view mode only. /// A view over the cached summary data that covers the specified time range. @@ -262,6 +264,17 @@ public void OnInstantViewRangeChanged(TimeInterval viewRange) return this.FindStreamSummaryManager(streamBinding).ReadSummary(streamBinding, viewMode, DateTime.MinValue, DateTime.MaxValue, interval, 0, tailRange); } + /// + /// Gets originating time of the message in a stream that's closest to a given time. + /// + /// The stream binding indicating which stream to read from. + /// The time for which to return the message with the closest originating time. + /// The originating time of the message closest to time. + public DateTime? GetOriginatingTimeOfNearestInstantMessage(StreamBinding streamBinding, DateTime time) + { + return this.FindDataStoreReader(streamBinding).GetOriginatingTimeOfNearestInstantMessage(streamBinding, time); + } + /// /// Runs a task to read instant data, and continues to run read tasks until there are no read requests left. /// @@ -281,7 +294,19 @@ private void RunReadInstantDataTask() // If there's a cursor time, initiate an instant read request on each data store reader, otherwise we're done. if (taskCursorTime.HasValue) { - Parallel.ForEach(this.GetDataStoreReaderList(), dataStoreReader => dataStoreReader.ReadInstantData(taskCursorTime.Value)); + try + { + Parallel.ForEach(this.GetDataStoreReaderList(), dataStoreReader => dataStoreReader.ReadInstantData(taskCursorTime.Value)); + } + catch (Exception ex) + { + new MessageBoxWindow( + Application.Current.MainWindow, + "Instant Data Push Error", + $"An error occurred while attempting to push instant data to the visualization objects{Environment.NewLine}{Environment.NewLine}{ex.Message}", + "Close", + null).ShowDialog(); + } } else { @@ -293,7 +318,7 @@ private void RunReadInstantDataTask() /// /// Disposes of an instance of the class. /// - /// Indicates wheter the method call comes from a Dispose method (its value is true) or from its destructor (its value is false). + /// Indicates whether the method call comes from a Dispose method (its value is true) or from its destructor (its value is false). private void Dispose(bool disposing) { if (this.disposed) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataStoreReader.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataStoreReader.cs index 8ff997181..3cf6b834c 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataStoreReader.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/DataStoreReader.cs @@ -83,7 +83,7 @@ public void Dispose() { // Get the stream reader. Note that we don't care about the stream reader's stream adapter // because with instant data we always read raw data and adapt the stream later. - IStreamReader streamReader = this.GetStreamReader(target.StreamName, null, true); + IStreamReader streamReader = this.GetOrCreateStreamReader(target.StreamName, null); // Register the target with the stream reader streamReader.RegisterInstantDataTarget(target, viewRange); @@ -115,7 +115,7 @@ internal void UpdateInstantDataTargetEpsilon(Guid registrationToken, RelativeTim } /// - /// Notifies the data store the the view range of instant data has changed. + /// Notifies the data store the view range of instant data has changed. /// /// The new view range of the navigator. internal void OnInstantViewRangeChanged(TimeInterval viewRange) @@ -155,7 +155,7 @@ internal void ReadInstantData(DateTime cursorTime) /// TailRange - sliding dynamic range that includes the tail of the underlying data based on function. /// /// The type of the message to read. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Mode the view will be created in. /// Start time of messages to read. /// End time of messages to read. @@ -170,7 +170,18 @@ internal void ReadInstantData(DateTime cursorTime) uint tailCount, Func tailRange) { - return this.GetStreamReader(streamBinding.StreamName, streamBinding.StreamAdapter, true).ReadStream(viewMode, startTime, endTime, tailCount, tailRange); + return this.GetOrCreateStreamReader(streamBinding.StreamName, streamBinding.StreamAdapter).ReadStream(viewMode, startTime, endTime, tailCount, tailRange); + } + + /// + /// Gets originating time of the message in a stream that's closest to a given time. + /// + /// The stream binding indicating which stream to read from. + /// The time for which to return the message with the closest originating time. + /// The originating time of the message closest to time. + internal DateTime? GetOriginatingTimeOfNearestInstantMessage(StreamBinding streamBinding, DateTime time) + { + return this.GetExistingStreamReader(streamBinding.StreamName, null).GetOriginatingTimeOfNearestInstantMessage(time); } /// @@ -247,21 +258,25 @@ internal void DispatchData() this.streamReaders.ForEach(sr => sr.DispatchData()); } - private IStreamReader GetStreamReader(string streamName, IStreamAdapter streamAdapter, bool createIfNecessary) + private IStreamReader GetExistingStreamReader(string streamName, IStreamAdapter streamAdapter) + { + IStreamReader streamReader = this.streamReaders.Find(sr => sr.StreamName == streamName && sr.StreamAdapterType == streamAdapter?.GetType()); + if (streamReader == null) + { + throw new ArgumentException("No stream reader exists for the stream."); + } + + return streamReader; + } + + private IStreamReader GetOrCreateStreamReader(string streamName, IStreamAdapter streamAdapter) { var streamReader = this.streamReaders.Find(sr => sr.StreamName == streamName && sr.StreamAdapterType == streamAdapter?.GetType()); if (streamReader == null) { - if (createIfNecessary) - { - streamReader = new StreamReader(streamName, streamAdapter); - this.streamReaders.Add(streamReader); - } - else - { - throw new ArgumentException("No stream reader exists for the stream binding."); - } + streamReader = new StreamReader(streamName, streamAdapter); + this.streamReaders.Add(streamReader); } return streamReader; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/EpsilonInstantStreamReader{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/EpsilonInstantStreamReader{T}.cs index 1969a39e9..14172f174 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/EpsilonInstantStreamReader{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/EpsilonInstantStreamReader{T}.cs @@ -6,7 +6,6 @@ namespace Microsoft.Psi.Visualization.Data using System; using System.Collections.Generic; using System.Linq; - using System.Threading.Tasks; using Microsoft.Psi.Data; using Microsoft.Psi.Persistence; using Microsoft.Psi.Visualization.Collections; @@ -20,7 +19,7 @@ namespace Microsoft.Psi.Visualization.Data public class EpsilonInstantStreamReader { /// - /// Flag indicating whether type paramamter T is Shared{} or not. + /// Flag indicating whether type parameter T is Shared{} or not. /// private readonly bool isSharedType = typeof(T).IsGenericType && typeof(T).GetGenericTypeDefinition() == typeof(Shared<>); @@ -91,7 +90,7 @@ public InstantDataTarget UnregisterInstantDataTarget(Guid registrationToken) if (target != null) { - // If the data provider now has no targets to call, remove it from the collecction + // If the data provider now has no targets to call, remove it from the collection if (!this.dataProviders[index].HasRegisteredTargets) { this.dataProviders.RemoveAt(index); @@ -126,19 +125,16 @@ public void ReadInstantData(ISimpleReader reader, DateTime cursorTime, Observabl data = reader.Read(indexEntry); } - // Notify all registered adapting data providers of the new data. If the data is Shared then perform a deep clone - // (which resolves to an AddRef() for this type) for each provider we call. The providers are responsible for releasing - // their reference to the data once they're done with it. - if (this.isSharedType && data != null) + // Notify each adapting data provider of the new data + foreach (IAdaptingInstantDataProvider adaptingInstantDataProvider in this.dataProviders.ToList()) { - Parallel.ForEach(this.dataProviders.ToList(), provider => provider.PushData(data.DeepClone(), indexEntry)); - - // Release the reference to the local copy of the data - (data as IDisposable).Dispose(); + adaptingInstantDataProvider.PushData(data, indexEntry); } - else + + // Release the reference to the local copy of the data if it's shared + if (this.isSharedType && data != null) { - Parallel.ForEach(this.dataProviders.ToList(), provider => provider.PushData(data, indexEntry)); + (data as IDisposable).Dispose(); } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IAdaptingInstantDataProvider{TSrc}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IAdaptingInstantDataProvider{TSrc}.cs index ca1987949..4f794e6af 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IAdaptingInstantDataProvider{TSrc}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IAdaptingInstantDataProvider{TSrc}.cs @@ -31,7 +31,7 @@ public interface IAdaptingInstantDataProvider void RegisterInstantDataTarget(InstantDataTarget target); /// - /// Unregisters an instant data target from receiving data fromt he provider. + /// Unregisters an instant data target from receiving data from he provider. /// /// The registration token that the target was given when it was initially registered. /// An instant data target representing the target that was unregistered, or null if no target with diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IStreamReader.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IStreamReader.cs index 9c02d7c20..c00ecca78 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IStreamReader.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/IStreamReader.cs @@ -89,9 +89,16 @@ public interface IStreamReader : IDisposable /// Reads instant data from the stream at the given cursor time and notifies all registered instant visualization objects of the new data. /// /// The reader to read from. - /// The currenttime at the cursor.. + /// The current time at the cursor.. void ReadInstantData(ISimpleReader reader, DateTime cursorTime); + /// + /// Gets originating time of the message in a stream that's closest to a given time. + /// + /// The time for which to return the message with the closest originating time. + /// The originating time of the message closest to time. + public DateTime? GetOriginatingTimeOfNearestInstantMessage(DateTime time); + /// /// Notifies the data store reader that the range of data that may be of interest to instant data targets has changed. /// @@ -99,7 +106,7 @@ public interface IStreamReader : IDisposable void OnInstantViewRangeChanged(TimeInterval viewRange); /// - /// Creates a view of the indices identified by the matching start and end times and asychronously fills it in. + /// Creates a view of the indices identified by the matching start and end times and asynchronously fills it in. /// /// Start time of indices to read. /// End time of indices to read. diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/PoolManager.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/PoolManager.cs index d4aa7b12e..4adb022a2 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/PoolManager.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/PoolManager.cs @@ -24,7 +24,7 @@ private PoolManager() this.sharedToPoolMap = new Dictionary> { { typeof(Shared), () => new Pool(() => new Image(0, 0, PixelFormat.Undefined)) }, - { typeof(Shared), () => new Pool(() => new EncodedImage()) }, + { typeof(Shared), () => new Pool(() => new EncodedImage(0, 0, PixelFormat.Undefined)) }, }; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamAdapter{TSrc,TDest}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamAdapter{TSrc,TDest}.cs index 5652b86c0..3175906f9 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamAdapter{TSrc,TDest}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamAdapter{TSrc,TDest}.cs @@ -15,10 +15,15 @@ namespace Microsoft.Psi.Visualization.Data public class StreamAdapter : IStreamAdapter { /// - /// Gets default stream adapater. + /// Gets default stream adapter. /// public static readonly IStreamAdapter Default = new StreamAdapter((src, env) => src); + /// + /// Flag indicating whether type parameter TSrc is Shared{} or not. + /// + public static readonly bool SourceIsSharedType = typeof(TSrc).IsGenericType && typeof(TSrc).GetGenericTypeDefinition() == typeof(Shared<>); + private readonly Func adapter; /// @@ -67,13 +72,7 @@ public Func Allocator /// Destination data. public TDest AdaptData(TSrc data) { - var dest = this.adapter(data, default(Envelope)); - if (data is IDisposable) - { - (data as IDisposable).Dispose(); - } - - return dest; + return this.adapter(data, default(Envelope)); } /// @@ -86,7 +85,9 @@ public TDest AdaptData(TSrc data) return (data, env) => { var dest = this.adapter(data, env); - if (data is IDisposable) + + // Release the reference to the source data if it's shared. + if (SourceIsSharedType && data != null) { (data as IDisposable).Dispose(); } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamBinding.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamBinding.cs index 9b2e2a228..dd5f96610 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamBinding.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamBinding.cs @@ -29,7 +29,7 @@ public class StreamBinding /// The simple reader type for the underlying store. /// The type of the stream adapter, null if there is none. /// The type of the stream summarizer, null if there is none. - /// The arguments used when constructing the stream summarizer, null if ther is none. + /// The arguments used when constructing the stream summarizer, null if there is none. public StreamBinding( string streamName, string partitionName, @@ -91,7 +91,7 @@ private StreamBinding() public IStreamMetadata StreamMetadata { get; private set; } /// - /// Gets stream adapater. + /// Gets stream adapter. /// [IgnoreDataMember] public IStreamAdapter StreamAdapter @@ -141,7 +141,9 @@ private set // update value and update type name this.streamAdapterType = value; - this.StreamAdapterTypeName = this.streamAdapterType?.FullName; + + // use assembly-qualified name as stream adapter may be in a different assembly + this.StreamAdapterTypeName = this.streamAdapterType?.AssemblyQualifiedName; } } @@ -215,7 +217,7 @@ private set public object[] SummarizerArgs { get; set; } /// - /// Gets summaraizer type. + /// Gets summarizer type. /// [IgnoreDataMember] public Type SummarizerType @@ -234,7 +236,9 @@ private set { // update value and update type name this.summarizerType = value; - this.SummarizerTypeName = this.summarizerType?.FullName; + + // use assembly-qualified name as simple reader may be in a different assembly + this.SummarizerTypeName = this.summarizerType?.AssemblyQualifiedName; } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamReader{T}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamReader{T}.cs index 9f5fdd55d..20df5a3e9 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamReader{T}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamReader{T}.cs @@ -13,6 +13,7 @@ namespace Microsoft.Psi.Visualization.Data using Microsoft.Psi.Data; using Microsoft.Psi.Persistence; using Microsoft.Psi.Visualization.Collections; + using Microsoft.Psi.Visualization.Helpers; using Microsoft.Psi.Visualization.Navigation; /// @@ -137,7 +138,7 @@ public void RegisterInstantDataTarget(InstantDataTarget target, TimeInt { this.InternalRegisterInstantDataTarget(target); - // Create the index view if one does not already exist + // Create the instant index view if one does not already exist if (this.instantIndexView == null) { this.OnInstantViewRangeChanged(viewRange); @@ -148,6 +149,14 @@ public void RegisterInstantDataTarget(InstantDataTarget target, TimeInt public void UnregisterInstantDataTarget(Guid registrationToken) { this.InternalUnregisterInstantDataTarget(registrationToken); + + // If no instant visualization objects are now using + // this stream reader, remove the instant index view + if (this.instantStreamReaders.Count <= 0) + { + this.instantIndexView = null; + this.currentIndexViewRange = new NavigatorRange(DateTime.MinValue, DateTime.MinValue); + } } /// @@ -158,7 +167,7 @@ public void UpdateInstantDataTargetEpsilon(Guid registrationToken, RelativeTimeI if (target != null) { - // Update the Ccursor epsilon + // Update the Cursor epsilon target.CursorEpsilon = epsilon; // Create the internal register method @@ -197,6 +206,18 @@ public void ReadInstantData(ISimpleReader reader, DateTime cursorTime) } } + /// + public DateTime? GetOriginatingTimeOfNearestInstantMessage(DateTime time) + { + int index = IndexHelper.GetIndexForTime(time, this.instantIndexView.Count, (idx) => this.instantIndexView[idx].OriginatingTime, SnappingBehavior.Nearest); + if (index >= 0) + { + return this.instantIndexView[index].OriginatingTime; + } + + return null; + } + /// public void Cancel() { @@ -397,7 +418,7 @@ private IList ComputeReadRequests(DateTime startTime, DateTime endT // compute read requests for first new range newReadRequests.AddRange(this.ComputeReadRequests(startTime, range.Item1, readIndicesOnly)); - // continue comptuing for second new range + // continue computing for second new range startTime = range.Item2; } } @@ -431,12 +452,6 @@ private InstantDataTarget InternalUnregisterInstantDataTarget(Guid registrationT this.instantStreamReaders.RemoveAt(index); } - // If there's no instant stream readers, remove the index view - if (this.instantStreamReaders.Count <= 0) - { - this.instantIndexView = null; - } - return target; } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummaryManager.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummaryManager.cs index 1930e1b98..cfb3ef780 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummaryManager.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummaryManager.cs @@ -49,7 +49,7 @@ public StreamSummaryManager(string storeName, string storePath, string streamNam public string StreamName { get; private set; } /// - /// Gets the stream adapater type. + /// Gets the stream adapter type. /// public Type StreamAdapterType { get; private set; } @@ -70,7 +70,7 @@ public void DispatchData() /// Finds the time of the next data point after the point indicated by the given time. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Time of current data point. /// The time interval each summary value covers. /// Time of the next data point. @@ -100,7 +100,7 @@ public DateTime FindNextDataPoint(StreamBinding streamBinding, DateTime time, /// Finds the time of the previous data point before the point indicated by the given time. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// Time of current data point. /// The time interval each summary value covers. /// Time of the previous data point. @@ -130,7 +130,7 @@ public DateTime FindPreviousDataPoint(StreamBinding streamBinding, DateTime t /// Gets a view over the specified time range of the cached summary data. /// /// The summary data type. - /// The stream binding inidicating which stream to read from. + /// The stream binding indicating which stream to read from. /// The view mode, which may be either fixed or live data. /// The start time of the view range. /// The end time of the view range. diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummary{TSrc,TDest}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummary{TSrc,TDest}.cs index 8c722d13b..e51048f1d 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummary{TSrc,TDest}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Data/StreamSummary{TSrc,TDest}.cs @@ -69,7 +69,7 @@ public StreamSummary(StreamBinding streamBinding, TimeSpan interval, uint maxCac public Type SummarizerType => this.streamBinding.SummarizerType; /// - /// Gets ths stream binding. + /// Gets the stream binding. /// public StreamBinding StreamBinding => this.streamBinding; @@ -296,7 +296,7 @@ public IntervalData Search(DateTime time, StreamSummarySearchMode // compute read requests for first new range newRangeRequests.AddRange(this.ComputeRangeRequests(startTime, range.Item1)); - // continue comptuing for second new range + // continue computing for second new range startTime = range.Item2; } } @@ -334,7 +334,7 @@ private void Data_CollectionChanged(object sender, NotifyCollectionChangedEventA /// Returns a view over the summary data and ensure that the view is preserved in the cache. /// /// The view mode. - /// Start stime of the view. + /// Start time of the view. /// End time of the view. /// Number of items to include in view. /// Tail duration function. @@ -419,7 +419,7 @@ private void OnReceiveData(List> items) int headMerge = this.summaryDataBuffer.FindIndex(list => this.itemComparer.Compare(list[list.Count - 1], items[0]) == 0); if (headMerge != -1) { - // Merge the tail of the predecesor with the head of the range being added + // Merge the tail of the predecessor with the head of the range being added var predecessor = this.summaryDataBuffer[headMerge]; // Use summarizer-specific method to combine the two IntervalData values diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/DataTypes/TimeIntervalHistory.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/DataTypes/TimeIntervalHistory.cs index 67a5892d2..e16c02d09 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/DataTypes/TimeIntervalHistory.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/DataTypes/TimeIntervalHistory.cs @@ -3,12 +3,27 @@ namespace Microsoft.Psi.Visualization.DataTypes { + using System; using System.Collections.Generic; + using System.Runtime.Serialization; /// /// Represents an indexed history of time intervals. /// + [Serializable] public class TimeIntervalHistory : Dictionary> { + /// + /// Initializes a new instance of the class with serialized data. + /// + /// + /// This is the serialization constructor. Satisfies rule CA2229: ImplementSerializationConstructors. + /// + /// The serialization info. + /// The streaming context. + protected TimeIntervalHistory(SerializationInfo serializationInfo, StreamingContext streamingContext) + : base(serializationInfo, streamingContext) + { + } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Extensions/VisualizationExtensions.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Extensions/VisualizationExtensions.cs index 0f9a813fa..26f7580a4 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Extensions/VisualizationExtensions.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Extensions/VisualizationExtensions.cs @@ -29,11 +29,11 @@ public static class VisualizationExtensions } /// - /// Converts stream of dictionarys of TKey and TValue to a stream of collections of TValue. + /// Converts stream of dictionaries of TKey and TValue to a stream of collections of TValue. /// /// The type of dictionary keys. /// The type of dictionary values. - /// The stream of dictionarys of TKey and TValue. + /// The stream of dictionaries of TKey and TValue. /// An optional delivery policy. /// A stream of the converted collections of TValue. public static IProducer.ValueCollection> Values(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) @@ -42,10 +42,10 @@ public static class VisualizationExtensions } /// - /// Converts stream of dictionarys of 2d points to a stream of list of named points. + /// Converts stream of dictionaries of 2d points to a stream of list of named points. /// /// The type of dictionary keys. - /// The stream of dictionarys of 2d points. + /// The stream of dictionaries of 2d points. /// An optional delivery policy. /// A stream of the converted list of named points. public static IProducer>> ToScatterPoints2D(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) @@ -54,10 +54,10 @@ public static class VisualizationExtensions } /// - /// Converts stream of dictionarys of rectangles to a stream of list of named rectangles. + /// Converts stream of dictionaries of rectangles to a stream of list of named rectangles. /// /// The type of dictionary keys. - /// The stream of dictionarys of rectangles. + /// The stream of dictionaries of rectangles. /// An optional delivery policy. /// A stream of the converted list of named rectangles. public static IProducer>> ToScatterRectangle(this IProducer> source, DeliveryPolicy> deliveryPolicy = null) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Helpers/IndexHelper.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Helpers/IndexHelper.cs index eea09f8c4..40b87f21f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Helpers/IndexHelper.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Helpers/IndexHelper.cs @@ -4,7 +4,6 @@ namespace Microsoft.Psi.Visualization.Helpers { using System; - using Microsoft.Psi.Visualization.Common; /// /// Represents helper methods for psi indices. @@ -18,53 +17,39 @@ public static class IndexHelper /// The time to find the index for. /// The number of items in the index collection. /// A function that returns the time for a given index id. + /// Timeline snapping behaviors. /// The index id closest to the specified time, using the specified interpolation style. - public static int GetIndexForTime(DateTime currentTime, int count, Func timeAtIndex) + public static int GetIndexForTime(DateTime currentTime, int count, Func timeAtIndex, SnappingBehavior snappingBehavior = SnappingBehavior.Nearest) { - return GetIndexForTime(currentTime, count, timeAtIndex, InterpolationStyle.Direct); - } - - /// - /// Gets the index id corresponding to a specified time for - /// a collection of indices that are ordered by time. - /// - /// The time to find the index for. - /// The number of items in the index collection. - /// A function that returns the time for a given index id. - /// The type of interpolation (Direct or Step) to use when resolving indices that don't exactly lie at the specified time. - /// The index id closest to the specified time, using the specified interpolation style. - public static int GetIndexForTime(DateTime currentTime, int count, Func timeAtIndex, InterpolationStyle interpolationStyle) - { - // Perform a binary search - SearchResult result = SearchIndex(currentTime, count, timeAtIndex); - if (result.ExactMatchFound) + // If there's only one point in the index, then return its index + if (count == 1) { - return result.ExactIndex; + return 0; } + // Perform a binary search // If no exact match, lo and hi indicate ticks that // are right before and right after the time we're looking for. - // If we're using Step interpolation, then we should return - // lo, otherwise we should return whichever value is closest - if (interpolationStyle == InterpolationStyle.Step) - { - return result.LowIndex; - } + SearchResult result = SearchIndex(currentTime, count, timeAtIndex); - // If the're only one point in the index, then return its index - if (count == 1) + if (result.ExactMatchFound) { - return 0; + return result.ExactIndex; } - // Return the index of whichever point is closest to the current time - if ((timeAtIndex(result.HighIndex) - currentTime) < (currentTime - timeAtIndex(result.LowIndex))) - { - return result.HighIndex; - } - else - { - return result.LowIndex; + switch (snappingBehavior) + { + case SnappingBehavior.Previous: + return result.LowIndex; + case SnappingBehavior.Next: + return result.HighIndex; + case SnappingBehavior.Nearest: + default: + // Return the index of whichever point is closest to the current time + return + (timeAtIndex(result.HighIndex) - currentTime) < (currentTime - timeAtIndex(result.LowIndex)) ? + result.HighIndex : + result.LowIndex; } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Microsoft.Psi.Visualization.Common.Windows.csproj b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Microsoft.Psi.Visualization.Common.Windows.csproj index dc53791dd..1b94b1d88 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Microsoft.Psi.Visualization.Common.Windows.csproj +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Microsoft.Psi.Visualization.Common.Windows.csproj @@ -11,7 +11,7 @@ true true - TRACE;RELEASE;NET47;COM_SERVER + TRACE;RELEASE;NET47 bin\Debug\net472\Microsoft.Psi.Visualization.Common.Windows.xml @@ -19,7 +19,7 @@ true true - TRACE;DEBUG;NET47;COM_SERVER + TRACE;DEBUG;NET47 @@ -221,12 +221,17 @@ - + + + all + runtime; build; native; contentfiles; analyzers + - - + + + @@ -235,7 +240,6 @@ - @@ -247,7 +251,7 @@ - + diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/CursorModeChangedEventArgs.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/CursorModeChangedEventArgs.cs index c086c3c83..0f2b5334f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/CursorModeChangedEventArgs.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/CursorModeChangedEventArgs.cs @@ -9,7 +9,7 @@ namespace Microsoft.Psi.Visualization.Navigation /// Represents the method that will handle an event that has cursor mode changed event data. /// /// The source of the event. - /// An object that contains cursot mode changed event data. + /// An object that contains cursor mode changed event data. public delegate void CursorModeChangedHandler(object sender, CursorModeChangedEventArgs e); /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/Navigator.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/Navigator.cs index 16f99e239..f612f5073 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/Navigator.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Navigation/Navigator.cs @@ -318,17 +318,34 @@ public bool RepeatPlayback } /// - /// Moves the cursor to the start of the selection. + /// Moves the cursor to the given datetime. /// - public void MoveToSelectionStart() + /// Time to which to move cursor. + public void MoveTo(DateTime dateTime) { if (this.CursorMode != CursorMode.Live) { - this.Cursor = this.SelectionRange.StartTime; + this.Cursor = dateTime; this.EnsureCursorVisible(); } } + /// + /// Moves the cursor to the start of the selection. + /// + public void MoveToSelectionStart() + { + this.MoveTo(this.SelectionRange.StartTime); + } + + /// + /// Moves the cursor to the end of the selection. + /// + public void MoveToSelectionEnd() + { + this.MoveTo(this.SelectionRange.EndTime); + } + /// /// Remove an audio playback stream. /// @@ -351,18 +368,6 @@ public void AddAudioPlaybackStream(StreamBinding streamBinding) this.RaisePropertyChanged(nameof(this.AudioPlaybackStreams)); } - /// - /// Moves the cursor to the end of the selection. - /// - public void MoveToSelectionEnd() - { - if (this.CursorMode != CursorMode.Live) - { - this.Cursor = this.SelectionRange.EndTime; - this.EnsureCursorVisible(); - } - } - /// /// Sets the cursor mode. /// @@ -526,7 +531,7 @@ public void ZoomToSelection() } /// - /// Animates the navigator curor based on indicated speed. + /// Animates the navigator cursor based on indicated speed. /// private void StartPlayback() { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/DatasetViewModel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/DatasetViewModel.cs index 70302dc6c..0f2110ea1 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/DatasetViewModel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/DatasetViewModel.cs @@ -211,7 +211,7 @@ public static Task LoadAsync(string filename) } /// - /// Creates a new dataset from an exising data store. + /// Creates a new dataset from an existing data store. /// /// The name of the data store. /// The path of the data store. @@ -223,7 +223,7 @@ public static DatasetViewModel CreateFromExistingStore(string storeName, string } /// - /// Asynchronously creates a new dataset from an exising data store. + /// Asynchronously creates a new dataset from an existing data store. /// /// The name of the data store. /// The path of the data store. @@ -240,7 +240,7 @@ public static Task CreateFromExistingStoreAsync(string storeNa } /// - /// Sets a session to be the currrent session being visualized. + /// Sets a session to be the current session being visualized. /// /// The SessionViewModel to visualize. public void VisualizeSession(SessionViewModel sessionViewModel) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PartitionViewModel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PartitionViewModel.cs index 15c4ef6c8..e54331620 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PartitionViewModel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PartitionViewModel.cs @@ -135,7 +135,7 @@ public string Name public string LastMessageOriginatingTimeString => DateTimeFormatHelper.FormatDateTime(this.LastMessageOriginatingTime); /// - /// Gets the orginating time interval (earliest to latest) of the messages in this session. + /// Gets the originating time interval (earliest to latest) of the messages in this session. /// [Browsable(false)] public TimeInterval OriginatingTimeInterval => this.streamTreeRoot.OriginatingTimeInterval; @@ -312,10 +312,10 @@ private void Monitor(object parameter) try { - // Keep waiting on messages until the partition exits live mode or we're signalled to stop + // Keep waiting on messages until the partition exits live mode or we're signaled to stop while (this.continueMonitoring && this.IsLivePartition && Application.Current != null) { - // If there's a new message, squirrel it away as the lastest recent message, otherwise sleep + // If there's a new message, squirrel it away as the latest recent message, otherwise sleep if (storeReader.MoveNext(out Envelope envelope)) { if ((!latestLiveMessageReceived.HasValue) || (envelope.Time > latestLiveMessageReceived.Value.Time)) @@ -380,7 +380,7 @@ private void UpdateStreamMetadata(IEnumerable metadata) // If we don't already have a node for this stream, add one if (!this.streamsById.ContainsKey(psiStreamMetadata.Id)) { - IStreamMetadata streamMetadata = new PsiLiveStreamMetadata(psiStreamMetadata.Name, psiStreamMetadata.Id, psiStreamMetadata.TypeName, this.StoreName, this.StorePath); + IStreamMetadata streamMetadata = new PsiLiveStreamMetadata(psiStreamMetadata.Name, psiStreamMetadata.Id, psiStreamMetadata.TypeName, psiStreamMetadata.SupplementalMetadataTypeName, this.StoreName, this.StorePath); this.streamsById[streamMetadata.Id] = this.StreamTreeRoot.AddPath(streamMetadata); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PsiLiveStreamMetadata.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PsiLiveStreamMetadata.cs index 9bb169725..1c4176814 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PsiLiveStreamMetadata.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/PsiLiveStreamMetadata.cs @@ -16,13 +16,15 @@ internal class PsiLiveStreamMetadata : IStreamMetadata /// The name of the application that generated the persisted files, or the root name of the files. /// The id of the data stream. /// The type of data of this stream. + /// The type of supplemental metadata for this stream. /// The name of th partition. /// The path of the partition. - public PsiLiveStreamMetadata(string name, int id, string typeName, string partitionName, string partitionPath) + public PsiLiveStreamMetadata(string name, int id, string typeName, string supplementalMetadataTypeName, string partitionName, string partitionPath) { this.Name = name; this.Id = id; this.TypeName = typeName; + this.SupplementalMetadataTypeName = supplementalMetadataTypeName; this.PartitionName = partitionName; this.PartitionPath = partitionPath; this.FirstMessageTime = DateTime.MinValue; @@ -40,6 +42,9 @@ public PsiLiveStreamMetadata(string name, int id, string typeName, string partit /// public string TypeName { get; private set; } + /// + public string SupplementalMetadataTypeName { get; private set; } + /// public string PartitionName { get; private set; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/SessionViewModel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/SessionViewModel.cs index 460c3c9d0..0167d8bd5 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/SessionViewModel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/SessionViewModel.cs @@ -113,7 +113,7 @@ public string Name public double UiElementOpacity => this.DatasetViewModel.CurrentSessionViewModel == this ? 1.0d : 0.5d; /// - /// Gets the orginating time interval (earliest to latest) of the messages in this session. + /// Gets the originating time interval (earliest to latest) of the messages in this session. /// [Browsable(false)] public TimeInterval OriginatingTimeInterval => @@ -183,7 +183,7 @@ public RelayCommand AddPartitionCommand } else { - throw new ApplicationException("Invalid file type selected when adding partition."); + throw new NotSupportedException("Invalid file type selected when adding partition."); } } }); @@ -267,7 +267,7 @@ public void AddStorePartition(string storeName, string storePath, string partiti /// /// The name of the annotation store. /// The path of the annotation store. - /// The annotated event definition to use when creating new annoted events in the newly created annotation partition. + /// The annotated event definition to use when creating new annotated events in the newly created annotation partition. /// The partition name. Default is null. public void CreateAnnotationPartition(string storeName, string storePath, AnnotatedEventDefinition definition, string partitionName = null) { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/StreamTreeNode.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/StreamTreeNode.cs index 4925db860..d569f4f78 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/StreamTreeNode.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/ViewModels/StreamTreeNode.cs @@ -14,7 +14,6 @@ namespace Microsoft.Psi.Visualization.ViewModels using GalaSoft.MvvmLight.CommandWpf; using Microsoft.Psi.Audio; using Microsoft.Psi.Diagnostics; - using Microsoft.Psi.PsiStudio.Common; using Microsoft.Psi.Visualization.Base; using Microsoft.Psi.Visualization.Common; using Microsoft.Psi.Visualization.Helpers; @@ -136,6 +135,14 @@ public StreamTreeNode(PartitionViewModel partition) [Description("The type of messages in the stream.")] public string TypeName { get; protected set; } + /// + /// Gets or sets the type of supplemental metadata for this stream tree node. + /// + [PropertyOrder(11)] + [DisplayName("SupplementalMetadataType")] + [Description("The type of supplemental metadata for the stream.")] + public string SupplementalMetadataTypeName { get; protected set; } + /// /// Gets the collection of children for the this stream tree node. /// @@ -254,7 +261,7 @@ public virtual string IconSource { if (this.IsStream) { - if (VisualizationContext.Instance.GetStreamType(this)?.Name == nameof(PipelineDiagnostics)) + if (VisualizationContext.Instance.GetDataType(this)?.Name == nameof(PipelineDiagnostics)) { return this.Partition.IsLivePartition ? IconSourcePath.DiagnosticsLive : IconSourcePath.Diagnostics; } @@ -262,7 +269,7 @@ public virtual string IconSource { return this.Partition.IsLivePartition ? IconSourcePath.StreamGroupLive : IconSourcePath.StreamGroup; } - else if (VisualizationContext.Instance.GetStreamType(this) == typeof(AudioBuffer)) + else if (VisualizationContext.Instance.GetDataType(this) == typeof(AudioBuffer)) { return this.Partition.IsLivePartition ? IconSourcePath.StreamAudioMutedLive : IconSourcePath.StreamAudioMuted; } @@ -279,7 +286,7 @@ public virtual string IconSource } /// - /// Gets the orginating time interval (earliest to latest) of the messages in this session. + /// Gets the originating time interval (earliest to latest) of the messages in this session. /// [Browsable(false)] public TimeInterval OriginatingTimeInterval @@ -381,13 +388,14 @@ private StreamTreeNode AddPath(string[] path, IStreamMetadata streamMetadata, in this.InternalChildren.Add(child); } - // if we are at the last segement of the path name then we are at the leaf node + // if we are at the last segment of the path name then we are at the leaf node if (path.Length == depth) { Debug.Assert(child.StreamMetadata == null, "There should never be two leaf nodes"); child.StreamMetadata = streamMetadata; - child.TypeName = streamMetadata.TypeName; child.StreamName = streamMetadata.Name; + child.TypeName = streamMetadata.TypeName; + child.SupplementalMetadataTypeName = streamMetadata.SupplementalMetadataTypeName; return child; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationContainerView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationContainerView.xaml.cs index ff7513b49..1ac5236e4 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationContainerView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationContainerView.xaml.cs @@ -37,7 +37,7 @@ public InstantVisualizationContainerView() } /// - /// Gets ths visualization panel. + /// Gets the visualization panel. /// protected InstantVisualizationContainer VisualizationPanel => (InstantVisualizationContainer)this.DataContext; @@ -70,9 +70,12 @@ private void InstantVisualizationContainerView_SizeChanged(object sender, SizeCh private void ResizeChildVisualizationPanels() { InstantVisualizationContainer containerPanel = this.DataContext as InstantVisualizationContainer; - foreach (VisualizationPanel panel in containerPanel.Panels) + if (containerPanel != null) { - panel.Width = this.ActualWidth / containerPanel.Panels.Count; + foreach (VisualizationPanel panel in containerPanel.Panels) + { + panel.Width = this.ActualWidth / containerPanel.Panels.Count; + } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationPlaceholderPanelView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationPlaceholderPanelView.xaml.cs index ba89a1327..ac29b3ac6 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationPlaceholderPanelView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/InstantVisualizationPlaceholderPanelView.xaml.cs @@ -19,7 +19,7 @@ public InstantVisualizationPlaceholderPanelView() } /// - /// Gets ths visualization panel. + /// Gets the visualization panel. /// protected InstantVisualizationPlaceholderPanel VisualizationPanel => (InstantVisualizationPlaceholderPanel)this.DataContext; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineSegmentView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineSegmentView.xaml.cs index e92f38caf..7ca41a909 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineSegmentView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineSegmentView.xaml.cs @@ -30,7 +30,7 @@ public TimelineSegmentView() /// /// Tick alignment. /// Number of divisions. - /// Timeline segment lable. + /// Timeline segment label. public TimelineSegmentView(VerticalAlignment tickAlignment, int numDivisions, string label) { this.InitializeComponent(); diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineView.xaml.cs index 1e259642e..ca2a40188 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/TimelineView.xaml.cs @@ -231,7 +231,7 @@ private void ComputeTicks() long segmentStart = (range.StartTime.Ticks - this.Navigator.DataRange.StartTime.Ticks) / tickZoomLevelDescriptorMajor.DurationInTicks; long segmentEnd = (range.EndTime.Ticks - this.Navigator.DataRange.StartTime.Ticks) / tickZoomLevelDescriptorMajor.DurationInTicks; - // remove all unnessary segments (due to scrolling out of view) + // remove all unnecessary segments (due to scrolling out of view) // segment start + duration < visible start time... no part of segment is visible // or segment start > visible range var segmentsToRemove = this.segments.Keys.Where(key => key.Item1 + key.Item2 < range.StartTime || key.Item1 > range.EndTime).ToList(); @@ -265,7 +265,7 @@ private void DataRange_RangeChanged(object sender, NavigatorTimeRangeChangedEven this.UpdateSelectionMarkers(); if (e.NewStartTime != e.OriginalStartTime) { - // our times are all expressed in ellapsed timne from DataRange.Start so we need to reset any timeline when this changes + // our times are all expressed in elapsed time from DataRange.Start so we need to reset any timeline when this changes this.Clear(); this.ComputeTicks(); } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/VisualizationContainerView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/VisualizationContainerView.xaml.cs index 1ece6d565..d6b526a14 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/VisualizationContainerView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/VisualizationContainerView.xaml.cs @@ -6,15 +6,12 @@ namespace Microsoft.Psi.Visualization.Views using System; using System.Collections.Generic; using System.Linq; - using System.Reflection; using System.Windows; using System.Windows.Controls; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; - using Microsoft.Psi.PsiStudio; - using Microsoft.Psi.PsiStudio.Common; using Microsoft.Psi.Visualization.Common; using Microsoft.Psi.Visualization.ViewModels; using Microsoft.Psi.Visualization.VisualizationObjects; @@ -72,8 +69,9 @@ private void Items_DragOver(object sender, DragEventArgs e) else if (dragOperation == DragDropOperation.DragDropStream) { StreamTreeNode streamTreeNode = e.Data.GetData(DragDropDataName.StreamTreeNode) as StreamTreeNode; + Type streamType = VisualizationContext.Instance.GetDataType(streamTreeNode); Point mousePosition = e.GetPosition(this.Items); - List metadatas = this.GetStreamDropCommands(streamTreeNode, this.GetVisualizationPanelUnderMouse(mousePosition)); + List metadatas = this.GetStreamDropCommands(streamType, this.GetVisualizationPanelUnderMouse(mousePosition)); e.Effects = metadatas.Count > 0 ? DragDropEffects.Move : DragDropEffects.None; e.Handled = true; } @@ -148,13 +146,16 @@ private void DropStream(DragEventArgs e) // Get the visualization panel (if any) that the mouse is above VisualizationPanel visualizationPanel = this.GetVisualizationPanelUnderMouse(mousePosition); + // Get the type of messages in the stream + Type dataType = VisualizationContext.Instance.GetDataType(streamTreeNode); + // Get the list of commands that are compatible with the user dropping the stream here - List metadatas = this.GetStreamDropCommands(streamTreeNode, visualizationPanel); + List metadatas = this.GetStreamDropCommands(dataType, visualizationPanel); - // If there's any compatible visualization commands, execute the first one + // If there's any compatible visualization commands, select the most appropriate one and execute it if (metadatas.Count > 0) { - VisualizationContext.Instance.VisualizeStream(streamTreeNode, metadatas[0], visualizationPanel); + VisualizationContext.Instance.VisualizeStream(streamTreeNode, VisualizerMetadata.GetClosestVisualizerMetadata(dataType, metadatas), visualizationPanel); } } } @@ -173,15 +174,10 @@ private VisualizationPanel GetVisualizationPanelUnderMouse(Point mousePosition) return this.hitTestResult != null ? this.hitTestResult.DataContext as VisualizationPanel : null; } - private List GetStreamDropCommands(StreamTreeNode streamTreeNode, VisualizationPanel visualizationPanel) + private List GetStreamDropCommands(Type messageType, VisualizationPanel visualizationPanel) { - List metadatas = new List(); - // Get all the commands that are applicable to this stream tree node and the panel it was dropped over - Type streamType = VisualizationContext.Instance.GetStreamType(streamTreeNode); - metadatas = VisualizationContext.Instance.VisualizerMap.GetByDataTypeAndPanelAboveSeparator(streamType, visualizationPanel); - - return metadatas; + return VisualizationContext.Instance.VisualizerMap.GetByDataTypeAndPanelAboveSeparator(messageType, visualizationPanel); } private HitTestFilterBehavior HitTestFilter(DependencyObject dependencyObject) @@ -211,7 +207,7 @@ private void CreateDragDropAdorner() private int FindPanelMoveIndices(VisualizationPanel droppedPanel, int panelVerticalCenter, out int currentPanelIndex) { - // Find the index of the panel whose vertical center is closest the the panel being dragged's vertical center + // Find the index of the panel whose vertical center is closest the panel being dragged's vertical center VisualizationContainer visualizationContainer = droppedPanel.Container; currentPanelIndex = -1; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationObjectView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationObjectView.xaml.cs index 4ce8bc558..a672e9e44 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationObjectView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationObjectView.xaml.cs @@ -3,6 +3,7 @@ namespace Microsoft.Psi.Visualization.Views.Visuals2D { + using System; using System.Collections.Generic; using System.ComponentModel; using System.Linq; @@ -19,7 +20,7 @@ namespace Microsoft.Psi.Visualization.Views.Visuals2D /// /// Interaction logic for DiagnosticsVisualizationObjectView.xaml. /// - public partial class PipelineDiagnosticsVisualizationObjectView : UserControl + public partial class PipelineDiagnosticsVisualizationObjectView : UserControl, IDisposable { private GraphViewer graphViewer = new GraphViewer() { LayoutEditingEnabled = false }; private Dictionary graphVisualPanZoom = new Dictionary(); @@ -43,6 +44,12 @@ public PipelineDiagnosticsVisualizationObjectView() /// public PipelineDiagnosticsVisualizationObject DiagnosticsVisualizationObject { get; private set; } + /// + public void Dispose() + { + this.graphViewer.Dispose(); + } + /// /// Update view. /// @@ -280,11 +287,27 @@ private void DiagnosticsVisualizationObject_PropertyChanged(object sender, Prope this.graphVisualPanZoom = new Dictionary(); this.FitGraphView(); } - else if (e.PropertyName == nameof(this.DiagnosticsVisualizationObject.CurrentValue) && this.DiagnosticsVisualizationObject.CurrentValue != null && this.DiagnosticsVisualizationObject.CurrentValue.Value.Data != null) + else if (e.PropertyName == nameof(this.DiagnosticsVisualizationObject.CurrentData) && this.DiagnosticsVisualizationObject.CurrentData != null) { - this.presenter.UpdateGraph(this.DiagnosticsVisualizationObject.CurrentValue.Value.Data, false); + this.presenter.UpdateGraph(this.DiagnosticsVisualizationObject.CurrentData, false); } - else + else if ( + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.ConnectorColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.EdgeColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.EdgeLineThickness) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.HeatmapColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.HeatmapStatistics) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.Highlight) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.HighlightColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.InfoTextSize) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.JoinColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.NodeColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.ShowDeliveryPolicies) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.ShowEmitterNames) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.ShowExporterConnections) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.ShowReceiverNames) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.SourceNodeColor) || + e.PropertyName == nameof(this.DiagnosticsVisualizationObject.SubpipelineColor)) { this.presenter.UpdateSettings(this.DiagnosticsVisualizationObject); } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationPresenter.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationPresenter.cs index 66f273e56..04a56fa03 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationPresenter.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/PipelineDiagnosticsVisualizationPresenter.cs @@ -1,727 +1,759 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Visualization.Views.Visuals2D -{ - using System; - using System.Collections.Generic; - using System.Linq; - using System.Text; - using Microsoft.Msagl.Drawing; - using Microsoft.Psi.Diagnostics; - using Microsoft.Psi.PsiStudio.TypeSpec; - using Microsoft.Psi.Visualization.VisualizationObjects; - - /// - /// Interaction logic for DiagnosticsVisualizationObjectView.xaml. - /// - public partial class PipelineDiagnosticsVisualizationPresenter - { - private readonly PipelineDiagnosticsVisualizationModel model; - private readonly PipelineDiagnosticsVisualizationObjectView view; - - /// - /// Initializes a new instance of the class. - /// - /// Diagnostics view. - /// Visualization object for this presenter. - public PipelineDiagnosticsVisualizationPresenter(PipelineDiagnosticsVisualizationObjectView view, PipelineDiagnosticsVisualizationObject visualizationObject) - { - this.model = new PipelineDiagnosticsVisualizationModel(); - this.view = view; - this.UpdateSettings(visualizationObject); - } - - /// - /// Gets diagnostics graph. - /// - public PipelineDiagnostics DiagnosticsGraph => this.model.Graph; - - /// - /// Gets visual graph. TODO: arg to update. - /// - public Graph VisualGraph { get; private set; } - - /// - /// Gets details of selected edge. - /// - public string SelectedEdgeDetails => this.model.SelectedEdgeDetails; - - /// - /// Gets edge color. - /// - public Color HighlightColor { get; private set; } - - /// - /// Gets edge color. - /// - public Color EdgeColor { get; private set; } //// TODO: private - - /// - /// Gets node color. - /// - public Color NodeColor { get; private set; } - - /// - /// Gets source node color. - /// - public Color SourceNodeColor { get; private set; } - - /// - /// Gets subpipeline color. - /// - public Color SubpipelineColor { get; private set; } - - /// - /// Gets connector node color. - /// - public Color ConnectorColor { get; private set; } - - /// - /// Gets join node color. - /// - public Color JoinColor { get; private set; } - - /// - /// Gets label color (light). - /// - public Color LabelColorLight { get; private set; } - - /// - /// Gets label color (dark). - /// - public Color LabelColorDark { get; private set; } - - /// - /// Gets heatmap color (base). - /// - public Color HeatmapColorBase { get; private set; } - - /// - /// Gets info text size. - /// - public double InfoTextSize { get; private set; } - - /// - /// Gets a value indicating whether to show exporter connections. - /// - public bool ShowExporterConnections { get; private set; } - - /// - /// Gets breadcrumb graph IDs. - /// - public IEnumerable Breadcrumbs - { - get - { - return this.model.NavStack.Reverse(); - } - } - - /// - /// Update diagnostics configuration. - /// - /// Diagnostics visualization object. - public void UpdateSettings(PipelineDiagnosticsVisualizationObject visualizationObject) - { - // convert colors to MSAGL graph colors - Func colorFromMediaColor = (System.Windows.Media.Color color) => new Color(color.R, color.G, color.B); - this.model.VisualizationObject = visualizationObject; - this.EdgeColor = colorFromMediaColor(visualizationObject.EdgeColor); - this.HighlightColor = colorFromMediaColor(visualizationObject.HighlightColor); - this.NodeColor = colorFromMediaColor(visualizationObject.NodeColor); - this.SourceNodeColor = colorFromMediaColor(visualizationObject.SourceNodeColor); - this.SubpipelineColor = colorFromMediaColor(visualizationObject.SubpipelineColor); - this.ConnectorColor = colorFromMediaColor(visualizationObject.ConnectorColor); - this.JoinColor = colorFromMediaColor(visualizationObject.JoinColor); - this.HeatmapColorBase = colorFromMediaColor(visualizationObject.HeatmapColor); - this.InfoTextSize = visualizationObject.InfoTextSize; - this.ShowExporterConnections = visualizationObject.ShowExporterConnections; - this.LabelColorLight = Color.White; - this.LabelColorDark = Color.Black; - if (visualizationObject.ModelDirty) - { - this.model.Reset(); - visualizationObject.ModelDirty = false; - this.VisualGraph = null; - this.view.Update(true); - } - - if (this.model.Graph != null) - { - this.UpdateGraph(this.model.Graph, true); - } - } - - /// - /// Update diagnostics graph. - /// - /// Current diagnostics graph. - /// Force re-layout of graph (otherwise, updates labels, colors, etc. in place). - public void UpdateGraph(PipelineDiagnostics graph, bool forceRelayout) - { - this.model.Graph = graph; - if (graph == null) - { - this.VisualGraph = null; - } - else - { - var pipelineIdToPipelineDiagnostics = graph.GetAllPipelineDiagnostics().ToDictionary(p => p.Id); - var currentGraph = this.Breadcrumbs.Count() > 0 ? pipelineIdToPipelineDiagnostics[this.Breadcrumbs.Last()] : graph; - this.VisualGraph = this.BuildVisualGraph(currentGraph, pipelineIdToPipelineDiagnostics); - } - - this.view.Update(forceRelayout); - } - - /// - /// Update selected edge. - /// - /// Selected edge. - public void UpdateSelectedEdge(Edge edge) - { - if (edge == null) - { - // clear selected edge (if any) - this.model.SelectedEdgeId = -1; - this.view.Update(true); - return; - } - - var input = edge.UserData as PipelineDiagnostics.ReceiverDiagnostics; - if (input != null) - { - this.UpdateSelectedEdge(input, this.VisualGraph, true); - } - } - - /// - /// Navigate into subgraph. - /// - /// Subgraph ID into which to navigate. - public void NavInto(int subgraphId) - { - this.model.NavStack.Push(subgraphId); - this.UpdateGraph(this.model.Graph, true); - } - - /// - /// Navigate back one graph. - /// - public void NavBack() - { - if (this.model.NavStack.Count > 0) - { - this.model.NavStack.Pop(); - this.UpdateGraph(this.model.Graph, true); - } - } - - /// - /// Navigate back to givin graph. - /// - /// Graph Id. - public void NavBackTo(int id) - { - while (this.model.NavStack.Count > 0 && this.model.NavStack.Peek() != id) - { - this.model.NavStack.Pop(); - } - - this.UpdateGraph(this.model.Graph, true); - } - - /// - /// Navigate back to root graph. - /// - public void NavHome() - { - if (this.model.NavStack.Count > 0) - { - while (this.model.NavStack.Count > 0) - { - this.model.NavStack.Pop(); - } - - this.UpdateGraph(this.model.Graph, true); - } - } - - private static Edge GetEdgeById(int id, Graph graph) - { - foreach (var n in graph.Nodes) - { - foreach (var e in n.Edges) - { - if (e.UserData != null && ((PipelineDiagnostics.ReceiverDiagnostics)e.UserData).Id == id) - { - return e; - } - } - } - - return null; - } - - private static bool IsBridgeToExporter(PipelineDiagnostics.PipelineElementDiagnostics node) - { - var bridgeEmitters = node.ConnectorBridgeToPipelineElement.Emitters; - var typeName = bridgeEmitters.Length == 1 ? bridgeEmitters[0].PipelineElement.TypeName : string.Empty; - return typeName == "MessageConnector`1" || typeName == "MessageEnvelopeConnector`1"; - } - - private void UpdateSelectedEdge(PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, bool clicked) - { - var edge = GetEdgeById(input.Id, graph); - if (clicked && this.model.SelectedEdgeId == input.Id) - { - // toggle unselected - edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; // unselect current - this.model.SelectedEdgeDetails = string.Empty; - this.model.SelectedEdgeId = -1; - this.view.Update(true); - return; - } - - // new edge selected - if (this.model.SelectedEdgeId != -1) - { - var previousEdge = GetEdgeById(this.model.SelectedEdgeId, graph); - if (previousEdge != null) - { - previousEdge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; // unselect previous - } - } - - edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness * 2; // select current - this.model.SelectedEdgeId = input.Id; - var sb = new StringBuilder(); - sb.Append($"Type: {TypeSpec.Simplify(input.TypeName)}" + Environment.NewLine); - sb.Append($"Message Size (avg): {input.MessageSize:0}" + Environment.NewLine); - sb.Append($"Queue Size: {input.QueueSize:0.###}" + Environment.NewLine); - sb.Append($"Processed Count: {input.ProcessedCount}" + Environment.NewLine); - sb.Append($"Processed/Time: {input.ProcessedPerTimeSpan:0.###}" + Environment.NewLine); - sb.Append($"Dropped Count: {input.DroppedCount}" + Environment.NewLine); - sb.Append($"Dropped/Time: {input.DroppedPerTimeSpan:0.###}" + Environment.NewLine); - sb.Append($"Latency at Emitter (avg): {input.MessageLatencyAtEmitter:0.###}ms" + Environment.NewLine); - sb.Append($"Latency at Receiver (avg): {input.MessageLatencyAtReceiver:0.###}ms" + Environment.NewLine); - sb.Append($"Processing Time (avg): {input.ProcessingTime:0.###}ms" + Environment.NewLine); - sb.Append($"Delivery Policy: {input.DeliveryPolicyName}" + Environment.NewLine); - this.model.SelectedEdgeDetails = sb.ToString(); - this.view.Update(clicked); - } - - private Func StatsSelector(bool heatmap) - { - switch (this.model.VisualizationObject.HeatmapStatistics) - { - case PipelineDiagnosticsVisualizationObject.HeatmapStats.None: - return null; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.LatencyAtEmitter: - return i => i.MessageLatencyAtEmitter; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.LatencyAtReceiver: - return i => i.MessageLatencyAtReceiver; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.Processing: - return i => i.ProcessingTime; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.Throughput: - return i => i.ProcessedCount; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.QueueSize: - return i => i.QueueSize; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.DroppedCount: - return i => i.DroppedCount; - case PipelineDiagnosticsVisualizationObject.HeatmapStats.MessageSize: - return i => - { - var avg = i.MessageSize; - return heatmap && avg > 0 ? Math.Log(avg) : avg; - }; - default: - throw new ArgumentException($"Unknown visualization selector type."); - } - } - - private Color LabelColor(Color background) - { - var r = background.R / 255.0; - var g = background.G / 255.0; - var b = background.B / 255.0; - var brightness = Math.Sqrt((0.299 * r * r) + (0.587 * g * g) + (0.114 * b * b)); - return brightness < 0.55 ? this.LabelColorLight : this.LabelColorDark; - } - - private bool HilightEdge(PipelineDiagnostics.ReceiverDiagnostics receiverDiagnostics) - { - switch (this.model.VisualizationObject.Highlight) - { - case PipelineDiagnosticsVisualizationObject.HighlightCondition.None: - return false; - case PipelineDiagnosticsVisualizationObject.HighlightCondition.UnlimitedDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.Unlimited)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.LatestMessageDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.LatestMessage)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.ThrottleDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.Throttle)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.SynchronousOrThrottleDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.SynchronousOrThrottle)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.LatencyConstrainedDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.LatencyConstrained)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.QueueSizeConstrainedDeliveryPolicy: - return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.QueueSizeConstrained)); - case PipelineDiagnosticsVisualizationObject.HighlightCondition.ThrottledReceivers: - return receiverDiagnostics.Throttled; - default: - throw new ArgumentException($"Unknown hilight condition: {this.model.VisualizationObject.Highlight}"); - } - } - - private Color HeatmapColor(double stats, Color baseColor) - { - var baseR = baseColor.R * (1.0 - stats); - var baseG = baseColor.G * (1.0 - stats); - var baseB = baseColor.B * (1.0 - stats); - var heatR = this.HeatmapColorBase.R * stats; - var heatG = this.HeatmapColorBase.G * stats; - var heatB = this.HeatmapColorBase.B * stats; - return new Color((byte)(baseR + heatR), (byte)(baseG + heatG), (byte)(baseB + heatB)); // blend from base to heat color - } - - private void HeatmapStats(Graph graph, Func statsSelector, bool perNode) - { - if (graph.Edges.Count() == 0) - { - return; - } - - var edgeStats = graph.Edges.Where(e => e.UserData != null).Select(e => (e, statsSelector((PipelineDiagnostics.ReceiverDiagnostics)e.UserData))); - var max = edgeStats.Select(x => x.Item2).Max(); - - if (perNode) - { - // max edge per node - foreach (var node in graph.Nodes) - { - var inputs = node.InEdges; - if (inputs.Count() > 0) - { - var maxStats = node.InEdges.Where(e => e.UserData != null).Select(e => statsSelector((PipelineDiagnostics.ReceiverDiagnostics)e.UserData)).Max(); - var color = this.HeatmapColor(max > 0 ? maxStats / max : 0, this.NodeColor); - node.Attr.Color = color; - node.Attr.FillColor = color; - node.Label.FontColor = this.LabelColor(color); - } - } - } - else - { - // per edge - foreach (var (edge, stat) in edgeStats) - { - edge.Attr.Color = this.HeatmapColor(max > 0 ? stat / max : 0, this.EdgeColor); - } - } - } - - private void VisualizeEdgeColoring(Graph graph) - { - var selector = this.StatsSelector(true); - if (selector != null) - { - // visualize heatmap - var perNode = this.model.VisualizationObject.HeatmapStatistics == PipelineDiagnosticsVisualizationObject.HeatmapStats.Processing; - this.HeatmapStats(graph, selector, perNode); - } - - // overlay highlights - if (this.model.VisualizationObject.Highlight != PipelineDiagnosticsVisualizationObject.HighlightCondition.None) - { - foreach (var edge in graph.Edges) - { - if (edge.UserData != null && this.HilightEdge((PipelineDiagnostics.ReceiverDiagnostics)edge.UserData)) - { - edge.Attr.Color = this.HighlightColor; - } - } - } - } - - private bool IsConnectorBridge(PipelineDiagnostics.PipelineElementDiagnostics node) - { - return node.Kind == PipelineElementKind.Connector && (node.Receivers.Length == 0 || node.Emitters.Length == 0) && node.ConnectorBridgeToPipelineElement != null; - } - - private Node BuildVisualNode(PipelineDiagnostics.PipelineElementDiagnostics node) - { - var vis = new Node($"n{node.Id}"); - var fillColor = node.Kind == PipelineElementKind.Source ? this.SourceNodeColor : node.Kind == PipelineElementKind.Subpipeline ? this.SubpipelineColor : this.NodeColor; - var typ = TypeSpec.Simplify(node.TypeName); - var isStoppedSubpipeline = node.RepresentsSubpipeline != null && !node.RepresentsSubpipeline.IsPipelineRunning; - var stopped = isStoppedSubpipeline || !node.IsRunning ? " (stopped)" : string.Empty; - vis.LabelText = node.Kind == PipelineElementKind.Subpipeline ? $"{node.Name}{stopped}|{typ}" : typ; - vis.Label.FontColor = this.LabelColor(fillColor); - vis.Attr.Color = fillColor; - vis.Attr.FillColor = fillColor; - if (vis.LabelText == "Join") - { - this.SetJoinVisualAttributes(vis, node.Name); - } - - return vis; - } - - private string BuildVisualEdgeLabelText(string emitterName, string receiverName, string stats, string deliveryPolicyName) - { - var showEmitter = this.model.VisualizationObject.ShowEmitterNames; - var showReceiver = this.model.VisualizationObject.ShowReceiverNames; - var showDeliveryPolicy = this.model.VisualizationObject.ShowDeliveryPolicies; - emitterName = showEmitter ? emitterName : string.Empty; - receiverName = showReceiver ? receiverName : string.Empty; - deliveryPolicyName = showDeliveryPolicy && deliveryPolicyName.Length > 0 ? $" [{deliveryPolicyName}]" : string.Empty; - var arrow = showEmitter && showReceiver ? "→" : string.Empty; - return $" {emitterName}{arrow}{receiverName}{stats}{deliveryPolicyName} "; // extra padding to allow for stats changes without re-layout - } - - private Edge BuildVisualEdge(Node source, Node target, PipelineDiagnostics.ReceiverDiagnostics input, Func statsSelector) - { - var edge = new Edge(source, target, ConnectionToGraph.Connected); - edge.UserData = input; - edge.Attr.Color = this.EdgeColor; - edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; - var stats = statsSelector != null ? $" ({statsSelector(input):0.#})" : string.Empty; - edge.LabelText = this.BuildVisualEdgeLabelText(input.Source.Name, input.ReceiverName, stats, input.DeliveryPolicyName); - edge.Label.FontColor = this.LabelColorLight; - edge.Label.UserData = edge; - return edge; - } - - private bool AddVisualEdge(Node source, Node target, PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, Func statsSelector) - { - if (source != null && target != null) - { - var edge = this.BuildVisualEdge(source, target, input, statsSelector); - graph.AddPrecalculatedEdge(edge); - if (input.Id == this.model.SelectedEdgeId) - { - this.UpdateSelectedEdge(input, graph, false); - return true; - } - } - - return false; - } - - private bool AddVisualEdge(int sourceId, int targetId, PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, Func statsSelector) - { - return this.AddVisualEdge(graph.FindNode($"n{sourceId}"), graph.FindNode($"n{targetId}"), input, graph, statsSelector); - } - - private void SetVisualAttributes(Node vis, Shape shape, Color color, string symbol, string name) - { - vis.Attr.Color = color; - vis.Attr.FillColor = color; - vis.Label.FontColor = this.LabelColor(color); - vis.Attr.Shape = shape; - vis.LabelText = $"{symbol}|{name}"; - } - - private void SetConnectorVisualAttributes(Node vis, string label) - { - this.SetVisualAttributes(vis, Shape.Circle, this.ConnectorColor, "☍", label); - } - - private void SetJoinVisualAttributes(Node vis, string label) - { - this.SetVisualAttributes(vis, Shape.Circle, this.JoinColor, "+", label); - } - - private Graph BuildVisualGraph(PipelineDiagnostics diagnostics, Dictionary pipelineIdToPipelineDiagnostics) - { - var subpipelineIdToPipelineDiagnostics = diagnostics.SubpipelineDiagnostics.ToDictionary(p => p.Id); - var graph = new Graph($"{diagnostics.Name} (running={diagnostics.IsPipelineRunning})", $"g{diagnostics.Id}"); - switch (this.model.VisualizationObject.LayoutDirection) - { - case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.LeftToRight: - graph.Attr.LayerDirection = LayerDirection.LR; - break; - case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.TopToBottom: - graph.Attr.LayerDirection = LayerDirection.TB; - break; - case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.RightToLeft: - graph.Attr.LayerDirection = LayerDirection.RL; - break; - case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.BottomToTop: - graph.Attr.LayerDirection = LayerDirection.BT; - break; - } - - graph.UserData = diagnostics.Id; - graph.Attr.BackgroundColor = Color.Transparent; - var subpipelineNodes = new Dictionary(); - var connectorsWithinSubpipelines = new Dictionary(); - var statsSelector = this.StatsSelector(false); - - // add nodes - foreach (var node in diagnostics.PipelineElements.Where(n => !this.IsConnectorBridge(n))) - { - var vis = this.BuildVisualNode(node); - if (node.Kind == PipelineElementKind.Subpipeline && node.RepresentsSubpipeline != null) - { - vis.UserData = node.RepresentsSubpipeline; - subpipelineNodes.Add(node.RepresentsSubpipeline.Id, node); - foreach (var n in node.RepresentsSubpipeline.PipelineElements.Where(n => n.Kind == PipelineElementKind.Connector)) - { - connectorsWithinSubpipelines.Add(n.Id, n); - } - } - else if (node.Kind == PipelineElementKind.Connector) - { - this.SetConnectorVisualAttributes(vis, node.Name); - } - - graph.AddNode(vis); - } - - // add connectors - foreach (var node in diagnostics.PipelineElements.Where(this.IsConnectorBridge)) - { - var connectsToSubpipeline = subpipelineNodes.ContainsKey(node.ConnectorBridgeToPipelineElement.PipelineId); - if (!connectsToSubpipeline) - { - if (!this.ShowExporterConnections && IsBridgeToExporter(node)) - { - continue; - } - - var connector = new Node($"n{node.Id}"); - var bridgedPipeline = pipelineIdToPipelineDiagnostics[node.ConnectorBridgeToPipelineElement.PipelineId]; - this.SetConnectorVisualAttributes(connector, $"{node.Name} ({bridgedPipeline.Name})"); - graph.AddNode(connector); - } - } - - // add edges - var selectedEdgeUpdated = false; - foreach (var n in diagnostics.PipelineElements) - { - foreach (var i in n.Receivers) - { - if (i.Source != null) - { - if (this.AddVisualEdge(i.Source.PipelineElement.Id, n.Id, i, graph, statsSelector)) - { - selectedEdgeUpdated = true; - } - } - } - } - - // add connector bridge edges - foreach (var n in diagnostics.PipelineElements.Where(this.IsConnectorBridge)) - { - if (!this.ShowExporterConnections && IsBridgeToExporter(n)) - { - continue; - } - - // connector bridging to subpipeline? - if (subpipelineNodes.TryGetValue(n.ConnectorBridgeToPipelineElement.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics subNode)) - { - // edges from connector source directly to bridge target (subpipeline) - var sub = graph.FindNode($"n{subNode.Id}"); - if (sub != null) - { - foreach (var i in n.Receivers) - { - if (i.Source != null) - { - var source = graph.FindNode($"n{i.Source.PipelineElement.Id}"); - if (source != null) - { - if (this.AddVisualEdge(source, sub, i, graph, statsSelector)) - { - selectedEdgeUpdated = true; - } - } - } - } - - // edges from connector bridge source (subpipeline) to connector targets - foreach (var o in n.Emitters) - { - foreach (var t in o.Targets) - { - var target = graph.FindNode($"n{t.PipelineElement.Id}"); - if (target != null) - { - if (this.AddVisualEdge(sub, target, t, graph, statsSelector)) - { - selectedEdgeUpdated = true; - } - } - } - } - } - } - else - { - // connector bridging graphs - var bridgedPipeline = pipelineIdToPipelineDiagnostics[n.ConnectorBridgeToPipelineElement.PipelineId]; - var connector = graph.FindNode($"n{n.Id}"); - - // add dotted line edge representing connector bridged to descendant pipeline - var targetPipeline = bridgedPipeline; - while (targetPipeline != null) - { - if (subpipelineIdToPipelineDiagnostics.ContainsKey(targetPipeline.Id)) - { - var targetNode = graph.FindNode($"n{subpipelineNodes[targetPipeline.Id].Id}"); - var edge = new Edge(connector, targetNode, ConnectionToGraph.Connected); - edge.Attr.Color = this.EdgeColor; - edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; - edge.Attr.AddStyle(Style.Dotted); - edge.LabelText = this.BuildVisualEdgeLabelText(n.Name, bridgedPipeline.Name, string.Empty, string.Empty); - edge.Label.FontColor = this.LabelColorLight; - graph.AddPrecalculatedEdge(edge); - break; - } - - // walk up ancestor chain until we're at a direct child subpipeline - targetPipeline = targetPipeline.ParentPipelineDiagnostics; - } - } - } - - // add direct connections from one subpipeline (connector) to another - foreach (var c in connectorsWithinSubpipelines.Values) - { - if (c.ConnectorBridgeToPipelineElement != null) - { - if (c.ConnectorBridgeToPipelineElement.PipelineId == diagnostics.Id && c.ConnectorBridgeToPipelineElement.Receivers.Length == 1) - { - var i = c.ConnectorBridgeToPipelineElement.Receivers[0]; - if (i.Source != null && i.Source.PipelineElement.PipelineId == diagnostics.Id && i.Source.PipelineElement.ConnectorBridgeToPipelineElement != null) - { - if (subpipelineNodes.TryGetValue(i.Source.PipelineElement.ConnectorBridgeToPipelineElement.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics source) && - subpipelineNodes.TryGetValue(c.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics target)) - { - if (this.AddVisualEdge(source.Id, target.Id, i, graph, statsSelector)) - { - selectedEdgeUpdated = true; - } - } - } - } - } - } - - if (!selectedEdgeUpdated && this.model.SelectedEdgeId != -1) - { - // hide while in subpipeline - this.model.SelectedEdgeDetails = string.Empty; - } - - this.VisualizeEdgeColoring(graph); - return graph; - } - } -} +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.Views.Visuals2D +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Text; + using Microsoft.Msagl.Drawing; + using Microsoft.Psi.Diagnostics; + using Microsoft.Psi.PsiStudio.TypeSpec; + using Microsoft.Psi.Visualization.VisualizationObjects; + + /// + /// Interaction logic for DiagnosticsVisualizationObjectView.xaml. + /// + public partial class PipelineDiagnosticsVisualizationPresenter + { + private readonly PipelineDiagnosticsVisualizationModel model; + private readonly PipelineDiagnosticsVisualizationObjectView view; + + /// + /// Initializes a new instance of the class. + /// + /// Diagnostics view. + /// Visualization object for this presenter. + public PipelineDiagnosticsVisualizationPresenter(PipelineDiagnosticsVisualizationObjectView view, PipelineDiagnosticsVisualizationObject visualizationObject) + { + this.model = new PipelineDiagnosticsVisualizationModel(); + this.view = view; + this.UpdateSettings(visualizationObject); + } + + /// + /// Gets diagnostics graph. + /// + public PipelineDiagnostics DiagnosticsGraph => this.model.Graph; + + /// + /// Gets visual graph. TODO: arg to update. + /// + public Graph VisualGraph { get; private set; } + + /// + /// Gets details of selected edge. + /// + public string SelectedEdgeDetails => this.model.SelectedEdgeDetails; + + /// + /// Gets edge color. + /// + public Color HighlightColor { get; private set; } + + /// + /// Gets edge color. + /// + public Color EdgeColor { get; private set; } //// TODO: private + + /// + /// Gets node color. + /// + public Color NodeColor { get; private set; } + + /// + /// Gets source node color. + /// + public Color SourceNodeColor { get; private set; } + + /// + /// Gets subpipeline color. + /// + public Color SubpipelineColor { get; private set; } + + /// + /// Gets connector node color. + /// + public Color ConnectorColor { get; private set; } + + /// + /// Gets join node color. + /// + public Color JoinColor { get; private set; } + + /// + /// Gets label color (light). + /// + public Color LabelColorLight { get; private set; } + + /// + /// Gets label color (dark). + /// + public Color LabelColorDark { get; private set; } + + /// + /// Gets heatmap color (base). + /// + public Color HeatmapColorBase { get; private set; } + + /// + /// Gets info text size. + /// + public double InfoTextSize { get; private set; } + + /// + /// Gets a value indicating whether to show exporter connections. + /// + public bool ShowExporterConnections { get; private set; } + + /// + /// Gets breadcrumb graph IDs. + /// + public IEnumerable Breadcrumbs + { + get + { + return this.model.NavStack.Reverse(); + } + } + + /// + /// Update diagnostics configuration. + /// + /// Diagnostics visualization object. + public void UpdateSettings(PipelineDiagnosticsVisualizationObject visualizationObject) + { + // convert colors to MSAGL graph colors + Func colorFromMediaColor = (System.Windows.Media.Color color) => new Color(color.R, color.G, color.B); + this.model.VisualizationObject = visualizationObject; + this.EdgeColor = colorFromMediaColor(visualizationObject.EdgeColor); + this.HighlightColor = colorFromMediaColor(visualizationObject.HighlightColor); + this.NodeColor = colorFromMediaColor(visualizationObject.NodeColor); + this.SourceNodeColor = colorFromMediaColor(visualizationObject.SourceNodeColor); + this.SubpipelineColor = colorFromMediaColor(visualizationObject.SubpipelineColor); + this.ConnectorColor = colorFromMediaColor(visualizationObject.ConnectorColor); + this.JoinColor = colorFromMediaColor(visualizationObject.JoinColor); + this.HeatmapColorBase = colorFromMediaColor(visualizationObject.HeatmapColor); + this.InfoTextSize = visualizationObject.InfoTextSize; + this.ShowExporterConnections = visualizationObject.ShowExporterConnections; + this.LabelColorLight = Color.White; + this.LabelColorDark = Color.Black; + if (visualizationObject.ModelDirty) + { + this.model.Reset(); + visualizationObject.ModelDirty = false; + this.VisualGraph = null; + this.view.Update(true); + } + + if (this.model.Graph != null) + { + this.UpdateGraph(this.model.Graph, true); + } + } + + /// + /// Update diagnostics graph. + /// + /// Current diagnostics graph. + /// Force re-layout of graph (otherwise, updates labels, colors, etc. in place). + public void UpdateGraph(PipelineDiagnostics graph, bool forceRelayout) + { + this.model.Graph = graph; + if (graph == null) + { + this.VisualGraph = null; + } + else + { + var pipelineIdToPipelineDiagnostics = graph.GetAllPipelineDiagnostics().ToDictionary(p => p.Id); + var currentGraph = this.Breadcrumbs.Count() > 0 ? pipelineIdToPipelineDiagnostics[this.Breadcrumbs.Last()] : graph; + this.VisualGraph = this.BuildVisualGraph(currentGraph, pipelineIdToPipelineDiagnostics); + } + + this.view.Update(forceRelayout); + } + + /// + /// Update selected edge. + /// + /// Selected edge. + public void UpdateSelectedEdge(Edge edge) + { + if (edge == null) + { + // clear selected edge (if any) + this.model.SelectedEdgeId = -1; + this.view.Update(true); + return; + } + + var input = edge.UserData as PipelineDiagnostics.ReceiverDiagnostics; + if (input != null) + { + this.UpdateSelectedEdge(input, this.VisualGraph, true); + } + } + + /// + /// Navigate into subgraph. + /// + /// Subgraph ID into which to navigate. + public void NavInto(int subgraphId) + { + this.model.NavStack.Push(subgraphId); + this.UpdateGraph(this.model.Graph, true); + } + + /// + /// Navigate back one graph. + /// + public void NavBack() + { + if (this.model.NavStack.Count > 0) + { + this.model.NavStack.Pop(); + this.UpdateGraph(this.model.Graph, true); + } + } + + /// + /// Navigate back to given graph. + /// + /// Graph Id. + public void NavBackTo(int id) + { + while (this.model.NavStack.Count > 0 && this.model.NavStack.Peek() != id) + { + this.model.NavStack.Pop(); + } + + this.UpdateGraph(this.model.Graph, true); + } + + /// + /// Navigate back to root graph. + /// + public void NavHome() + { + if (this.model.NavStack.Count > 0) + { + while (this.model.NavStack.Count > 0) + { + this.model.NavStack.Pop(); + } + + this.UpdateGraph(this.model.Graph, true); + } + } + + private static Edge GetEdgeById(int id, Graph graph) + { + foreach (var n in graph.Nodes) + { + foreach (var e in n.Edges) + { + if (e.UserData != null && ((PipelineDiagnostics.ReceiverDiagnostics)e.UserData).Id == id) + { + return e; + } + } + } + + return null; + } + + private static bool IsBridgeToExporter(PipelineDiagnostics.PipelineElementDiagnostics node) + { + var bridgeEmitters = node.ConnectorBridgeToPipelineElement.Emitters; + var typeName = bridgeEmitters.Length == 1 ? bridgeEmitters[0].PipelineElement.TypeName : string.Empty; + return typeName == "MessageConnector`1" || typeName == "MessageEnvelopeConnector`1"; + } + + private void UpdateSelectedEdge(PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, bool clicked) + { + var edge = GetEdgeById(input.Id, graph); + if (clicked && this.model.SelectedEdgeId == input.Id) + { + // toggle unselected + edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; // unselect current + this.model.SelectedEdgeDetails = string.Empty; + this.model.SelectedEdgeId = -1; + this.view.Update(true); + return; + } + + // new edge selected + if (this.model.SelectedEdgeId != -1) + { + var previousEdge = GetEdgeById(this.model.SelectedEdgeId, graph); + if (previousEdge != null) + { + previousEdge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; // unselect previous + } + } + + edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness * 2; // select current + this.model.SelectedEdgeId = input.Id; + var sb = new StringBuilder(); + sb.Append($"Type: {TypeSpec.Simplify(input.TypeName)}" + Environment.NewLine); + sb.Append($"Message Size (avg): {input.MessageSize:0}" + Environment.NewLine); + sb.Append($"Queue Size: {input.QueueSize:0.###}" + Environment.NewLine); + sb.Append($"Processed Count: {input.ProcessedCount}" + Environment.NewLine); + sb.Append($"Processed/Time: {input.ProcessedPerTimeSpan:0.###}" + Environment.NewLine); + sb.Append($"Dropped Count: {input.DroppedCount}" + Environment.NewLine); + sb.Append($"Dropped/Time: {input.DroppedPerTimeSpan:0.###}" + Environment.NewLine); + sb.Append($"Latency at Emitter (avg): {input.MessageLatencyAtEmitter:0.###}ms" + Environment.NewLine); + sb.Append($"Latency at Receiver (avg): {input.MessageLatencyAtReceiver:0.###}ms" + Environment.NewLine); + sb.Append($"Processing Time (avg): {input.ProcessingTime:0.###}ms" + Environment.NewLine); + sb.Append($"Delivery Policy: {input.DeliveryPolicyName}" + Environment.NewLine); + this.model.SelectedEdgeDetails = sb.ToString(); + this.view.Update(clicked); + } + + private Func StatsSelector(bool heatmap) + { + switch (this.model.VisualizationObject.HeatmapStatistics) + { + case PipelineDiagnosticsVisualizationObject.HeatmapStats.None: + return null; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.LatencyAtEmitter: + return i => i.MessageLatencyAtEmitter; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.LatencyAtReceiver: + return i => i.MessageLatencyAtReceiver; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.Processing: + return i => i.ProcessingTime; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.Throughput: + return i => i.ProcessedCount; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.QueueSize: + return i => i.QueueSize; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.DroppedCount: + return i => i.DroppedCount; + case PipelineDiagnosticsVisualizationObject.HeatmapStats.MessageSize: + return i => + { + var avg = i.MessageSize; + return heatmap && avg > 0 ? Math.Log(avg) : avg; + }; + default: + throw new ArgumentException($"Unknown visualization selector type."); + } + } + + private Color LabelColor(Color background) + { + var r = background.R / 255.0; + var g = background.G / 255.0; + var b = background.B / 255.0; + var brightness = Math.Sqrt((0.299 * r * r) + (0.587 * g * g) + (0.114 * b * b)); + return brightness < 0.55 ? this.LabelColorLight : this.LabelColorDark; + } + + private bool HilightEdge(PipelineDiagnostics.ReceiverDiagnostics receiverDiagnostics) + { + switch (this.model.VisualizationObject.Highlight) + { + case PipelineDiagnosticsVisualizationObject.HighlightCondition.None: + return false; + case PipelineDiagnosticsVisualizationObject.HighlightCondition.UnlimitedDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.Unlimited)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.LatestMessageDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.LatestMessage)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.ThrottleDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.Throttle)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.SynchronousOrThrottleDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.SynchronousOrThrottle)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.LatencyConstrainedDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.LatencyConstrained)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.QueueSizeConstrainedDeliveryPolicy: + return receiverDiagnostics.DeliveryPolicyName.StartsWith(nameof(DeliveryPolicy.QueueSizeConstrained)); + case PipelineDiagnosticsVisualizationObject.HighlightCondition.ThrottledReceivers: + return receiverDiagnostics.Throttled; + default: + throw new ArgumentException($"Unknown highlight condition: {this.model.VisualizationObject.Highlight}"); + } + } + + private Color HeatmapColor(double stats, Color baseColor) + { + var baseR = baseColor.R * (1.0 - stats); + var baseG = baseColor.G * (1.0 - stats); + var baseB = baseColor.B * (1.0 - stats); + var heatR = this.HeatmapColorBase.R * stats; + var heatG = this.HeatmapColorBase.G * stats; + var heatB = this.HeatmapColorBase.B * stats; + return new Color((byte)(baseR + heatR), (byte)(baseG + heatG), (byte)(baseB + heatB)); // blend from base to heat color + } + + private void HeatmapStats(Graph graph, Func statsSelector, bool perNode) + { + if (graph.Edges.Count() == 0) + { + return; + } + + var edgeStats = graph.Edges.Where(e => e.UserData != null).Select(e => (e, statsSelector((PipelineDiagnostics.ReceiverDiagnostics)e.UserData))); + var max = edgeStats.Select(x => x.Item2).Max(); + + if (perNode) + { + // max edge per node + foreach (var node in graph.Nodes) + { + var inputs = node.InEdges; + if (inputs.Count() > 0) + { + var maxStats = node.InEdges.Where(e => e.UserData != null).Select(e => statsSelector((PipelineDiagnostics.ReceiverDiagnostics)e.UserData)).Max(); + var color = this.HeatmapColor(max > 0 ? maxStats / max : 0, this.NodeColor); + node.Attr.Color = color; + node.Attr.FillColor = color; + node.Label.FontColor = this.LabelColor(color); + } + } + } + else + { + // per edge + foreach (var (edge, stat) in edgeStats) + { + edge.Attr.Color = this.HeatmapColor(max > 0 ? stat / max : 0, this.EdgeColor); + } + } + } + + private void VisualizeEdgeColoring(Graph graph) + { + var selector = this.StatsSelector(true); + if (selector != null) + { + // visualize heatmap + var perNode = this.model.VisualizationObject.HeatmapStatistics == PipelineDiagnosticsVisualizationObject.HeatmapStats.Processing; + this.HeatmapStats(graph, selector, perNode); + } + + // overlay highlights + if (this.model.VisualizationObject.Highlight != PipelineDiagnosticsVisualizationObject.HighlightCondition.None) + { + foreach (var edge in graph.Edges) + { + if (edge.UserData != null && this.HilightEdge((PipelineDiagnostics.ReceiverDiagnostics)edge.UserData)) + { + edge.Attr.Color = this.HighlightColor; + } + } + } + } + + private bool IsConnectorBridge(PipelineDiagnostics.PipelineElementDiagnostics node) + { + return node.Kind == PipelineElementKind.Connector && (node.Receivers.Length == 0 || node.Emitters.Length == 0) && node.ConnectorBridgeToPipelineElement != null; + } + + private Node BuildVisualNode(PipelineDiagnostics.PipelineElementDiagnostics node) + { + var vis = new Node($"n{node.Id}"); + var fillColor = node.Kind == PipelineElementKind.Source ? this.SourceNodeColor : node.Kind == PipelineElementKind.Subpipeline ? this.SubpipelineColor : this.NodeColor; + var typ = TypeSpec.Simplify(node.TypeName); + var isStoppedSubpipeline = node.RepresentsSubpipeline != null && !node.RepresentsSubpipeline.IsPipelineRunning; + var stopped = isStoppedSubpipeline || !node.IsRunning ? " (stopped)" : string.Empty; + vis.LabelText = node.Kind == PipelineElementKind.Subpipeline ? $"{node.Name}{stopped}|{typ}" : typ; + vis.Label.FontColor = this.LabelColor(fillColor); + vis.Attr.Color = fillColor; + vis.Attr.FillColor = fillColor; + if (vis.LabelText == "Join") + { + this.SetJoinVisualAttributes(vis, node.Name); + } + + return vis; + } + + private string BuildVisualEdgeLabelText(string emitterName, string receiverName, string stats, string deliveryPolicyName) + { + var showEmitter = this.model.VisualizationObject.ShowEmitterNames; + var showReceiver = this.model.VisualizationObject.ShowReceiverNames; + var showDeliveryPolicy = this.model.VisualizationObject.ShowDeliveryPolicies; + emitterName = showEmitter ? emitterName : string.Empty; + receiverName = showReceiver ? receiverName : string.Empty; + deliveryPolicyName = showDeliveryPolicy && deliveryPolicyName.Length > 0 ? $" [{deliveryPolicyName}]" : string.Empty; + var arrow = showEmitter && showReceiver ? "→" : string.Empty; + return $" {emitterName}{arrow}{receiverName}{stats}{deliveryPolicyName} "; // extra padding to allow for stats changes without re-layout + } + + private Edge BuildVisualEdge(Node source, Node target, string emitterName, string receiverName, string stats, string deliveryPolicyName, Style style) + { + var edge = new Edge(source, target, ConnectionToGraph.Connected); + edge.Attr.Color = this.EdgeColor; + edge.Attr.LineWidth = this.model.VisualizationObject.EdgeLineThickness; + edge.Attr.AddStyle(style); + edge.LabelText = this.BuildVisualEdgeLabelText(emitterName, receiverName, stats, deliveryPolicyName); + edge.Label.FontColor = this.LabelColorLight; + return edge; + } + + private Edge BuildVisualEdge(Node source, Node target, PipelineDiagnostics.ReceiverDiagnostics input, Func statsSelector) + { + var stats = statsSelector != null ? $" ({statsSelector(input):0.#})" : string.Empty; + var edge = this.BuildVisualEdge(source, target, input.Source.Name, input.ReceiverName, stats, input.DeliveryPolicyName, Style.Solid); + edge.UserData = input; + edge.Label.UserData = edge; + return edge; + } + + private bool AddVisualEdge(Node source, Node target, PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, Func statsSelector) + { + if (source != null && target != null) + { + var edge = this.BuildVisualEdge(source, target, input, statsSelector); + graph.AddPrecalculatedEdge(edge); + if (input.Id == this.model.SelectedEdgeId) + { + this.UpdateSelectedEdge(input, graph, false); + return true; + } + } + + return false; + } + + private bool AddVisualEdge(int sourceId, int targetId, PipelineDiagnostics.ReceiverDiagnostics input, Graph graph, Func statsSelector) + { + return this.AddVisualEdge(graph.FindNode($"n{sourceId}"), graph.FindNode($"n{targetId}"), input, graph, statsSelector); + } + + private void SetVisualAttributes(Node vis, Shape shape, Color color, string symbol, string name) + { + vis.Attr.Color = color; + vis.Attr.FillColor = color; + vis.Label.FontColor = this.LabelColor(color); + vis.Attr.Shape = shape; + vis.LabelText = $"{symbol}|{name}"; + } + + private void SetConnectorVisualAttributes(Node vis, string label) + { + this.SetVisualAttributes(vis, Shape.Circle, this.ConnectorColor, "☍", label); + } + + private void SetJoinVisualAttributes(Node vis, string label) + { + this.SetVisualAttributes(vis, Shape.Circle, this.JoinColor, "+", label); + } + + private Graph BuildVisualGraph(PipelineDiagnostics diagnostics, Dictionary pipelineIdToPipelineDiagnostics) + { + var subpipelineIdToPipelineDiagnostics = diagnostics.SubpipelineDiagnostics.ToDictionary(p => p.Id); + var graph = new Graph($"{diagnostics.Name} (running={diagnostics.IsPipelineRunning})", $"g{diagnostics.Id}"); + switch (this.model.VisualizationObject.LayoutDirection) + { + case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.LeftToRight: + graph.Attr.LayerDirection = LayerDirection.LR; + break; + case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.TopToBottom: + graph.Attr.LayerDirection = LayerDirection.TB; + break; + case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.RightToLeft: + graph.Attr.LayerDirection = LayerDirection.RL; + break; + case PipelineDiagnosticsVisualizationObject.GraphLayoutDirection.BottomToTop: + graph.Attr.LayerDirection = LayerDirection.BT; + break; + } + + graph.UserData = diagnostics.Id; + graph.Attr.BackgroundColor = Color.Transparent; + var subpipelineNodes = new Dictionary(); + var connectorsWithinSubpipelines = new Dictionary(); + var statsSelector = this.StatsSelector(false); + + // add nodes + foreach (var node in diagnostics.PipelineElements.Where(n => !this.IsConnectorBridge(n))) + { + var vis = this.BuildVisualNode(node); + if (node.Kind == PipelineElementKind.Subpipeline && node.RepresentsSubpipeline != null) + { + vis.UserData = node.RepresentsSubpipeline; + subpipelineNodes.Add(node.RepresentsSubpipeline.Id, node); + foreach (var n in node.RepresentsSubpipeline.PipelineElements.Where(n => n.Kind == PipelineElementKind.Connector)) + { + connectorsWithinSubpipelines.Add(n.Id, n); + } + } + else if (node.Kind == PipelineElementKind.Connector) + { + this.SetConnectorVisualAttributes(vis, node.Name); + } + + graph.AddNode(vis); + } + + // add connectors + foreach (var node in diagnostics.PipelineElements.Where(this.IsConnectorBridge)) + { + var connectsToSubpipeline = subpipelineNodes.ContainsKey(node.ConnectorBridgeToPipelineElement.PipelineId); + if (!connectsToSubpipeline) + { + if (!this.ShowExporterConnections && IsBridgeToExporter(node)) + { + continue; + } + + var connector = new Node($"n{node.Id}"); + var bridgedPipeline = pipelineIdToPipelineDiagnostics[node.ConnectorBridgeToPipelineElement.PipelineId]; + this.SetConnectorVisualAttributes(connector, $"{node.Name} ({bridgedPipeline.Name})"); + graph.AddNode(connector); + } + } + + // add edges + var selectedEdgeUpdated = false; + foreach (var n in diagnostics.PipelineElements) + { + foreach (var i in n.Receivers) + { + if (i.Source != null) + { + if (this.AddVisualEdge(i.Source.PipelineElement.Id, n.Id, i, graph, statsSelector)) + { + selectedEdgeUpdated = true; + } + } + } + } + + // add connector bridge edges + foreach (var n in diagnostics.PipelineElements.Where(this.IsConnectorBridge)) + { + if (!this.ShowExporterConnections && IsBridgeToExporter(n)) + { + continue; + } + + // connector bridging to subpipeline? + if (subpipelineNodes.TryGetValue(n.ConnectorBridgeToPipelineElement.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics subNode)) + { + // edges from connector source directly to bridge target (subpipeline) + var sub = graph.FindNode($"n{subNode.Id}"); + if (sub != null) + { + foreach (var i in n.Receivers) + { + if (i.Source != null) + { + var source = graph.FindNode($"n{i.Source.PipelineElement.Id}"); + if (source != null) + { + if (this.AddVisualEdge(source, sub, i, graph, statsSelector)) + { + selectedEdgeUpdated = true; + } + } + } + } + + // edges from connector bridge source (subpipeline) to connector targets + foreach (var o in n.Emitters) + { + foreach (var t in o.Targets) + { + var target = graph.FindNode($"n{t.PipelineElement.Id}"); + if (target != null) + { + if (this.AddVisualEdge(sub, target, t, graph, statsSelector)) + { + selectedEdgeUpdated = true; + } + } + } + } + } + } + else + { + // connector bridging graphs + var bridgedPipeline = pipelineIdToPipelineDiagnostics[n.ConnectorBridgeToPipelineElement.PipelineId]; + var connector = graph.FindNode($"n{n.Id}"); + + // add dotted line edge representing connector bridged to descendant pipeline + var targetPipeline = bridgedPipeline; + while (targetPipeline != null) + { + if (subpipelineIdToPipelineDiagnostics.ContainsKey(targetPipeline.Id)) + { + var targetNode = graph.FindNode($"n{subpipelineNodes[targetPipeline.Id].Id}"); + graph.AddPrecalculatedEdge(this.BuildVisualEdge(connector, targetNode, n.Name, bridgedPipeline.Name, string.Empty, string.Empty, Style.Dotted)); + break; + } + + // walk up ancestor chain until we're at a direct child subpipeline + targetPipeline = targetPipeline.ParentPipelineDiagnostics; + } + } + } + + // add connector bridge edges between descendants (shown between current-level subpiplines) + int? TryFindCurrentLevelAncestorSubpipelineId(int id) + { + foreach (var ancestor in pipelineIdToPipelineDiagnostics[id].AncestorPipelines) + { + if (subpipelineNodes.TryGetValue(ancestor.Id, out PipelineDiagnostics.PipelineElementDiagnostics subpipeline)) + { + return subpipeline.Id; + } + } + + return null; + } + + foreach (var descendantConnector in diagnostics.GetAllPipelineElementDiagnostics().Where(this.IsConnectorBridge)) + { + if (descendantConnector.Emitters.Length == 0 /* source-side of connector pair */) + { + var sourceId = descendantConnector.PipelineId; + var targetId = descendantConnector.ConnectorBridgeToPipelineElement.PipelineId; + var sourceCurrentLevelId = TryFindCurrentLevelAncestorSubpipelineId(sourceId); + var targetCurrentLevelId = TryFindCurrentLevelAncestorSubpipelineId(targetId); + if (sourceCurrentLevelId != null && targetCurrentLevelId != null && sourceCurrentLevelId != targetCurrentLevelId) + { + var sourceNode = graph.FindNode($"n{sourceCurrentLevelId}"); + var targetNode = graph.FindNode($"n{targetCurrentLevelId}"); + graph.AddPrecalculatedEdge(this.BuildVisualEdge(sourceNode, targetNode, string.Empty, descendantConnector.Name, string.Empty, string.Empty, Style.Dotted)); + } + } + } + + // add direct connections from one subpipeline (connector) to another + foreach (var c in connectorsWithinSubpipelines.Values) + { + if (c.ConnectorBridgeToPipelineElement != null) + { + if (c.ConnectorBridgeToPipelineElement.PipelineId == diagnostics.Id && c.ConnectorBridgeToPipelineElement.Receivers.Length == 1) + { + var i = c.ConnectorBridgeToPipelineElement.Receivers[0]; + if (i.Source != null && i.Source.PipelineElement.PipelineId == diagnostics.Id && i.Source.PipelineElement.ConnectorBridgeToPipelineElement != null) + { + if (subpipelineNodes.TryGetValue(i.Source.PipelineElement.ConnectorBridgeToPipelineElement.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics source) && + subpipelineNodes.TryGetValue(c.PipelineId, out PipelineDiagnostics.PipelineElementDiagnostics target)) + { + if (this.AddVisualEdge(source.Id, target.Id, i, graph, statsSelector)) + { + selectedEdgeUpdated = true; + } + } + } + } + } + } + + if (!selectedEdgeUpdated && this.model.SelectedEdgeId != -1) + { + // hide while in subpipeline + this.model.SelectedEdgeDetails = string.Empty; + } + + this.VisualizeEdgeColoring(graph); + return graph; + } + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/TypeSpec.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/TypeSpec.cs index 73a31ad99..50e03acac 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/TypeSpec.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/DiagnosticsVisualization/TypeSpec.cs @@ -116,7 +116,7 @@ private static IEnumerable Lex(string value) { if (i > j) { - yield return new Token(TokenKind.Name, value.Substring(j, i - j)); // preceeding name, if any + yield return new Token(TokenKind.Name, value.Substring(j, i - j)); // preceding name, if any } j = i + 1; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/ImageVisualizationObjectView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/ImageVisualizationObjectView.xaml.cs index b7ffc73b5..d9f440ceb 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/ImageVisualizationObjectView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/ImageVisualizationObjectView.xaml.cs @@ -16,7 +16,7 @@ namespace Microsoft.Psi.Visualization.Views.Visuals2D public partial class ImageVisualizationObjectView : UserControl { // A blank image to display when there is no data. - private static Shared blankImage = Shared.Create(new Imaging.Image(0, 0, Imaging.PixelFormat.Undefined)); + private static readonly Shared BlankImage = Shared.Create(new Imaging.Image(100, 100, Imaging.PixelFormat.BGR_24bpp)); /// /// Initializes a new instance of the class. @@ -66,20 +66,16 @@ private void ShowCurrentImage() if (image == null) { // There's no data, so use the default blank image - using (Shared defaultImage = blankImage.AddRef()) - { - this.DisplayImage.UpdateImage(defaultImage); - } + using Shared defaultImage = BlankImage.AddRef(); + this.DisplayImage.UpdateImage(defaultImage); } else if (this.ImageVisualizationObject.HorizontalFlip) { // Flip the image before displaying it - Bitmap bitmap = image.Resource.ToManagedImage(true); + Bitmap bitmap = image.Resource.ToBitmap(true); bitmap.RotateFlip(RotateFlipType.RotateNoneFlipX); - using (Shared flippedImage = Shared.Create(Imaging.Image.FromManagedImage(bitmap))) - { - this.DisplayImage.UpdateImage(flippedImage); - } + using Shared flippedImage = Shared.Create(Imaging.Image.FromBitmap(bitmap)); + this.DisplayImage.UpdateImage(flippedImage); } else { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/PlotVisualizationObjectView{TPlotVisualizationObject,TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/PlotVisualizationObjectView{TPlotVisualizationObject,TData}.cs index c58b18226..b2c65730f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/PlotVisualizationObjectView{TPlotVisualizationObject,TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/PlotVisualizationObjectView{TPlotVisualizationObject,TData}.cs @@ -456,7 +456,7 @@ public Segment(PlotVisualizationObjectView pare public DateTime? StartTime { get; set; } - public DateTime EndTime { get; set; } = DateTime.MaxValue; // Start out at MaxValue so that an intial empty segment doesn't get removed from a data range change + public DateTime EndTime { get; set; } = DateTime.MaxValue; // Start out at MaxValue so that an initial empty segment doesn't get removed from a data range change public void AddPoint(DateTime time, Point point) { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/TimeIntervalHistoryVisualizationObjectViewItem.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/TimeIntervalHistoryVisualizationObjectViewItem.cs index 8d1ec7b25..44ec6739f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/TimeIntervalHistoryVisualizationObjectViewItem.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals2D/TimeIntervalHistoryVisualizationObjectViewItem.cs @@ -77,7 +77,7 @@ internal TimeIntervalHistoryVisualizationObjectViewItem(TimeIntervalHistoryVisua /// The data to update the item from. internal void Update(TimeIntervalHistoryVisualizationObject.TimeIntervalVisualizationObjectData data) { - // determine the correspondance of 5 pixels in the time space + // determine the correspondence of 5 pixels in the time space var offset = 10.0 / this.parent.ScaleTransform.ScaleX; var verticalSpace = this.parent.VisualizationObject.LineWidth * 4 / this.parent.ScaleTransform.ScaleY; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectDepthVisual.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/DepthImageAs3DMeshVisual3D.cs similarity index 85% rename from Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectDepthVisual.cs rename to Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/DepthImageAs3DMeshVisual3D.cs index 5384eb5a1..543e53598 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectDepthVisual.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/DepthImageAs3DMeshVisual3D.cs @@ -10,30 +10,29 @@ namespace Microsoft.Psi.Visualization.Views.Visuals3D using System.Windows.Media; using System.Windows.Media.Media3D; using Microsoft.Psi.Imaging; - using Microsoft.Psi.Visualization.Extensions; using Microsoft.Psi.Visualization.VisualizationObjects; /// - /// Represents a Kinect depth visual. + /// Represents a depth image visual. /// - public class KinectDepthVisual : ModelVisual3D + public class DepthImageAs3DMeshVisual3D : ModelVisual3D { - private KinectDepth3DVisualizationObject visualizationObject; + private DepthImageAs3DMeshVisualizationObject visualizationObject; private MeshGeometry3D meshGeometry; private Point3D[] depthFramePoints; private int[] rawDepth; /// - /// Initializes a new instance of the class. + /// Initializes a new instance of the class. /// - /// The Kinect depth 3D visualization object. - public KinectDepthVisual(KinectDepth3DVisualizationObject visualizationObject) + /// The depth image 3D visualization object. + public DepthImageAs3DMeshVisual3D(DepthImageAs3DMeshVisualizationObject visualizationObject) { this.visualizationObject = visualizationObject; this.visualizationObject.PropertyChanged += this.VisualizationObject_PropertyChanged; } - private void UpdateMesh(Image depthImage) + private void UpdateMesh(DepthImage depthImage) { if (this.depthFramePoints?.Length != (depthImage.Width * depthImage.Height)) { @@ -68,18 +67,6 @@ private void VisualizationObject_PropertyChanged(object sender, PropertyChangedE } } - private double ConvertRawDepthToMeters(int rawDepth) - { - // http://nicolas.burrus.name/index.php/Research/KinectCalibration - // http://www.ros.org/wiki/kinect_node - if (rawDepth < 2047) - { - return 1.0 / ((rawDepth * -0.0030711016) + 3.3309495161); - } - - return 0; - } - private void CreateMesh(int width, int height, double depthDifferenceTolerance = 200) { this.meshGeometry = new MeshGeometry3D(); @@ -127,7 +114,7 @@ private void CreateMesh(int width, int height, double depthDifferenceTolerance = (this.Content as GeometryModel3D).BackMaterial = material; } - private void UpdateDepth(Image depthImage) + private void UpdateDepth(DepthImage depthImage) { int width = depthImage.Width; int height = depthImage.Height; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectBodiesVisual.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectBodiesVisual.cs deleted file mode 100644 index 36d76a962..000000000 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/Visuals3D/KinectBodiesVisual.cs +++ /dev/null @@ -1,313 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Visualization.Views.Visuals3D -{ - using System; - using System.Collections.Generic; - using System.ComponentModel; - using System.Linq; - using System.Windows.Media; - using System.Windows.Media.Media3D; - using HelixToolkit.Wpf; - using MathNet.Spatial.Euclidean; - using Microsoft.Kinect; - using Microsoft.Psi.Kinect; - using Microsoft.Psi.Visualization.VisualizationObjects; - - /// - /// Represents a Kinect bodies visual. - /// - public class KinectBodiesVisual : ModelVisual3D - { - private static readonly int SphereDiv = 5; - private static readonly int PipeDiv = 7; - private Plane horizontalPlane = new Plane(new MathNet.Spatial.Euclidean.Point3D(0, 0, 0), UnitVector3D.Create(0, 0, 1)); - private int numBodies = 0; - - private KinectBodies3DVisualizationObject visualizationObject; - private Dictionary> bodyJoints = new Dictionary>(); - private Dictionary> bodyJointsTracked = new Dictionary>(); - private Dictionary, PipeVisual3D>> bodyBones = new Dictionary, PipeVisual3D>>(); - private Dictionary, bool>> bodyBonesTracked = new Dictionary, bool>>(); - private Dictionary trackingIdBillboards = new Dictionary(); - - private Brush trackedEntitiesBrush = new SolidColorBrush(); - private Brush untrackedEntitiesBrush = new SolidColorBrush(); - - /// - /// Initializes a new instance of the class. - /// - /// - /// The Kinect bodies 3D visualization object. - public KinectBodiesVisual(KinectBodies3DVisualizationObject visualizationObject) - { - this.visualizationObject = visualizationObject; - this.visualizationObject.PropertyChanged += this.VisualizationObject_PropertyChanged; - } - - private void ClearAll() - { - this.bodyJoints.Clear(); - this.bodyJointsTracked.Clear(); - this.bodyBones.Clear(); - this.bodyBonesTracked.Clear(); - this.trackingIdBillboards.Clear(); - this.Children.Clear(); - } - - private void AddBody(ulong trackingId) - { - List joints = new List(); - List jointsTracked = new List(); - for (int i = 0; i < Body.JointCount; i++) - { - var bodyPart = new SphereVisual3D() - { - ThetaDiv = SphereDiv, - PhiDiv = SphereDiv, - }; - - joints.Add(bodyPart); - jointsTracked.Add(true); - this.Children.Add(bodyPart); - } - - this.bodyJoints.Add(trackingId, joints); - this.bodyJointsTracked.Add(trackingId, jointsTracked); - - var bones = new Dictionary, PipeVisual3D> - { - { Tuple.Create(JointType.HandTipLeft, JointType.HandLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ThumbLeft, JointType.HandLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.HandLeft, JointType.WristLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.WristLeft, JointType.ElbowLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ElbowLeft, JointType.ShoulderLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ShoulderLeft, JointType.SpineShoulder), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.HandTipRight, JointType.HandRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ThumbRight, JointType.HandRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.HandRight, JointType.WristRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.WristRight, JointType.ElbowRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ElbowRight, JointType.ShoulderRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.ShoulderRight, JointType.SpineShoulder), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.Head, JointType.Neck), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.Neck, JointType.SpineShoulder), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.SpineShoulder, JointType.SpineBase), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.SpineBase, JointType.HipRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.HipRight, JointType.KneeRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.KneeRight, JointType.AnkleRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.AnkleRight, JointType.FootRight), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.SpineBase, JointType.HipLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.HipLeft, JointType.KneeLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.KneeLeft, JointType.AnkleLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - { Tuple.Create(JointType.AnkleLeft, JointType.FootLeft), new PipeVisual3D() { ThetaDiv = PipeDiv } }, - }; - var bonesTracked = new Dictionary, bool> - { - { Tuple.Create(JointType.HandTipLeft, JointType.HandLeft), true }, - { Tuple.Create(JointType.ThumbLeft, JointType.HandLeft), true }, - { Tuple.Create(JointType.HandLeft, JointType.WristLeft), true }, - { Tuple.Create(JointType.WristLeft, JointType.ElbowLeft), true }, - { Tuple.Create(JointType.ElbowLeft, JointType.ShoulderLeft), true }, - { Tuple.Create(JointType.ShoulderLeft, JointType.SpineShoulder), true }, - { Tuple.Create(JointType.HandTipRight, JointType.HandRight), true }, - { Tuple.Create(JointType.ThumbRight, JointType.HandRight), true }, - { Tuple.Create(JointType.HandRight, JointType.WristRight), true }, - { Tuple.Create(JointType.WristRight, JointType.ElbowRight), true }, - { Tuple.Create(JointType.ElbowRight, JointType.ShoulderRight), true }, - { Tuple.Create(JointType.ShoulderRight, JointType.SpineShoulder), true }, - { Tuple.Create(JointType.Head, JointType.Neck), true }, - { Tuple.Create(JointType.Neck, JointType.SpineShoulder), true }, - { Tuple.Create(JointType.SpineShoulder, JointType.SpineBase), true }, - { Tuple.Create(JointType.SpineBase, JointType.HipRight), true }, - { Tuple.Create(JointType.HipRight, JointType.KneeRight), true }, - { Tuple.Create(JointType.KneeRight, JointType.AnkleRight), true }, - { Tuple.Create(JointType.AnkleRight, JointType.FootRight), true }, - { Tuple.Create(JointType.SpineBase, JointType.HipLeft), true }, - { Tuple.Create(JointType.HipLeft, JointType.KneeLeft), true }, - { Tuple.Create(JointType.KneeLeft, JointType.AnkleLeft), true }, - { Tuple.Create(JointType.AnkleLeft, JointType.FootLeft), true }, - }; - - foreach (var b in bones) - { - this.Children.Add(b.Value); - } - - this.bodyBones.Add(trackingId, bones); - this.bodyBonesTracked.Add(trackingId, bonesTracked); - - // add the billboard - var billboard = new BillboardTextVisual3D() - { - // Background = new SolidColorBrush(Color.FromArgb(255, 70, 85, 198)), - Background = Brushes.Gray, - Foreground = new SolidColorBrush(Colors.White), - Padding = new System.Windows.Thickness(5), - Text = $"Kinect Id: {this.numBodies++}", - }; - this.trackingIdBillboards.Add(trackingId, billboard); - this.Children.Add(billboard); - - this.UpdateProperties(); - } - - private void RemoveBody(ulong trackingId) - { - // remove joints - foreach (var joint in this.bodyJoints[trackingId]) - { - this.Children.Remove(joint); - } - - // remove bones - foreach (var bone in this.bodyBones[trackingId].Values) - { - this.Children.Remove(bone); - } - - this.Children.Remove(this.trackingIdBillboards[trackingId]); - - this.bodyJoints.Remove(trackingId); - this.bodyJointsTracked.Remove(trackingId); - this.bodyBones.Remove(trackingId); - this.bodyBonesTracked.Remove(trackingId); - this.trackingIdBillboards.Remove(trackingId); - } - - private void UpdateProperties() - { - this.trackedEntitiesBrush = new SolidColorBrush(this.visualizationObject.Color); - var alphaColor = Color.FromArgb( - (byte)(this.visualizationObject.InferredJointsOpacity * 255), - this.visualizationObject.Color.R, - this.visualizationObject.Color.G, - this.visualizationObject.Color.B); - this.untrackedEntitiesBrush = new SolidColorBrush(alphaColor); - - foreach (var body in this.bodyJoints) - { - for (int i = 0; i < body.Value.Count; i++) - { - body.Value[i].Radius = this.visualizationObject.Size; - body.Value[i].Fill = this.bodyJointsTracked[body.Key][i] ? this.trackedEntitiesBrush : this.untrackedEntitiesBrush; - } - } - - foreach (var body in this.bodyBones) - { - foreach (var bone in body.Value) - { - bone.Value.Diameter = this.visualizationObject.Size; - bone.Value.Fill = this.bodyBonesTracked[body.Key][bone.Key] ? this.trackedEntitiesBrush : this.untrackedEntitiesBrush; - } - } - - foreach (var billboard in this.trackingIdBillboards) - { - if (this.visualizationObject.ShowTrackingBillboards) - { - if (!this.Children.Contains(billboard.Value)) - { - this.Children.Add(billboard.Value); - } - } - else - { - if (this.Children.Contains(billboard.Value)) - { - this.Children.Remove(billboard.Value); - } - } - } - } - - private void UpdateBodies(List kinectBodies) - { - if (kinectBodies == null) - { - this.ClearAll(); - return; - } - - // add any missing bodies - foreach (var body in kinectBodies) - { - if (!this.bodyJoints.ContainsKey(body.TrackingId)) - { - this.AddBody(body.TrackingId); - } - } - - // remove any non-necessary bodies - var currentIds = this.bodyJoints.Select(kvp => kvp.Key).ToArray(); - foreach (var id in currentIds) - { - if (!kinectBodies.Any(b => b.TrackingId == id)) - { - this.RemoveBody(id); - } - } - - // populate the bodies with information - for (int body = 0; body < kinectBodies.Count; body++) - { - var trackingId = kinectBodies[body].TrackingId; - var bodyTracked = kinectBodies[body].IsTracked; - for (int joint = 0; joint < Body.JointCount; joint++) - { - var jointTracked = - kinectBodies[body].Joints.ContainsKey((JointType)joint) && kinectBodies[body].Joints[(JointType)joint].TrackingState != TrackingState.NotTracked; - if (body < kinectBodies.Count && bodyTracked && jointTracked) - { - var jointPosition = kinectBodies[body].Joints[(JointType)joint].Position; - this.bodyJoints[trackingId][joint].Transform = new TranslateTransform3D(jointPosition.X, jointPosition.Y, jointPosition.Z); - this.bodyJointsTracked[trackingId][joint] = kinectBodies[body].Joints[(JointType)joint].TrackingState == TrackingState.Tracked; - } - else - { - this.bodyJointsTracked[trackingId][joint] = false; - } - } - - foreach (var bone in this.bodyBones[trackingId].Keys) - { - var boneTracked = kinectBodies[body].Joints[bone.Item1].TrackingState != TrackingState.NotTracked && kinectBodies[body].Joints[bone.Item2].TrackingState != TrackingState.NotTracked; - if (body < kinectBodies.Count && bodyTracked && boneTracked) - { - var joint1Position = kinectBodies[body].Joints[bone.Item1].Position; - var joint2Position = kinectBodies[body].Joints[bone.Item2].Position; - this.bodyBones[trackingId][bone].Point1 = new System.Windows.Media.Media3D.Point3D(joint1Position.X, joint1Position.Y, joint1Position.Z); - this.bodyBones[trackingId][bone].Point2 = new System.Windows.Media.Media3D.Point3D(joint2Position.X, joint2Position.Y, joint2Position.Z); - this.bodyBonesTracked[trackingId][bone] = kinectBodies[body].Joints[bone.Item1].TrackingState == TrackingState.Tracked && kinectBodies[body].Joints[bone.Item2].TrackingState == TrackingState.Tracked; - } - else - { - this.bodyBonesTracked[trackingId][bone] = false; - } - } - - // set billboard position - var spineBasePosition = kinectBodies[body].Joints[JointType.SpineBase].Position; - this.trackingIdBillboards[trackingId].Position = new System.Windows.Media.Media3D.Point3D( - spineBasePosition.X, - spineBasePosition.Y, - spineBasePosition.Z + 1); - } - } - - private void VisualizationObject_PropertyChanged(object sender, PropertyChangedEventArgs e) - { - if (e.PropertyName == nameof(KinectBodies3DVisualizationObject.CurrentValue)) - { - this.UpdateBodies(this.visualizationObject.CurrentValue.GetValueOrDefault().Data); - } - else if (e.PropertyName == nameof(this.visualizationObject.Size) || - e.PropertyName == nameof(this.visualizationObject.Color) || - e.PropertyName == nameof(this.visualizationObject.InferredJointsOpacity)) - { - this.UpdateProperties(); - } - } - } -} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYVisualizationPanelView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYVisualizationPanelView.xaml.cs index b42ba6911..b87e5dbd3 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYVisualizationPanelView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYVisualizationPanelView.xaml.cs @@ -20,7 +20,7 @@ public XYVisualizationPanelView() } /// - /// Gets ths visualization panel. + /// Gets the visualization panel. /// protected XYVisualizationPanel VisualizationPanel => (XYVisualizationPanel)this.DataContext; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml index be90c57b8..a96ebb384 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml @@ -17,12 +17,12 @@ - + - + diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml.cs index d38c145d3..337087a09 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/Views/XYZVisualizationPanelView.xaml.cs @@ -7,11 +7,8 @@ namespace Microsoft.Psi.Visualization.Views using System.Collections.Generic; using System.Collections.Specialized; using System.Windows; - using System.Windows.Input; - using System.Windows.Media; using System.Windows.Media.Animation; using System.Windows.Media.Media3D; - using Microsoft.Psi.Visualization.Views.Visuals3D; using Microsoft.Psi.Visualization.VisualizationObjects; using Microsoft.Psi.Visualization.VisualizationPanels; @@ -20,8 +17,6 @@ namespace Microsoft.Psi.Visualization.Views /// public partial class XYZVisualizationPanelView : VisualizationPanelViewBase { - private AnimatedModelVisual cameraLocation; - private bool follow = false; private Storyboard cameraStoryboard; /// @@ -31,13 +26,6 @@ public XYZVisualizationPanelView() { this.InitializeComponent(); this.DataContextChanged += this.XYZVisualizationPanelView_DataContextChanged; - CompositionTarget.Rendering += this.CompositionTarget_Rendering; - - // Register the view camera's name in the namescope so that it can be referenced in storyboard timelines - NameScope.SetNameScope(this, new NameScope()); - this.RegisterName(nameof(this.ViewCamera), this.ViewCamera); - - this.ViewPort3D.CameraChanged += this.ViewPort3D_CameraChanged; // Add a handler for the storyboard completed event. this.cameraStoryboard = this.FindResource("CameraStoryboard") as Storyboard; @@ -45,7 +33,7 @@ public XYZVisualizationPanelView() } /// - /// Gets ths visualization panel. + /// Gets the visualization panel. /// protected XYZVisualizationPanel VisualizationPanel => this.DataContext as XYZVisualizationPanel; @@ -61,19 +49,6 @@ private void RemoveVisualForVisualizationObject(VisualizationObject visualizatio this.SortingVisualRoot.Children.Remove(visual); } - private void ViewPort3D_CameraChanged(object sender, RoutedEventArgs e) - { - this.UpdateCameraInfoInPanel(); - } - - private void UpdateCameraInfoInPanel() - { - // Update the camera position info in the visualization panel for display in the property browser - this.VisualizationPanel.CameraPosition = this.ViewPort3D.Camera.Position; - this.VisualizationPanel.CameraLookDirection = this.ViewPort3D.Camera.LookDirection; - this.VisualizationPanel.CameraUpDirection = this.ViewPort3D.Camera.UpDirection; - } - private void XYZVisualizationPanelView_DataContextChanged(object sender, DependencyPropertyChangedEventArgs e) { if (this.VisualizationPanel != null) @@ -85,7 +60,6 @@ private void XYZVisualizationPanelView_DataContextChanged(object sender, Depende } this.VisualizationPanel.PropertyChanged += this.VisualizationPanel_PropertyChanged; - this.UpdateCameraInfoInPanel(); } } @@ -126,53 +100,6 @@ private void VisualizationObjects_CollectionChanged(object sender, NotifyCollect } } - private void CompositionTarget_Rendering(object sender, EventArgs e) - { - if (this.cameraLocation != null && this.follow) - { - var cameraController = this.ViewPort3D.CameraController; - var up = this.cameraLocation.TransformToAncestor(this.Root).Transform(this.cameraLocation.CameraTransform.Transform(new Point3D(0, 0, 1))); - var lookAt = this.cameraLocation.TransformToAncestor(this.Root).Transform(this.cameraLocation.CameraTransform.Transform(new Point3D(1, 0, 0))); - var lookFrom = this.cameraLocation.TransformToAncestor(this.Root).Transform(this.cameraLocation.CameraTransform.Transform(new Point3D(0, 0, 0))); - cameraController.CameraPosition = lookFrom; - cameraController.CameraLookDirection = lookAt - cameraController.CameraPosition; - this.ViewPort3D.Camera.UpDirection = new Vector3D(0, 0, 1); - } - } - - private void Grid_KeyDown(object sender, KeyEventArgs e) - { - // Follow mode - if (e.Key == Key.F) - { - this.follow = !this.follow; - this.cameraLocation = this.FindCameraLocation(this.Root); - } - } - - private AnimatedModelVisual FindCameraLocation(ModelVisual3D modelVisual3D) - { - if (modelVisual3D is AnimatedModelVisual model && model.IsCameraLocation) - { - return model; - } - - var children = modelVisual3D.Children; - foreach (var child in children) - { - if (child is ModelVisual3D) - { - var first = this.FindCameraLocation((ModelVisual3D)child); - if (first != null) - { - return first; - } - } - } - - return null; - } - private void AnimateCamera() { // Stop any existing storyboard that's executing. @@ -194,7 +121,7 @@ private void AnimateCamera() private void CameraStoryboard_Completed(object sender, EventArgs e) { - this.VisualizationPanel.CameraStoryboardCompleted(); + this.VisualizationPanel.NotifyCameraAnimationCompleted(); } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationContext.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationContext.cs index c50f56614..7b200bf66 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationContext.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationContext.cs @@ -34,7 +34,7 @@ namespace Microsoft.Psi.Visualization public class VisualizationContext : ObservableObject { private VisualizationContainer visualizationContainer; - private DatasetViewModel datasetViewModel; + private DatasetViewModel datasetViewModel = null; private DispatcherTimer liveStatusTimer = null; private List typeVisualizerActions = new List(); @@ -49,16 +49,13 @@ static VisualizationContext() private VisualizationContext() { - this.RegisterCustomSerializers(); - var booleanSchema = new AnnotationSchema("Boolean"); booleanSchema.AddSchemaValue(null, System.Drawing.Color.Gray); booleanSchema.AddSchemaValue("false", System.Drawing.Color.Red); booleanSchema.AddSchemaValue("true", System.Drawing.Color.Green); AnnotationSchemaRegistry.Default.Register(booleanSchema); - this.DatasetViewModel = new DatasetViewModel(); - this.DatasetViewModels = new ObservableCollection { this.datasetViewModel }; + this.DatasetViewModels = new ObservableCollection(); // Periodically check if there's any live partitions in the dataset this.liveStatusTimer = new DispatcherTimer(TimeSpan.FromSeconds(10), DispatcherPriority.Normal, new EventHandler(this.OnLiveStatusTimer), Application.Current.Dispatcher); @@ -137,7 +134,7 @@ public bool OpenLayout(string path, string name) this.VisualizationContainer = visualizationContainer; // zoom into the current session if there is one - SessionViewModel sessionViewModel = this.DatasetViewModel.CurrentSessionViewModel; + SessionViewModel sessionViewModel = this.DatasetViewModel?.CurrentSessionViewModel; if (sessionViewModel != null) { // Zoom to the current session extents @@ -158,19 +155,19 @@ public bool OpenLayout(string path, string name) } /// - /// Gets the message type for a stream. If the message type is unknown (i.e. the assembly + /// Gets the message data type for a stream. If the data type is unknown (i.e. the assembly /// that contains the message type is not currently being referenced by PsiStudio, then /// we return the generic object type. /// /// The stream tree node. /// The type of messages in the stream. - public Type GetStreamType(StreamTreeNode streamTreeNode) + public Type GetDataType(StreamTreeNode streamTreeNode) { return TypeResolutionHelper.GetVerifiedType(streamTreeNode.TypeName) ?? TypeResolutionHelper.GetVerifiedType(streamTreeNode.TypeName.Split(',')[0]) ?? typeof(object); } /// - /// Visualizes a streamin the visualization container. + /// Visualizes a streaming the visualization container. /// /// The stream to visualize. /// The visualizer metadata to use. @@ -221,7 +218,7 @@ public void VisualizeStream(StreamTreeNode streamTreeNode, VisualizerMetadata vi Type streamAdapterType = visualizerMetadata.StreamAdapterType; if (visualizerMetadata.VisualizationObjectType == typeof(MessageVisualizationObject)) { - streamAdapterType = typeof(ObjectAdapter<>).MakeGenericType(this.GetStreamType(streamTreeNode)); + streamAdapterType = typeof(ObjectAdapter<>).MakeGenericType(this.GetDataType(streamTreeNode)); } // Update the binding @@ -239,36 +236,45 @@ public void VisualizeStream(StreamTreeNode streamTreeNode, VisualizerMetadata vi /// Asynchronously opens a previously persisted dataset. /// /// Fully qualified path to dataset file. + /// Indicates whether to show the status window. /// A task that represents the asynchronous operation. - public async Task OpenDatasetAsync(string filename) + public async Task OpenDatasetAsync(string filename, bool showStatusWindow = true) { - // Window that will be used to indicate that an open operation is in progress. - // Progress notification and cancellation are not yet fully supported. - var statusWindow = new LoadingDatasetWindow(Application.Current.MainWindow, filename); - - // progress reporter for the load dataset task - var progress = new Progress<(string s, double p)>(t => + var loadDatasetTask = default(Task); + if (showStatusWindow) { - statusWindow.Status = t.s; - if (t.p == 1.0) + // Window that will be used to indicate that an open operation is in progress. + // Progress notification and cancellation are not yet fully supported. + var statusWindow = new LoadingDatasetWindow(Application.Current.MainWindow, filename); + + // progress reporter for the load dataset task + var progress = new Progress<(string s, double p)>(t => { + statusWindow.Status = t.s; + if (t.p == 1.0) + { // close the status window when the task reports completion statusWindow.Close(); - } - }); + } + }); - // start the load dataset task - var loadDatasetTask = this.LoadDatasetOrStoreAsync(filename, progress); + // start the load dataset task + loadDatasetTask = this.LoadDatasetOrStoreAsync(filename, progress); - try - { - // show the modal status window, which will be closed once the load dataset operation completes - statusWindow.ShowDialog(); + try + { + // show the modal status window, which will be closed once the load dataset operation completes + statusWindow.ShowDialog(); + } + catch (InvalidOperationException) + { + // This indicates that the window has already been closed in the async task, + // which means the operation has already completed, so just ignore and continue. + } } - catch (InvalidOperationException) + else { - // This indicates that the window has already been closed in the async task, - // which means the operation has already completed, so just ignore and continue. + loadDatasetTask = this.LoadDatasetOrStoreAsync(filename); } try @@ -324,7 +330,7 @@ public bool IsDatasetLoaded() public void ToggleLiveMode() { // Only enter live mode if the current session contains live partitions - if (this.DatasetViewModel.CurrentSessionViewModel.ContainsLivePartitions && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live) + if (this.DatasetViewModel != null && this.DatasetViewModel.CurrentSessionViewModel.ContainsLivePartitions && this.VisualizationContainer.Navigator.CursorMode != CursorMode.Live) { this.VisualizationContainer.Navigator.SetCursorMode(CursorMode.Live); } @@ -365,7 +371,7 @@ internal WpfControls.ContextMenu GetDatasetStreamMenu(StreamTreeNode streamTreeN // Get the message type. Type of object will be returned if we don't reference the // assembly that contains the message type. This will allow us to still display // the visualize messages and visualize latency menuitems. - Type messageType = this.GetStreamType(streamTreeNode); + Type messageType = this.GetDataType(streamTreeNode); // Get the list of visualization commands for this stream tree node List metadataItems = this.VisualizerMap.GetByDataType(messageType); @@ -473,17 +479,12 @@ private void OnLiveStatusTimer(object sender, EventArgs e) // Update the list of live partitions this.DatasetViewModel.UpdateLivePartitionStatuses(); - // If the're no longer any live partitions, exit live mode + // If there are no longer any live partitions, exit live mode if ((this.DatasetViewModel.CurrentSessionViewModel?.ContainsLivePartitions == false) && (this.VisualizationContainer.Navigator.CursorMode == CursorMode.Live)) { this.VisualizationContainer.Navigator.SetCursorMode(CursorMode.Manual); } } } - - private void RegisterCustomSerializers() - { - KnownSerializers.Default.Register>(null); - } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/AnimatedModel3DVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/AnimatedModel3DVisualizationObject.cs index 599d15d9e..7c0d1120e 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/AnimatedModel3DVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/AnimatedModel3DVisualizationObject.cs @@ -8,7 +8,7 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects using Microsoft.Psi.Visualization.Views.Visuals3D; /// - /// Represnts an animated model 3D visualization object. + /// Represents an animated model 3D visualization object. /// [VisualizationObject("Visualize Animated Model")] public class AnimatedModel3DVisualizationObject : Instant3DVisualizationObject diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/CoordinateSystemVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/CoordinateSystemVisualizationObject.cs index 568bcfb59..262515c4f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/CoordinateSystemVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/CoordinateSystemVisualizationObject.cs @@ -5,7 +5,6 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects { - using System; using System.Runtime.Serialization; using System.Windows.Media; using HelixToolkit.Wpf; @@ -30,8 +29,6 @@ public class CoordinateSystemVisualizationObject : ModelVisual3DVisualizationObj new ArrowVisual3D() { ThetaDiv = ThetaDivDefault, Fill = Brushes.Blue }, }; - private CoordinateSystem currentData = null; - private double size = 35; /// @@ -59,7 +56,7 @@ public override void NotifyPropertyChanged(string propertyName) if (propertyName == nameof(this.Size)) { this.UpdateDiameters(); - this.UpdateData(this.currentData, default); + this.UpdateData(); } else if (propertyName == nameof(this.Visible)) { @@ -68,34 +65,29 @@ public override void NotifyPropertyChanged(string propertyName) } /// - public override void UpdateData(CoordinateSystem coordinateSystem, DateTime originatingTime) + public override void UpdateData() { - if (!Equals(this.currentData, coordinateSystem)) + if (this.CurrentData != null) { - this.currentData = coordinateSystem; - - if (this.currentData != null) - { - var length = this.Size / 200.0; - var x = this.currentData.Origin + (length * this.currentData.XAxis.Normalize()); - var y = this.currentData.Origin + (length * this.currentData.YAxis.Normalize()); - var z = this.currentData.Origin + (length * this.currentData.ZAxis.Normalize()); - - this.axes[XAxisIndex].BeginEdit(); - this.axes[XAxisIndex].Point1 = new Win3D.Point3D(this.currentData.Origin.X, this.currentData.Origin.Y, this.currentData.Origin.Z); - this.axes[XAxisIndex].Point2 = new Win3D.Point3D(x.X, x.Y, x.Z); - this.axes[XAxisIndex].EndEdit(); - - this.axes[YAxisIndex].BeginEdit(); - this.axes[YAxisIndex].Point1 = new Win3D.Point3D(this.currentData.Origin.X, this.currentData.Origin.Y, this.currentData.Origin.Z); - this.axes[YAxisIndex].Point2 = new Win3D.Point3D(y.X, y.Y, y.Z); - this.axes[YAxisIndex].EndEdit(); - - this.axes[ZAxisIndex].BeginEdit(); - this.axes[ZAxisIndex].Point1 = new Win3D.Point3D(this.currentData.Origin.X, this.currentData.Origin.Y, this.currentData.Origin.Z); - this.axes[ZAxisIndex].Point2 = new Win3D.Point3D(z.X, z.Y, z.Z); - this.axes[ZAxisIndex].EndEdit(); - } + var length = this.Size / 200.0; + var x = this.CurrentData.Origin + (length * this.CurrentData.XAxis.Normalize()); + var y = this.CurrentData.Origin + (length * this.CurrentData.YAxis.Normalize()); + var z = this.CurrentData.Origin + (length * this.CurrentData.ZAxis.Normalize()); + + this.axes[XAxisIndex].BeginEdit(); + this.axes[XAxisIndex].Point1 = new Win3D.Point3D(this.CurrentData.Origin.X, this.CurrentData.Origin.Y, this.CurrentData.Origin.Z); + this.axes[XAxisIndex].Point2 = new Win3D.Point3D(x.X, x.Y, x.Z); + this.axes[XAxisIndex].EndEdit(); + + this.axes[YAxisIndex].BeginEdit(); + this.axes[YAxisIndex].Point1 = new Win3D.Point3D(this.CurrentData.Origin.X, this.CurrentData.Origin.Y, this.CurrentData.Origin.Z); + this.axes[YAxisIndex].Point2 = new Win3D.Point3D(y.X, y.Y, y.Z); + this.axes[YAxisIndex].EndEdit(); + + this.axes[ZAxisIndex].BeginEdit(); + this.axes[ZAxisIndex].Point1 = new Win3D.Point3D(this.CurrentData.Origin.X, this.CurrentData.Origin.Y, this.CurrentData.Origin.Z); + this.axes[ZAxisIndex].Point2 = new Win3D.Point3D(z.X, z.Y, z.Z); + this.axes[ZAxisIndex].EndEdit(); } this.UpdateVisibility(); @@ -115,7 +107,7 @@ private void UpdateVisibility() { foreach (ArrowVisual3D axis in this.axes) { - this.UpdateChildVisibility(axis, this.Visible && this.currentData != null); + this.UpdateChildVisibility(axis, this.Visible && this.CurrentData != null); } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectDepth3DVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DMeshVisualizationObject.cs similarity index 72% rename from Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectDepth3DVisualizationObject.cs rename to Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DMeshVisualizationObject.cs index 259ea4d0e..8ceb5f5a2 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectDepth3DVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DMeshVisualizationObject.cs @@ -9,10 +9,10 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects using Microsoft.Psi.Visualization.Views.Visuals3D; /// - /// Represents a Kinect depth 3D visualization object. + /// Represents a depth image 3D mesh visualization object. /// - [VisualizationObject("Visualize Kinect Depth Data")] - public class KinectDepth3DVisualizationObject : Instant3DVisualizationObject> + [VisualizationObject("Visualize Depth Image as 3D Mesh")] + public class DepthImageAs3DMeshVisualizationObject : Instant3DVisualizationObject> { private Color color = Colors.Navy; @@ -29,7 +29,7 @@ public Color Color /// protected override void InitNew() { - this.Visual3D = new KinectDepthVisual(this); + this.Visual3D = new DepthImageAs3DMeshVisual3D(this); base.InitNew(); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DPointCloudVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DPointCloudVisualizationObject.cs new file mode 100644 index 000000000..bdbf86470 --- /dev/null +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/DepthImageAs3DPointCloudVisualizationObject.cs @@ -0,0 +1,138 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT license. + +namespace Microsoft.Psi.Visualization.VisualizationObjects +{ + using System.ComponentModel; + using System.Runtime.Serialization; + using System.Windows.Media; + using System.Windows.Media.Media3D; + using HelixToolkit.Wpf; + using Microsoft.Psi.Imaging; + + /// + /// Represents a depth image 3D point cloud visualization object. + /// + [VisualizationObject("Visualize Depth Image as 3D Point Cloud")] + public class DepthImageAs3DPointCloudVisualizationObject : Instant3DVisualizationObject> + { + private Color color = Colors.Green; + private double pointSize = 1.0; + private int sparsity = 1; + + private PointsVisual3D pointsVisual; + + /// + /// Gets or sets the point cloud color. + /// + [DataMember] + [DisplayName("Color")] + [Description("Color used for point cloud points.")] + public Color Color + { + get { return this.color; } + set { this.Set(nameof(this.Color), ref this.color, value); } + } + + /// + /// Gets or sets the point size. + /// + [DataMember] + [DisplayName("Point Size")] + [Description("Size of each point cloud point.")] + public double PointSize + { + get { return this.pointSize; } + set { this.Set(nameof(this.PointSize), ref this.pointSize, value); } + } + + /// + /// Gets or sets the point cloud sparsity. + /// + [DataMember] + [DisplayName("Sparsity")] + [Description("Sparseness of the cloud point.")] + public int Sparsity + { + get { return this.sparsity; } + set { this.Set(nameof(this.Sparsity), ref this.sparsity, value); } + } + + /// + protected override void InitNew() + { + this.PropertyChanged += this.VisualizationPropertyChanged; + this.Visual3D = this.pointsVisual = new PointsVisual3D() + { + Size = this.PointSize, + Color = this.Color, + }; + base.InitNew(); + } + + private void UpdatePoints(DepthImage depthImage) + { + var points = this.pointsVisual.Points; + this.pointsVisual.Points = null; // "unhook" for performance + points.Clear(); + + int width = depthImage.Width; + int height = depthImage.Height; + int cx = width / 2; + int cy = height / 2; + + const double fxinv = 1.0 / 366; + const double fyinv = 1.0 / 366; + const double scale = 0.001; // millimeters + + const ushort tooNearDepth = 500; + const ushort tooFarDepth = 10000; + const ushort unknownDepth = 0; + + unsafe + { + ushort* depthFrame = (ushort*)((byte*)depthImage.ImageData.ToPointer()); + + for (int iy = 0; iy < height; iy += this.sparsity) + { + for (int ix = 0; ix < width; ix += this.sparsity) + { + int i = (iy * width) + ix; + var d = depthFrame[(iy * width) + ix]; + if (d != unknownDepth && d > tooNearDepth && d < tooFarDepth) + { + double zz = d * scale; + double x = (cx - ix) * zz * fxinv; + double y = zz; + double z = (cy - iy) * zz * fyinv; + points.Add(new Point3D(x, y, z)); + } + } + } + } + + this.pointsVisual.Points = points; + } + + private void VisualizationPropertyChanged(object sender, PropertyChangedEventArgs e) + { + switch (e.PropertyName) + { + case nameof(this.CurrentValue): + case nameof(this.Sparsity): + if (this.CurrentData != null) + { + this.UpdatePoints(this.CurrentData.Resource); + } + + break; + case nameof(this.Color): + this.pointsVisual.Color = this.Color; + break; + case nameof(this.PointSize): + this.pointsVisual.Size = this.PointSize; + break; + } + } + } +} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IModelVisual3DVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IModelVisual3DVisualizationObject.cs index f1d0d04d8..2a9610da4 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IModelVisual3DVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IModelVisual3DVisualizationObject.cs @@ -3,8 +3,6 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects { - using System.Collections.Generic; - /// /// Represents a model visual 3d visualization object. /// @@ -26,10 +24,10 @@ public interface IModelVisual3DVisualizationObject void OnChildPropertyChanged(string path, object value); /// - /// Called when we need to notify the child visualization objects that a proeprty has changed. + /// Called when we need to notify the child visualization objects that a property has changed. /// /// The path from the current visualization object to the property to set. - /// The new value for the proeprty. + /// The new value for the property. void SetDescendantProperty(string path, object newValue); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IStreamVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IStreamVisualizationObject.cs index 64d5c472c..546b0277f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IStreamVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/IStreamVisualizationObject.cs @@ -6,6 +6,7 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects using System; using Microsoft.Psi.Data; using Microsoft.Psi.Visualization.Data; + using Microsoft.Psi.Visualization.Helpers; /// /// Represents a stream visualization object. @@ -42,7 +43,8 @@ public interface IStreamVisualizationObject /// Gets a snapped time based on a given time. /// /// The input time. + /// Timeline snapping behavior. /// The snapped time. - DateTime? GetSnappedTime(DateTime time); + DateTime? GetSnappedTime(DateTime time, SnappingBehavior snappingBehavior = SnappingBehavior.Nearest); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ImageVisualizationObjectBase{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ImageVisualizationObjectBase{TData}.cs index 37ae31257..8a5c58467 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ImageVisualizationObjectBase{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ImageVisualizationObjectBase{TData}.cs @@ -8,7 +8,7 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects /// /// Represents an image visualization object. /// - /// The type of the image visualzation object. + /// The type of the image visualization object. public abstract class ImageVisualizationObjectBase : Instant2DVisualizationObject { /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/InstantVisualizationObject{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/InstantVisualizationObject{TData}.cs index 1208bd603..04690d736 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/InstantVisualizationObject{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/InstantVisualizationObject{TData}.cs @@ -5,12 +5,12 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects { using System; using System.ComponentModel; - using System.Reflection; using System.Runtime.Serialization; using System.Windows; using System.Windows.Threading; using Microsoft.Psi.Persistence; using Microsoft.Psi.Visualization.Data; + using Microsoft.Psi.Visualization.Helpers; /// /// Represents an instant visualization object. @@ -88,6 +88,12 @@ private set } } + /// + public override DateTime? GetSnappedTime(DateTime time, SnappingBehavior snappingBehavior) + { + return DataManager.Instance.GetOriginatingTimeOfNearestInstantMessage(this.StreamBinding, time); + } + /// /// Called when the current instant data has changed. This method is called on a worker thread. /// @@ -103,30 +109,26 @@ public void OnInstantDataChanged(object data, IndexEntry indexEntry) // Abort the task if (this.lastDataChangedTask.Abort()) { - // If the data is shared, release the reference + // If the data that was to be used as the current data is shared, then decrement its reference count if (this.IsShared && (this.lastDataChangedTaskData != null)) { - (this.lastDataChangedTaskData as IDisposable).Dispose(); + (this.lastDataChangedTaskData as IDisposable).Dispose(); } } } - // Squirrel away the data object in case we need to abort the task before it gets scheduled - this.lastDataChangedTaskData = (TData)data; - - // Queue up the task on the UI thread - this.lastDataChangedTask = Application.Current.Dispatcher.BeginInvoke( - (Action)(() => - { - // Construct a message for the current value - this.CurrentValue = new Message((TData)data, indexEntry.OriginatingTime, indexEntry.Time, 0, 0); - - // If the data is shared, release the reference - if (this.IsShared && (data != null)) - { - (data as IDisposable).Dispose(); - } - }), DispatcherPriority.Render); + // Because this method is called on a worker thread and some of our Model3DVisual visualization objects directly touch UI elements + // when we call SetCurrentValue, we need to invoke the dispatcher, and we do so asynchronously for performance reasons. If data is + // a shared object then the code that called this method will release its reference to the shared data object shortly after this + // method returns. Consequently we need to add a new reference to the shared data object now instead of the usual pattern where + // the code in SetCurrentValue adds the new reference. We therefore pass false as the incrementSharedRefCount parameter to indicate + // that we've already incremented the reference count and the code in SetCurrentValue should not. We also squirrel away a reference + // to data in case we get called again before the dispatcher has had a chance to execute our task. If this happens, the currently + // pending dispatcher task is obsolete so we'll cancel it. If we cancel the task and the data for the task was a shared object then + // we'll need to dereference the data ourselves to ensure it gets released properly, see code above. + Message newValue = new Message((this.IsShared && data != null) ? ((TData)data).DeepClone() : (TData)data, indexEntry.OriginatingTime, indexEntry.Time, 0, 0); + this.lastDataChangedTaskData = newValue.Data; + this.lastDataChangedTask = Application.Current.Dispatcher.BeginInvoke((Action)(() => this.SetCurrentValue(newValue, false)), DispatcherPriority.Render); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectBodies3DVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectBodies3DVisualizationObject.cs deleted file mode 100644 index e9b6437cc..000000000 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/KinectBodies3DVisualizationObject.cs +++ /dev/null @@ -1,71 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT license. - -namespace Microsoft.Psi.Visualization.VisualizationObjects -{ - using System.Collections.Generic; - using System.Runtime.Serialization; - using System.Windows.Media; - using Microsoft.Psi.Kinect; - using Microsoft.Psi.Visualization.Views.Visuals3D; - - /// - /// Represents a Kinect bodies 3D visualization object. - /// - [VisualizationObject("Visualize Kinect Bodies")] - public class KinectBodies3DVisualizationObject : Instant3DVisualizationObject> - { - private Color color = Colors.White; - private double inferredJointsOpacity = 0; - private double size = 0.03; - private bool showTrackingBillboards = false; - - /// - /// Initializes a new instance of the class. - /// - public KinectBodies3DVisualizationObject() - { - this.Visual3D = new KinectBodiesVisual(this); - } - - /// - /// Gets or sets the color. - /// - [DataMember] - public Color Color - { - get { return this.color; } - set { this.Set(nameof(this.Color), ref this.color, value); } - } - - /// - /// Gets or sets the inferred joints opacity. - /// - [DataMember] - public double InferredJointsOpacity - { - get { return this.inferredJointsOpacity; } - set { this.Set(nameof(this.InferredJointsOpacity), ref this.inferredJointsOpacity, value); } - } - - /// - /// Gets or sets the size. - /// - [DataMember] - public double Size - { - get { return this.size; } - set { this.Set(nameof(this.Size), ref this.size, value); } - } - - /// - /// Gets or sets a value indicating whether to show tracking billboards. - /// - [DataMember] - public bool ShowTrackingBillboards - { - get { return this.showTrackingBillboards; } - set { this.Set(nameof(this.ShowTrackingBillboards), ref this.showTrackingBillboards, value); } - } - } -} diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisObj,TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisualizationObject,TData}.cs similarity index 89% rename from Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisObj,TData}.cs rename to Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisualizationObject,TData}.cs index 638557359..fe5a92866 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisObj,TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectCollectionBase{TVisualizationObject,TData}.cs @@ -12,10 +12,10 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects /// /// Represents the base class for collections of 3D model visualization objects. /// - /// The type of visualization objects in the collection. + /// The type of visualization objects in the collection. /// The type of data being represented. - public abstract class ModelVisual3DVisualizationObjectCollectionBase : ModelVisual3DVisualizationObject - where TVisObj : VisualizationObject, IModelVisual3DVisualizationObject, ICustomTypeDescriptor, new() + public abstract class ModelVisual3DVisualizationObjectCollectionBase : ModelVisual3DVisualizationObject + where TVisualizationObject : VisualizationObject, IModelVisual3DVisualizationObject, ICustomTypeDescriptor, new() { // The list of properties of the prototype (including the prototype's children) // that have changed since this collection was created. After we create a new @@ -28,7 +28,7 @@ public abstract class ModelVisual3DVisualizationObjectCollectionBase [Browsable(false)] - [IgnoreDataMember] - public TVisObj Prototype { get; private set; } + [DataMember] + public TVisualizationObject Prototype { get; private set; } /// /// Gets or sets the collection of child visualization objects. /// - protected IEnumerable Items { get; set; } + protected IEnumerable Items { get; set; } /// public override void OnChildPropertyChanged(string path, object value) @@ -59,7 +59,7 @@ public override void OnChildPropertyChanged(string path, object value) if (nextSegment == nameof(this.Prototype)) { // Set the new property value on all of the children - foreach (TVisObj item in this.Items) + foreach (TVisualizationObject item in this.Items) { item.SetDescendantProperty(pathRemainder, value); } @@ -79,7 +79,7 @@ public override void NotifyPropertyChanged(string propertyName) // Properties in prototype do not appear in the property browser // if they have the same name as a property in this parent collection // class. This applies to all public properties that were defined - // high up in the heirarchy. Of these hidden properties, the only + // high up in the hierarchy. Of these hidden properties, the only // one we want to propagate to the children is the visible property // and setting it on the prototype will cause it to automatically // be propagated to the children. @@ -91,13 +91,13 @@ public override void NotifyPropertyChanged(string propertyName) /// /// Creates a new created TVisObj and then copies the values of all properties - /// of the proptotype (and its children) that have changed into the new object. + /// of the prototype (and its children) that have changed into the new object. /// /// The newly created TVisObj. - protected TVisObj CreateNew() + protected TVisualizationObject CreateNew() { // Create the new object - TVisObj visualizationObject = new TVisObj(); + TVisualizationObject visualizationObject = new TVisualizationObject(); // Copy all of the prototype's updated properties into the new object. foreach (var property in this.updatedPrototypeProperties) @@ -138,7 +138,7 @@ protected override void GenerateCustomPropertyDescriptors() private void CopyPropertyValue(object destination, string path, object value) { - // Get the next segemnt in the path to the property + // Get the next segment in the path to the property this.GetNextPropertyPathSegment(path, out string nextSegment, out string pathRemainder); // Get the property info for the next segment in the path diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectEnumerable{TVisObj,TData,TColl}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectEnumerable{TVisObj,TData,TColl}.cs index e97f644b6..6ddfb5537 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectEnumerable{TVisObj,TData,TColl}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObjectEnumerable{TVisObj,TData,TColl}.cs @@ -3,7 +3,6 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects { - using System; using System.Collections.Generic; /// @@ -11,7 +10,7 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects /// /// The type of visualization objects in the collection. /// The type of data of each item in the collection. - /// The type of of the collection. + /// The type of the collection. public abstract class ModelVisual3DVisualizationObjectEnumerable : ModelVisual3DVisualizationObjectCollectionBase where TVisObj : ModelVisual3DVisualizationObject, new() where TColl : IEnumerable @@ -28,21 +27,21 @@ public ModelVisual3DVisualizationObjectEnumerable() } /// - public override void UpdateData(TColl currentData, DateTime originatingTime) + public override void UpdateData() { - if (currentData != null) + if (this.CurrentData != null) { int index = 0; - foreach (TData datum in currentData) + foreach (TData datum in this.CurrentData) { - // If we don't have enought visualization objects, create a new one + // If we don't have enough visualization objects, create a new one while (index >= this.ModelView.Children.Count) { this.AddNew(datum); } // Get the child visualization object to update itself - this.children[index].UpdateData(datum, originatingTime); + this.children[index].SetCurrentValue(this.SynthesizeMessage(datum)); index++; } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObject{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObject{TData}.cs index eb7e2b90d..1f70737de 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObject{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/ModelVisual3DVisualizationObject{TData}.cs @@ -24,7 +24,7 @@ public abstract class ModelVisual3DVisualizationObject : Instant3DVisuali // The list of properties that should not appear in the property browser if // this visualization object is the child of another visualization object. - private List hiddenChildProperties = new List() { "CursorEpsilonMs", "InterpolationStyle", "Name" }; + private List hiddenChildProperties = new List() { nameof(CursorEpsilonMs), nameof(Name), nameof(StreamAdapterType), nameof(SummarizerType) }; /// /// Gets the model view of this visualization object. @@ -67,9 +67,7 @@ public void RegisterChildPropertyChangedNotifications(IModelVisual3DVisualizatio /// /// Called when the current stream data has changed. /// - /// The data for the current value. - /// The originating time of the current data. - public abstract void UpdateData(TData currentData, DateTime originatingTime); + public abstract void UpdateData(); /// /// Called when a property other than CurrentValue changes. @@ -98,7 +96,7 @@ public void SetDescendantProperty(string path, object newValue) PropertyInfo property = this.GetType().GetProperty(nextSegment); // If we're at the end of the path, we're at the property and we can - // set it, otherwise we still need to drill further into the heirarchy + // set it, otherwise we still need to drill further into the hierarchy if (string.IsNullOrEmpty(remainder)) { property.SetValue(this, newValue); @@ -206,12 +204,10 @@ protected override void InitNew() /// protected override void OnPropertyChanged(object sender, PropertyChangedEventArgs e) { - Message currentValue = this.CurrentValue.GetValueOrDefault(); - if (e.PropertyName == nameof(this.CurrentValue)) { // Notify of the change to the current data - this.UpdateData(currentValue == default ? default : currentValue.Data, currentValue.OriginatingTime); + this.UpdateData(); } else { @@ -255,6 +251,17 @@ protected void UpdateChildVisibility(Visual3D child, bool visible) } } + /// + /// Creates a new struct suitable for passing to a child . + /// + /// The type of the new message. + /// The new message data. + /// A newly created message using the envelope of the current value. + protected Message SynthesizeMessage(T data) + { + return new Message(data, this.CurrentValue.Value.OriginatingTime, this.CurrentValue.Value.Time, this.CurrentValue.Value.SourceId, this.CurrentValue.Value.SequenceId); + } + /// /// Gets the next path segment from a property path. /// @@ -290,7 +297,7 @@ protected virtual void GenerateCustomPropertyDescriptors() /// private void RemoveHiddenChildProperties() { - // TypeDeescriptor.GetProperties returns a readonly collction, so we need to copy + // TypeDeescriptor.GetProperties returns a readonly collection, so we need to copy // the properties to a new collection, skipping those properties we wish to hide. PropertyDescriptorCollection updatedPropertyDescriptors = new PropertyDescriptorCollection(null); diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PipelineDiagnosticsVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PipelineDiagnosticsVisualizationObject.cs index 790ab6c97..570574af3 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PipelineDiagnosticsVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PipelineDiagnosticsVisualizationObject.cs @@ -260,7 +260,7 @@ public HighlightCondition Highlight } /// - /// Gets or sets hilight color. + /// Gets or sets highlight color. /// [DataMember] [DisplayName("Highlight Color")] @@ -269,7 +269,7 @@ public HighlightCondition Highlight public Color HighlightColor { get { return this.highlightColor; } - set { this.Set(nameof(this.highlightColor), ref this.highlightColor, value); } + set { this.Set(nameof(this.HighlightColor), ref this.highlightColor, value); } } /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PlotVisualizationObject{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PlotVisualizationObject{TData}.cs index e5a4e07a2..e30e451a8 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PlotVisualizationObject{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/PlotVisualizationObject{TData}.cs @@ -231,7 +231,7 @@ public double YMin /// protected override int GetIndexForTime(DateTime currentTime, int count, Func timeAtIndex) { - return IndexHelper.GetIndexForTime(currentTime, count, timeAtIndex, this.InterpolationStyle); + return IndexHelper.GetIndexForTime(currentTime, count, timeAtIndex); } /// diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/Rect3DVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/Rect3DVisualizationObject.cs index 6b34d91e0..99c37bf40 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/Rect3DVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/Rect3DVisualizationObject.cs @@ -24,9 +24,6 @@ public class Rect3DVisualizationObject : ModelVisual3DVisualizationObject /// Initializes a new instance of the class. /// @@ -67,19 +64,18 @@ public double Opacity } /// - public override void UpdateData(Rect3D data, DateTime originatingTime) + public override void UpdateData() { - this.currentData = data; - if (this.currentData != default) + if (this.CurrentData != default) { - var p0 = new Point3D(data.Location.X, data.Location.Y, data.Location.Z); - var p1 = new Point3D(data.Location.X + data.SizeX, data.Location.Y, data.Location.Z); - var p2 = new Point3D(data.Location.X + data.SizeX, data.Location.Y + data.SizeY, data.Location.Z); - var p3 = new Point3D(data.Location.X, data.Location.Y + data.SizeY, data.Location.Z); - var p4 = new Point3D(data.Location.X, data.Location.Y, data.Location.Z + data.SizeZ); - var p5 = new Point3D(data.Location.X + data.SizeX, data.Location.Y, data.Location.Z + data.SizeZ); - var p6 = new Point3D(data.Location.X + data.SizeX, data.Location.Y + data.SizeY, data.Location.Z + data.SizeZ); - var p7 = new Point3D(data.Location.X, data.Location.Y + data.SizeY, data.Location.Z + data.SizeZ); + var p0 = new Point3D(this.CurrentData.Location.X, this.CurrentData.Location.Y, this.CurrentData.Location.Z); + var p1 = new Point3D(this.CurrentData.Location.X + this.CurrentData.SizeX, this.CurrentData.Location.Y, this.CurrentData.Location.Z); + var p2 = new Point3D(this.CurrentData.Location.X + this.CurrentData.SizeX, this.CurrentData.Location.Y + this.CurrentData.SizeY, this.CurrentData.Location.Z); + var p3 = new Point3D(this.CurrentData.Location.X, this.CurrentData.Location.Y + this.CurrentData.SizeY, this.CurrentData.Location.Z); + var p4 = new Point3D(this.CurrentData.Location.X, this.CurrentData.Location.Y, this.CurrentData.Location.Z + this.CurrentData.SizeZ); + var p5 = new Point3D(this.CurrentData.Location.X + this.CurrentData.SizeX, this.CurrentData.Location.Y, this.CurrentData.Location.Z + this.CurrentData.SizeZ); + var p6 = new Point3D(this.CurrentData.Location.X + this.CurrentData.SizeX, this.CurrentData.Location.Y + this.CurrentData.SizeY, this.CurrentData.Location.Z + this.CurrentData.SizeZ); + var p7 = new Point3D(this.CurrentData.Location.X, this.CurrentData.Location.Y + this.CurrentData.SizeY, this.CurrentData.Location.Z + this.CurrentData.SizeZ); this.UpdateLinePosition(this.edges[0], p0, p1); this.UpdateLinePosition(this.edges[1], p1, p2); @@ -139,7 +135,7 @@ private void UpdateVisibility() { foreach (PipeVisual3D line in this.edges) { - this.UpdateChildVisibility(line, this.Visible && this.currentData != default); + this.UpdateChildVisibility(line, this.Visible && this.CurrentData != default); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/StreamVisualizationObject{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/StreamVisualizationObject{TData}.cs index cae2e41fb..5d852778f 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/StreamVisualizationObject{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/StreamVisualizationObject{TData}.cs @@ -64,7 +64,7 @@ public StreamVisualizationObject() /// [Browsable(false)] [IgnoreDataMember] - public override bool CanSnapToStream => true; + public override bool CanSnapToStream => this.IsBound; /// /// Gets the snap to stream command. @@ -95,11 +95,7 @@ public RelayCommand ZoomToStreamCommand { if (this.zoomToStreamCommand == null) { - this.zoomToStreamCommand = new RelayCommand( - () => - { - this.Container.Navigator.Zoom(this.StreamBinding.StreamMetadata.FirstMessageOriginatingTime, this.StreamBinding.StreamMetadata.LastMessageOriginatingTime); - }); + this.zoomToStreamCommand = new RelayCommand(() => this.Container.Navigator.Zoom(this.StreamBinding.StreamMetadata.FirstMessageOriginatingTime, this.StreamBinding.StreamMetadata.LastMessageOriginatingTime)); } return this.zoomToStreamCommand; @@ -107,37 +103,25 @@ public RelayCommand ZoomToStreamCommand } /// - /// Gets or sets the current value. + /// Gets the current value. /// [Browsable(false)] [IgnoreDataMember] - public Message? CurrentValue - { - get => this.currentValue; - protected set - { - if (this.currentValue != value) - { - this.RaisePropertyChanging(nameof(this.CurrentValue)); + public Message? CurrentValue => this.currentValue; - if (this.IsShared) - { - if (this.currentValue.HasValue) - { - ((IDisposable)this.currentValue.Value.Data)?.Dispose(); - } - - value.DeepClone(ref this.currentValue); - } - else - { - this.currentValue = value; - } + /// + /// Gets the current data. + /// + [Browsable(false)] + [IgnoreDataMember] + public TData CurrentData => this.currentValue.HasValue ? this.currentValue.Value.Data : default; - this.RaisePropertyChanged(nameof(this.CurrentValue)); - } - } - } + /// + /// Gets the originating time of the current data. + /// + [Browsable(false)] + [IgnoreDataMember] + public DateTime CurrentOriginatingTime => this.currentValue.HasValue ? this.currentValue.Value.OriginatingTime : default; /// /// Gets or sets the data view. @@ -165,7 +149,7 @@ protected set } else { - this.CurrentValue = null; + this.SetCurrentValue(null); } } } @@ -187,22 +171,25 @@ public StreamBinding StreamBinding /// [Browsable(true)] [IgnoreDataMember] - public Type StreamAdapterType => this.StreamBinding.StreamAdapterType; + public Type StreamAdapterType => this.StreamBinding?.StreamAdapterType; /// /// Gets the summarizer type. /// [Browsable(true)] [IgnoreDataMember] - public Type SummarizerType => this.StreamBinding.SummarizerType; + public Type SummarizerType => this.StreamBinding?.SummarizerType; /// - /// Gets a value indicating whether the visualization object is currenty bound to a datasource. + /// Gets a value indicating whether the visualization object is currently bound to a datasource. /// [Browsable(false)] [IgnoreDataMember] public bool IsBound => this.StreamBinding != null ? this.StreamBinding.IsBound : false; + /// + public override bool ShowZoomToStreamMenuItem => this.IsBound; + /// [Browsable(false)] [IgnoreDataMember] @@ -256,10 +243,47 @@ public bool IsLive } /// - /// Gets a value indicating whether type paramamter T is Shared{} or not. + /// Gets a value indicating whether type parameter T is Shared{} or not. /// protected bool IsShared { get; private set; } + /// + /// Sets the current value for the visualization object. + /// + /// The value to use as the new current value. + /// True if the reference count of shared data should be automatically incremented, otherwise false. + public void SetCurrentValue(Message? value, bool incrementSharedRefCount = true) + { + if (this.currentValue != value) + { + this.RaisePropertyChanging(nameof(this.CurrentValue)); + this.RaisePropertyChanging(nameof(this.CurrentData)); + this.RaisePropertyChanging(nameof(this.CurrentOriginatingTime)); + + // If we're handling shared objects, decrement the reference count of the current value + if (this.IsShared && this.currentValue.HasValue) + { + ((IDisposable)this.currentValue.Value.Data)?.Dispose(); + } + + // If we're handling shared objects, increment the reference count if requested. + // NOTE: InstantVisualzationObject will pass false for incrementSharedRefCount + // because it already increased the reference count itself before calling us. + if (this.IsShared && incrementSharedRefCount) + { + this.currentValue = value.DeepClone(); + } + else + { + this.currentValue = value; + } + + this.RaisePropertyChanged(nameof(this.CurrentValue)); + this.RaisePropertyChanged(nameof(this.CurrentData)); + this.RaisePropertyChanged(nameof(this.CurrentOriginatingTime)); + } + } + /// public override void ToggleSnapToStream() { @@ -293,10 +317,7 @@ public override void ToggleSnapToStream() /// public void UpdateStreamBinding(Session session) { - this.RaisePropertyChanging(nameof(this.IconSource)); - this.RaisePropertyChanging(nameof(this.CanSnapToStream)); - - // Keep a note of whether we're currenty bound to a source + // Keep a note of whether we're currently bound to a source bool wasBound = this.StreamBinding.IsBound; // Attempt to rebind to the underlying dataset @@ -308,6 +329,10 @@ public void UpdateStreamBinding(Session session) return; } + this.RaisePropertyChanging(nameof(this.IconSource)); + this.RaisePropertyChanging(nameof(this.CanSnapToStream)); + this.RaisePropertyChanging(nameof(this.ShowZoomToStreamMenuItem)); + // Notify that we're no longer bound to the previous data source if (wasBound) { @@ -322,12 +347,14 @@ public void UpdateStreamBinding(Session session) this.RaisePropertyChanged(nameof(this.IconSource)); this.RaisePropertyChanged(nameof(this.CanSnapToStream)); + this.RaisePropertyChanged(nameof(this.ShowZoomToStreamMenuItem)); } /// - public virtual DateTime? GetSnappedTime(DateTime time) + public virtual DateTime? GetSnappedTime(DateTime time, SnappingBehavior snappingBehavior = SnappingBehavior.Nearest) { - return this.GetTimeOfNearestMessage(time, this.Data?.Count ?? 0, (idx) => this.Data[idx].OriginatingTime); + // TODO + return this.GetTimeOfNearestMessage(time, this.Data?.Count ?? 0, (idx) => this.Data[idx].OriginatingTime, snappingBehavior); } /// @@ -336,10 +363,11 @@ public void UpdateStreamBinding(Session session) /// The time underneath the mouse cursor. /// Number of entries to search within. /// Function that returns an index given a time. + /// Timeline snapping behavior. /// The timestamp of the message that's temporally closest to currentTime. - protected DateTime? GetTimeOfNearestMessage(DateTime currentTime, int count, Func timeAtIndex) + protected DateTime? GetTimeOfNearestMessage(DateTime currentTime, int count, Func timeAtIndex, SnappingBehavior snappingBehavior) { - int index = IndexHelper.GetIndexForTime(currentTime, count, timeAtIndex); + int index = IndexHelper.GetIndexForTime(currentTime, count, timeAtIndex, snappingBehavior); if (index >= 0) { return timeAtIndex(index); @@ -378,7 +406,7 @@ protected virtual void OnDataCollectionChanged(NotifyCollectionChangedEventArgs var last = this.Data.LastOrDefault(); if (last != default(Message)) { - this.CurrentValue = last; + this.SetCurrentValue(last); } } } @@ -405,6 +433,12 @@ protected virtual void OnStreamBound() protected virtual void OnStreamUnbound() { this.Data = null; + + // If this is the stream currently being snapped to, disable snap to stream. + if (this.IsSnappedToStream) + { + this.ToggleSnapToStream(); + } } private void OnDataDetailedCollectionChanged(object sender, NotifyCollectionChangedEventArgs e) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimeIntervalHistoryVisualizationObject.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimeIntervalHistoryVisualizationObject.cs index b2fcfa2cf..be7b958ac 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimeIntervalHistoryVisualizationObject.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimeIntervalHistoryVisualizationObject.cs @@ -46,12 +46,12 @@ public class TimeIntervalHistoryVisualizationObject : TimelineVisualizationObjec private string legendValue = string.Empty; /// - /// Gets the data to de displayed in the control. + /// Gets the data to be displayed in the control. /// public List DisplayData { get; private set; } = new List(); /// - /// Gets the data to de displayed in the control. + /// Gets the data to be displayed in the control. /// public int TrackCount => Math.Max(1, this.trackNames.Count); @@ -220,7 +220,7 @@ private Brush GetBrush(Color color) private void GenerateLegendValue() { - // For now the legend value is simmply a list of all the track names + // For now the legend value is simply a list of all the track names StringBuilder legend = new StringBuilder(); foreach (string trackName in this.trackNames) { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimelineVisualizationObject{TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimelineVisualizationObject{TData}.cs index 85685216c..bebef8d84 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimelineVisualizationObject{TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/TimelineVisualizationObject{TData}.cs @@ -11,6 +11,7 @@ namespace Microsoft.Psi.Visualization.VisualizationObjects using System.Windows.Media; using Microsoft.Psi.Visualization.Collections; using Microsoft.Psi.Visualization.Data; + using Microsoft.Psi.Visualization.Helpers; using Microsoft.Psi.Visualization.Navigation; using Microsoft.Psi.Visualization.VisualizationPanels; @@ -109,24 +110,21 @@ public string LegendFormat } } - /// - public override bool ShowZoomToStreamMenuItem => true; - /// /// Gets a value indicating whether the visualization object is using summarization. /// protected bool IsUsingSummarization => this.StreamBinding.SummarizerType != null; /// - public override DateTime? GetSnappedTime(DateTime time) + public override DateTime? GetSnappedTime(DateTime time, SnappingBehavior snappingBehavior) { if (this.IsUsingSummarization) { - return this.GetTimeOfNearestMessage(time, this.SummaryData?.Count ?? 0, (idx) => this.SummaryData[idx].OriginatingTime); + return this.GetTimeOfNearestMessage(time, this.SummaryData?.Count ?? 0, (idx) => this.SummaryData[idx].OriginatingTime, snappingBehavior); } else { - return base.GetSnappedTime(time); + return base.GetSnappedTime(time, snappingBehavior); } } @@ -238,7 +236,7 @@ protected virtual void OnSummaryDataCollectionChanged(NotifyCollectionChangedEve IntervalData last = this.SummaryData.LastOrDefault(); if (last != default(IntervalData)) { - this.CurrentValue = Message.Create(last.Value, last.OriginatingTime, last.EndTime, 0, 0); + this.SetCurrentValue(Message.Create(last.Value, last.OriginatingTime, last.EndTime, 0, 0)); } } } @@ -254,11 +252,11 @@ protected override void OnCursorChanged(object sender, NavigatorTimeChangedEvent if (index != -1) { var interval = this.SummaryData[index]; - this.CurrentValue = Message.Create(interval.Value, interval.OriginatingTime, interval.EndTime, 0, 0); + this.SetCurrentValue(Message.Create(interval.Value, interval.OriginatingTime, interval.EndTime, 0, 0)); } else { - this.CurrentValue = null; + this.SetCurrentValue(null); } } else @@ -266,11 +264,11 @@ protected override void OnCursorChanged(object sender, NavigatorTimeChangedEvent int index = this.GetIndexForTime(currentTime, this.Data?.Count ?? 0, (idx) => this.Data[idx].OriginatingTime); if (index != -1) { - this.CurrentValue = this.Data[index]; + this.SetCurrentValue(this.Data[index]); } else { - this.CurrentValue = null; + this.SetCurrentValue(null); } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableModelVisual3DVisualizationObjectDictionary{TVisObj,TKey,TData}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableModelVisual3DVisualizationObjectDictionary{TVisObj,TKey,TData}.cs index 8f2cd9249..c5da4c6be 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableModelVisual3DVisualizationObjectDictionary{TVisObj,TKey,TData}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableModelVisual3DVisualizationObjectDictionary{TVisObj,TKey,TData}.cs @@ -26,6 +26,9 @@ public class UpdatableModelVisual3DVisualizationObjectDictionary visibilityPredicate = null; + /// /// Initializes a new instance of the class. /// @@ -65,6 +68,11 @@ public UpdatableModelVisual3DVisualizationObjectDictionary() // Add the visual to the collection and to the model visual this.visuals[key] = visual; + if (this.visibilityPredicate != null) + { + this.visuals[key].Visible = this.visibilityPredicate(key); + } + this.ModelView.Children.Add(visual.ModelView); } @@ -113,9 +121,9 @@ public void EndUpdate() } /// - public override void UpdateData(Dictionary currentData, DateTime originatingTime) + public override void UpdateData() { - if (currentData == null) + if (this.CurrentData == null) { this.RemoveAll(); } @@ -123,9 +131,9 @@ public override void UpdateData(Dictionary currentData, DateTime or { this.BeginUpdate(); - foreach (var datum in currentData) + foreach (var datum in this.CurrentData) { - this[datum.Key].UpdateData(datum.Value, originatingTime); + this[datum.Key].SetCurrentValue(this.SynthesizeMessage(datum.Value)); } this.EndUpdate(); @@ -138,6 +146,19 @@ public override void NotifyPropertyChanged(string propertyName) base.NotifyPropertyChanged(propertyName); } + /// + /// Set the visibility of the 3D visualization objects based on a specified predicate. + /// + /// A predicate that determines whether the visualization object corresponding to a given key is visible. + public void SetVisibility(Predicate visibilityPredicate) + { + this.visibilityPredicate = visibilityPredicate; + foreach (var key in this.visuals.Keys) + { + this.visuals[key].Visible = visibilityPredicate(key); + } + } + private void RemoveAll() { this.visuals.Clear(); diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DDictionary{TKey,TVisual}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DDictionary{TKey,TVisual}.cs index 53de32702..7578e4665 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DDictionary{TKey,TVisual}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DDictionary{TKey,TVisual}.cs @@ -31,7 +31,7 @@ public class UpdatableVisual3DDictionary : ModelVisual3D /// /// Initializes a new instance of the class. /// - /// The delegate that identifies the method to call whne a new TVisual + /// The delegate that identifies the method to call when a new TVisual /// needs to be initialized. This parameter can be null if no initialization is required. public UpdatableVisual3DDictionary(NewVisualHandler newVisualHandler) { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DList{TVisual}.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DList{TVisual}.cs index 043e47629..006ede9e6 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DList{TVisual}.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/UpdatableVisual3DList{TVisual}.cs @@ -29,7 +29,7 @@ public class UpdatableVisual3DList : ModelVisual3D /// /// Initializes a new instance of the class. /// - /// The delegate that identifies the method to call whne a new TVisual + /// The delegate that identifies the method to call when a new TVisual /// needs to be initialized. This parameter can be null if no initialization is required. public UpdatableVisual3DList(NewVisualHandler newVisualHandler) { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/VisualizationContainer.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/VisualizationContainer.cs index c790e2cda..2b4794567 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/VisualizationContainer.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationObjects/VisualizationContainer.cs @@ -46,7 +46,7 @@ public class VisualizationContainer : ObservableObject private ObservableCollection panels; /// - /// multithreaded collection lock. + /// Multithreaded collection lock. /// private object panelsLock; @@ -391,7 +391,7 @@ public void NotifyLivePartitionStatus(string storePath, bool isLive) } /// - /// Zoom to the spcified time interval. + /// Zoom to the specified time interval. /// /// Time interval to zoom to. public void ZoomToRange(TimeInterval timeInterval) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/TimelineVisualizationPanel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/TimelineVisualizationPanel.cs index bd10b063f..730818d32 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/TimelineVisualizationPanel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/TimelineVisualizationPanel.cs @@ -221,7 +221,7 @@ public ContextMenu ContextMenu VisualizationObject snappedVisualizationObject = this.Container.SnapToVisualizationObject; // Work out how many visualization objects we could potentially snap to. If one of - // this panel's visualization objects is currenlty being snapped to, then this total + // this panel's visualization objects is currently being snapped to, then this total // is actually one fewer, and we'll also need to add an "unsnap" menu item. int snappableVisualizationObjectsCount = this.VisualizationObjects.Count; if ((snappedVisualizationObject != null) && this.VisualizationObjects.Contains(snappedVisualizationObject)) diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanel.cs index f0d5716e0..b6ec52c64 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanel.cs @@ -63,7 +63,7 @@ public abstract class VisualizationPanel : ObservableTreeNodeObject private VisualizationObject currentVisualizationObject; /// - /// multithreaded collection lock. + /// Multithreaded collection lock. /// private object visualizationObjectsLock; diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelFactory.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelFactory.cs index 4f7012b24..4094d283b 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelFactory.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelFactory.cs @@ -26,7 +26,7 @@ public static VisualizationPanel CreateVisualizationPanel(VisualizationPanelType case VisualizationPanelType.XYZ: return Activator.CreateInstance(); default: - throw new ArgumentException(string.Format("Unknown visualiation panel type {0}.", visualizationPanelType.ToString())); + throw new ArgumentException(string.Format("Unknown visualization panel type {0}.", visualizationPanelType.ToString())); } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelType.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelType.cs index e720707af..b6a3de1dc 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelType.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelType.cs @@ -4,7 +4,7 @@ namespace Microsoft.Psi.Visualization.VisualizationPanels { /// - /// Visualizationo panel types. + /// Visualization panel types. /// public enum VisualizationPanelType { diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelTypeAttribute.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelTypeAttribute.cs index b36710eee..83012bb15 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelTypeAttribute.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/VisualizationPanelTypeAttribute.cs @@ -7,7 +7,7 @@ namespace Microsoft.Psi.Visualization.VisualizationPanels using System.Windows.Media; /// - /// represetns a visualization panel type attribute. + /// Represents a visualization panel type attribute. /// [AttributeUsage(AttributeTargets.Class)] public class VisualizationPanelTypeAttribute : Attribute diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/XYZVisualizationPanel.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/XYZVisualizationPanel.cs index 6165d5d82..801f1b536 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/XYZVisualizationPanel.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizationPanels/XYZVisualizationPanel.cs @@ -23,35 +23,30 @@ public class XYZVisualizationPanel : VisualizationPanel private double minorDistance = 5; private double thickness = 0.01; - /// - /// The extents of the models in the scene. - /// - private Rect3D sceneExtents; - /// /// The current plan for moving the camera. /// private Dictionary cameraAnimation; /// - /// The point towards which the camera looks. + /// The current camera position. /// - private Point3D modelCenterOffset; + private Point3D cameraPosition = new Point3D(15, 15, 15); /// - /// The curreent camera position. + /// The current camera look direction. /// - private Point3D cameraPosition; + private Vector3D cameraLookDirection = new Vector3D(-15, -15, -15); /// - /// The current camera look direction. + /// The current camera up direction. /// - private Vector3D cameraLookDirection; + private Vector3D cameraUpDirection = new Vector3D(0, 0, 1); /// - /// The current camera up direction. + /// The current camera field of view. /// - private Vector3D cameraUpDirection; + private double cameraFieldOfView = 45; /// /// Initializes a new instance of the class. @@ -113,31 +108,12 @@ public double Thickness set { this.Set(nameof(this.Thickness), ref this.thickness, value); } } - /// - /// Gets or sets the extents of the models in the scene. - /// - public Rect3D SceneExtents - { - get { return this.sceneExtents; } - set { this.Set(nameof(this.SceneExtents), ref this.sceneExtents, value); } - } - - /// - /// Gets or sets the offset of the center of the model from the origin. - /// - [IgnoreDataMember] - [ExpandableObject] - public Point3D ModelCenterOffset - { - get { return this.modelCenterOffset; } - set { this.Set(nameof(this.ModelCenterOffset), ref this.modelCenterOffset, value); } - } - /// /// Gets or sets the view's camera position. /// [IgnoreDataMember] [ExpandableObject] + [Description("The view camera position.")] public Point3D CameraPosition { get { return this.cameraPosition; } @@ -149,6 +125,7 @@ public Point3D CameraPosition /// [IgnoreDataMember] [ExpandableObject] + [Description("The view camera look direction.")] public Vector3D CameraLookDirection { get { return this.cameraLookDirection; } @@ -160,16 +137,29 @@ public Vector3D CameraLookDirection /// [IgnoreDataMember] [ExpandableObject] + [Description("The view camera up direction.")] public Vector3D CameraUpDirection { get { return this.cameraUpDirection; } set { this.Set(nameof(this.CameraUpDirection), ref this.cameraUpDirection, value); } } + /// + /// Gets or sets the view's camera field of view (in degrees). + /// + [IgnoreDataMember] + [ExpandableObject] + [Description("The view camera field of view.")] + public double CameraFieldOfView + { + get { return this.cameraFieldOfView; } + set { this.Set(nameof(this.CameraFieldOfView), ref this.cameraFieldOfView, value); } + } + /// /// Called when the views's current camera animation has completed. /// - public void CameraStoryboardCompleted() + public void NotifyCameraAnimationCompleted() { this.CameraAnimationCompleted?.Invoke(this, EventArgs.Empty); } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMap.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMap.cs index 3dffce980..d2a9db5bd 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMap.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMap.cs @@ -24,11 +24,6 @@ public class VisualizerMap // The list of default assemblies in which the visualizer mapper will search for visualization objects, adapter, and summarizers. private string[] defaultAssemblies = new[] { "Microsoft.Psi.Visualization.Common.Windows.dll" }; - /// - /// True if the Initialize() has been called, otherwise false. - /// - private bool isInitialized = false; - // The list of summarizers that were found during discovery. private Dictionary summarizers = new Dictionary(); @@ -38,6 +33,11 @@ public class VisualizerMap // The list of visualization objects that were found during discovery. private List visualizers = new List(); + /// + /// Gets a value indicating whether or not Initialize() has been called. + /// + public bool IsInitialized { get; private set; } = false; + /// /// Initializes the visualizer map. /// @@ -56,7 +56,7 @@ public void Initialize(List additionalAssembliesToSearch, string visuali // Load all the visualizers, summarizers, stream adapters this.DiscoverVisualizerObjects(assembliesToSearch, visualizerLoadLogFilename); - this.isInitialized = true; + this.IsInitialized = true; } /// @@ -159,7 +159,7 @@ private void DiscoverVisualizerObjects(List assemblies, string visualize // Get the list of types in the assembly Type[] types = this.GetTypesFromAssembly(assemblyPath, logWriter); - // Look for attributes denoting visualziation objects, summarizers, and stream adapters. + // Look for attributes denoting visualization objects, summarizers, and stream adapters. foreach (Type type in types) { if (type.GetCustomAttribute() != null) @@ -280,9 +280,9 @@ private void AddStreamAdapter(Type adapterType, VisualizationLogWriter logWriter private void EnsureInitialized() { - if (!this.isInitialized) + if (!this.IsInitialized) { - throw new InvalidOperationException("VisualizerMap.Initialize() must be called before calling this method."); + throw new InvalidOperationException($"{nameof(this.Initialize)} must be called before calling this method."); } } } diff --git a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMetadata.cs b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMetadata.cs index 8f806c063..f9fb57703 100644 --- a/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMetadata.cs +++ b/Sources/Visualization/Microsoft.Psi.Visualization.Common.Windows/VisualizerMetadata.cs @@ -5,6 +5,7 @@ namespace Microsoft.Psi.Visualization { using System; using System.Collections.Generic; + using System.Linq; using System.Reflection; using Microsoft.Psi.Visualization.Adapters; using Microsoft.Psi.Visualization.Summarizers; @@ -168,6 +169,60 @@ public static List Create(Type visualizationObjectType, Dict return metadatas; } + /// + /// Gets the visualizer metadata whose data type is hierarchically closest to a stream's data type. + /// Metadata objects that don't use an adapter are prioritized first. + /// + /// The data type of messages in the stream. + /// A list of metadatas to select from. + /// The metadata whose data type is closest (hierarchically, prioritizing non-adapters) to the message data type. + public static VisualizerMetadata GetClosestVisualizerMetadata(Type dataType, IEnumerable metadatas) + { + // Get the collection of metadatas that don't use an adapter + var nonAdaptedMetadatas = metadatas.Where(m => m.StreamAdapterType == null); + + // If there are any metadata objects that don't use an adapter, return the one + // whose data type is closest to the message data type in the derivation hierarchy. + if (nonAdaptedMetadatas.Any()) + { + VisualizerMetadata metadata = GetVisualizerMetadataOfNearestBaseType(dataType, nonAdaptedMetadatas); + if (metadata != default) + { + return metadata; + } + } + + // Return the metadata object whose data type is closest + // to the message data type in the derivation hierarchy. + return GetVisualizerMetadataOfNearestBaseType(dataType, metadatas); + } + + /// + /// Gets the visualizer metadata whose data type is hierarchically closest to a stream's data type. + /// + /// The data type of messages in the stream. + /// A collection of metadatas to select from. + /// The metadata whose data type is hierarchically closest to the message data type. + private static VisualizerMetadata GetVisualizerMetadataOfNearestBaseType(Type dataType, IEnumerable metadatas) + { + Type type = dataType; + do + { + VisualizerMetadata metadata = metadatas.FirstOrDefault(m => m.DataType == type); + if (metadata != default) + { + return metadata; + } + + type = type.BaseType; + } + while (type != null); + + // The collection of metadata objects passed to this method should be guaranteed + // to find a match. If that failed, then there's a bug in our logic. + throw new ApplicationException("No compatible metadata could be found for the message type"); + } + private static void Create(List metadatas, Type dataType, Type visualizationObjectType, VisualizationObjectAttribute visualizationObjectAttribute, VisualizationPanelTypeAttribute visualizationPanelTypeAttribute, StreamAdapterMetadata adapterMetadata) { // Create the metadata for the "visualize" menu command if required diff --git a/Sources/Visualization/Test.Psi.Visualization/Properties/AssemblyInfo.cs b/Sources/Visualization/Test.Psi.Visualization/Properties/AssemblyInfo.cs index ec753ed75..f6f556489 100644 --- a/Sources/Visualization/Test.Psi.Visualization/Properties/AssemblyInfo.cs +++ b/Sources/Visualization/Test.Psi.Visualization/Properties/AssemblyInfo.cs @@ -10,6 +10,6 @@ [assembly: AssemblyCopyright("Copyright (C) Microsoft Corporation. All rights reserved.")] [assembly: ComVisible(false)] [assembly: Guid("7cd463a8-61bb-4937-aa96-02da13e622d0")] -[assembly: AssemblyVersion("0.11.82.2")] -[assembly: AssemblyFileVersion("0.11.82.2")] -[assembly: AssemblyInformationalVersion("0.11.82.2-beta")] +[assembly: AssemblyVersion("0.12.53.2")] +[assembly: AssemblyFileVersion("0.12.53.2")] +[assembly: AssemblyInformationalVersion("0.12.53.2-beta")] diff --git a/Sources/Visualization/Test.Psi.Visualization/Test.Psi.Visualization.csproj b/Sources/Visualization/Test.Psi.Visualization/Test.Psi.Visualization.csproj index f4c9b4d2c..17676ef73 100644 --- a/Sources/Visualization/Test.Psi.Visualization/Test.Psi.Visualization.csproj +++ b/Sources/Visualization/Test.Psi.Visualization/Test.Psi.Visualization.csproj @@ -73,11 +73,16 @@ + + 2.9.8 + runtime; build; native; contentfiles; analyzers + all + - 2.1.0 + 2.1.1 - 2.1.0 + 2.1.1 1.1.118 diff --git a/build.sh b/build.sh index 4b5aed93d..4912762ca 100755 --- a/build.sh +++ b/build.sh @@ -12,6 +12,7 @@ (cd ./Sources/Integrations/CognitiveServices/Microsoft.Psi.CognitiveServices.Speech/ && . ./build.sh) (cd ./Sources/Integrations/CognitiveServices/Test.Psi.CognitiveServices.Speech/ && . ./build.sh) (cd ./Sources/Integrations/ROS/Microsoft.ROS/ && . ./build.sh) +(cd ./Sources/Kinect/Microsoft.Psi.AzureKinect/ && . ./build.sh) (cd ./Sources/Language/Microsoft.Psi.Language/ && . ./build.sh) (cd ./Sources/Media/Microsoft.Psi.Media.Linux/ && . ./build.sh) (cd ./Sources/Speech/Microsoft.Psi.Speech/ && . ./build.sh) @@ -21,5 +22,6 @@ (cd ./Sources/Runtime/Test.Psi/ && . ./build.sh) (cd ./Sources/Toolkits/FiniteStateMachine/Microsoft.Psi.FiniteStateMachine/ && . ./build.sh) (cd ./Sources/Tools/PsiStoreTool/ && . ./build.sh) +(cd ./Samples/AzureKinectSample/ && . ./build.sh) (cd ./Samples/RosTurtleSample/ && . ./build.sh) (cd ./Samples/LinuxSpeechSample/ && . ./build.sh)