Stream through wifi?

[sleepyeye]

Hi,

If I want to scan large space such as a room, I have to connect ipad to laptop.
This is quite cumbersome so I want to stream RGBD images through wifi.

Is there any way to stream RGBD image through wifi or any chances to add this feature in the future?

[marek-simonik]

Hi @sleepyeye ,

I have tried to implement Wi-Fi streaming, but due to very high bandwidth requirements, I decided not to add this feature (there were additional issues) — the average Wi-Fi speeds were not enough to reliably transmit the data in real time.

If you do not require live streaming, then you can use the new Shareable/Internal Format export option in the latest version of Record3D. The exported .r3d file is in fact a ZIP file containing all frames of the 3D video in form of pairs of JPEG files and .depth files.

The .depth files contain the corresponding depth map for the JPEG file. They are compressed using LZFSE — see the source code of the record3d library to see how to decompress it.

You could alter the Unity demo project to load the pairs of JPEG images and depth maps instead of loading the color and depth data from the iOS device.

[sleepyeye]

Hi @marek-simonik , thanks for responding.

Unfortunately, I'm working on a project based on live-streamed point cloud.
For validation purposes, I can use .r3d files but the validation has already been done.
So I need to test my project with the actual live-streamed point cloud. (And some frame losses is okay.)

I thought real-time and lossless Wi-Fi streaming is possible because iPad pro 4 supports Wi-Fi 6 which has almost similar bandwidth to USB 3.1.
But it seems like there are practical issues that I can't imagine.

Do you have any plan for adding this feature but left it as an experimental feature?

[marek-simonik]

I am going to do new Wi-Fi streaming tests at the end of the next week — feel free to drop me an email (support@record3d.app) on Friday (2020/10/02) to ask for the results.

[sleepyeye]

Okay. Thanks a lot.

[adrelino]

I am also interested in Wi-Fi streaming. Are there any updates on the progress?

If bandwidth is still a problem frame-skipping would be ok (I imagine 10Hz would be enough).

Alternatively or additionally, one can think about a (more) lossy compression. I looked in the code and it seems that for:

1. RGB stream you already use JPEG based (de)compression:
   

   record3d/src/Record3DStream.cpp

   Line 157 in 8127e38

   uint8_t* rgbPixels = stbi_load_from_memory( rawMessageBuffer.data() + offset, currSize, &loadedWidth, &loadedHeight, &loadedChannels, STBI_rgb );

   
   Is there a way to increase the compression ratio? I know jpeg has options to control the quantization.

2. Depth stream you already use the lossless LZFSE algorihtm:
   

   record3d/src/Record3DStream.cpp

   Line 184 in 8127e38

   size_t outSize = lzfse_decode_buffer( $destinationBuffer, depthBufferSize_, $compressedDepthBuffer,

   
   Was this picked because there is already a fast compression routine (possibly in Hardware) from Apple on the iOS side? Maybe a more tailed lossless compression for depth images achieves higher compression?:
   • Fast Lossless Depth Image Compression (RVL) (Microsoft, 2016) (https://www.microsoft.com/en-us/research/uploads/prod/2018/09/p100-wilson.pdf) and see a videos https://www.youtube.com/watch?v=DhT8NFzbGrY and https://www.youtube.com/watch?v=WYU2upBs2hA

For a lossy depth compression, a quick search revealed the following options:

• Temporal RVL (Stanford, 2020) (https://vhil.stanford.edu/pubs/2020/temporal-rvl-a-depth-stream-compression-method) code here https://github.com/hanseuljun/temporal-rvl
• Adapting Standard Video Codecs for Depth Streaming (UCL, 2011) http://reality.cs.ucl.ac.uk/projects/depth-streaming/depth-streaming.pdf
• Effective Compression of Range Data Streams for Remote Robot Operations using H.264 (Freiburg University, 2014) (http://www2.informatik.uni-freiburg.de/~spinello/nenciIROS14.pdf)

I would be happy to collaborate with you to get the required Mbps down so Wi-Fi streaming becomes a reality soon.

[marek-simonik]

Hello @adrelino,

apologies for not responding sooner and thank you for your reply.

My intention with Record3D streaming was to get as accurate depth data as possible, so I opted for lossless depth map compression of float32 depth values, for which LZFSE was the most viable option (mainly due to native iOS support and compression speed).

The goal of depth accuracy was the reason why I abandoned Wi-Fi streaming after performing tests that showed a large bandwidth would be required for the streaming to work without frame drops. However, there seems to be growing interest in Wi-Fi streaming, so I am going to start working on it in about two weeks time. Please note that lossy compression will be necessary to achieve real-time Wi-Fi streaming without significant frame drops, thus the Wi-Fi streaming feature will be suited just and only for entertainment purposes.

To comment on your two remarks:

1. There is indeed the option to choose the JPEG compression ratio, I can change that parameter.

2. RVL seems to be compression method for float16 depth images, which would explain why you got only half of the image when using it in your pull request (fix compile error on linux, add cstring include #5). Record3D is not using float16 depth values, because 16bit floats cannot reliably store sub-centimeter depth values.

[andybak]

Hi. I'm interested in this. I can measure speeds of up to 830Mbps on my domestic Wifi. I've also got access to some 5g test equipment and I would very much like to see how far we can push that for WAN access.

What kind of bandwidth requirements are you seeing - both currently and with your planned lossy version?

[andybak]

In comparison - the other app I'm playing with is https://github.com/keijiro/Rcam2 by @keijiro

That packs the RGB, depth and a stencil into a single uncompressed 1080p NDI frame. I believe that uses about 100Mbps and the results are very satisfactory for the purposes I have in mind.

On reflection I think the key point is "people have different requirements". USB is a great fallback but wireless should work if either a) you've got fantastic wifi b) you don't need great depth accuracy c) you can lose framerate/depth resolution etc etc

It's only in the case where someone needs all of this at once and doesn't have the network bandwidth to support it - but most people will be happy to sacrifice one of more dimensions to suit their network conditions.

[marek-simonik]

Hi @andybak,

I'll start working on the Wi-Fi streaming feature at the end of the next week, so I can't guarantee a specific bandwidth that I will end up with.

Your domestic Wi-Fi would for sure handle streaming of even the non-lossy stream (~38 Mbps for LiDAR @60fps and ~85 Mbps for the selfie FaceID camera @30 FPS). However, similar to what you wrote, my goal is to offer the Wi-Fi streaming also to people with worse network conditions, so I plan to reduce the required bandwidth hopefully to sub-10 Mbps.

[andybak]

Crikey. That's going to make even WAN-streaming viable. You can potentially live stream RGB and depth to a WebVR client like Mozilla Hubs. I'd love to get a proof of concept going of that...

[andybak]

Good luck.

If you need a beta tester (especially with regards to Unity integration) let me know.

[marek-simonik]

Thank you, I'll post an update after I'll achieve some progress with the Wi-Fi streaming feature.

[adrelino]

My intention with Record3D streaming was to get as accurate depth data as possible, so I opted for lossless depth map compression of float32 depth values, for which LZFSE was the most viable option (mainly due to native iOS support and compression speed).

2. RVL seems to be compression method for float16 depth images, which would explain why you got only half of the image when using it in your pull request (fix compile error on linux, add cstring include #5).

Thanks for your hint, I was indeed not aware of these differences. But I got around the issue and succeeded in saving the complete depth image as 16 bit depth in RVL format by first loading and uncompressing the 32 bit .depth image with lzfse, then using openCV to convert from CV_32FC1 to CV_16FC1 and then saving the resulting bytes using RVL, see
my code from here :

void compressAndSave(std::string filename, cv::Mat depth){  //16 bit depth
    RvlCodec codec;
    int nPixels = depth.total();
    short buf[nPixels]; //usually too large
    int nBytes2 = codec.CompressRVL((const unsigned short*) depth.data, (unsigned char*) buf,nPixels);
    write_buffer_to_disk(filename, (char*) buf,nBytes2);
}

This is then used as follows:

cv::Mat depth32 = readDepth(cv::utils::fs::join(rgbd,std::to_string(i)+".depth"),m.w,m.h);
depth32.convertTo(depth16,CV_16FC1);
compressAndSave(rvlFile,depth16);

Of course, I asked myself how much space can be saved and information is lost by this conversion process from 32 to 16 bit float depth images so I ran it on a sample sequence I recorded with the iPad Pro 2018 (3rd Gen) TrueDepth camera.
Visually, there is no difference, so in the last diff image on the right the colorscale is scaled between 0 and max, where max is still below 1mm (see code below) and only happens in the background (white):

So I also looked at the raw values:

frame: 0
lzfse   size:   249.825 kB       range [0.27925,3.89213] m
RVL     size:   205.06 kB        range [0.279297,3.89258] m
diff    mean:   0.179536 mm      max: 0.968933 mm

Record3D is not using float16 depth values, because 16bit floats cannot reliably store sub-centimeter depth values.

The saving in space (250 vs 200kb) is not too large, but note that 16 bit can store sub-centimeter depth values, since the maximum difference between the 32 and 16 bit image is below 1 millimeter (this was consistent for my whole sequence).

[marek-simonik]

Thank you for the analysis, I guess I rushed the assumption that 16bit floats cannot store sub-millimeter measurements in a reliable way; I based that assumption on a very simple test with numpy:

import numpy as np

num = 0.9765
np.float32(num)  # 0.9765
np.float16(num)  # 0.9766

Anyway, I am going to start working on the Wi-Fi streaming feature by the end of this week, so I will try to push the file size down as much as I can.

I don't want to promise something I would not be able to deliver, but I hope to make the Wi-Fi streaming feature publicly available in about 3 weeks from now.

[adrelino]

For 16 bit floats it really depends how many bits are used for the mantissa and how many for the exponent. Might be different between numpy and CV_16F. But I just noticed that RVL, though it works, is actually not made for a floating point datatype but an integer type, namely 16 bit unsigned shorts (CV_16U in OpenCV).

These allow to store 256*256 = 65.536 distinct values. For Kinect style sensors, people usually save discretize in mm, so the 32bit float meter value is multiplied by 1.000 before saved to the unsigned short integer type. This allows to store ranges from 0 m up to 65,5 meters without running into overflow.

Since in the sequence that I recorded, the maximum depth value is around 4 meters, I wanted to increase the depth resolution furthermore and discretized to 1/10 of a millimeter, so I multiplied by 10.000 and can save ranges of up to 6,55m.

I summarized my results in the following table:

CV_32F	CV_16F	CV_16U 1000	CV_16U 10000
diff image _
size 249.825 kB (lzfse)	205.06 kB (RVL)	188.452 kB (RVL)	249.152 kB (RVL)
range [0.27925,3.89213] m	[0.279297,3.89258] m	[0,3.892] m	[0,3.8921] m
diff mean	0.179536 mm	0.255711 mm	0.0236755 mm
diff max	0.968933 mm	0.500202 mm	0.0503063 mm

So it seems that for float16, the mean is lower, while the max is higher than for uint16 with mm discretization.

[marek-simonik]

After quite some time, I have finally managed to implement Wi-Fi streaming. I am sorry for the delay.

The Wi-Fi streaming feature is implemented via WebRTC (works only on local Wi-Fi network); that way, I could ensure that streaming would be possible even on very low bandwidth networks. RGBD data are encoded in lossy mp4 videos, so Wi-Fi streaming will be intended mainly for entertainment purposes.

The feature is described in more detail at https://record3d.app/features. There are also two JS demos:

• Simple Wi-Fi Streaming demo
• Wi-Fi Streaming & RGBD mp4 3D Video demo

Wi-Fi Streaming is a part of a paid Extension Pack, but I am giving you the Extension Pack for free — I already sent a Promo Code to @adrelino (via email).

However, I could not find your email address — @adrelino — please send me an email at support@record3d.app and I will give you a Promo Code too as a thanks :).

[andybak]

I've got pretty good Wifi. I'd like to stream with low compression rates and the depth packed into 24 or 32 bits of video. Is this possible? What's the best quality you've implemented over wireless?

[marek-simonik]

I may add lossless (or near-lossless) Wi-Fi streaming as an additional Settings in a future update. That Wi-Fi streaming option would be based on a completely different principle — likely by utilizing RVL, as proposed by @adrelino. I sent you an email.

[sssynk]

Will Wi-Fi streaming ever be added to this C++/Python library? I tried to implement it in Python with the aiortc Python library with no success. Just wondering if this will ever be added.

[marek-simonik]

Will Wi-Fi streaming ever be added to this C++/Python library? I tried to implement it in Python with the aiortc Python library with no success. Just wondering if this will ever be added.

This library in particular (i.e. record3d) is specific to USB, so there will be no support for Wi-Fi.

However, you can use the existing Wi-Fi streaming feature of Record3D, which streams RGBD video via WebRTC (the depth values are encoded into the Hue component of HSV, so it there is lossy compression). Here you can see a minimal working example of how to connect to Record3D's WebRTC stream (comments in the JS code describe the specifics you need to follow for successful connection): https://github.com/marek-simonik/record3d-simple-wifi-streaming-demo

Since WebRTC is a standard, there should be WebRTC implementations in major languages with the same API, which should allow you to replicate the JS demo linked above.

[graycrawford]

Hi! Is it possible to stream depth >3m? Would that be at the expense of depth accuracy given the dynamic range of the mapping? I see the max depth was recently adjusted for recorded mp4s.

[marek-simonik]

Hi,
the color-based depth encoding is able to store values in the range 0–1, so a mapping that converts the values in 0–1 to absolute depth values in meters is required. By default, a static mapping [0; 1] -> [0; 3] meters is used for mp4 videos and Wi-Fi streaming.

After enabling "Settings > Export options > RGBD mp4 video dynamic max. depth", the exported mp4 files will include the rangeOfEncodedDepth key inside the JSON string stored in the mp4 metadata (it stores the min and max depth of the whole video). That way, a dynamic depth mapping [0; 1] -> [rangeOfEncodedDepth[0]; rangeOfEncodedDepth[1]] meters can be used. This is possible thanks to the fact that the distance of the farthest depth value of the whole video sequence is known before the depth data is encoded.

However, when streaming via Wi-Fi, there is (of course) no way to tell what will be the maximum depth value observed throughout the whole Wi-Fi streaming session. A possible solution would be to associate a rangeOfEncodedDepth with each encoded frame, but I am not sure how to do this reliably (i.e. how to achieve perfect metadata synchronization) while using WebRTC.

[andybak]

Would a compomise be to have a "depth" option that switched between say 0-3m, 0-6m and 0-12m? (can't remember the max LIDAR depth but you get my point)

[andybak]

Another thought - use a logarithmic mapping or some other non-linear mapping?

[marek-simonik]

I'm not sure if logarithmic/nonlinear mapping would work well; my main concern is that the WebRTC video compression can introduce significant noise (especially with low bandwidth) and that the nonlinearity would only amplify already existing noise (but I haven't tested this, so I might be wrong).

However, if it would be enough for your purposes (@graycrawford and @andybak) to have the option to manually specify the maximum depth value used for depth encoding during Wi-Fi streaming, then I will implement such option into the next Record3D update.

[graycrawford]

hi @marek-simonik, manually specifying the maximum depth value would be perfect!
For context, I'm doing visuals for a live music show November 19th (in about a week) where I'm using the streamed (either wifi or USB) depth data from an audience to influence a fluid sim projected on the wall.

As it doesn't need to be hyper precise, only gesturally expressive, we'll take the noise in stride, likely just clamping the data from the far wall to isolate only the figures. Our space is max 10m, but audience occlusion will mean we're likely to be mostly measuring people 2-7m distant.

Thank you for your rapid response! It's such a cool tool and we haven't found a better tool for our needs; (something like Zig Sim has worked in the past but has also very limited range by default.)

[marek-simonik]

Thanks for sharing the screenshots, @graycrawford :)!

Please download the latest update of Record3D (version 1.8.3, released yesterday) — there is a new option in the Settings tab > Export options > Wi-Fi Streaming max depth.

You can modify the value to adjust the maximum depth range used for depth encoding during Wi-Fi streaming (the depth range will be [0; <your-custom-value>] meters]). I'd recommend using USB streaming instead of Wi-FI streaming whenever possible; with USB streaming, the depth maps are lossless, which will avoid unnecessary H.264 compression noise.

A tip for using Wi-Fi streaming

increase the maximum depth value so that it's a bit higher than the maximum distance you need to measure (e.g. if you need to measure people 7 meters away, then set the max. depth to 8–10 meters).

The depth is encoded into the Hue value of the HSV color space, which is circular. That means depth values too close to the camera (near 0) and values near the maximum depth value will be encoded into similar colors. Increasing the maximum depth will allow you to better discern between those two extremes and thus better filter the depth values.

[graycrawford]

Incredible! On the USB streaming front, does the wifi max depth value apply to USB streaming too?

[marek-simonik]

Incredible! On the USB streaming front, does the wifi max depth value apply to USB streaming too?

No, it does not apply. When streaming via USB, there is no need to encode the depth image into HSV, so there is no maximal depth value to be applied. Instead, the raw float32 depth values (in meters) are compressed in a lossless way (LZFSE), so when using this library, you get the original (non-clipped) depth values.