Best way of copying (unmanaged) memory

[ldematte]

Hi clr team!
The basic question seems easy, but there are many variant and differences so bear with me :)
I have this situation: unmanaged memory (float*), a ReadOnlySpan over it, and the need to copy the memory to managed.
The memory is not huge, but it is not small either (somewhere south of 1Mbytes), and of course I would like to avoid unnecessary allocations.
Given these three scenarios:

1. No managed memory allocated. I could do just a .ToArray() on the Span, but is this the fastest way? Or is there a better (faster) way?
2. Managed memory already allocated. I am thinking about fixed(float* p = span), pin the managed memory, and then memcpy. I am not fond of pinvoking memcpy but it seems the fastest. Alternatives?
3. Copy a Span to a double[]. Of course I can do a for() over the span casting each item into a double, but are there better alternatives? Maybe something in Intrinsics, like CVTPS2PD?

To add a layer of complication, the code will run both con Core (3.1, eventually also 5) and Framework 4.x.
IIRC span performance are different (worse) in Framework. Looking at the decompiled assemblies I am concerned about .ToArray() and [] (indexed access). Is it true, or is it a myth?

Any suggestion or pointers in the right direction is really welcome!
L

[gfoidl]

1. Use ToArray() -- it's the easiest and most readable way to do it. It it proves that it's too slow, then file an issue and it can be improved (although it's basically just the allocation of the final array + memmove).
   If the source is really large -- not the case here -- you could allocate an uninitialized array, then copy the unmanaged memory to the managed one. But this won't work for .NET Full.

2. Use ReadOnlySpan<T>.CopyTo(Span<T>). No need to pin and p/invoke.

3. For .NET Core 3.1+ you could use a vectorized approach / intrinsics.
   For .NET Full I don't think there's a vectorized approach, as System.Numerics.Vectors has no conversion from float to double (except I missed something).

Is it true, or is it a myth?

Measure, then you'll know (for sure and for your workload).

[jkotas]

For ~1MB block of memory, all memory copying methods in recent runtimes (Span.ToArray, Span.CopyTo, Array.Copy, Buffer.BlockCopy, Buffer.MemoryCopy, ...) end up calling memcpy. We depend on memcpy to provide efficient memory copying for large memory blocks.

For .NET Framework, you can call Buffer.MemoryCopy to avoid PInvoking. Buffer.MemoryCopy always calls memcpy in .NET Framework. The other methods such as Span.CopyTo may not.

.NET Core 3.1+ you could use a vectorized approach / intrinsics.

Implementing memory copying that is as efficient as possible on variety of machines, for all combinations of block sizes, source and target alignment, is very complex. The typical memcpy implementation has number of strategies, and picks one based on variety of factors. Unless you know the exact machine and conditions that this will run on, it is probably not worth for you to spend time on this.

[gfoidl]

Just to note: the quote (i.e. point 3) is for reading native float-buffer and converting to double-data on the managed side (that's at least how I understood the question).

[ldematte]

Yep, exactly: copy and converting.

[jkotas]

[] (indexed access). Is it true, or is it a myth?

Yes, Span<T> implementation for .NET Framework is quite a bit slower compared to .NET Core. We have done a lot of work on optimizing Span<T> in .NET Core.

[ldematte]

First of all, thank you for your answers, very informative!

Measure, then you'll know (for sure and for your workload).

Yeah, I know, I was just being lazy :)
Anyhow, today during race there was not much I could do so I coded the benchmark. I can confirm what you two said, with some surprises too.

First, .NET Full 32bit

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
  [Host]     : .NET Framework 4.8 (4.8.4200.0), X86 LegacyJIT
  DefaultJob : .NET Framework 4.8 (4.8.4200.0), X86 LegacyJIT


|              Method |      Mean |    Error |   StdDev |
|-------------------- |----------:|---------:|---------:|
|  ReadOnlySpanCopyTo |  65.93 μs | 1.049 μs | 0.981 μs |
| ReadOnlySpanToArray | 267.73 μs | 4.201 μs | 3.508 μs |
|    ReadOnlySpanLoop | 196.55 μs | 2.040 μs | 1.908 μs |

CopyTo is the clear winner. ToArray is like allocation + CopyTo. Looping is not so good, as expected.
Switching to 64 bit

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
  [Host]     : .NET Framework 4.8 (4.8.4200.0), X64 RyuJIT
  DefaultJob : .NET Framework 4.8 (4.8.4200.0), X64 RyuJIT


|              Method |      Mean |    Error |    StdDev |
|-------------------- |----------:|---------:|----------:|
|  ReadOnlySpanCopyTo |  96.15 μs | 0.922 μs |  0.862 μs |
| ReadOnlySpanToArray | 374.65 μs | 7.381 μs | 14.043 μs |
|    BufferMemoryCopy |  68.08 μs | 1.317 μs |  1.168 μs |
|    ReadOnlySpanLoop | 190.39 μs | 1.706 μs |  1.596 μs |

Suprinsingly (maybe) CopyTo is slightly worse. I added Buffer.MemoryCopy, and it seems to be the best alternative.
Finally, Core 3.1

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.100-preview.5.20279.10
  [Host]     : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT
  DefaultJob : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT


|              Method |      Mean |    Error |    StdDev |
|-------------------- |----------:|---------:|----------:|
|  ReadOnlySpanCopyTo |  67.78 μs | 0.938 μs |  0.877 μs |
| ReadOnlySpanToArray | 341.45 μs | 6.797 μs | 14.776 μs |
|    BufferMemoryCopy |  68.10 μs | 1.210 μs |  1.132 μs |
|    ReadOnlySpanLoop | 133.20 μs | 1.887 μs |  1.673 μs |

CopyTo is like MemoryCopy (expeted) and lopping over the ReadOnlySpan is better (expected)

Finally, copy and convert. Of corse Core here is better than Full because indexed access on ReadOnlySpan is better:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.100-preview.5.20279.10
  [Host]     : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT
  DefaultJob : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT


|            Method |     Mean |   Error |  StdDev |
|------------------ |---------:|--------:|--------:|
|  CopyAndCastLoop | 183.7 μs | 2.84 μs | 2.65 μs |

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
  [Host]     : .NET Framework 4.8 (4.8.4200.0), X64 RyuJIT
  DefaultJob : .NET Framework 4.8 (4.8.4200.0), X64 RyuJIT


|            Method |     Mean |   Error |  StdDev |
|------------------ |---------:|--------:|--------:|
|  CopyAndCastLoop | 250.2 μs | 4.72 μs | 4.85 μs |

The clear winner here is using Intrinsics:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i5-4300U CPU 1.90GHz (Haswell), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.100-preview.5.20279.10
  [Host]     : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT
  DefaultJob : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT


|        Method |      Mean |    Error |   StdDev |
|-------------- |----------:|---------:|---------:|
| LoadStoreSse2 |  59.56 μs | 1.075 μs | 0.953 μs |
|  LoadStoreAvx |  63.99 μs | 0.454 μs | 0.402 μs |

It costs as much as just copying. Quite impressed! Too bad we are not fully on Core yet :/

I will try to see if 5.0 has any differences and if I can get Avx2 to work too.

[Clockwork-Muse]

CopyTo is the clear winner. ToArray is like allocation + CopyTo.

.... you haven't posted your benchmark code, but my first question would be "are you also including the cost of allocating the array used in just-CopyTo?". ToArray will win for simplicity of implementation, whereas using just CopyTo is only a benefit if you're copying (into the same array, or into a pooled array) multiple times. This is an area where a microbenchmark can distort results.

[ldematte]

Sorry, it was clear in my head :) but I should have written

ToArray is like allocation + CopyTo, which is totally expected as I am reusing the array in CopyTo

In a benchmark I have not published here I just did allocation + CopyTo, and the cost was exactly the same as ToArray (at least in .NET Full). Totally expected.
I have cases in which CopyTo is a benefit as I mentioned in my opening question:

2. Managed memory already allocated.

I am now using CopyTo in those cases.

To wrap it up: I do not have a AVX512 machine :/ so I cannot try it (but it was only curiosity), and running the benchmark on .NET 5 leads to the same results as Core 3.1 on my machine.