having an SMB backend directly on restic will have less overhead

By what metrics?

Memory and CPU overhead for http processing due the local webserver required with rclone.

Memory and CPU overhead for http processing due the local webserver required with rclone.

Can you quantify that a bit more? Yes, obviously an additional process adds some overhead, but adding 1k+ lines of code to restic is not free either. So we have to weigh a possible performance benefit (that's all it is up to now) against the added complexity and maintenance overhead. Reviewing such a large PR is also a significant time investment which competes with other features that provide larger benefits for restic (e.g. features which cannot be properly implemented outside of restic, e.g. a restore progress bar).

Additional backends increase the work necessary to maintain them and to e.g. evolve the internal interfaces. It also adds even more new dependencies, which is also somewhat problematic (see e.g, #4119 ). So, in general we are very reluctant to add new backends. Even more so if those are already available via rclone.

Thank you for the insights.

I can run some benchmarks and try to provide some more details and speed comparisons in a couple of days.

Some other minor considerations -

1. The rest backend which has to be used with rclone does not support atomic replace during save, while the smb backend follows the same logic as the local backend along with pooling of smb connections and does support atomic replace (some refactoring may be possible to reduce duplicate code with the local backend and improve maintainability, but I thought that could be done as a subsequent step. That would significantly reduce the loc. Also the test setup and documentation added many lines, I was trying to be thorough).
2. The smb backend does call sync on the files and marks them readonly to avoid accidental modifications. I believe those functionalities would be skipped when using the rest api with rclone.

But I do understand that this needs to be weighed against the big picture for restic.

Here are the benchmarks:

Test data used:
Files - 23130
Dirs - 3599
Total size - 54.076 GiB

Note: All tests were performed on the same environment.

Test Environment:
Source:
Windows 10
Processor Intel(R) Core(TM) i7-6650U CPU @ 2.20GHz, 2201 Mhz, 2 Core(s), 4 Logical Processor(s)
Installed Physical Memory (RAM) 16.0 GB

Destination: SMB server (SMB3) on an Asustor NAS device.

Restic version: Build from #4186 commit - d062a82
Rclone version: Release v1.61.1 Intel/AMD - 64 Bit

Performance Monitor was used to capture Memory (in Bytes converted to MBs) and CPU usage (in % Processor time) for restic and rclone processes every 5 seconds.
For SMB backend rclone process was not running.
For Rclone backend, the usage of rclone and restic processes is summed up.

The Average CPU and Memory gives an idea about the average resource usage at a given instant.
However, since Rclone backend completes faster than SMB backend, the Total CPU and Memory usage difference is not as large.
The total CPU and Memory usage readings are just for a comparison between the two and do not have any significance on their own.

Benchmarks -
ResticRcloneBenchmark1.xlsx
ResticRcloneBenchmark2.xlsx
ResticRcloneBenchmark3.xlsx
ResticRcloneBenchmark4.xlsx

ResticSMBBenchmarkC5-1.xlsx
ResticSMBBenchmarkC5-2.xlsx
ResticSMBBenchmarkC5-3.xlsx
ResticSMBBenchmarkC5-4.xlsx

Summary -
BenchmarksSummary.xlsx

So the SMB backend is 10% faster than the Rclone backend when same max connections are used and it saves 11% total CPU usage, 26% average memory usage and 40% total memory usage.
These numbers might increase as the data size increases.

Besides this, the SMB backend makes Sync calls on files like the local backend and has ability to mark files as read-only to avoid accidental modifications like the local backend, which would not happen with the rclone+SMB backend. SMB backend also supports atomic replace during save.

The test results below indicate the pattern for performance characteristics with increasing data:

Test data used:
Files - 12985
Dirs - 139
Total size - 1.238 TiB

Benchmarks:

ResticSMBRcloneBenchmark-1TB .xlsx

Summary:

BenchmarksSummary.xlsx

For 1.24 TiB, the SMB backend is 18% faster than the Rclone SMB backend and it saves 8% total CPU usage, 20% average memory usage and 34% total memory usage.

I know that adding backends adds to the maintenance burden, but SMB is apparently very popular, judging by the number of bug reports about the Linux CIFS driver. The speedup is also impressive.

I agree with SMB being very popular. In our environment, we would benefit with SMB support. I had many problems with cifs. Also it adds complexity with mount, or with rclone, that I would like to avoid with backup solution.