Implemented a web server to browse a repository (snapshots, files) #4276

ducalex · 2023-03-29T20:45:46Z

What does this PR change? What problem does it solve?

It adds a web server to browse a repository (snapshots, files). Overall it is a very simple patch and requires no additional dependencies but it might be very useful for people who can't use fuse mount.

It allows to browse snapshots, view files, and cherry pick files to download an archive.

Usage: restic -r repo serve and open your web browser to http://localhost:3080/

demo.2.mp4

Was the change previously discussed in an issue or on the forum?

It is related to #60 but after reading the discussion I don't know if this PR would be enough to close it.

Checklist

I have read the contribution guidelines.
I have enabled maintainer edits.
I have added tests for all code changes.
I have added documentation for relevant changes (in the manual).
There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
I have run gofmt on the code in all commits.
All commit messages are formatted in the same style as the other commits in the repo.
I'm done! This pull request is ready for review.

rawtaz · 2023-03-29T20:49:58Z

I like the idea. I've actually started the very same thing myself a while ago. Nothing to show though.

ducalex · 2023-04-02T21:17:13Z

I like the idea. I've actually started the very same thing myself a while ago. Nothing to show though.

I think I've now accomplished everything I wanted for this PR! Seems to work pretty well so far.

Do you mind having a quick look at the code to see if I did something obviously stupid before I undraft it?

Enrico204

Interesting feature! LGTM!

cmd/restic/cmd_serve.go

Enrico204 · 2023-04-03T08:36:14Z

cmd/restic/cmd_serve.go

+<link rel="stylesheet" href="/style.css">
+<title>{{.Title}} :: restic</title>


Uhm, what about adding viewport meta tags, so that mobile browsers are happy?
https://developer.mozilla.org/en-US/docs/Web/HTML/Viewport_meta_tag

<meta name="viewport" content="width=device-width, initial-scale=1" />

I haven't tested the current code, though.

ducalex · 2023-05-05T03:30:32Z

Anyone willing to review? I'm happy to make any changes needed.

isarrider · 2023-05-12T15:06:46Z

This will be awesome for usability - Thank you @ducalex

isarrider · 2023-07-04T08:16:23Z

is there an update for the review?
It would increase the family compability by a lot ;)

ducalex · 2023-08-01T18:46:16Z

is there an update for the review? It would increase the family compability by a lot ;)

It's been 5 months so clearly not a priority or maybe my particular implementation is beyond salvaging? Would be nice to have an answer from a maintainer either way! I'm still available to complete the work once a direction has been chosen.

zanna-37 · 2024-01-28T13:43:08Z

@MichaelEischer any chances this can be reviewed? I tested it and works well. It would be a great improvement for windows users that cannot mount the repo easily.

I was able to rebase it quickly without knowledge of Go or restic, so pretty straightforward to integrate.
Thank you very much.

fd0 · 2024-04-11T18:09:02Z

I like this approach a lot! Before we can get this merged (and the implementation to handle HTML files correctly and fixes the XSS issue I've discovered by briefly testing it), the challenges mentioned in #4754 need to be addressed. :)

ducalex · 2024-04-13T18:49:34Z

I like this approach a lot! Before we can get this merged (and the implementation to handle HTML files correctly and fixes the XSS issue I've discovered by briefly testing it), the challenges mentioned in #4754 need to be addressed. :)

I think it's reasonable to delay the addition of WebDAV until you're done with your work and my quick draft clearly isn't production-worthy!

But, depending on the timeline you have in mind, it might still be worth merging this web server PR. Adding a random token to the URL and forcing html/svg to render as text/plain should negate any potential XSS issues.

I can appreciate that you have a vision and my implementation might not fit. But perhaps you can reshape my code into something that you can live with until your better version comes out?

I hope I do not come across as pushy, I just think we're really overdue for a convenient way to browse snapshots!

fd0 · 2024-04-15T16:25:10Z

I can appreciate that you have a vision and my implementation might not fit. But perhaps you can reshape my code into something that you can live with until your better version comes out?

Yes, that's something we can indeed do. Especially if we just need to swap out the backend implementation, that's quite easy to do and won't be noticed by users at all.

I hope I do not come across as pushy, I just think we're really overdue for a convenient way to browse snapshots!

For my personal use fuse was sufficient, but I can understand that it's inconvenient on macOS and not available on Windows, so I can see your need!

So, from my point of view we can start tackling the web server in parallel to the WebDAV server (and unify them once the latter is merged). The access control layer (e.g. a random password or so) we can even build in the context of the web server and then also use for the WebDAV server as well.

I can think of the following that needs to be done:

Build an access control layer
Move the code for the server to a new package below internal/restic somewhere, so that within the cmd_serve.go the server is only configured and run
Split the code for the server that handles serving blobs and the website that is returned
Embed the templates using the suggested embed functionality (that's really easy to do!)

Please let me know how we should split the work, I'd like to contribute. :)

ducalex · 2024-04-22T19:29:08Z

I can certainly split the code and add an authentication layer that can be shared by the servers, do something like:

package server:
    internal/server/auth.go         some shared authenticator that handles .htpasswd, maybe like rest-server?
    internal/server/web.go          the current server (html)
    internal/server/webdav.go    the webdav server

Not quite sure about naming. I also don't quite know where to put the HTML templates, internal/server/assets?

Otherwise feel free to take the lead and lay down the structure that you have in mind!

The code is pretty bad and unorganized, I am very new at Go. But it works!

fd0 · 2024-04-25T19:41:29Z

I've started reworking the server but then ran out of time (for today). It does not work yet, I'll need to debug this next.

fd0 · 2024-04-26T17:37:33Z

The server works now, I've split it out into its own package (directory) and embedded the assets, which are stored in their own files now.

MichaelEischer

Just a few random remarks from my side.

cmd/restic/cmd_serve.go

internal/server/assets/fs.go

MichaelEischer · 2024-04-28T16:24:55Z

internal/server/server.go

+	}
+
+	mux.HandleFunc("/tree/", func(rw http.ResponseWriter, req *http.Request) {
+		snapshotID, curPath, _ := strings.Cut(req.URL.Path[6:], "/")


The path surgery can be replaced with https://pkg.go.dev/net/http#StripPrefix

ducalex force-pushed the add-web-server branch from 73a742a to 4673916 Compare April 2, 2023 21:11

ducalex force-pushed the add-web-server branch from 85d47b5 to 8e940aa Compare April 2, 2023 22:26

Enrico204 reviewed Apr 3, 2023

View reviewed changes

ducalex marked this pull request as ready for review April 7, 2023 20:55

konidev20 mentioned this pull request Sep 7, 2023

Please allow for restic to be used as a "Go library" #4406

Open

fd0 force-pushed the add-web-server branch from 505074a to bbe1c09 Compare February 17, 2024 19:40

ducalex closed this Apr 2, 2024

ducalex force-pushed the add-web-server branch from b80cdaf to 831fc44 Compare April 2, 2024 21:55

ducalex reopened this Apr 2, 2024

ducalex mentioned this pull request Apr 2, 2024

Add WebDAV server as alternative for fuse #485

Open

fd0 mentioned this pull request Apr 11, 2024

Add WebDAV server (ie "mount" on Windows) #4754

Draft

8 tasks

ducalex added 4 commits April 25, 2024 20:35

Implemented a web server to browse a repository (snapshots, files)

3541a3a

The code is pretty bad and unorganized, I am very new at Go. But it works!

fixed compilation issues

e977c9f

improved readability

5fc118f

satisfy the linter

993eb70

fd0 force-pushed the add-web-server branch from 0c20d04 to 993eb70 Compare April 25, 2024 18:36

Rework server

8a5ac6d

serve: Properly shut down the server

d463996

Fix server

303dda6

MichaelEischer reviewed Apr 28, 2024

View reviewed changes

fd0 added 3 commits April 28, 2024 22:00

serve: Only include html and css files

fb45bd5

serve: Use non-exclusive read lock

cd72014

serve: Change assets embedding

c374a4e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented a web server to browse a repository (snapshots, files) #4276

Implemented a web server to browse a repository (snapshots, files) #4276

ducalex commented Mar 29, 2023 •

edited

rawtaz commented Mar 29, 2023

ducalex commented Apr 2, 2023

Enrico204 left a comment

Enrico204 Apr 3, 2023

ducalex commented May 5, 2023

isarrider commented May 12, 2023

isarrider commented Jul 4, 2023

ducalex commented Aug 1, 2023

zanna-37 commented Jan 28, 2024

fd0 commented Apr 11, 2024

ducalex commented Apr 13, 2024

fd0 commented Apr 15, 2024 •

edited

ducalex commented Apr 22, 2024 •

edited

fd0 commented Apr 25, 2024

fd0 commented Apr 26, 2024

MichaelEischer left a comment

MichaelEischer Apr 28, 2024

		<link rel="stylesheet" href="/style.css">
		<title>{{.Title}} :: restic</title>

Implemented a web server to browse a repository (snapshots, files) #4276

Are you sure you want to change the base?

Implemented a web server to browse a repository (snapshots, files) #4276

Conversation

ducalex commented Mar 29, 2023 • edited

What does this PR change? What problem does it solve?

Was the change previously discussed in an issue or on the forum?

Checklist

rawtaz commented Mar 29, 2023

ducalex commented Apr 2, 2023

Enrico204 left a comment

Choose a reason for hiding this comment

Enrico204 Apr 3, 2023

Choose a reason for hiding this comment

ducalex commented May 5, 2023

isarrider commented May 12, 2023

isarrider commented Jul 4, 2023

ducalex commented Aug 1, 2023

zanna-37 commented Jan 28, 2024

fd0 commented Apr 11, 2024

ducalex commented Apr 13, 2024

fd0 commented Apr 15, 2024 • edited

ducalex commented Apr 22, 2024 • edited

fd0 commented Apr 25, 2024

fd0 commented Apr 26, 2024

MichaelEischer left a comment

Choose a reason for hiding this comment

MichaelEischer Apr 28, 2024

Choose a reason for hiding this comment

ducalex commented Mar 29, 2023 •

edited

fd0 commented Apr 15, 2024 •

edited

ducalex commented Apr 22, 2024 •

edited