Debian packaging (repository)

[xor-gate]

Hi all,

I started to create a debian package of restic which builds and runs fine and is installable. In some weeks I will hopefully setup a debian wheezy repository. Currently I have my draft in my master branch, look at the debian folder and the Makefile deb target:
https://github.com/xor-gate/restic

The tests are disabled because I could not get them work.
Also when using build.go and /tmp is mounted as noexec it fails in a painfull way :_).

I'm no real debian developer but have many experience on linux and unix systems. So it is possible it doesn't meet the full debian packaging guidelines (yet).

[fd0]

Thanks for the effort. You can set the temporary directory via the environment variable TMPDIR.

[xor-gate]

I will look into it and when it is acceptable I will document, squash and create a pull request.

[mika]

https://github.com/xor-gate/restic does no longer exist, is anyone (still) working on the Debian packaging?

[Mebus]

A Debian repo for this would be great.

Mebus

[xor-gate]

@mika, i had screwed up the repository. Now it is "polished" and reproducable.

I also added a little documentation: https://github.com/xor-gate/restic/tree/debian#debian-packaging

https://github.com/xor-gate/restic/tree/debian
https://github.com/xor-gate/restic/blob/debian/Makefile
https://github.com/xor-gate/restic/releases/tag/v0.0.1-debian

Please use the makefile in a container or VM. No guarantees :-)

[xor-gate]

If @fd0 is interessted I can clean it up a bit and create a pull request of this ? Actual signing and setup of a repository can be done with e.g. http://bintray.com . It is not the nicest solution but it works and there doesn't need to be someone hosting a package repository (pain). Optional you could run the debian repository yourself without the nasty reprepro tools and stuffs: http://www.aptly.info/

[sjoerd-ccu]

@xor-gate Do you intend to push/maintain it into debian? Assuming my first test with restic work out nicely I'd be happy to sponsor an upload :)

[xor-gate]

@sjoerd-ccu i'm not a official debian developer and maintaining a package can be time consuming. My current debian packaging will need some love to actual make it into the debian packaging system.

[sjoerd-ccu]

I'm a Debian Developer, hence i can sponsor an upload and review your packaging. But yes the real choice you need to make is whether you want to maintain it properly :)

[fd0]

I was a Debian Developer myself once, but did not have the time any more.

I'd very much appreciate someone stepping up and maintaining a proper package, either inside of Debian or outside. I can also host a repository for packages outside of Debian (e.g. on GitHub), but this would need someone else to manage the packaging (I'll be glad to assist of course) :)

If there is interest, we can e.g. also add a repository for group-maintaining, e.g. at github.com/restic/pkg-debian or so.

[xor-gate]

I would like to maintain the package (outside of debian) but for having it upstream in Debian it would be much more effort and community contribution because of the guideliness and people not breaking their setups. Then I would setup it under http://bintray.com which just works out of the box and nobody needs to run their own server. Keep in mind that we maybe want a restic/packaging repository so also archlinux and other distros could benefit. Maybe also RPM specfile. I had a look at http://packager.io but the packages it emits are bloated.

[jmallach]

Hi there,

As another Debian Developer with an eye put on restic, I can only urge you to reconsider your view of maintaining restic outside of Debian is much effort. It's basically the same effort, or in my experience, actually less effort.

Debian would give you visibility, mirrors, continous testing of upgrades, build reproducibilty, a gateway for inclusion in Ubuntu and so many other things I cannot list here.

You already have a Debian developer offering sponsorship. Here's another one.

I'd be glad to see your packaging effort in Debian officially. If it's not you, it's quite easy someone else will end up doing it sooner or later.

[fd0]

So, as far as I can see we have enough people who are interested in maintaining this for Debian. You can have a repository in the restic organisation if you like, e.g. github.com/restic/pkg-Debian, and have access to that repo. Are you interested in group-maintaining?

From my point of view (main author) It'd be awesome to have a group of people maintaining the packages for the different distributions, I'm glad to help :)

[xor-gate]

@fd0 a seperate repo would be handy. And I would like to kickstart it and with multiple people involved it would be nice as most people do this in their spare time (me) and quality will be better. Hopefully it will make it someday into the official debian infrastructure.

[fd0]

@xor-gate great, I've created https://github.com/restic/pkg-Debian and enabled push access for you. When you have someone else who should have write access, please let me know. PRs can be created, and I've enabled the issue tracker.

[mika]

@xor-gate thanks for sharing your work! Sadly your approach with the "go get" inside the build process (https://github.com/xor-gate/restic/blob/debian/Makefile) isn't what's considered best practices for Debian packaging and has no chance in resulting in an official Debian package (as in "within the Debian project"), especially since the official build infrastructure doesn't even allow network access during build time. Of course this shouldn't prevent anyone from providing Debian packages outside of Debian, so jftr && jfyi. :)

A few days ago I started to look at the Debian packaging of restic and this is what I have in my shell history what should be relevant as a starting point (disclaimer: that might not be 100% correct nor complete, consider it as a starting point where I just left off):

 dh-make-golang github.com/restic/restic
 dh-make-golang github.com/jessevdk/go-flags-dev
 dh-make-golang github.com/juju/errors-dev
 dh-make-golang github.com/restic/chunker
 dh-make-golang github.com/juju/errors
 dh-make-golang github.com/jessevdk/go-flags
 dh-make-golang github.com/minio/minio-go
 dh-make-golang github.com/juju/testing
 dh-make-golang gopkg/check.v1 golang-gopkg-check.v1-dev 
 dh-make-golang github.com/juju/loggo
 dh-make-golang github.com/juju/utils 

JFTR: I'm a Debian Developer as well and would be happy to also sponsor any uploads (but I currently lack time to take care of the packaging and maintaining the packages properly myself).

[xor-gate]

@mika Yes I know, it was just a dirty fix to get it working. Can you point me to instructions how golang tags/commits could be used in the debian golang packaging process?

[mika]

@xor-gate: check out dh-make-golang, it's supposed to make most of the work for you (PS: I'm mika, not mike, JFYI :))

[digitalcrab]

Hello, guys. Any success? ;)
Hope this can help you a little bit
https://github.com/laher/goxc

[schorsch3000]

Any news in here?
Is there a reliable source for a debian packe or is building your own the way to go?

[jannic]

Just to mention it here: An ITP has been filed to the debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846176

[xor-gate]

Hi guys, sorry for my non-responsiveness as I'm the author if this "issue". I have not been using restic for a long time now (fairly because borgbackup is currently more stable, and has a same set of features). Therefor I have not invested time and effort to continue debian packaging. I was waiting to have some sort of stable restic release (which I use myself) to finish debian packaging and creating a repository.

If people don't want to maintain a debian repository (with own servers) it could be considered to use Bintray for repo/bandwidth hosting.

Again, I'm sorry for the silence on this topic and hopefully you understand.

[fd0]

Thanks for letting us know. I'm glad you found a backup solution that works for you :)

[legrostdg]

Woops, I had not seen this bug (and the packaging repository), I've started this from scratch, and sent an ITP yesterday (which resulted in a link to this bug :-)). I can share the result soon, and would be totally open to group maintenance.

[legrostdg]

I've packaged the two missing dependencies (golang-github-elithrar-simple-scrypt and golang-github-restic-chunker) and I'm looking for sponsors to get these in the archive before I can share the restic package, preferably in a working state :-).

[legrostdg]

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846346
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846347

[fd0]

Thanks @mika for sponsoring the packages :)

[legrostdg]

Update: golang-github-minio-minio-go seems too old for restic. At least the build fails with:

restic/backend/s3/s3.go:39: assignment count mismatch: 2 = 1

I'm trying to get 2.0.2 in Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846969 (I've also contacted the last uploader privately)

[jannic]

@legrostdg: Alternatively, you could try reverting 06b23ed for the debian package. Not nice, but may be a workaround if updating the minio package takes too long.
(See minio/minio-go@5621899 for the API change in minio. AFAIKT they only changed the way the result is reported back to the caller. But I didn't check if there are other incompatibilities between minio 2.0.1 and 2.0.2.)

[fd0]

Further review comments from my side:

• Modifying build.go as you do in 0001-replace-vendor-src-by-usr-share-gocode-src.patch is a bad idea: During build, the program creates a new GOPATH constructed from the restic source code and the vendor directory in a temporary directory. After the patch is applied, all the source code available in /usr/share/gocode/src is copied to the temp dir, that's unnecessary and slows down the build.
• I was astonished to see that you removed a section from the manual regarding installation on other OS/Distributions in 0002-Debianize-doc-Manual.md.patch. I must say that I don't like this change at all. There's no reason to hide the installation instructions, and you also removed quite a lot of "getting started" text which shows the user how to get help for restic commands. If you do not like the section about Debian, I'm sure we can work something out.

If you need help getting the binary to build with go build, please reach out, I'm glad to assist.

[fd0]

Here's a sample debian/rules: https://gist.github.com/fd0/c3ec442e3b77be0f959425469f6b7583

[jmallach]

Please avoid including build dates and the like in binaries. They make it impossible to reproduce the build and obtain a byte-for-byte identical package. https://reproducible-builds.org/docs/timestamps/

[jmallach]

Hi @stapelberg! I propose that when @legrostdg addresses the GPL-3 thing in debian/* (I also noticed, seems I forgot to add that one), the Manual.md thing, the install code bit and add the pristiner-tar bit, the first of us two who gets to re-review does the upload.

Switching to pkg-go can happen after restic has transitioned once to testing, and before February 5th. The less steps that can delay the upload, the better. @legrostdg, let us know if you can work on these in the short term.

[legrostdg]

I replied to @stapelberg concerns, on the debian bug report.

I also imported 0.3.2.

Concerning 0002-Debianize-doc-Manual.md.patch I did remove the installation paragraph because these manual installation instructions are not suitable for Debian, it is usual to remove these from Debian packages. I did also remove the end of the paragraph with the basic usage instructions because I felt like this was just the output of --help. I think this should probably in a new paragraph but in the meantime, I just renamed the paragraph "basic usage" and restored this part of the Manual.md.

Concerning 0001-replace-vendor-src-by-usr-share-gocode-src.patch, you are aware that the vendor/src dir has to be removed from the package, right? I added this patch because go run build.go failed if vendor/src was not existent. I'm of course open to something else. Do you think it would be better to remove everything which mentions vendor?

I just uploaded a new package to mentors. Thanks for your reviews!

[fd0]

@legrostdg For building, I'd suggest using go build with GOPATH set, see here: https://gist.github.com/fd0/c3ec442e3b77be0f959425469f6b7583

You can also leave the main.compiledAt variable unset, this will get you reproduceable builds, I'll remove the timestamp from restic shortly (as suggested by @jmallach).

[legrostdg]

Thanks, I've updated the package with your debian/rules, and removed my patch for modifying vendor in build.go.

[fd0]

Looks good from my side!

[jmallach]

Same here!

[jmallach]

I have signed & uploaded this. It will hopefully appear soon in the NEW queue. Good work Félix!

[legrostdg]

Thanks! I hope it will reach unstable within 5 days, so that restic can be in stretch. I would have preferred to get a feedback from @stapelberg but I guess you're right, we can fix the package to use dh-golang and set the maintainer to pkg-go later.

[jmallach]

Yeah, I wanted to get the clock ticking, and the package is more than good enough for the archive already. I am strongly aligned with @stapelberg regarding debian/*'s GPL+3. It is becoming more and more standard practice that the packaging just follows the upstream license, or at least it is dual licensed, unless the upstream license is non-free, of course.

Regarding the manual.md patch, I also don't see the point in patching it, but I think it would be less of an issue if you could add a manpage for restic that can be even contributed upstream. Starting with what help2man outputs is a good start and gets rid of much of the grunt work.

I also noticed the source is in section utils, while the binary is in section misc. I'd settle for utils, or admin, or whatever, but not misc.

Finally, for the nitpicker in me, use https on the "Format" URL in d/copyright. :)

[jmallach]

restic_0.3.2-2_amd64.changes ACCEPTED into unstable, unstable

The hardest is now over. Congrats @legrostdg and @fd0! Let's see if RC bugs leave us alone and we make it in testing before the 5th!

[fd0]

Awesome!

[jmallach]

And to probably close this issue,

https://tracker.debian.org/news/829517

restic 0.3.2-2 MIGRATED to testing

@fd0, it'd be nice if a new release could be made during the next week or so, that includes the latest fixes, and in particular e1fc455. Last restic updates can go into stretch in if uploaded around Jan 22th or so.

[fd0]

Cool, thanks for letting us know. I'll do a release next week, probably 0.3.3 (no larger changes, just bugfixes).

[fd0]

Done: restic 0.3.3

[fd0]

I've announced the debian package in the blog, together with the new release: https://restic.github.io/blog/2017-01-08/restic-0.3.3-released

[legrostdg]

I've seen that the code for rest-server has been moved to another directory. Unfortunately it won't be possible to have another package enter stretch.

What can be done is to add a patch to add rest-server back, or to have stretch without this functionality. What do you think?

[fd0]

The REST server is a separate package now. And it is not built automatically by build.go, so it technically wasn't in the package before. It is also not necessary for running restic, only if you want to use a different transport than sftp or s3.

I'd leave it out for now.

[legrostdg]

OK! I've pushed the updated package to mentors.

[fd0]

Hey, restic 0.4.0 was released recently and I think it should be uploaded to Debian ASAP. @legrostdg can you please have a look?

I also noticed that 0.3.3 wasn't uploaded either. Is there anything I can do to speed up uploading new versions?

[legrostdg]

I've seen the recent releases, but Debian has entered full freeze, so there won't be any non rc fixing upload to unstable, before stretch is released.

[stapelberg]

You could upload to experimental in the meantime.

[fd0]

Ah, that explains it. Hm, bummer.

[jannic]

BTW, 0.3.3 was uploaded and entered testing before full freeze:
https://tracker.debian.org/pkg/restic

[fd0]

Thanks for the correction, I just looked at the "news" box.

[SnakeO]

Is there a PPA I can use to install the latest version of restic on Ubuntu 16.04LTS ?

[stapelberg]

Not answering your question directly, but in case there is no PPA: the Debian package should be installable on Ubuntu without any issues. It only depends on libc6.