Merge pull request #5 from Deflix-tv/feature/fiber-v2

Update to Fiber v2

README.md

@@ -119,48 +119,60 @@ Some reasons why you might want to consider developing an addon in Go with this
 
 Criterium|Node.js addon|Go addon
 ---------|-------------|--------
-Direct SDK dependencies|9|5
-Transitive SDK dependencies|85|8
-Size of a runnable addon|27 MB¹|15 MB
-Number of artifacts to deploy|depends²|1
+Direct SDK dependencies|9|4
+Transitive SDK dependencies|90¹|35²
+Size of a runnable addon|27 MB³|11-15 MB⁴
+Number of artifacts to deploy|depends⁵|1
 Runtime dependencies|Node.js|-
 Concurrency|Single-threaded|Multi-threaded
 
-¹) `du -h --max-depth=0 node_modules`  
-²) All your JavaScript files and the `package.json` if you can install the depencencies with `npm` on the server, otherwise (like in a Docker container) you also need all the `node_modules`, which are hundreds to thousands of files.  
+¹) `ls -l node_modules | wc -l` - 1  
+²) `go list -m all | wc -l` - 1 - (number of direct dependencies)  
+³) `du -h --max-depth=0 node_modules`  
+⁴) The smaller binary is easily achieved by compiling with `-ldflags "-s -w"`  
+⁵) All your JavaScript files and the `package.json` if you can install the depencencies with `npm` on the server, otherwise (like in a Docker container) you also need all the `node_modules`, which are hundreds to thousands of files.  
 
 Looking at the performance it depends a lot on what your addon does. Due to the single-threaded nature of Node.js, the more CPU-bound tasks your addon does, the bigger the performance difference will be (in favor of Go). Here we compare the simplest possible addon to be able to compare just the SDKs and not any additional overhead (like DB access):
 
+On a [DigitalOcean](https://www.digitalocean.com/) "Droplet" of type "Basic" (shared CPU) with 2 cores and 2 GB RAM, which costs $15/month:
+
+Criterium|Node.js addon|Go addon
+---------|-------------|--------
+Startup time to 1st request¹|400ms-4s|20-30ms
+Max rps² @ 1000 connections|Local³: 1,000<br>Remote⁴: 1,000|Local³: 17,000<br>Remote⁴: 29,000
+Memory usage @ 1000 connections|Idle: 42 MB<br>Load⁵: 73 MB|Idle: 11 MB<br>Load⁵: 45 MB
+
+On a [DigitalOcean](https://www.digitalocean.com/) "Droplet" of type "CPU-Optimized" (dedicated CPU) with 2 cores and 4 GB RAM, which costs $40/month:
+
 Criterium|Node.js addon|Go addon
 ---------|-------------|--------
-Startup time to 1st request¹|150-230ms|5-20ms
-Max rps² @ 1000 connections|Local³: 6,000<br>Remote⁴: 3,000|Local³: 59,000<br>Remote⁴: 58,000
-Memory usage @ 1000 connections|Idle: 35 MB<br>Load⁵: 70 MB|Idle: 10 MB<br>Load⁵: 45 MB
+Startup time to 1st request¹|200-400ms|9-20ms
+Max rps² @ 1000 connections|Local³: 5,000<br>Remote⁴: 1,000|Local³: 39,000<br>Remote⁴: 39,000
+Memory usage @ 1000 connections|Idle: 42 MB<br>Load⁵: 90 MB|Idle: 11 MB<br>Load⁵: 47 MB
 
 ¹) Measured using [ttfok](https://github.com/doingodswork/ttfok) and the code in [benchmark](benchmark). This metric is relevant in case you want to use a "serverless functions" service (like [AWS Lambda](https://aws.amazon.com/lambda/) or [Vercel](https://vercel.com/) (former ZEIT Now)) that doesn't keep your service running between requests.  
 ²) Max number of requests per second where the p99 latency is still < 100ms  
-³) The load testing tool ran on a different server, but in the same datacenter and the requests were sent within a private network  
+³) The load testing tool ran on a different server, but in the same datacenter and the requests were sent within a private network. Note that DigitalOcean seems to have performance issues with their local "VPC Network" (which didn't affect the Node.js service as it maxed out the CPU, but the Go service maxed out the network before the CPU).  
 ⁴) The load testing tool ran on a different server *in a different datacenter of another cloud provider in another city* for more real world-like circumstances  
-⁵) At a request rate *half* of what we measured as maximum  
+⁵) Resident size (`RES` in `htop`) at a request rate *half* of what we measured as maximum  
 
 The load tests were run under the following circumstances:
 
 - We used the addon code, load testing tool and setup described in [benchmark](benchmark)
-- We ran the service on a [DigitalOcean](https://www.digitalocean.com/) "Droplet" with 2 cores and 2 GB RAM, which costs $15/month
-- The load tests ran for 60s, with previous warmup  
-- Virtualized servers of cloud providers vary in performance throughout the week and day, even when using the exact same machines, because the CPU cores of the virtualization host are shared between multiple VPS. We conducted the Node.js and Go service tests at the same time so their performance difference doesn't come from the performance variation due to running at different times.  
-- The client server used to run the load testing tool was high-powered (4-8 *dedicated* cores, 8-32 GB RAM)
+- We ran the Node.js and Go service on the same Droplet and conducted the benchmark on the same day, several minutes apart, so that the resource sharing of the VPS is about the same. Note that when you try to reproduce the benchmark results, a different VPS could be subject to more or less resource sharing with other VPS on the virtualization host. Other times of day can also lead to differing benchmark results (e.g. low traffic on a Monday morning, high traffic on a Saturday evening).
+- The load tests ran for 60s (to have a somewhat meaningful p99 value), with previous warmup
+- The client servers (both the one in the same DC and the one in a different DC of another cloud provider in another city) used to run the load testing tool were high-powered (8 *dedicated* cores, 32 GB RAM)
 
 Additional observations:
 
-- The Go service's response times were generally lower across all request rates  
-- The Go service's response times had a much lower deviation, i.e. they were more stable. With less than 60s of time for the load test the Node.js service fared even worse, because outliers lead to a higher p99 latency.  
-- We also tested on a lower-powered server by a cheap cloud provider (also 2 core, 2 GB RAM, but the CPU was generally worse). In this case the difference between the Node.js and the Go service was even higher. The Go service is perfectly fitted for scaling out with multiple cheap servers.  
-- We also tested with different amounts of connections. With more connections the difference between the Node.js and the Go service was also higher. In a production deployment you want to be able to serve as many users as possible, so this goes in favor of the Go service as well.  
+- The Go service's response times were generally lower across all request rates
+- The Go service's response times had a much lower deviation, i.e. they were more stable. With less than 60s of time for the load test the Node.js service fared even worse, because outliers lead to a higher p99 latency.
+- We also tested on a lower-powered server by a cheap cloud provider (also 2 core, 2 GB RAM, but the CPU was generally worse). In this case the difference between the Node.js and the Go service was even higher. The Go service is perfectly fitted for scaling out with multiple cheap servers.
+- We also tested with different amounts of connections. With more connections the difference between the Node.js and the Go service was also higher. In a production deployment you want to be able to serve as many users as possible, so this goes in favor of the Go service as well.
 
 > Note:
 >
-> - This Go SDK is at its very beginning. Some features will be added in the future that might decrease its performance, while others will increase it.  
+> - This Go SDK is still young. Some features will be added in the future that might decrease its performance, while others will increase it.  
 > - The Node.js addon was run as a single instance. You can do more complex deployments with a load balancer like [HAProxy](https://www.haproxy.org/) and multiple instances of the same Node.js service on a single machine to take advantage of multiple CPU cores. But then you should also activate preforking in the Go addon for using several OS processes in parallel, which we didn't do.  
 
 ## Related projects

addon.go

@@ -13,11 +13,13 @@ import (
 	"syscall"
 	"time"
 
-	"github.com/deflix-tv/go-stremio/pkg/cinemeta"
-	"github.com/gofiber/adaptor"
-	"github.com/gofiber/fiber"
-	"github.com/gofiber/fiber/middleware"
+	"github.com/gofiber/adaptor/v2"
+	"github.com/gofiber/fiber/v2"
+	"github.com/gofiber/fiber/v2/middleware/filesystem"
+	"github.com/gofiber/fiber/v2/middleware/recover"
 	"go.uber.org/zap"
+
+	"github.com/deflix-tv/go-stremio/pkg/cinemeta"
 )
 
 // ManifestCallback is the callback for manifest requests, so mostly addon installations.
@@ -197,17 +199,17 @@ func (a *Addon) Run(stoppingChan chan bool) {
 	// Fiber app
 
 	logger.Info("Setting up server...")
-	app := fiber.New(&fiber.Settings{
-		ErrorHandler: func(ctx *fiber.Ctx, err error) {
+	app := fiber.New(fiber.Config{
+		ErrorHandler: func(c *fiber.Ctx, err error) error {
 			code := fiber.StatusInternalServerError
 			if e, ok := err.(*fiber.Error); ok {
 				code = e.Code
 				logger.Error("Fiber's error handler was called", zap.Error(e))
 			} else {
 				logger.Error("Fiber's error handler was called", zap.Error(err))
 			}
-			ctx.Set(fiber.HeaderContentType, fiber.MIMETextPlainCharsetUTF8)
-			ctx.Status(code).SendString("An internal server error occurred")
+			c.Set(fiber.HeaderContentType, fiber.MIMETextPlainCharsetUTF8)
+			return c.Status(code).SendString("An internal server error occurred")
 		},
 		DisableStartupMessage: true,
 		BodyLimit:             0,
@@ -219,7 +221,7 @@ func (a *Addon) Run(stoppingChan chan bool) {
 
 	// Middlewares
 
-	app.Use(middleware.Recover())
+	app.Use(recover.New())
 	if !a.opts.DisableRequestLogging {
 		app.Use(createLoggingMiddleware(logger, a.opts.LogIPs, a.opts.LogUserAgent, a.opts.LogMediaName, a.manifest.BehaviorHints.ConfigurationRequired))
 	}
@@ -244,9 +246,9 @@ func (a *Addon) Run(stoppingChan chan bool) {
 	if a.opts.Profiling {
 		group := app.Group("/debug/pprof")
 
-		group.Get("/", func(c *fiber.Ctx) {
+		group.Get("/", func(c *fiber.Ctx) error {
 			c.Set(fiber.HeaderContentType, fiber.MIMETextHTML)
-			adaptor.HTTPHandlerFunc(netpprof.Index)(c)
+			return adaptor.HTTPHandlerFunc(netpprof.Index)(c)
 		})
 		for _, p := range pprof.Profiles() {
 			group.Get("/"+p.Name(), adaptor.HTTPHandler(netpprof.Handler(p.Name())))
@@ -280,16 +282,18 @@ func (a *Addon) Run(stoppingChan chan bool) {
 		app.Get("/:userData/stream/:type/:id.json", streamHandler)
 	}
 	if a.opts.ConfigureHTMLfs != nil {
-		// fsmw := middleware.FileSystem(a.opts.ConfigureHTMLfs)
-		app.Use("/configure", middleware.FileSystem(a.opts.ConfigureHTMLfs))
+		fsConfig := filesystem.Config{
+			Root: a.opts.ConfigureHTMLfs,
+		}
+		app.Use("/configure", filesystem.New(fsConfig))
 		// When a Stremio user has the addon already installed and configures it again, this endpoint is called,
 		// theoretically enabling the addon to deliver a website with the configuration fields populated with the currently configured values.
 		// The Fiber filesystem middleware currently doesn't work with parameters in the route (see https://github.com/gofiber/fiber/issues/834),
 		// so we'll just redirect to the original one, as we don't use the existing configuration anyway.
 		// TODO: At some point we should populate the config fields with the existing configuration.
-		app.Get("/:userData/configure", func(c *fiber.Ctx) {
+		app.Get("/:userData/configure", func(c *fiber.Ctx) error {
 			c.Set("Location", c.BaseURL()+"/configure")
-			c.SendStatus(fiber.StatusMovedPermanently)
+			return c.SendStatus(fiber.StatusMovedPermanently)
 		})
 	}
 

benchmark/README.md

@@ -7,7 +7,7 @@ The code in `addon.js` is from the main README from the official Stremio addon S
 On a fresh Ubuntu 20.04 machine run it with:
 
 ```bash
-# Install Node.js 12, which is the latest LTS release as of writing this
+# Install Node.js 12, which is the latest LTS release as of writing this.
 curl -sL https://deb.nodesource.com/setup_12.x | bash -
 apt install -y nodejs
 
@@ -23,9 +23,10 @@ On a fresh Ubuntu 20.04 machine run it with:
 
 ```bash
 # Install Go 1.15, which is the latest version as of writing this
-curl -sL -o go.tar.gz https://dl.google.com/go/go1.15.linux-amd64.tar.gz
+curl -sL -o go.tar.gz https://golang.org/dl/go1.15.3.linux-amd64.tar.gz
 tar -C /usr/local -xzf go.tar.gz
 echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
+echo 'export PATH=$PATH:$(go env GOPATH)/bin' >> ~/.bashrc
 . ~/.bashrc
 
 apt install -y git
@@ -93,12 +94,13 @@ apt install -y nodejs
 npm install stremio-addon-sdk
 
 # Set up Go
-curl -sL -o go.tar.gz https://dl.google.com/go/go1.15.linux-amd64.tar.gz
+curl -sL -o go.tar.gz https://golang.org/dl/go1.15.3.linux-amd64.tar.gz
 tar -C /usr/local -xzf go.tar.gz
-echo 'export PATH="/usr/local/go/bin:~/go/bin:$PATH"' >> ~/.bashrc
-set -ux
-. ~/.bashrc
+echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
+echo 'export PATH=$PATH:$(go env GOPATH)/bin' >> ~/.bashrc
 set +ux
+. ~/.bashrc
+set -ux
 go build -v
 
 # Set up wrk2

benchmark/go.sum

@@ -7,19 +7,13 @@ github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XL
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/gofiber/adaptor v0.2.0 h1:OJtST958Zc6WTXHTJA/9d464VgakAWKBjzM3lmkKjsc=
-github.com/gofiber/adaptor v0.2.0/go.mod h1:0M9Agxi+L3uS5oKd6pN317h35JQhfvrfD4H+c4a5qck=
-github.com/gofiber/cors v0.2.2 h1:NQgLeNq8SWCKsdGotodyFCqLdSnxGLISsp9OU01k/cs=
-github.com/gofiber/cors v0.2.2/go.mod h1:lAXoymRHZKASLfydSAtsRGVrukWi3KefFnfxmCEAH5o=
-github.com/gofiber/fiber v1.13.3/go.mod h1:KxRvVkqzfZOO6A7mBu+j7ncX2AcT6Sm6F7oeGR3Kgmw=
-github.com/gofiber/fiber v1.14.6 h1:QRUPvPmr8ijQuGo1MgupHBn8E+wW0IKqiOvIZPtV70o=
-github.com/gofiber/fiber v1.14.6/go.mod h1:Yw2ekF1YDPreO9V6TMYjynu94xRxZBdaa8X5HhHsjCM=
-github.com/gofiber/utils v0.0.9/go.mod h1:9J5aHFUIjq0XfknT4+hdSMG6/jzfaAgCu4HEbWDeBlo=
+github.com/gofiber/adaptor/v2 v2.0.1 h1:OC9BbDP115atsYXnUeJrFZNWWXr9noL2TLtMY6VKqmw=
+github.com/gofiber/adaptor/v2 v2.0.1/go.mod h1:Hn255OIJFLVAfGmievRC9CBE49FiKlf4LkgnEqi7mxc=
+github.com/gofiber/fiber/v2 v2.1.0 h1:gvEQJDxVHFLY4bNb4HSu7nqVWeLeXry8P4tA4zPKfhQ=
+github.com/gofiber/fiber/v2 v2.1.0/go.mod h1:aG+lMkwy3LyVit4CnmYUbUdgjpc3UYOltvlJZ78rgQ0=
 github.com/gofiber/utils v0.0.10 h1:3Mr7X7JdCUo7CWf/i5sajSaDmArEDtti8bM1JUVso2U=
 github.com/gofiber/utils v0.0.10/go.mod h1:9J5aHFUIjq0XfknT4+hdSMG6/jzfaAgCu4HEbWDeBlo=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
-github.com/gorilla/schema v1.1.0 h1:CamqUDOFUBqzrvxuz2vEwo8+SUdwsluFh7IlzJh30LY=
-github.com/gorilla/schema v1.1.0/go.mod h1:kgLaKoK1FELgZqMAVxx/5cbj0kT+57qxUrAlIO2eleU=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.10.7 h1:7rix8v8GpI3ZBb0nSozFRgbtXKv+hOe+qfEpZqybrAg=
 github.com/klauspost/compress v1.10.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
@@ -28,10 +22,6 @@ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORN
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
-github.com/mattn/go-colorable v0.1.7 h1:bQGKb3vps/j0E9GfJQ03JyhRuxsvdAanXlT9BTw3mdw=
-github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
-github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
-github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
 github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
@@ -43,7 +33,6 @@ github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJy
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
-github.com/valyala/fasthttp v1.15.1/go.mod h1:YOKImeEosDdBPnxc0gy7INqi3m1zK6A+xl6TwOBhHCA=
 github.com/valyala/fasthttp v1.16.0 h1:9zAqOYLl8Tuy3E5R6ckzGDJ1g8+pw15oQp2iL9Jl6gQ=
 github.com/valyala/fasthttp v1.16.0/go.mod h1:YOKImeEosDdBPnxc0gy7INqi3m1zK6A+xl6TwOBhHCA=
 github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a h1:0R4NLDRDZX6JcmhJgXi5E4b8Wg84ihbmUKp/GvSPEzc=
@@ -68,11 +57,11 @@ golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 h1:OjiUf46hAmXblsZdnoSXsEUSKU8r1UEzcL5RVZ4gO9Y=
 golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211 h1:9UQO31fZ+0aKQOFldThf7BKPMJTiBfWycGh/u3UoO88=
+golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=