API server error when run from docker

[andyedison]

Similar issue to #505, however this is not Docker on Mac, rather Docker running on a CentOS 7 Linux server.

I will note however, it appears everything works right??

The issue:

==> docker run --rm -i loadimpact/k6 run --vus 10 --duration 10s -<k6_google.js

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

  execution: local--------------------------------------------------]   servertor
     output: -
     script: -

    duration: 10s, iterations: -
         vus: 10,  max: 10

time="2019-04-02T16:16:33Z" level=warning msg="Error from API server" error="listen tcp 10.81.107.117:6565: bind: cannot assign requested address"
time="2019-04-02T16:16:34Z" level=info msg=Running i=0 t=929.070901ms
time="2019-04-02T16:16:35Z" level=info msg=Running i=0 t=1.929081512s
time="2019-04-02T16:16:36Z" level=info msg=Running i=0 t=2.929080371s
time="2019-04-02T16:16:37Z" level=info msg=Running i=10 t=3.929069433s
time="2019-04-02T16:16:38Z" level=info msg=Running i=10 t=4.929125503s
time="2019-04-02T16:16:39Z" level=info msg=Running i=10 t=5.929073206s
time="2019-04-02T16:16:40Z" level=info msg=Running i=20 t=6.929073878s
time="2019-04-02T16:16:41Z" level=info msg=Running i=20 t=7.929080316s
time="2019-04-02T16:16:42Z" level=info msg=Running i=20 t=8.929083925s
time="2019-04-02T16:16:43Z" level=info msg=Running i=30 t=9.929079308s
time="2019-04-02T16:16:43Z" level=info msg="Test finished" i=30 t=10.000086132s

    data_received..............: 440 kB 44 kB/s
    data_sent..................: 17 kB  1.7 kB/s
    http_req_blocked...........: avg=32.19ms min=391ns   med=467ns   max=166.63ms p(90)=163.6ms  p(95)=164.5ms
    http_req_connecting........: avg=6.92ms  min=0s      med=0s      max=29.85ms  p(90)=27.82ms  p(95)=29.13ms
    http_req_duration..........: avg=58.36ms min=35.06ms med=61.73ms max=221.53ms p(90)=72.51ms  p(95)=77.28ms
    http_req_receiving.........: avg=36.88µs min=16.58µs med=30.99µs max=119.7µs  p(90)=50.42µs  p(95)=66.98µs
    http_req_sending...........: avg=86.23µs min=47.34µs med=57.91µs max=452.08µs p(90)=142.92µs p(95)=162.26µs
    http_req_tls_handshaking...: avg=25.03ms min=0s      med=0s      max=135.6ms  p(90)=134.63ms p(95)=135.08ms
    http_req_waiting...........: avg=58.24ms min=34.99ms med=61.61ms max=221.44ms p(90)=72.43ms  p(95)=77.18ms
    http_reqs..................: 80     7.999931/s
    iteration_duration.........: avg=3.2s    min=3.1s    med=3.11s   max=3.48s    p(90)=3.37s    p(95)=3.38s
    iterations.................: 30     2.999974/s
    vus........................: 10     min=10 max=10
    vus_max....................: 10     min=10 max=10

example file from above (k6_google.js)

import http from "k6/http";
import { sleep } from "k6";

export default function() {
  http.get("https://google.com");
  sleep(3);
};

docker version:

==> docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:33:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 06:02:24 2019
  OS/Arch:          linux/amd64
  Experimental:     false

host info

Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.10.1.el7.x86_64
Architecture: x86-64

k6 image info

==> docker image inspect loadimpact/k6:latest
[
    {
        "Id": "sha256:834ff5a07521a27c8a4033dec0d5b7a0d9c48ee68f1b4490b040bb2d6c88fde5",
        "RepoTags": [
            "loadimpact/k6:latest"
        ],
        "RepoDigests": [
            "loadimpact/k6@sha256:8016f2cd4b9452f764d347a84ad0d2306dd2c9d4b1edcb80dc87e56bd7f9e41c"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2019-03-20T10:20:43.711380321Z",
        "Container": "8fc3c87056261b4066ffe53f810cdf6df79020b07af4fef30d93cabfc2df6a07",
        "ContainerConfig": {
            "Hostname": "8fc3c8705626",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "ENTRYPOINT [\"k6\"]"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:0a097f857d835a409c01ea263ec427f610fc7600b15fc532b04387de512d0a42",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "k6"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "17.11.0-ce",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "ArgsEscaped": true,
            "Image": "sha256:0a097f857d835a409c01ea263ec427f610fc7600b15fc532b04387de512d0a42",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "k6"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 26201090,
        "VirtualSize": 26201090,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/d4719583e2f6ded0a8b0a7dcb7c232d65f4d98f66370d6e5422e344b89e40ba1/diff:/var/lib/docker/overlay2/164b9ea65be0c64a54198ddbef9c3d53b15b36817a94fceff1e45808875c239f/diff",
                "MergedDir": "/var/lib/docker/overlay2/edfada309ffe8d1b7a9d021f01f16817be88fe7b73489b4fd277018c07b3e114/merged",
                "UpperDir": "/var/lib/docker/overlay2/edfada309ffe8d1b7a9d021f01f16817be88fe7b73489b4fd277018c07b3e114/diff",
                "WorkDir": "/var/lib/docker/overlay2/edfada309ffe8d1b7a9d021f01f16817be88fe7b73489b4fd277018c07b3e114/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:3fc64803ca2de7279269048fe2b8b3c73d4536448c87c32375b2639ac168a48b",
                "sha256:489c4fd0c14cb43c3cc4b364b99b3dc9f8ddc882bb9010b0d9734fa11e525086",
                "sha256:2d49514babf048461eb017a2dc090940d64ce43dd172d76fe1eab8d23987cd77"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[mstoykov]

Hi,
It looks like for some reason we couldn't open a port for the k6 REST API - it is not required for you to run your scripts so it is not an issue. This is probably due to you running in docker than, maybe something else already uses that port.

Also maybe the rest API should not be started unless we call it with a flag ... @na-- what do you think ?

[na--]

If we started from scratch, I'd say that we probably should start the REST API server only when users request it. But since it has been enabled by default for quite some time, and since it binds to localhost, maybe leaving it as it is is the best compromise - try to spin it up, but if there's an error, don't abort the script, just log the error.

What I'd like to fix though is to be able to actually disable the API server. Currently, I don't think there's a way to do that... k6 run --address="" script.js will just result in a Error from API server error="listen tcp :80: bind: permission denied"... 🤦‍♂️ A simple check here should suffice, I think.

[mstoykov]

I would argue that given that:

1. I doubt anyone uses it all that much - it requires additional tooling and as I have shown it is easy to generate panics with it
2. Us probably mangling it around Arrival-rate based VU executor #550
3. Probable changes around Remove dependencies for the API that are not necessary #911
   That having it enabled by default isn't really necessary. I would argue (as I have previously) that we will need to completely rewrite if it will be usable after Arrival-rate based VU executor #550 for most use cases and us such we will (probably) break it in order to make it work.
   But yeah it should at least be disyllable although I vote for it to be enable-able :)

[na--]

it is easy to generate panics with it

It's currently easy to generate panics with it because of bugs and shitty code... It hopefully won't be when I'm done with the current massive refactoring 😄

Probable changes around #911

Any changes that remove dependencies by causing backward incompatibilities are probably not what we want 😉

Regarding the usage, we don't have any data, but you're probably right, likely not a ton of people use the REST API. Still, I stand by my earlier claim that it's probably better to err on the side of caution and leave it as it is, as much as possible. True, the new schedulers (including arrival-rate) will force us to make some backwards incompatible changes, but a lot of the current API functionality will still work. Most if it, if you run k6 with --manual-execution...

And it's not just the REST API directly. Probably the biggest consumer of the API is the k6 binary itself. There are a lot of k6 sub-commands which internally use the same API. Here's a list (subset of k6 help):

  pause       Pause a running test
  resume      Resume a paused test
  scale       Scale a running test
  stats       Show test metrics
  status      Show test status

All of them will mostly continue to work with --manual-execution. But even without it, stats, status will continue to work and resume will work when the script is started with --paused. Even if we ignore any frustration to an unknowable number of current REST API users, breaking the backwards compatibility by disabling the API by default is not worth it simply because we'd have add an asterisk to all of these commands and mention that they only work when the API is enabled...

[mstoykov]

Not certain if anyone uses this commands as well ... like .. those are IMO not useful in MOST cases ... maybe status/stats and even them ... ? And even that ... why would most people use it to implement #140 I would think. And yes I think that we should think about how people use our tool and shape it in a way, including breaking or removing things that we think are bad idea.

Secondly I would've liked around #911 to change the PUT/POST methods because currently they are AWFUL to use, as well as everything else but the post/put are specifically curl unfriendly for no reason. We can obviously keep the commands compatible after the changes :) and if someone uses the golang api calls they will just need to update k6. Not that I think anyone does this. Don't get me wrong I still think nobody uses those as they probably would've asked why the api looks like this :)

I would argue that the commands above should have never existed as I don't see their point without #140 and as that in particular will probably add a totally different API as it will be needed (probably GRPC instead of REST one but still). Of course I suppose the command exist because calling the API with curl requires you to wrap and unwrap data for no added benefit ?

Being backwards compatible is important and I am very much against changing things that will break script computability like change to k6/* but not changing things that are clearly not good design especially prior to 1.0.0 seems unreasonable to me and just as user unfriendly as constantly changing things. I again like to point out that if we are not going to break anything and we would like to communicate this with our users we should be 1.0.0.
If not I would like to be able to break things where the cost-benefit is on the side of the breaking.

Maybe the solution is to deprecate everything API related next release and remove/replace it in a future release. With a lot of warnings that point out that the API in it's current form won't be supported or will be severely changed.

cc @robingustafsson

[na--]

I agree that the current API doesn't look very good. IIRC, the strange data format stems from the fact that the JSONAPI library was used. I don't know why that decision was taken, it was before I started working on k6, and I haven't looked very much into the pros and cons of it. Still, in my mind, this is a minor and very isolated issue that doesn't deserve for us to spend too much time focusing on it.

And we've basically hijacked a very unrelated issue to discuss it at great length already 😉 And you've further shifted the discussion from "should we start the API server by default" to "what the API should look like"...

For the record, I'm not necessarily opposed to making backwards compatibility breaks, but we should do it only where it makes sense to do so. With the new schedulers, there will be breaking changes in both the API (unless --manual-execution is enabled, but even then stages won't work) and in the script options. But that's because there simply wasn't another choice, and we still warned users in v0.24.0 when they were using option combinations that will be deprecated in the next version.

Apparently, with regards to changing the API format or disabling the API server by default, my cost-benefit analysis is different from yours - to me, it's not worth it to do a breaking change. We won't gain all that much, and even if few people use them, they're likely power users. So if in my mind, something like this looks like the correct way to handle things:

• leave the API server running by default, but fix it so it can be disabled
• leave the current API as it is, as much as possible, i.e. GET / POST / PUT of /v1/setup will behave like it currently does
• add a new API at /v2 that does things better - both in terms of API design, and in terms of Go code (i.e. don't use tons of unnecessary dependencies)
• switch the Go API client and the k6 sub-commands to use the v2 API
• after 1-2 k6 versions, simply delete the old code and dependencies, make sure that the /v1 API returns HTTP errors

[olegbespalov]

It seems like more of a docker issue rather than the k6. I've checked with the latest available grafana/k6 image and have no issues. Closing the task, and if more details appear, we could re-open.