Run/import scripts by URL

[liclac]

Something we talked about last week is to allow you to do something like:

k6 run https://docs.k6.io/some_script.js

As well as:

import lib from "https://example.com/some-library.js";

Being able to import stuff from URLs is one of my favorite Go features, and I'm 100% behind introducing it to k6. #136 was a prerequisite for doing this in a sane way, I'd like to get #137 done first as well to get an easier point of integration + avoid duplicating work.

The catch of this is that we'd pretty much have to kill the JS runner, unless we want to change the syntax for it to something like k6 run GET http://example.com/… but honestly, I personally wouldn't miss it.

[liclac]

This is my proposed implementation for this.

When a user does one of the following:

import { name } from "path";

$ k6 run [path]

There are currently two separate behaviors. On the commandline:

• path is parsed as a URL; if it looks like one, it's passed to the simple runner.
• If not, it's read, and passed to the js runner.

If it's an import:

• If path is k6 or starts with k6/, the named host module is imported.
• If not, the named file is imported.

What I want to do is make this a single, unified behavior:

1. If path is k6 or starts with k6/, the named host module is imported.
   • This doesn't make any sense on the commandline, and will fail there.
   • This intentionally means you can't name your own modules this.
2. If not, attempt, in order:
   1. ${path}
   2. ${path}.js
   3. ${path}/index.js
3. Try essentially the same semantics as a go package:
   1. Matched against a set of predefined providers (Github, Bitbucket, Launchpad, …)
      • This is where we can add support for custom providers, e.g. CDNJS.
   2. Failing that, a type suffix can be used to indicate a specific vcs scheme.
   3. Try downloading the page and looking for a meta tag pointing to the real location.
      • This is how we will make imports from k6.io work (see gopherpit).
   4. Finally, assume the response is just straight-up a JS file and try loading it.

(Note: all fetching of URLs will be over HTTPS.)

Non-local imports will be written relative to the root of the virtual filesystem, e.g.: github.com/loadimpact/k6/samples/es6sample.js -> /github.com/loadimpact/k6/samples/es6sample.js

There should be a way to cache downloaded scripts, but I don't think it should be the default - we're just loading JS files, primarily off various CDNs, the load time shouldn't be much worse than loading an average web page.

[micsjo]

Questions to start with:

q1: Wouldn't 3.iv already be covered by 2.i or 2.ii?

q2: 3.i A lot of execution in most controlled corporate environments do simply not allow network traffic like that or limit in other ways (bandwidth, request, limited authorization and so on). Wouldn't this lead to scripts "breaking" and needing refactoring so dependencies are always locally loaded? It's probably not an issue we can (or will want to) solve but awareness can change the order of precedence.

q3: 3.i & 3.iv Assuming limited public network access it will fail accessing a predefined provider (how are they predefined, ordered and documented?) it will fall back (in order of precedence) to local loading. But executing the same script with public network access at a later time will result in executing different code and can yield vastly different results. So can I override default order of precedence to always make sure I get a known (local or network) resource loaded?

q4: Will this be valid for cloud execution? I.e. can a cloud executed script load resources from the network? If so - consider the case that 1000's of tests are executed daily with 10.000's load generators all loading "themostusedstandardlibraryfork6" from let's say github. A couple of weeks of 10k+ daily downloads of the same thing using the same identity from the same place (cloud execution) I suspect it would be raising a lot of interesting flags for the providers. Worst case to the extent execution from the cloud is blacklisted. Cloud execution will also have to take into consideration network usage effects (including costs).

[liclac]

Ah, case 2 refers specifically local files, so it's the distinction between importing "./jquery.js" and "code.jquery.com/jquery.js".

But networks with limited access is an interesting point. We respect proxies from the environment by default, but I kinda want some way of caching remote resources too, I'm just not sure what the best way to do so would be. Caching by default, and offering some way to fetch up-to-date resources? A manual k6 fetch command to make a local stash? Global (temporary directory) or local cache?

It shouldn't cause a problem for distributed execution though - assuming we go with my proposed implementation (#140), we'd share a single virtual filesystem between all nodes through a key-value store of some description.

[ragnarlonn]

I really like this solution. I'm sure there will be roadblocks somewhere, but it seems very powerful without being hard to understand. I just have a couple of minor questions:

• is "index.js" used somewhere else, or why that particular file name? If it is alluding to index.html it feels slightly wrong to me. "index" there comes from "directory index" - i.e. showing a file list. Most executable programs use "main" or similar to denote entry points. Is there some other name that would make more sense, like maybe "default.js" or "main.js"?

• I assume {path} will be relative to the calling file, or to CWD in case of a command-line option?

[liclac]

Oh, I forgot to reply, but:

• index.js is used by Node when importing a directory name.
• Yup, all relative to the current working directory or importing file.

[ppcano]

Some thoughts regarding importing URL resources in the scripts.

K6 user will write ES6 code. I think users will expect the import API to mimic the current ES6 import API.

import incrementer from './lib2';

Perhaps, I don’t have enough info but right now, I don’t agree on mixin the import ES6 API with a go get syntax. http://stackoverflow.com/questions/34607252/es6-import-module-from-url

I am trying to understand how it will work and would like to ask some questions:

1. Which is the non-ES6 type of module we will support?

2. Will the user be able to place locally the file with the non-ES6 module, and use the ES6 import syntax to import the module? Does it make sense to use the import ES6 API to import different type of modules?

http://exploringjs.com/es6/ch_modules.html

If you clarify these questions, I could reason more in this subject. But at this moment, I think, it would be better to have different API syntax for different modules (ES6 vs AMD or Browser global) and/or actions (importing vs loader).

Quick proposal:

import ES6modulesAPI

require 'cdns.faker.js'

require './vendors/faker.js'

[ppcano]

@liclac; now, how can I import local ES6 modules?

Giving a print function at ./lib/print.js, the example below shows the error:

GoError: Get https://lib/print?_k6=1: dial tcp: lookup lib: no such host

import print from './lib/print';
export default function() {
  http.get("http://test.loadimpact.com/");
  print();
}

I also tried to place the module file at ./print/index.js and importing as:

import print from 'print';

GoError: Get https://print?_k6=1: dial tcp: lookup print: no such host

[ppcano]

@liclac Please, could you tell me if the above error is something I am doing wrong or a possible bug.

Running like k6 -v run -t js2 libraries.js

[liclac]

import print from "./lib/print.js"

[ppcano]

@liclac Using current master, it doesn't work.

I tried another example under samples/REST_API_testsuite/

cd samples/REST_API_testsuite/
k6 -v run v3_account_login.js

DEBU[0001] Babel: Transformed                            t=261.775845ms
DEBU[0001] Fetching source...                            url="https://common?_k6=1"
DEBU[0001] Fetching source...                            url="https://common"
ERRO[0001] GoError: Get https://common?_k6=1: dial tcp: lookup common: no such host
	at native
	at v3_account_login.js:40:122(62)
 
GoError: Get https://common?_k6=1: dial tcp: lookup common: no such host

Should I open a new issue?

[liclac]

Ah, sorry, I keep glossing this over because it's closed - yeah, please open a new issue

[Stumblor]

@ppcano Could you link to the new issue please? Also getting this error.

[ppcano]

@Stumblor This issue was already fixed.

You may likely receive the error because of a syntax error. The Modules documentation article has been recently published; it describes the feature, APIs, and examples.

There is a related open issue Transforming libraries takes a long time

Feedback is appreciated!

[Stumblor]

Hi @ppcano - thanks for getting back. Yes I've read both of those.

My syntax is:
import { checkAll } from "./Common/index.js";

Also tried
import { checkAll } from "./Common";

And
import common from "./index.js";

All result in variations of:

ERRO[0000] GoError: parse https://........\LoadTests\Users\Common\index.js?_k6=1: invalid character "\\" in host name
        at native
        at ........\LoadTests/Users/list.js:60:121(54)

GoError: parse https://........\LoadTests\Users\Common\index.js?_k6=1: invalid character "\\" in host name

(........ directory structure omitted, but contains Z:\ so should be file location, not url)

I'm using 0.14.0 of the windows 64bit build.

[liclac]

OH, I see, it's making a faulty assumption that the path separator is always /, but on Windows, it's \, leading it to construct faulty paths. Could you make a new issue about this?

[Stumblor]

Sure will do

[arichiardi]

Sorry to write in a closed issue, but is this taking into consideration Node.js modules? I have just opened an issue (#360)