Clarification on versioning and API compatibilty

[MasonM]

I'm currently evaluating if k6 is a right fit for my organization, and I really like what I've seen so far, but I haven't been able to find any information on the versioning policy or compatibility guarantees (if any) for the Javascript API. I'm guessing k6 follows semantic versioning, and since the major version is 0, that means anything can change at any point:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

But the docs have a note under "k6/net/grpc" that says:

The k6 gRPC API is currently considered in beta and is subject to change.

which implies that the other APIs are not considered beta, so I'm confused. Is there anything in the API that can be considered "stable", or could every release potentially force us to rewrite all our tests? If the latter, are there any plans to establish compatibility guarantees?

Thanks in advance!

[MattDodsonEnglish]

Hello! Thanks for this callout! You are absolutely right that this is a bit unclear. The core maintainers can give you the best answer, but I will try to provide a documentation-focused answer before sending this to the main repo.

At this moment, k6 is not following SemVer. Releases happen every two months, so the version number is more a record of time. Rather, k6 versioning is more like that of React, which also is still in a 0. version, even though React is obviously a mature, heavily used piece of software.

In effect, k6 is 0ver (that website is a joke, but there's an element of truth). Whether k6 should stay in 0ver or when it When this will change is still an open question and certainly a subject of much internal discussion.

other APIs are not considered beta

I think this is mostly a safe assumption: if the docs don't explicitly mention something is experimental or beta, it is quite unlikely to break.

If it's not marked beta, many users are probably already using the functionality in production, so a breaking change would be something that the maintainers would take quite seriously.

If you'd like to see what kind of breaking changes happen, you can take a look at the Release notes. You'll see they are almost always minor. In the latest release, we also included a Roadmap to speculate on how we would deal with breaking changes.

[na--]

TLDR version of the wall of text below:

• the existing k6 JS APIs are mostly stable, though not completely stable yet 😞
• the k6 Go APIs (for xk6 extensions) are unstable, we have absolutely no stability guarantees there and probably won't offer any for a long while
• there are a few reasons we haven't released k6 v1.0.0 and JS API stability is not the biggest one
• with new JS APIs, we're going to be much more explicit about what's stable and unstable, and we'll have a gradual API stabilization process (unstable xk6 extension -> k6/experimental built-in semi-stable module -> stable built-in k6 module)

Long version 😅

0ver is cool, though the biggest reason we haven't cut a k6 v1.0.0 release and adopted semver yet is probably connected to Go modules and how Go deals with major versions in dependencies. We have xk6 extensions, which are basically Go modules that import parts of k6 as a Go library and are built with the xk6 tool into a single k6 binary. If we adopted semver and released k6 v1.0.0, as soon as we make some breaking change (even in our Go API, not just the JS one!), we will need to release a v2.0.0, but that will mess up everything because of the Go module import paths being different... 😞

The solution there is to probably split apart the k6 codebase into at least 2 separate Go modules, the semver-based k6 binary and a 0ver common library that's imported by both the k6 binary and xk6 extensions. That would require some significant refactoring though, to both k6 and xk6 and xk6 extensions... 😞 So we're mostly focusing on other k6 improvements and on refactoring things that have more immediate benefits for us, for now, but they will also make the eventual split easier too 🤞

The second biggest reason we haven't released k6 v1.0.0 yet is that, unfortunately, sometimes we still have to make minor breaking changes in the k6 JS APIs or options... 😞 As @MattDodsonEnglish said, you can look through the release notes of previous k6 versions to see that:

1. We try really, really hard to avoid breaking changes whenever possible! Unfortunately, sometimes we have no good choice but to correct a bad decision we made, usually some years ago, but...
2. Such breaking changes are relatively rare and we try to make them as minor and inconsequential as possible
3. When they really need to be done, we try to give as much advance notice as possible. For at least one k6 version (~8 weeks), and we try to have both warning messages in k6 (when the old behavior we will break is used) and mention them in the release notes of previous k6 versions.

We know that people use k6 in production environments and are understandably annoyed when their script breaks due to a k6 update, so we try to avoid that at almost all costs. People sometimes even rely on the bugs and undefined/undocumented behavior that k6 has (#2623 is a recent example), so we are even labeling some of these bugfixes as "breaking changes", to potentially warn people that rely on the broken behavior... 😅

As we explained in the "Roadmap and future plans" section of the k6 v0.39.0 release notes, we have quite a lot of major refactoring and big changes planned for the near future. Besides the ones explicitly mentioned there, native support for distributed execution (#140) and test suites (#1342) are also hopefully going to be started soon. I don't think we'll need to make any breaking changes for them, but we might, so it makes some sense to consider releasing k6 v1.0.0 only after they are done.

Finally, as we also mention in the v0.39.0 release notes, we are trying to be more deliberate and gradual in how we develop and stabilize new APIs. We'll start developing and incubate them in an unstable xk6 extension, then move them to a k6/experimental built-in (but still somewhat unstable) module for a while, and finally stabilize them when we've gotten all of the feedback we can get.

The gRPC warning in the docs is probably a bit over-cautious at this point, but still somewhat relevant. The original gRPC implementation was done before we had xk6 extensions or explicitly experimental modules, so it was directly merged into the core with that warning. Because it was done before k6 had event loops, it was also merged without support for streaming RPC calls, so that still has to be added (#2020). I think we won't need to break anything in the existing APIs to support streaming, but I can't guarantee that 100%, so the warning should probably stay.

Anyway, sorry for the wall of text, but I hope that detailed explanation was useful to you and anyone else sharing your concerns.

[MasonM]

Wow, thanks for the great reply @na-- ! That does answer my question, so I'll go ahead and close this.

I recently finished my evaluation, and I'm going to recommend my team switch from JMeter to k6 for load testing. One of the reasons we've stuck with JMeter has been its relative stability: we almost never need to update our test scripts when upgrading. But JMeter doesn't have compatibility guarantees either, so it's still something we have to worry about. To me, the advantages k6 has over JMeter (particularly with respect to developer-friendliness and performance) are well worth the occasional breaking change. Hopefully I'll be able to convince the rest of my team of that!