My Own Cluster

This is an experiment to provide a full blown yet scalable, durable, simple and flexible computing platform based on new standards and able to run and integrate legacy POSIX applications.

It is designed from the beginning to be simple to operate, especially disaster recovery is easy and always possible. Backups is managed by the software itself. Giving an instance the backuped storages puts the environment in production again.

Elements :

WASM
WASI
KVM
WebGPU
...

For a computer to be useful, it needs a model for :

Computing,
Storage,
Network and IO.

Building and running

Clone this repository :

git clone git@github.com:ltearno/my-own-cluster.git

cd my-own-cluster

The first time, download the dependencies :

make build-prepare

Then to run it, call this :

# Some values will be asked for https autosigned certificates generation,
# you can type [ENTER] until the end to use default values
make build run-serve

The program should build and start. It will listen on port 8443 with https protocol.

Testing

In another terminal, go to the repository directory and type :

# to build the executable binary (the previous step used "go run")
make build

# the samples directory
cd _samples

# this will upload some webassembly compiled sources to the service instance
make register

# this will call the uploaded code and show some results
make call

Executable formats

My-own-cluster executes :

web assembly byte code thanks to the wasm3 interpreter,
javascript code thanks to the duktape interpreter (there is no nodejs runtime).
glsl OpenGL shader language, by using OpenGL NVidia/AMD drivers loaded by EGL.

Each function will be executed in a jailed and sandboxed very light VM instance (requires KVM, not yet fully implemented, code in experiments/kvm).

The functions can interact with the platform through the Guest API which is described just below.

Exchange Buffers

Exchange buffers is a concept merging files, http requests and responses, web sockets and so on...

In memory it looks like this (data structure)

[ EXCHANGE BUFFER ] Lock : mutex // a lock to access the buffer Headers : map[string]string StatusCode : int WriteFinished : boolean // if closed, an exchangebuffer will not accept new parts. Once closed, it cannot be reopened Parts : [][]byte // a queue of not consumed parts One only reader at a time One only writer at a time

Usually an exchange buffer is manipulated through two interfaces serving two different purposes : a reader and a writer.

TODO : tranform ExhcnageBuffer to be a memory data structure TODO : a http request creates a input buffer (closed if http, still opened if websocket) TODO : request processing finishes when the output buffer is closed

Guest API

From the guest to the my-own-cluster host.

To support legacy POSIX applications, the host implements most of its API through WASI. There is also a direct binding for applications not using libc. partially implemented.

File system :

api://input : application input payload
api://output : application output payload
http:// and https:// : used by the guest application to issue a request to some JSON REST Service services.

There is also an API that my-own-cluster provides for guests. It does not require POSIX at all.

The bindings are available in :

rust : in this file assets/my-own-cluster-guest-api.rs
C/C++ : in this file assets/my-own-cluster-guest-api.rs

Those functions can be called by whatever language which is targetting WASM and has an FFI (all serious ones).

Core REST API

A basic set of REST api is provided through the core module. This module is loaded at startup like other functions and use the guest api to provide the REST API that the cli program uses.

The core api module implementation can be found in the assets/rest-default-api.js file.

Automatic module binding

You can import a wasm module and my-own-cluster will bind a stub to module registered with same name if it exists. The importing module can then call the imported module as if it was in the same VM instance. In reality they execute in separate KVM vcpu instance.

IFC (Inter Function Call)

Different modes possibles, based on low level directives.

a function has an array of input buffers (by default 1)
a function output is done in one output buffer

The orchestrator knows what to do with the input and output buffers when it executes the function. There is an API to control the execution plan done by the orchestrator.

The most basic execution plan is this one : the caller gives one input buffer, the callee executes and the output buffer is given back to the caller.

One more advanced execution plan is the map-reduce execution plan model.

Both modes should operate through the virtual file api.

Hooks and customization

The platforms should be very open and ease the creation of a community ecosystem of plugins and tools.

Hooks and customization allow to customize web requests processing, filter, implement ACLs and security and so on...

Samples

The samples directory contains littles samples compiled from different languages, using wasi or not.

Dashboard

This sample is an application written in javascript that uses the Guest API to provide a Web page listing all the resources and statistics actually available from the my own cluster platform.

Have the server running and run :

cd samples/dashboard
make

This will upload the required files and plug them to /dashboard/ url.

You can then visit https://localhost:8443/dashboard/ in your browser.

Watchdog

This sample is written in Rust. It is a simple Web application that uses the platform persistence utility (a key-value storage) to store services health checks timestamps.

Have the server running and run :

cd sample/watchdog
make

This will upload the required files and plug them to /watchdog/ url.

You can then visit https://localhost:8443/watchdog/ in your browser.

At the beginning no service is declared, they are added as they call their heart-beat url.

You can simulate a heart beat by opening this url in another tab of your brower : https://localhost:8443/watchdog/update/SERVICE_NAME (with any SERVICE_NAME).

Then refresh the web page to see the updated status.

WASI Rust demo

This is a demo of a POSIX program written in Rust running in my own cluster with the WASI ad-hoc implementation.

cd samples/wasi-rust-demo
make

OpenGL compute shader demo

This is a demo using the OpenGL GLSL compute shader language. The shader simply squares float elements of an input array.

This will only work woth a real GPU (tested with NVIDIA GTX960M and GTX1060).

cd samples/gpu-compute
make

Other demos

Other demos are available in the samples/other directory.

Here the list :

C : api-demo-c (without WASI), http-request (with WASI), uppercase (with WASI)
WebAssembly (native) : wasi-write (with WASI)
Go : hello-go (with WASI) can be compiled with tinygo and normal go compiler.

cd samples/other
make

More samples are coming...

Cleaning database

Run make clean-db to remove all persistence files. Your state will be lost.

Done

provide way to publish files with persistence
allow the function to call another function
run code: golang (tinygo)
run code: rust
run code: C/C++ (wasicc)
run code: javascript (bare duktape runtime)
define guest/host api over wasi/libc
provide persistent key-value storage
allow function to read/write in key-value storage
Provide GPU access (through WebGPU or other)
finish a mandelbrot sample with opengl
integrate esgl3 for broader compatibility

TODO

auto reload
update process
multi input buffers
sequencer/orchestrator, cost estimation and exec plan.
ACME protocol for TLS certs auto upgrade with Let'sEncrypt
use host name from http request in routing
integrate client machine into the exec model (a client with webgl or js engine or wasm engine could execute some processing for local speed-up with the same processing being launched also on server later for verification. With a consensus algo maybe blockchain like to prevent fraud).
upgrading, hot-reloading
demo of ACL filter
begin distribution of nodes
sqlite engine
include project templates in the CLI : rust, c, js, ...
isolate wasm3 execution in KVM
run code: compile GNU core-utils and run them to see improve WASI POSIX compatibility issues (https://github.com/coreutils/coreutils)
run code: C#
run code: Lua
run code: AssemblyScript
run code: Python
run code: Ruby
run code: Kotlin/Native
run code: PHP
allow for easy backup and disaster recovery, by default start on previous state
implement cron like scheduling api
think about distributing, replicating and scaling
export and import to and from a tar.gz file (even if leveldb files should be enough...)
SSO, User Federation, and security implemented as custom hooks

TO SORT Usefult links

KVM, ELF, ...

Rust

WebAssembly

Duktape:

Golang Web assembly

https://github.com/golang/go/wiki/WebAssembly

Name		Name	Last commit message	Last commit date
Latest commit History 323 Commits
_samples		_samples
api-generator		api-generator
apicore		apicore
apigpu		apigpu
apijwt		apijwt
assets		assets
assetsgen		assetsgen
common		common
enginejs		enginejs
enginewasm		enginewasm
enginewasmer		enginewasmer
experiments		experiments
tools		tools
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.ubuntu		Dockerfile.ubuntu
Makefile		Makefile
README.md		README.md
build-releases.sh		build-releases.sh
cli.go		cli.go
go.mod		go.mod
go.sum		go.sum
image.png		image.png
kvm.go		kvm.go
kvm_101.md		kvm_101.md
main.go		main.go
my-own-cluster.service		my-own-cluster.service
update-dependencies.sh		update-dependencies.sh
version.go		version.go
webserver.go		webserver.go

ltearno/my-own-cluster

Folders and files

Latest commit

History

Repository files navigation