Image processing with Virgil

[kalwalt]

I'm playing with Virgil and image processing and i have created a simple repository vrgl-imgproc. My goal is to develop a small library to perform some basic image processing task with Virgil. Because there aren't yet libraries for Virgil dedicated to load images or video-streams, it's "easier" to deploy a WASM file that handle Virgil Pointer's to load and return RGBA data.
This is the method i used to accomplish this task:

Before all you need to "allocate" some space in the WASM linear memory:

def malloc_u8(size: int) -> Pointer {
    var Arr: Array<byte> = Array<byte>.new(size);
    return Pointer.atContents(Arr);
}

the function will return a Pointer ( an address of the memory) this will be used soon. In the javascript code after instantiated the WASM code you invoke that function:

// First Get the pixels as Uint8Array from the image with the getImgData function:       
const idata = getImgData(img, canvas, WIDTH, HEIGHT)
               

const mallocU8 = instance.exports.malloc_u8(idata.data.length);

at this point you can:

// Now we are ready to set some values in the memory.
// Note that we instantiate the Uint8Array with the buffer from the WebAssembly memory and the memory offset from mallocU8.
// Very important!!! Size defined in malloc_u8(1228800) need match here : (640 * 480 * 4)
const offset_invertData = new Uint8Array(instance.exports.memory.buffer, mallocU8, WIDTH * HEIGHT * 4);
offset_invertData.set(idata.data)

then invoke the imgProc_invert function

// This is where the magic happens imgProc_invert accept (width: int, height: int, channels: u8, inputData: Pointer)
const offset_invert = instance.exports.imgProc_invert(WIDTH, HEIGHT, 4, mallocU8);

the imgProc_invert function is defined as:

//from FilterWasm.v3
def imgProc_invert(width: int, height: int, channels: u8, inputData: Pointer) -> Pointer {
    var imgProc = ImgProc.new(width, height, channels);
    var inputData_size = width * height * channels;
    var internal_data: Array<byte> = Array<byte>.new(inputData_size);
    for (i = 0; i < inputData_size; i++) {
        internal_data[i] = (inputData + i).load<byte>();
    }
    return Pointer.atContents(imgProc.invert(internal_data));
}

note that we are invokeing another function instantiating the ImgProc class and we get the Pointer.atContents thanks to the invert function:

def invert(src: Array<byte>) -> Array<byte> {
        var out = Array<byte>.new(this.width * this.height * this.channels);
        var imageSize = this.width * this.height * this.channels;
        def invert_const: byte = 255;
        var cn = this.channels;
        var q: int = 0;
        if (this.channels == 4) {
            for(p = 0; p < imageSize / 4; p++) {
                out[q + 0] = invert_const - src[q + 0];
                out[q + 1] = invert_const - src[q + 1];
                out[q + 2] = invert_const - src[q + 2];
                out[q + 3] = 255;
                q += cn;
            }
        } else {
            for(p = 0; p < imageSize; p++) {
                out[p] = invert_const - src[p];
            }
        }
        
        return out;
    }

we are using an Array since we can't get a Pointer address with a Vector type? So we can't resize or shrink our array, better will be using a Vector.

so we can easily display the result:

const img_invertData = ctx_invert.createImageData(WIDTH, HEIGHT);
// Ready to show the inverted values, as we did for offset_invertData we do for linearMemory_invert
const linearMemory_invert = new Uint8Array(instance.exports.memory.buffer, offset_invert,  WIDTH * HEIGHT * 4);
// This will print the Uint8Array
console.log(linearMemory_invert);

var inputData = linearMemory_invert;
console.log(inputData);
for (let i = 0, j = 0; i < inputData.length; i++) {
    img_invertData.data[i] = inputData[i];
}
ctx_invert.fillRect(0, 0, WIDTH, HEIGHT)
console.log(img_invertData);
ctx_invert.putImageData(img_invertData, 0, 0);

At the end we have our inverted image....
For more test the examples at test/Filters

[titzer]

Hey, really nice work. I'm impressed you figured out how to do this with how little documentation there is here :-) Your approach mirrors some tinkering I did with the WebMandelbrot "app" here: https://github.com/titzer/virgil/tree/master/apps/WebMandelbrot

I love these kinds of demos and would happily accept (small) demos and improvements to documentation that make it easier to get started and do what you've done here.

One improvement I didn't do in the WebMandelbrot app that I wanted to is to use a layout to define the RGB pixel data:

layout RGB {
    +0    red: byte;
    +1    green: byte;
    +2    blue: byte;
    +3    alpha: byte;
}

You can then define an object that wraps up the Array<byte> and gives access to individual pixels:

class PixelData(width: u32, height: u32) {
    def [x: int, y: int] -> Ref<RGB> { ... }
    def start() -> Pointer;
    def size() -> u32;
}

def foo() {
    var p = PixelData.new(320, 640);
    p[33,44].red = 99;
    ...
}

[kalwalt]

I love these kinds of demos and would happily accept (small) demos and improvements to documentation that make it easier to get started and do what you've done here.

I would like to contribute in this sense, maybe we can add a simple demo to explain how to handle Pointer as arguments as i did in my code and maybe improve the mandelbrot demo.
Have you thought to deploy the documentation with github pages?

[titzer]

I think it'd be great to improve documentation accessibility with GitHub pages, but haven't had time to look into it.

[kalwalt]

Because most of the documentation is in the doc folder you could deploy a ghpage from the main branch and choosing the doc folder. It's possible also to setup a github action to automate and fine tuning the whole process.

[kalwalt]

Hey, really nice work. I'm impressed you figured out how to do this with how little documentation there is here :-) Your approach mirrors some tinkering I did with the WebMandelbrot "app" here: https://github.com/titzer/virgil/tree/master/apps/WebMandelbrot

I love these kinds of demos and would happily accept (small) demos and improvements to documentation that make it easier to get started and do what you've done here.

One improvement I didn't do in the WebMandelbrot app that I wanted to is to use a layout to define the RGB pixel data:

layout RGB {
    +0    red: byte;
    +1    green: byte;
    +2    blue: byte;
    +3    alpha: byte;
}

You can then define an object that wraps up the Array<byte> and gives access to individual pixels:

class PixelData(width: u32, height: u32) {
    def [x: int, y: int] -> Ref<RGB> { ... }
    def start() -> Pointer;
    def size() -> u32;
}

def foo() {
    var p = PixelData.new(320, 640);
    p[33,44].red = 99;
    ...
}

Hi Ben i looked a lot at your Mandelbrot example that was useful to get the Pointer but the problem for me was to solve how to fill the data as an argument. I had read that in the Pointer documentation exist the load method but wasn't clear to me how to handle this. I realized that i need to imcrement it to load every byte of data ( or what type is our data).
I developed the malloc_u8 function thinking about how Emscripten works in a similar scenario ( the _malloc function), i' don't know if it is a better way or a better approach to manage the data in memory...
Thank you for the input for the layout stuff i think i will implement in my code, this is very useful!

My idea is also to use ImgProc class with linux. The great obstacle is to provide some amount of reasonable data. I will try maybe to load raw data as binary converting an image in an external tool to binary and then try to load with Virgil and the DataReader, this will be my to do list.

About the documentation: You can find a resource in the test folder, i discovered this fact after writing my code, it's a bit hard but you can find some interesting snippets of code.

In the end i'm very impressed how you developed Virgil, i hope you continue on this road...

[kalwalt]

@titzer i created a PixelData class as you suggested you can find it here, i implemented a Pixels enum to store different PixelsType (RGB, RGBA and so on)

enum Pixels(numPixels: int) {
    RGB(3),
    RGBA(4),
    BGR(3),
    BGRA(4),
    GRAY(1)
}

and for every PIxels type a different layout, then my PixelData class is:

class PixelData {
    private var data:  Array<byte>;
    new(width: u32, height: u32, pixelType: Pixels) {
        match(pixelType) {
            RGB => data = Array<byte>.new(int.!(width) * int.!(height) * Pixels.RGB.numPixels);
            RGBA => data = Array<byte>.new(int.!(width) * int.!(height) * Pixels.RGBA.numPixels);
            BGR => data = Array<byte>.new(int.!(width) * int.!(height) * Pixels.BGR.numPixels);
            BGRA => data = Array<byte>.new(int.!(width) * int.!(height) * Pixels.BGRA.numPixels);
            GRAY => data = Array<byte>.new(int.!(width) * int.!(height) * Pixels.GRAY.numPixels);
        }
    }
    def [x: int, y: int] -> Ref<RGB> { return Ref<RGB>.at(data, x * y); }
    def start() -> Pointer { return Pointer.atContents(data); }
    def size() -> u32 { return u32.!(data.length); }
}

maybe this is not elegant and probably there is a better way to implement the constructor?
Anyway the problem is that the code can handle only RGB Pixels because Ref. I would add composability to the class but if i change the class in this way:

class PixelData<T> {
// constructor and other same code
// ....

def [x: int, y: int] -> Ref<T> { return Ref<T>.at(data, x * y); }
// ...
}

i got this errors:

[testPixels.v3 @ 4:23] UnresolvedLayout: layout "T" cannot be found
    def [x: int, y: int] -> Ref<T> { return Ref<T>.at(data, x * y); }
                                ^
[testPixels.v3 @ 4:23] UnresolvedType: type "RGB" cannot be found
    var p = PixelData<RGB>.new(width, height, Pixels.RGB);

I think because it should be declared a type RGB but i don't know if this is possible...

[titzer]

Nice. Because the enum has a field, you can just do pixelType.numPixels. The error you are getting is because Ref<T> is, confusingly, not like other things that accept parameters. The T must be the name of a layout.

Thinking about your design problem, one idea would be to separate the RGB image data from the gray scale (by having a separate class for each), and then having a setRGB(x: int, y: int, r: byte, g: byte, b: byte) and getRGB(x: int, y: int) -> (byte, byte, byte) and then internally switching on the pixel type.

[kalwalt]

Thinking about your design problem, one idea would be to separate the RGB image data from the gray scale (by having a separate class for each), and then having a setRGB(x: int, y: int, r: byte, g: byte, b: byte) and getRGB(x: int, y: int) -> (byte, byte, byte) and then internally switching on the pixel type.

i have thought also this, maybe this is the best solution. Maybe a PixelData class as a base class that extends a PixelDataRGB and so on.

[kalwalt]

I modified the code making PixelData a base class and creating extended classes PixelDataRGB, PixelDataRGBA and so on for other pixel types, i think it's a better design. I need to tests for every classes but i think my code should be almost correct.

class PixelData {
    private var data: Array<byte>;
    private def width: u32;
    private def height: u32;
    private var pixelType: PixelType;
    private var channels: int;

    new(width: u32, height: u32) {}

    def start() -> Pointer { return Pointer.atContents(data); }
    def size() -> u32 { return u32.!(data.length); }
    def getPixelType() -> PixelType { return pixelType; }
    def getWidth() -> u32 { return width; }
    def getHeight() -> u32 { return height; }
    def getChannels() -> int { return channels; }
}

class PixelDataRGB extends PixelData {
    new(width: u32, height: u32) super(width, height) {
        pixelType = PixelType.RGB;
        channels = pixelType.numPixels;
        data = Array<byte>.new(int.!(width) * int.!(height) * channels);
    }
    def [x: int, y: int] -> Ref<RGB> { return Ref<RGB>.at(data, x * y); }
}

// other classes PixelDataRGBA and so on.. 

One issue can arise if a developer want instantiate the PixelData class directly, and not the extended class, question: virgil does have a mechanism to prevent this?

[k-sareen]

Very cool library! To answer your question, yes it is possible by doing two things: (i) declaring the classes in a new file; and (ii) making the definition of PixelData.new into private.

For example, this will not compile:

$ cat NN.v3
class A {
        private def i: int;
        private def j: float;

        private new(i: int, j: float) {}

        def foo() {
                System.puts("A: ");
                System.puti(i);
                System.ln();
        }
}

class B extends A {
        private def k: long;
        new(i: int, j: float, k: long) super(i, j) {}

        def bar() {
                System.puts("B: ");
                System.puti(int.!(k));
                System.ln();
        }
}

$ cat NM.v3
def main() {
        def x = A.new(10, 4.3f);
        def y = B.new(15, 4.1f, 159);

        x.foo();
        y.bar();
}

You get errors like:

$ v3i NN.v3 NM.v3
[NM.v3 @ 2:19] UnresolvedMember: type A has no such member "new"
        def x = A.new(10, 4.3f);
                  ^^^
[NM.v3 @ 2:17] TypeError: expected function for call, got ?
        def x = A.new(10, 4.3f);
                ^^^^^
[NM.v3 @ 5:11] UnresolvedMember: expression of type ? has no such member "foo"
        x.foo();
          ^^^
[NM.v3 @ 5:9] TypeError: expected function for call, got ?
        x.foo();

Essentially private means private to current file (so akin to static in C/C++ if you come from that). So if I had inserted the main function into the same file, then it would have happily compiled (which is personally a bit counter-intuitive).