Can windowjs run headless?

[guzmonne]

I've been playing with windowjs and found it amazing, but I was wondering if it could be made to run in headless mode. I would love to create animations as code that I can then export as a gif or mpeg4 video.

With the current API, I'm capable of generating screenshots on every animation frame that I can then process with FFmpeg to get the output I want. The problem is that the window must be visible to capture the screenshots. That is why it would be great to avoid running the screen on export and have it just for development.

Thank you, and keep up the excellent work!

[joaodasilva]

Hi, and thanks for the feedback!

IIUC, you'd like to capture screenshots very often, maybe on every frame, and export them automatically; and ideally, skip showing the main window while doing that. Is the understanding correct?

I guess you've already tried setting window.visible = false; the screenshot capturing doesn't work when you do that? If that's the case then it's a bug to be fixed. I'm surprised that it doesn't work, but will have to investigate why.

Here's 3 different ways to capture a screenshot:

1. canvas.encode(). This is probably the best solution, and should work even when the window is invisible (there might be a bug in there though)

2. canvas.getImageData() + imageData.encode(). This copies the canvas texture to RAM first, and might be a temporary workaround for the bug in (1)

3. Using the F8 key. I guess this doesn't work without the window, and doesn't fit well an automated tool :-)

Could you give these details to help diagnose the problem:

• Which of the three variants are you using?
• What platform are you running on?
• Did you build from source or are you using a prebuilt binary?

[guzmonne]

Many thanks for the excellent response. I was using the canvas.encode() method but had utterly disregarded the existence of the window.visible option.

As you said, running just canvas.encode() to store the images has strange behaviors. Most of the time, the process gets stuck after running for a few cycles.

Using canvas.getImageData() plus imageData.encode() gives much better result. I made a naive implementation that isn't useful because some images are stored with weird artifacts. I've added an image for reference.

I believe that a more robust implementation, with fixed framerate and other guards to handle the creation of the images might fix most problems. Maybe even find a way to control the JavaScript time engine to control when each frame gets rendered before moving to the next. I wouldn't mind sacrificing time in exchange for a complete render of all the frames necessary to create a perfect animation during the export phase.

Here are the answers to your questions:

• Which of the three variants are you using?
  I was using the first variant.

• What platform are you running on?
  macOS

• Did you build from source or are you using a prebuilt binary?
  I used a prebuilt binary.

[joaodasilva]

Thanks for the context.

Those artifacts are definitely a bug. The trouble is likely that reading the texture for the screenshot isn't synchronizing as it should with the Skia operations on the canvas.

You could try to await on canvas.encode() before rendering the next frame. Something like this:

function draw() {
  // Draw a frame...
}

async function drawAndScreenshot() {
  draw();
  const screenshot = await window.canvas.encode(...);
  // Save to file, etc.

  // Schedule the next frame *after* awaiting:
  requestAnimationFrame(drawAndScreenshot);
}

requestAnimationFrame(drawAndScreenshot);

[guzmonne]

So, I tried your suggestion, and it is working a little strangely.

When I run the script with window.visible equals true, the animation only happens if the windows have focus and events are triggered. Nothing happens if I run it with window.visible equal false.

Here is a Gist with my current script

Am I doing something wrong here?

[joaodasilva]

Thanks for the help Guzman.

I could reproduce the original issues at v0.1.8, and also verified that they're now fixed at HEAD:

• no corrupted images
• draws even if the windows isn't focused and doesn't have input
• draws even if the window is invisible i.e. with window.visible = false

I'll prepare a new signed release with the fixes (v0.1.9).

If you give this another try then please post back any findings. Many thanks!

[guzmonne]

It works!

Here is my updated code: Gist

Just run it with windowjs test.js let it run for a few seconds, and then run the following ffmpeg command to build the mp4 file:

ffmpeg -framerate 60 -i ./tmp/%06d.png -vf format=yuv420p output.mp4

Make sure to create the tmp folder before running each of these commands.

[joaodasilva]

Thanks for posting back! Very cool that this workflow works.

Do you know if it's possible to stream images to ffmpeg, instead of passing them all in the command line?

The idea would be to spawn ffmpeg from window.js, via Process.spawn(), and pass the images as they get generated via a pipe. Another option would to be make the processes talk to each other via a local socket; do you know if ffmpeg supports anything like that?

[guzmonne]

I know that ffmpeg supports consuming images from a pipe. Here is a quick example I found online using python:

import Image
from subprocess import Popen, PIPE

fps, duration = 24, 100
p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'mjpeg', '-r', '24', '-i', '-', '-vcodec', 'mpeg4', '-qscale', '5', '-r', '24', 'video.avi'], stdin=PIPE)
for i in range(fps * duration):
    im = Image.new("RGB", (300, 300), (i, 1, 1))
    im.save(p.stdin, 'JPEG')
p.stdin.close()
p.wait()

We would need windowjs to pipe each screenshot to a spawned ffmpeg process. Can we configure windowjs so that the screenshots get sent to stdout. If so, we could just pipe it using the command line, and it will work as long as windowjs outputs an EOF once it's done sending files.

If you can give me a brief outline of how to configure windowsjs to do this I can set-up a quick example.

[joaodasilva]

This isn't possible right now: window.js can only spawn copies of itself, and it doesn't give a direct handle to write to stdout.

I think it could have an extended Process.spawn that supports spawning any commands, and then expose "child.stdin" in the returned object to allow the parent to write images.

Closing that fd and waiting for the child process would have to be introduced as well.

I think this is a nifty idea but it requires some hours to implement and test for correctness.

A simpler approach would be to just enable writing to stdout from window.js; then your main loop could just write an image to stdout and await on that before iterating. You could then pipe windowjs to ffmpeg in the shell.