spike: investigate if Fluttium can run on CI

[wolfenrain]

Description

I haven't actively explored if Fluttium can run on CI, in theory it should but it might need some minor code changes on the CLI side to make that a more streamlined experience.

Requirements

• Test Fluttium on the CI
• Make the visual experience useful on the CI

[matthiasn]

@wolfenrain I'm using Fluttium in CI, in Buildkite and it's working well, even when other tests are running in parallel. This is the pipeline step running Fluttium.

My only wish at the moment would be to get higher resolution screenshots, as they are pixelated and low-resolution now. Or even better, capture in the same way that Shift-Command-4 and then space on a Mac would give me, with a pleasing shadow and space around the window, and with an alpha channel.

[matthiasn]

Thanks for the library, super helpful.

[wolfenrain]

@wolfenrain I'm using Fluttium in CI, in Buildkite and it's working well, even when other tests are running in parallel. This is the pipeline step running Fluttium.

My only wish at the moment would be to get higher resolution screenshots, as they are pixelated and low-resolution now. Or even better, capture in the same way that Shift-Command-4 and then space on a Mac would give me, with a pleasing shadow and space around the window, and with an alpha channel.

Oh that is interesting! Glad to know it is running in some CI situations already!!

The screenshots are being taken directly from the device (using the RepaintBoundary) so I am not sure why the resolution would be lower? Can you show some examples related to it?

[matthiasn]

Sure, here's an example:

I think I found a fix, see this commit in the fork I just created. Came across that in this Flutter issue. Testing now, will know in a few if working.

[wolfenrain]

Ah yes! We should probably use https://api.flutter.dev/flutter/widgets/MediaQueryData/devicePixelRatio.html if we can access it somehow!

[matthiasn]

I can confirm that it works fine after just editing the respective file in .pub-cache, as I didn't want to learn how to assemble the fluttium-cli right now.

It also works fine when setting to pixelRatio: 3, with the images larger, as expected. Maybe use 2 as a sensible default and make it configurable? Or 3 as a default, as long as the quality is good enough. Would you accept a PR just changing the default? Didn't figure out at a quick glance how parameters are passed to actions.

[wolfenrain]

I would opt to default to the pixel ratio of the device and then have people pass it as a parameter if they want to override it.

Parameters are automatically passed to actions, so if you add it to the constructor it will be automatically available in the yaml.

We can access the pixel ratio if we expose media query data on the Tester:

class Tester {
  ...

  MediaQueryData get mediaQuery => MediaQueryData.fromWindow(_bindings.instance.window); 

  ...
}

[matthiasn]

Thanks, where would I add that in my project?

[wolfenrain]

That is meant to go into Fluttium itself haha, to expose information about the device, so we can access the devicePixelRatio in the screenshot action

[matthiasn]

Hehe cool, yeah once that lands the pixel ratio from the device will probably suffice anyway. Until then my hack editing the file directly on my CI is fine, thanks 🙏

[wolfenrain]

Will be available once #229 lands!

[matthiasn]

Wow, that was fast, thanks a lot!

[matthiasn]

@wolfenrain there's a potential improvement that could be made for running in CI. Currently, my test flow running in CI (Buildkite) results in roughly 142K lines of log output, which is a bit excessive (and might also slow things down noticeably). Buildkite deals rather gracefully by only showing the last 1MB of logs in the UI and still allowing to download the entire log file.

Could the excessive logging be reduced? I did try the --no-verbose switch suggested by the CLI but that's already used in the scenario reported above.

[wolfenrain]

@wolfenrain there's a potential improvement that could be made for running in CI. Currently, my test flow running in CI (Buildkite) results in roughly 142K lines of log output, which is a bit excessive (and might also slow things down noticeably). Buildkite deals rather gracefully by only showing the last 1MB of logs in the UI and still allowing to download the entire log file.

Could the excessive logging be reduced? I did try the --no-verbose switch suggested by the CLI but that's already used in the scenario reported above.

Are you running with the expanded reporter? fluttium test --reporter expanded (which on a old-school CI would be the default. That reporter is pretty non-verbose as it only prints for each new step and an additional one if it fails:

[matthiasn]

@wolfenrain great thanks, that works well, now I only have just over 200 lines of logs for the same test flow.

[matthiasn]

@wolfenrain when I use --reporter expanded, screenshots are no longer being taken though.

[wolfenrain]

@wolfenrain when I use --reporter expanded, screenshots are no longer being taken though.

Can you open a new issue for that with a reproducible example? The reporter should not influence that

[matthiasn]

@wolfenrain done, thanks 🤓