GitHub Workflows - Binaries Auto-download

[hassandraga]

Every wrapper uses a different method to embed the WebUI binaries, some download it at runtime, some has a copy, and other build it from source... To unify the method and makes thing simple and secure, I created and tested a simple GitHub Workflow that downloads the binaries and push them to the repo whenever the binaries get updated. It's working fine for the Python wrapper right now.

1. Add .github/workflows/update_binaries.yml to the wrapper repo
2. Edit WRAPPER_WEBUI_BIN_PATH and set the folder path where binaries should be (Example)
3. Edit WRAPPER_WEBUI_BIN_VERSION and set the version to download, which should be 2.4.0 now
4. Make the wrapper use those binaries
5. Make sure NOT to accept any PR that has modified binaries

Result:

/include
/webui-linux-clang-x64
/webui-linux-gcc-arm
/webui-linux-gcc-arm64
/webui-linux-gcc-x64
/webui-macos-clang-arm64
/webui-macos-clang-x64
/webui-windows-gcc-x64
/webui-windows-msvc-x64

@webui-dev/webui-maintainers-team

[ttytm]

Won't this add a vast amount of binary data to the repositories over time?

What is wrong with the method e.g. V-WebUI or Odin-WebUI uses? One command to execute a script that does the full setup in a breeze of time, pulling the latest library file that is needed for a platform. I'm hesitant when including compiled binary data into repositories.

[hassandraga]

Won't this add a vast amount of binary data to the repositories over time?

Yes, it will make the repo bigger with every new WebUI release.

One command to execute a script that does the full setup

I see 3 issues with the script:

1. Let's say a user A developed a project X that has 3 dependencies, one of the dependencies is V-WebUI, end-user B want to use the project X, he has no idea how it works, he simply wants to use it. So, end-user B will clone project X, all dependencies will be cloned and ready to use, except V-WebUI, end-user B will need to cd folders... and sh script.
2. There is a very high chance that user A will NOT use V-WebUI simply because it needs extras steps... which is complicated for some people (end-users).
3. Probably the script for Linux (wget) and macOS (curl) are simple, but script for Windows it should be PowerShell, need to fix first disabled rights by default... So, the Windows script won't be easy to use. And replace the script with an executable is not a good idea.

I don't mind keeping script. I created this CI downloader just for testing purposes and comparison. We are discussing here what is the best option that can help the end-user easily use wrappers. With less headache as possible.

[ttytm]

Won't this add a vast amount of binary data to the repositories over time?

Yes, it will make the repo bigger with every new WebUI release.

One command to execute a script that does the full setup

I see 3 issues with the script:

1. Let's say a user A developed a project X that has 3 dependencies, one of the dependencies is V-WebUI, end-user B want to use the project X, he has no idea how it works, he simply wants to use it. So, end-user B will clone project X, all dependencies will be cloned and ready to use, except V-WebUI, end-user B will need to cd folders... and sh script.
2. There is a very high chance that user A will NOT use V-WebUI simply because it needs extras steps... which is complicated for some people (end-users).
3. Probably the script for Linux (wget) and macOS (curl) are simple, but script for Windows it should be PowerShell, need to fix first disabled rights by default... So, the Windows script won't be easy to use. And replacing the script with an executable is not a good idea.

I don't mind keeping script. I created this CI downloader just for testing purposes and comparison. We are discussing here what is the best option that can help the end-user easily use wrappers. With less headache as possible.

• Regarding 1, it's one command without the need to change directories.

  ~/.vmodules/vwebui/setup.vsh

  When having a setup step, this is as simple as it gets. It's simpler than setting up a lot of projects for first development usage. Having something precompiled shipped makes it simpler, but it's not worth it imho. A library is a tool for developers after all.

• Regarding 2. If there is a very high chance that someone won't use WebUI due to this one extra step, then I probably don't know the audience for the library.

• On Windows, the script for v-webui is also just one command, similar to the one above.

There are enough devs who prefer having a setup step over the binaries in a project, or to build from source - such concerns were stated from multiple parties in the project before. Sometimes there are even company rules that would disallow using the precompiled files due to security concerns.

So being honest, I think the contrary what is stated in the points is true. Adding such things can become the dealbreakers for developers who seek quality and look that standards are followed. There is unfortunately a reoccurring scheme in the library of too low standards and unesseccary redundant data. It's not too long ago, there were binary files everywhere and build direcoritires and redundancies for every platform and architecture. It was required to switch directories just be able to use the Makefile for another compiler.

Another thing is testing, before updating the CI there was nothing that would check if things even run. Now there is a little bit. The V wrapper got some actual tests, but over-all it's still way too little. The C lib has race conditions, and at least for my use cases - in a proper multithreaded application - it's impossible to use WebUI. So there are dealbreakers, but imo they are not in the setup process. For me personally, the lack of testing of the parent library is the biggest concern. For other developers and users I think that it's probably that versioning that is off and unreliably. Which is okay for a library in development but not for major version 2 project.

Now seeing the code quality of the CI downloader would make me even more skeptical about the product.

Sorry if this is harsh, but I think being fully honest will help most to progress towards something.

[GiuseppeCesarano]

I also agree with most of the comments in this post. The versioning of the project is not an issue for me because is such an arbitrary thing that I don't see a problem with that, but the lack of testing and documentation is something that needs to be changed, I am willing to write and help with the testing and documentation of the C++ wrapper, but I need the basics of the C API to be covered.

Binary in the repository is also a big no go for me, there are too many issues for little to no gain, my point of view on the 3 issues enanunched by @hassandraga:

1. Let's say a user A developed a project X that has 3 dependencies, one of the dependencies is V-WebUI, end-user B want to use the project X, he has no idea how it works, he simply wants to use it. So, end-user B will clone project X, all dependencies will be cloned and ready to use, except V-WebUI, end-user B will need to cd folders... and sh script.

WebUI is a framework intended for the end user to publish a desktop application, this example is already a corner case that I don't think will be that common, but for the sake of argument let's go with that; why would such a complex framework/dependency based on webui not have an installation or setup step, is it really that common to directly clone dependencies and not run any installation/setup step?

2. There is a very high chance that user A will NOT use V-WebUI simply because it needs extras steps... which is complicated for some people (end-users)

To be fair, if the user finds it hard to run a setup script, he will find it much harder to write the application and package it for distribution, I don't want to sound like an elitist in any way, but the focus of the project should not be on users who are now learning what a setup/build script is.

3. Probably the script for Linux (wget) and macOS (curl) are simple, but script for Windows it should be PowerShell, need to fix first disabled rights by default... So, the Windows script won't be easy to use. And replace the script with an executable is not a good idea.

I somewhat agree with this point, the windows script will be the ugly one, manually downloading the compiled binary or some sort of worst counterpart to curl will happen. But by the logic of not embedding an executable in the repository, why is a precompiled binary any better? In fact, I would argue that the precompiled library has more potential for damage if malicious code is injected.

But I also agree that a standard way of distributing the binary needs to be enablied

[hassandraga]

I already responded that we will keep those extra steps. But I would like to clarify things here for more transparency.

Sorry if this is harsh, but I think being fully honest will help most to progress towards something

Don't worry @ttytm 👍, we are all here discussing wisely to make the open source world even better ✌️

There is unfortunately a reoccurring scheme in the library of too low standards and unesseccary redundant data.

Usually, in the Beta version stage, we don't really try to make it perfect but to make it just work as a proof of concept. After that, in the pre-release, we remove all unnecessary redundant copies, and we make things cleaner and faster. And to be honest, I did what I could @ttytm, and I'm not saying the code is perfect, sure there is a large room for improvement, so feel free to improve it @ttytm ✌️

The C lib has race conditions

Ops!, my bad, sorry. I added mutex where flags are shared by threads, except for WS Open/Close events I bypass it as I want the app to close/refresh as fast as possible... however, I could miss some flags, so again, for sure there is a large room for improvement, so, feel free to improve it @ttytm ✌️ and thank you in advance

Now seeing the code quality of the CI downloader would make me even more skeptical about the product

Sorry @ttytm, I'm a little bit confused. There is no code in the CI downloader, just a bunch of Linux commands to download the archives and extract it. And I did it in a specific way to fix the GNUSparseFile.0 folder issue, otherwise, the commands will be just a few 🤷‍♂️ However, this CI will be thrown to the garbage and get replaced by a setup scripts 👍

but I need the basics of the C API to be covered

Understand, you are right. I just haven't found free time for the release documentation, and if you hadn't implemented the C++ wrapper a few months ago, I would not even included C++ in the release, so, thank you for that @GiuseppeCesarano. However, a list of all available APIs is here in webui.h. I will try my best to finish the C release documentation soon.

I would argue that the precompiled library has more potential for damage if malicious code is injected

To clarify, all the precompiled binaries in the repo are generated by GitHub CI script developed by @ttytm, which he did a very remarkable work by the way, so let me thank him for it 👍

[ttytm]

Thanks for addressing the feedback in the way that you do @hassandraga.

I'm going to try to contribute where possible. Though currently it's often just have a few free minutes. It's not enough to start working into getting the right overview and familiarity of the C code to solve things like the race condition.

If the chance opens up, I thought of further improving the Odin wrapper next and finish a nice example with a modern web framework.

[fibodevy]

I guess this doesnt apply to Pascal? Its just a matter of downloading latest release, or nightly build, by the user.

[ttytm]

It's the same for every wrapper as far as I can tell, it's just the processes how users get the latest library files.

[hassandraga]

Sure, but as I already explain in my previous comment, the issue is with end-user who don't want extras steps, except if the user who used Pascal-WebUI already downloaded and embedded the whole Pascal-WebUI repo into his project (repo). Please correct me if I'm wrong.

I don't mind keeping this extras-download step. I created this CI downloader just for testing purposes and comparison. We are discussing here what is the best option that can help the end-user easily use wrappers. With less headache as possible.

[fibodevy]

Pascal is very much like C. Will you add this to the C(++) wrapper? Its good for interpreted langs, like Python. Pascal users are used to get required libs separately, no need to provide them with the wrapper. A link to the compiled binaries is enough.

So for Python and the like - its ok, for compiled langs, rather not. But thats just my opinion.

[hassandraga]

Thank you @ttytm, @fibodevy and @GiuseppeCesarano, for taking the time to read this topic and sharing your ideas and point of view. This open-source project is driven by the community, and it is clear that the majority voted for keeping extra steps. So, let's do that, and I will remove the GitHub compiled CI binaries in the Python repo.

To ensure we are all on the same page, I would like everyone to agree on how the wrappers should "get" the binaries. And let's use this topic as a reference.

Build From Source Wrappers

Nim WebUI:

• Binaries Usage: Build from source
• Binaries Source: Already embedded in the Nim repo
• Status: ✅ Working

Static Library Wrappers

V WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Need to run ~/.vmodules/vwebui/setup.vsh
• Status: ✅ Working

Pascal WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Need to manually download binaries
• Status: ✅ Working

Go WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Right now, CI binaries are embedded. Let's remove it
• Status: ❔ Unknown (_Need the new implementation)

Odin WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Need to run webui/setup.sh
• Status: ☑️ Working but needs Windows implementation

PureBasic WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Need to manually download binaries
• Status: ✅ Working

Rust WebUI:

• Binaries Usage: Static lib compiling
• Binaries Source: Need to manually download binaries
• Status: ❔ Unknown (_Need implementation)

Dynamic Library Wrappers

Python WebUI:

• Binaries Usage: Direct dynamic lib use
• Binaries Source: Right now, CI binaries are embedded. Let's switch to downloading at runtime
• Status: 🚨 Won't work for offline cases

Deno WebUI:

• Binaries Usage: Direct dynamic lib use
• Binaries Source: Right now, it is downloading it at runtime
• Status: 🚨 Won't work for offline cases

Bun WebUI:

• Binaries Usage: Direct dynamic lib use
• Binaries Source: We will make it downloading it at runtime
• Status: 🚨 Won't work for offline cases

[ttytm]

Thanks for the overview. I think it's a great help to see it visualized and things just become small steps that can be checked off.

The script for Go-WebUI can either be run in a folder or via a webscript:

# Linux & macOS
sh -c "$(curl -fsSL https://raw.githubusercontent.com/webui-dev/go-webui/main/setup.sh)"

# Windows Powershell
irm https://raw.githubusercontent.com/webui-dev/go-webui/main/setup.ps1 | iex

[hassandraga]

The web script looks great.

[hassandraga]

I guess we can create one web script for all wrappers, every wrapper has the same base script, but with different settings of course.