The whole system depends solely on looo #158
Comments
|
ping @looooo @yorikvanhavre |
|
looooo is doing a tremendous job AND we do need more people involved that understand the whole conda workflow. |
|
I'd be all for moving stuff to the FreeCAD org repo, if @looooo finds it a good idea. But it won't magically make things go faster. |
|
I wonder if the whole packaging area woudn't be something where the FPA could help... |
|
Between this thread https://forum.freecad.org/viewtopic.php?t=76777&start=20 and this thread https://forum.freecad.org/viewtopic.php?t=76795&start=10 user "oursland" may be able to help. I've been able to build FreeCAD on macOS ARM using his conda recipe. |
|
We do need a packaging team that has a way to communicate. For obvious reasons but specifically to keep all builds and their dependencies in unison (a quick example would be keeping all OCC libs ideally be at the same version across all permutations) |
|
Well yes I am trying to help where I can with conda (but don't have enough knowledge to be a proper maintainer) I'm open to accepting a role as a fallback guy for merging PRs or other mundane activities. An additional benefit is that it makes those repos discoverable, I can't even remember how I found that compilation is happening from his repo but it sure as hell wasn't straightforward to find! A discoverable repo might attract more contributors. |
|
User ppd was documenting how to maintain the Snap packaging. But it still incomplete. Nevertheless, it has been somewhat helpful, for example at release time. I think it would be important to document the workflow of all package building/managing (PPD, Conda (AppImage, Windows Installer, macOS), Snap, Flatpak, Docker (GUI, CLI)). |
|
case in point: https://forum.freecad.org/viewtopic.php?p=670177#p670177
we need to merge a change in looo's repo to fix current issues the proper way, if he's not available there's no telling when/if that is ever getting merged. A hacky solution can be made on the bundle side (this repo) but that is not ideal and might not always be possible. Who else has merge access to this repo that can merge my PR as workaround in the meantime? |
|
I have merge access here, so I can do the local merge, but of course I can't do the feedstock. Next time @looooo has some availability we should discuss moving the feedstock to under the FreeCAD git account so that other maintainers can help out. |
|
Note that the actual build config is not on @looooo 's repo (he has only fork) but on conda itself: https://github.com/conda-forge/freecad-feedstock . But AFAICS it only configures how to build FreeCAD on the conda forge, similarly to other CI platforms. I can't find anything there that says how the different release packages are built |
regarding macOS, loooo still uses his own fork per #171 (comment) |
that's not entirely correct, yes some builds are made on the conda-forge repo, those get uploaded here: https://anaconda.org/conda-forge/freecad/files when I opened this issue only stable releases where being uploaded there, but it seems it has now enabled dev builds there too: conda-forge/freecad-feedstock@b0af988#diff-f4fdc7679c30abcace9506f9d2cb7c79b22b5bfb24787e0ffb18f37349a0a178 could you comment on that @looooo ? on the other hand there are scheduled builds running at: https://github.com/looooo/freecad-feedstock which get uploaded to: https://anaconda.org/freecad/freecad/files and these are the ones we are using in our bundles AFAIK, these are the ones I'm worried about because it's using what looks like an official account using our name but it seems nobody besides lorenz has the credentials to access that and the builds are configured on his personal repo. Maybe @sgrogan knows somethings? Although I haven't seen him around for a long time either. |
|
Trying to understand how the system works, its seems you can generate and get CI running on a repo using https://github.com/conda-forge/conda-smithy Perhaps the FPA can contact anaconda and try to get access to it, it's listed as an organization account. |
|
that's not entirely correct, yes some builds are made on the conda-forge repo, those get uploaded here: https://anaconda.org/conda-forge/freecad/files when I opened this issue only stable releases where being uploaded there, but it seems it has now enabled dev builds there too: conda-forge/freecad-feedstock@b0af988#diff-f4fdc7679c30abcace9506f9d2cb7c79b22b5bfb24787e0ffb18f37349a0a178 could you comment on that @looooo ? Sorry for my late response. Actually I tried to address this issue by moving the builds to conda-forge. So there is now the conda-forge/freecad-feedstock development branch which is a replacement for my fork. I also moved the script which builds freecad twice a week to this repo. So to keep the conda-infrastructure for freecad running some work is needed. I tried to do this but I failed several times with the mentioned tasks. Also I don't see why this all depends only on me. All the scripts are openly available and everyone can experiment with the builds. But maybe new forces are needed to keep this running. |
|
Thank you for all your work looo, sorry to be bothering you
are there not changes from your fork that need to be pushed to the conda forge repo? I can see about making a PR for this next week. Will this account/repo be disabled then: https://anaconda.org/freecad/ ? I ask because I think only you have credentials for that one, and it looks kinda official. If we'll keep that online I think it's important that the FPA and/or other maintainers also have access to the credentials for that account.
well the one in your repo seems to be working on windows, that's why I say there's probably stuff that needs to be pushed from your repo to the conda-forge one. I tried to work today on fixing the build error and it got really confusing trying to track which repo I should be trying to fix, and why they are different.
doesn't azure have arm machines? why do we need cross-compilation? For example occt-feedstock appears to be configured to build in arm too. We can tackle this problem later though, I don't think we're in a rush on that front.
I will try to help fix the compilation problems.
yes, here on the bundle repo we have other people who can merge stuff, but on the feedstock side where the actual compilation happens only you had write access, I want to contribute but there's nothing I can do if there's nobody around to accept my contributions, that's why I think we need more people with write access on the relevant repos and perhaps to the anaconda and azure accounts associated. I mean we could start from scratch if it really comes down to it but I think it would be better to have more people on the current setup. Now that you are moving the compilation to conda-forge, who else has write access there that we can contact when you are not around? If there's nobody else do you have the authority to give someone else in our team access there as fallback? |
Yeah if we move everything to conda-forge we should be able to disable this account. But I am not sure how to upload local builds to conda-forge. So for osx-arm and Linux-aarch64 the freecad account is still needed.
The one in my repo is not updated to boost 1.78.
Afaik azure has no arm machines. The occt-feedstock is also cross-compiled. There are 2 options: building with qemu (which is too slow for freecad) and cross-compiling.
Maintainers can add them self by:
Thanks, solving this issue would be really great. Additionally there is also a crash with osx-arm64. I reported in the forum. There it would be nice to check if the brew-build also has this issue. The crash can be reduced by drawing a line in sketcher and toggle the constraint type to blue and "constraint" the length of the line. To me it seems like a coin callback issue, but I am not sure. This crash was introduced by some osx-updates... |
|
Could you describe briefly how the appimages are produced from the conda builds @looooo ? I couldn't find anywhere something that shows how it is done... |
|
@yorikvanhavre The bundling is done via a script which gathers all the necessary dependencies via the conda package manager. Then it copies the files and compress them to a appimage/DMG/zip file. |
|
Okay found it: https://github.com/FreeCAD/FreeCAD-Bundle/blob/master/conda/linux/create_bundle.sh Thanks! I'd like to make a script that one can run locally, and that creates an appimage build, without conda. I'll have a look at how that one works |
|
Running locally is certainly possible but not using conda would mean we have to get all the other dependencies from some other source. We would have to use some form of package manager either way (unless you want to compile everything, we're partially doing that for flatpak), using a distro's repo would be more stable but also less flexible, when there are problems in a 3rd party package it'll be harder to solve, and if we use Debian or Ubuntu as a base updating things like opencascade would be kind of a pain. One not frequently considered advantage of running conda builds is that we can detect early problems with new versions of our deps, so we solve them before they become a problem on other distros. |
|
If using a standard distro like debian, most of the packages are there already, and there are scripts like https://github.com/AppImageCommunity/pkg2appimage that can use it. The rest can be obtained via pip. My idea here is not to substitute the conda workflow for our releases. Rather have an alternative in case, for some reason, the conda workflow is not available. Basically something that can be documented and used by someone else to either produce a release package or, who knows, adapt to something else... It's mostly to build knowledge ;) |
@looooo could you then pass along credentials for this account to @yorikvanhavre so that the FPA has access to it and can then give access to others as required? I will not do it right now but eventually I'd like to setup a feedstock repo in our org account where I would pin all or most of the dependencies so that we can easily make stable releases without having to chase after new library releases every time. My idea is to make the process smoother so that we can increase our release frequency without adding absurd amounts of work to volunteer packagers. It seems to me that for this type of usage that conda repository is the most appropriate. |
|
i transferred my fork of the freecad-feedstock to the freecad organisation. Still not sure if this makes a lot of sense as osx and linux are now build via conda-forge/freecad-feedstock (development-branch). |
|
Thank you, that still leaves the question of credentials to the anaconda account though, I've seen that circle CI has mac m1 machines available so I was thinking of moving the mac builds to that as I recall that you told me in azure we are doing cross compilation (which seems to not be working) and bundling for arm from x86 had some issues. (but we probably can't move everything to circle CI as there's a limit for free use) About using conda-forge, that is indeed good and I think it's a good strategy for the weekly builds, however from what I remember it's been problematic to keep freecad up to date with the constantly updated dependencies on conda specially for point releases. I hope that if we pin most of the dependencies that would not be a problem and we would be able to more easily make releases, which would help if we aim to increase our release frequency. So my idea would be to get things working on the conda-forge repo, use that for our weekly builds and once we make a major release pin all dependencies to what is working at the moment then use that for all subsequent point releases in that series. |
@looooo I've created issues for some of the meaningful repos, I don't know if you are able to approve these: are there other significant feedstocks where we could benefit from having more people from our team? |
|
I hate to keep nagging you @looooo but now that the feedstock repo has been moved to the org account it seems the automated builds are not running on it anymore, I suspect it has to be registered again with azure and anaconda, could you please provide the account and password to the anaconda account? (perhaps via forum PM) and if you can please also accept the issues I created to add myself to the conda-forge smesh and freecad feedstocks, and thank you for approving the one for occt. |
@yorikvanhavre did you have the time to make any progress regarding building AppImages locally? |
|
I shamefully have to say no 😅 will try ASAP |
|
Great to hear that but no pressure! |
|
given that the people of conda-forge don't want weekly builds done on their infrastructure: conda-forge/freecad-feedstock#99 I think we have to put our freecad-feedstock repo back in action @looooo, I essentially have it working already on my personal repo. And we don't need to use azure since conda-smithy also supports github actions, plus I figured out how to use cirrus-ci for native arm builds (although this might not actually be needed: https://github.com/adrianinsaval/freecad-feedstock/actions/runs/5502596625, I'll have to make a bundle out of the cross compiled packages and have some people test them to see if no functionality is lost, particularly I'm uncertain if the spacemouse driver will work on the cross compiled version). I'm just missing access to the anaconda.org/freecad channel. |
|
How does it work to put our stable releases to conda-forge? |
|
I have a checklist here: @yorikvanhavre Btw. Thanks for continuing the work on the feedstocks @adrianinsaval |
|
Not yet, but soon we'll have a release candidate |
For some weeks now the conda builds have been stuck, I've been asking around trying to find the problem and solutions but looo seems to not have much availability lately (can't blame him for that) so things move slow, it might be the case that we manage to fix this without his intervention or that later he has time and can get it working again, but this is very fragile and completely depends on this one person.
The actual compilation of freecad for conda even happens on a personal repo from him: https://github.com/looooo/freecad-feedstock
so if something needs fixing there we can't even push changes without his intervention.
I also don't know who else has access to the anaconda freecad account, is it also just him?
https://anaconda.org/freecad
and how does the uploading of those packages work? how to set up another repo to upload stuff there, how is the authentication done and who can approve it?
What can we do to improve this? Can we move the feedstocks repos for the packages in that account into the github org account?
The text was updated successfully, but these errors were encountered: