Download sources automatically?

[zakalawe]

This is a question, not an issue really - why not use ExternalProjectAdd to download the sources automatically? (Don't have to use the build / install feature, it can be download only). Either from a zip or from source control with a stable branch name.

[bjornblissing]

Certainly a good question, with many answers...

The first answer is that the function ExternalProjectAdd was unknown to me when I started this project. Secondly, I have tried using it and it never seems to do exactly what I want it to do relating to where files end up after compilation. Maybe it could be a good idea to use it for just downloading the different sources. This on the other hand would require the users to have svn and/or git configured correctly for use from inside of Cmake, otherwise it would fail.

So for now I will stick with the current manual solution. But I am always open for good pull requests.

[SrTobi]

I think it's a very nice idea to download the source automatically. So I started a fork using ExternalProject_add. I successfully added minizip, libjpeg, libpng, zlib and curl. You can find it here. All those projects have git repositories on github. I use ExternalProject_add to download the source. After that the already existing cmake targets are executed... That was a little bit tricky but now it works 😌

However, as so often in life one begins with something before thinking 😄 now I realize, that it would have been much easier and more failsave to just add those repositories as submodules ❗

Then we could simply use git submodule init --update and the source is there.

We could also use the ExternalProject approach and simply let the command configure, build and install the projects. All from the projects I added - except libjpeg - had a CMakeLists.txt which makes the building process extremely easy.

Only the approach I choose makes no sense 😭 ...

[SrTobi]

I think the submodule approach is much easier. I will try to implement it tomorrow.

[bjornblissing]

@SrTobi The drawbacks with the submodule approach is that it will lock this the repo to a specific version of the respective library and it will also force the users to download ALL supported libraries, even though they only need a subset.

[BurningEnlightenment]

@SrTobi FYI

it will lock this the repo to a specific version of the respective library

The user can locally checkout any commit he wants within the submodule and versions referenced by this repository could represent the latest ones tested.

it will also force the users to download ALL supported libraries, even though they only need a subset.

git clone only clones submodules when the --recursive switch is given. Furthermore you can specify a submodule if you do git submodule update using its relative path which then "updates" (i.e. clone & checkout) only the specified submodule.

The real no-no is that quite many people just barely know about submodules and how to use them correctly which makes it inconvenient for them to use this convenience project 😉
And obviously the submodule approach only works for projects which use git as vcs.

[AlessandroMenti]

Hi, is there any update on this? I am currently working on upgrading the third-party dependencies used by the FlightGear project (which uses these build scripts as a basis) and it would be great to have this functionality.

I'm also willing to prepare a pull request for this - just let me know the approach you would prefer (I am slightly in favor of downloading released .zip files instead of submodules, as not all the dependencies we rely on use Git as their VCS, but I am fine with both choices - I just need to know which one to implement).

[zakalawe]

Based on the feedback I received, I decided not to pursue this approach, the effort of download the zips is not bad. Of course, Cmake is capable to download zips using libCurl - that's what ExternalProjectAdd does.

[bjornblissing]

@AlessandroMenti There exists another fork which also have done some work regarding automatic downloading of sources. You might want to checkout:
https://github.com/chrisjaquet/osg-3rdparty-cmake/tree/feature/AddAutoDownload

Have not tried this my self, so I don't know if this branch is working or is work in progress. @chrisjaquet would you mind comment on this?

[chrisjaquet]

Hello, the branch does work, but it is, unfortunately, a quick "hack" that I did to quickly accomplish what I needed at the time. Please note the following differences from upstream:

• I changed the way that that the various libraries are enabled/added. upstream checks that the source directories exist whereas my branch uses cached "USE" variables. See the root CMakeLists.txt for the precise variables (around line 53).
• The logic for determining whether a particular library can be used (e.g. libpng depends on zlib) has been moved to the CMakeLists.txt of the relevant library.
  • I also make use of "target" names when calling TARGET_LINK_LIBRARIES as this sorts out some of the release/debug issues I experienced and automatically adds necessary dependencies. This is not 100% correctly done yet as I still need to correctly specify the include directories for the dependent files.
• I have added the d postfix for debug libraries (since I generally build OSG in debug mode).
• I added my own custom function to download and extract the sources into the structure that upstream expects. Details can be seen CMakeModules/FunctionDefinitions.cmake
• I added support for installing the built libraries to the CMAKE_INSTALL_PREFIX as well - this changes the default *_OUTPUT_PATHs specified in the root and, as a result, requires that the 3rdParty dir be correctly specified when building OSG. I needed to do this to support building multiple versions of OSG.
• The sources for specific commits from github are downloaded as zip files, not cloned from the repos.

I would, however, like to look at support for ExternalProjectAdd, but until I have the need to work on this again, it probably will not happen. It was also a case of having not used the function before.

I hope this provides some clarity, but feel free to ask any questions you may have.

[bjornblissing]

I have added support for AppVeyor builds in the following commit: 8d8edd6 .

This means that there will be prebuilt version of these dependencies available to download for the latest commit for Visual Studio 2015 & 2017, both 32-bit and 64-bit. The download links are posted on the readmepage.

For anyone desiring to write your own batch script for automating the build, the command section in the appveyor.yml file may be a good starting point.