Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"autogen.pl --force" fails in master tarball due to missing hwloc dirs #7363

Closed
jsquyres opened this issue Feb 4, 2020 · 5 comments · Fixed by #7449
Closed

"autogen.pl --force" fails in master tarball due to missing hwloc dirs #7363

jsquyres opened this issue Feb 4, 2020 · 5 comments · Fixed by #7449
Assignees
Labels

Comments

@jsquyres
Copy link
Member

jsquyres commented Feb 4, 2020

If you run ./autogen.pl --force in a distribution tarball, it fails because there are hwloc directories not present:

opal/mca/hwloc/hwloc2/hwloc/Makefile.am:23: error: required directory opal/mca/hwloc/hwloc2/hwloc/netloc does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:25: error: required directory opal/mca/hwloc/hwloc2/hwloc/utils does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:25: error: required directory opal/mca/hwloc/hwloc2/hwloc/tests does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:25: error: required directory opal/mca/hwloc/hwloc2/hwloc/contrib/systemd does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:25: error: required directory opal/mca/hwloc/hwloc2/hwloc/contrib/hwloc-ps.www does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:25: error: required directory opal/mca/hwloc/hwloc2/hwloc/doc does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:35: error: required directory opal/mca/hwloc/hwloc2/hwloc/contrib/windows does not exist
opal/mca/hwloc/hwloc2/hwloc/Makefile.am:37: error: required directory opal/mca/hwloc/hwloc2/hwloc/netloc does not exist
autoreconf: automake failed with exit status: 1
Command failed: autoreconf -ivf --warnings=all,no-obsolete,no-override -I config

We used to include these directories -- along with dummy Makefile.am files -- before we made hwloc be a git submodule.

make distcheck works on a master git checkout, but something is not translating correctly to the generated tarball such that if you run autogen in it, 💥.

@jsquyres jsquyres added the bug label Feb 4, 2020
@jsquyres jsquyres self-assigned this Feb 4, 2020
@bgoglin
Copy link
Contributor

bgoglin commented Feb 5, 2020

autoreconf -f fails the same in hwloc embedded tarball (with additional errors for missing *.in files).
@ggouaillardet looked at fixing this in the past open-mpi/hwloc#235 but it wasn't really needed at that point.
Not sure how to fix this. We could try changing the hwloc dist script to add all those dummy directories and Makefile.am?

bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…d tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However these tarballs cannot be re-autogen'ed unless they contain
all directories needed by automake even in non-embedded mode
(autogen doesn't know we're configuring hwloc in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because it needs many more standalone files
(everything used by configure, basically) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit 9af4e4d361c0f9e0e44a587c4b9b364b724265a6)
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…d tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However these tarballs cannot be re-autogen'ed unless they contain
all directories needed by automake even in non-embedded mode
(autogen doesn't know we're configuring hwloc in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because it needs many more standalone files
(everything used by configure, basically) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit ba6d413)
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…d tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However these tarballs cannot be re-autogen'ed unless they contain
all directories needed by automake even in non-embedded mode
(autogen doesn't know we're configuring hwloc in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because it needs many more standalone files
(everything used by configure, basically) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit ba6d413)
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 19, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit 9af4e4d361c0f9e0e44a587c4b9b364b724265a6)
@bgoglin
Copy link
Contributor

bgoglin commented Feb 19, 2020

This should be fixed in hwloc commit bgoglin/hwloc@18caedf (from hwloc v2.1)
It just creates the missing directories that autogen/automake complained about.
It added tests to hwloc CI so that such issues are now detected before reaching OMPI's submodule.
My quick OMPI test looks good but I'd like a double-check.

bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 20, 2020
…d tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However these tarballs cannot be re-autogen'ed unless they contain
all directories needed by automake even in non-embedded mode
(autogen doesn't know we're configuring hwloc in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because it needs many more standalone files
(everything used by configure, basically) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
bgoglin added a commit to bgoglin/hwloc that referenced this issue Feb 20, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
@jsquyres
Copy link
Member Author

@bgoglin I confirm that bgoglin/hwloc@18caedf fixes the issue. Would it be possible to get this merged to the hwloc v2.1 branch?

@bgoglin
Copy link
Contributor

bgoglin commented Feb 21, 2020

I'll decide after seeing your votes this morning :)

@bgoglin
Copy link
Contributor

bgoglin commented Feb 21, 2020

OK, zapproved.

bgoglin added a commit to open-mpi/hwloc that referenced this issue Feb 21, 2020
…ded tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However it means their tarballs cannot be re-autogen'ed because automake
fails when those directories are missing
(automake doesn't know if we're in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because they needs many more standalone files
(e.g. all .in files used by configure) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
bgoglin added a commit to open-mpi/hwloc that referenced this issue Feb 21, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
bgoglin added a commit to open-mpi/hwloc that referenced this issue Feb 21, 2020
…ded tarball

Embedders don't get utils,tests,contrib/*,doc,... directories in their
tarball because they build hwloc in embedded mode.
However it means their tarballs cannot be re-autogen'ed because automake
fails when those directories are missing
(automake doesn't know if we're in embedded or standalone mode).

OMPI didn't have any issue in the past because they manually copied
hwloc embedded directories AND created standalone subdirectories that
their automake would dist.
Now that they use hwloc as git submodule, bring the proper fix inside hwloc
so that OMPI doesn't have to hardwire that list of subdirectories anymore.

Refs open-mpi/ompi#7363

Note that pure-hwloc embedded tarballs still cannot be re-autogen'ed
because they needs many more standalone files
(e.g. all .in files used by configure) but we do not care.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit b332c93)
bgoglin added a commit to open-mpi/hwloc that referenced this issue Feb 21, 2020
…dded tarball

Build an embedded tarball from the original tarball and use it instead
for all tests.

The intend is to have our CI detect problems such as open-mpi/ompi#7363
whenever the list of non-embedded dist directories changes.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit 88fc5b4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants