Don't require .buckconfig and/or fix nested projects #939
Comments
|
I can't even fix this using the workaround of re-creating the rules in the higher directories right now. I'm going to have to use an EDIT: the workaround: # XXX temporary - see facebook/buck#939
import os
import os.path as path
def punlink(*args, **kwargs):
try:
os.unlink(*args, **kwargs)
except:
pass
punlink('ext/arua-bootstrap-grammar/BUCK')
punlink('ext/arua-bootstrap-grammar/.buckconfig')
punlink('ext/arua-bootstrap-grammar/ext/BUCK')
# XXX end temporary |
|
You've hit the nail on the head. I looked at using Buck or Bazel for managing HDL environments (Verilog, SystemVerilog, VHDL, etc.), but lack of support for nesting projects was a show stopper. In Verilog, the basic unit of "code" is actually called a module, so one would think it would be easy to adapt Buck to it. Not so! I'm unable to reuse a module from another Buck project if that modules also happened to reuse sub-modules Thanks for the write-up here. |
|
I believe the undocumented Cells feature solves this problem. It's undocumented because it's still evolving as the team figures out edge cases that don't work well with it. I believe the Gerrit project uses these to manage some of their third-party dependencies (@davido has filed issues too that has led to improvements of the Cell UI). @mzlee and @philipjameson have also been playing with them and have been helping to improve the Cell UI. |
|
@sdwilsh I'm not sure I'm following. What are cells? Where can I read about them? |
|
The best there is right now is probably the integration tests for it (so documentation by example only): https://github.com/facebook/buck/tree/master/test/com/facebook/buck/crosscell/testdata |
|
From what I can gather about cells, no, this is not what I'm addressing. From what I can tell (since there is no documentation) cells are just sibling directories with Buck projects inside that allow you to reference another sibling. This doesn't solve this problem. |
|
Buck was initially developed for use in a monorepo, and is still pretty biased towards it. We currently use cross cell with a flat hierarchy of cells, so I'm not sure how well it works with nested cells, but at least in theory it should as long as each cell only refers to its child cells, and not other descendants or ancestors. In your example repo, I applied this patch: Now you can build it: Does this feature fit your needs? |
|
If Buck is to remain a monorepo-biased build system then yes, that does work. Thank you for the followup @Coneko :) |
I know there was some discussion a while back about
.buckconfigfiles being mandatory.This anti-nesting paradigm is ultimately the reason why I have personally strayed away from build systems like Bazel and (now, potentially) Buck.
Let's start with the needless empty
.buckconfigfile:I'm still getting the following
This whole concept of "project roots" throws a wrench into the whole modularity thing (link is to factor 2).
Why? Because you can't nest modules anymore. At least, not in any clear or direct way. Projects/repositories will (should) never know if they're really the "project root".
Here's an example. Let's take this sample repository that has a dependency within a dependency.
At a glance, here is the directory structure for each dependency:
In the real world, each of the above would be a dependency of some sort, either via a submodule or some package manager. Each dependency is completely unaware of their consumers (downstream dependents) and merely expose a
'PUBLIC'-ly visible member for the outer codebase (downstream dependent) to consume.Let's take a look at the compilation chain here, starting first with the inner-most dependency. All it does is export a single executable script using
genrule()(sinceexport_file()doesn't support executables).Compiling it by itself is, of course, just fine.
Now let's go up a directory and try the
namesproject. This project will "build" the compiler if it isn't already built (copied), and will proceed to use it to compile all of the names into a single file.Sweet, it appears to have found the compiler and has executed it (the output
compiling 2 filesresulting from our 'compiler' script).Let's go back up to the main repository and try to build it. It uses the names dependency to get a list of all names, and then reverses them. We're in the business of reversing things so this is really our intended output. We don't care that it uses a compiler dependency (the product manager has approved, etc.) thus we're just after its main
all.namesfile.Wait a second! Buck is looking for the nested compiler dependency in
<repo>/ext. That's not right!Maybe we need relative paths. Let's apply this diff:
Cool, now we'll look for
//./ext/compiler:compilerinstead of the rootextfolder.Alright, what if we drop the
//altogether? Maybe that means 'relative':Let's try that.
Well, what do we do now? Do we really have to re-adjust all of those dependencies to use the root of our repository?
What if I (hypothetically) have the source code for Awesomium complete with a vendored
BUCKfile by their team, and it in turn has Chromium inside of it with a vendoredBUCKfile vendored by Google.If I build Awesomium by itself, great! It would work, assuming Google doesn't use any nested dependencies. But, when I include it in my own project and use Awesomium as a nested dependency, the whole thing will fail because it will look for Chromium relative to my "project root".
And thus we have a completely unusable nested dependency system in Buck.
CMake has a very similar problem, one which I addressed a while back, where
CMAKE_SOURCE_DIRdoesn't refer to the directory that housesCMakeLists.txtbut instead the some/dir when runningcmake some/dir.The developers insisted all projects update their CMakeLists.txt to use
CMAKE_CURRENT_SOURCE_DIRinstead, which was pretty poorly documented (completely unocumented?) until version 3.0.2. While that isn't a problem now, most CMakeLists.txt for projects older than 2014 that haven't been updated in a while (there are a lot!) will fall fate to this flaw.This renders
add_subdirectory()very, very spotty in terms of intended functionality.CMake isn't the only culprit. Several build systems fall short in this area, which is troubling since this directly hinders any usable package managers from spawning that can rely on a battle-tested build system instead of using their own and (ab)using strange tricks like symlinks or path variables to do what they do:
subninjafunction.Tupfile.-Coption, which is fine I suppose but you lose out on just about every directed graph benefit with an added cost of increased confusion of how dependencies interact.The whole concept of "project root" kind of sullies non-standard dependency management (e.g. not Java). Substitute the above submodule hierarchy with some package manager (e.g.
npm@2, which we know is still used a lot sincenpm@3doesn't work with node 0.10 andnvmautomatically configuresnpm@2all the way up through node 4.0.0!) and you'll run into these problems very quickly.There are two proposals here - one is arguably more "philosophically correct" but more-breaky, and the other doesn't get rid of that pesky empty
.buckconfigfile -- but won't break as much.Proposal 1
I propose the requirement for an empty
.buckconfigbe dropped and instead count on the user runningbuckfrom within a (sub)directory of aBUCKfile (i.e. traverse upward until Buck finds aBUCKfile and run from there). Existing.buckconfigfiles are treated normally.This is pretty much exactly how Git does it.
Secondly, I propose
//-prefixed paths refer to theBUCKfile they are used in.How do I reference rules above the current
BUCKfile? Simple: you don't. This would philosophically enforce thatBUCKfiles are considered an upstream dependency and thus should never depend on a downstream dependency (parent directory, in this case) which would ultimately break the dependency chain.This results in one less messy empty dotfile and makes building up project dependencies much cleaner and easier to maintain.
Proposal 2
I propose that
.buckconfigactually marks a directory as a Buck project (officially). This means that all//...paths in all recursiveBUCKfiles (that do not, in turn, have a.buckconfigsibling) refer to the next parent up directory with a.buckconfigfile.This is almost identical to the current functionality, though any nested projects that also have a
.buckconfigfile will have their target paths resolve to those respective directories instead of the "project root".I understand either of these would introduce a breaking change, though without them, it makes Buck very hard to adopt by ad-hoc languages or environments, especially in the world of native development.
Please consider at least addressing this issue and discussing ways to handle such use cases. 💃
The text was updated successfully, but these errors were encountered: