Allow to configure source and output paths

[doron-cohen]

I noticed that some paths and suffixes are hardcoded.

First, the required src dir which I didn't see any convention about it while reading about protobuf. Actually, I didn't see any convention anywhere. Right now I am using protoconf with Buf which looks very popular. My directory structure looks like this:

config
├── materialized_config
│  └── main
│     └── domain1
│        └── v1
│           └── config.materialized_JSON
└── src
   └── main
      ├── domain1
      │  └── v1
      │     ├── config.pconf
      │     ├── net.pinc
      │     ├── mem.pinc
      │     └── schema.proto
      └── domain2
         └── v1
            ├── net.pinc
            ├── mem.pinc
            └── schema.proto

I run protoconf from the root of the repo like this: protoconf compile config/ config/src/main/domain1/v1/config.pconf. To lint with Buf I have to run it from config/src, otherwise it will not be able to compile the imports or package declarations. For example this is how config/src/main/domain1/v1/schema.proto starts:

syntax = "proto3";

package main.domain1.v1;

import "main/domain1/v1/schema.proto";

Second, the materialized_config dir and the .materialized_JSON suffix. The first issue is that the file is not recognised as a JSON file by IDEs which is kind of annoying. I would suggest file.materialized.json. Second, I wish I could set the output path for example to pipe it to jq for editing, sorting keys or pretty printing.

[smintz]

@doron-cohen I really want to discuss all of the great issues you raised here but I feel it will be more appropriate to break this issue into multiple issues so we can have contextual discussion. Sounds good?

[doron-cohen]

Yes no problems at all.

[doron-cohen]

Let's focus on the src dir issue. Correct me if I'm wrong but it looks like the tools expects this directory to hold the source code. I am not against it but I think it might be better to make this configurable. I think what I am missing here is a Protobuf package file definition like Go has go.mod but we can't solve this problem. Instead I believe it will be better to define the source (now set to src) and output (now set to materialized_config) directories.

[doron-cohen]

I personally like the idea of a configuration file. I imagine a .protoconf.toml (or whatever format) that looks like:

source_dir="src"
materialized_config_dir="configuration"

# more configurations in the future?

That way it all protoconf invocations will know where to look for the files.

[smintz]

Yes, a workspace level config file is something I was intended to put for a long time and make the src dir configurable was on my agenda, I can prioritize this.

I am not sure it will be a wise idea to allow the value to be .. Having the target dir (materialized_config) inside the src dir might cause problems, I am not sure which, compilation performance (which I should improve anyway) is one problem I can think of.

On a different note, I am still trying to wrap my head around what format this workspace configuration file should be. I am trying to advocate at tool that uses a specific configuration method, so it feels awkward to me to use toml/json/yaml for this configuration file, however using starlark seems like an overkill at the moment. Need to think about it, would love your feedback here:

The config format will be json, defined by a proto file. If one would like to use starlark to configure it, she could create a config based on the ProtoconfWorkspaceConfig message. The compiler will detect it and will compare the configs, if there is a diff between them, to compiler will prompt the user to confirm the new config.

[doron-cohen]

I understand how you feel. Using JSON might be weird while building this machine but if these two parameters I listed are the only ones for now I think it's an overkill. I do like TOML as it has a good balance between readability and type safety.

[smintz]

@doron-cohen, regarding this:

Second, I wish I could set the output path for example to pipe it to jq for editing, sorting keys or pretty printing.

I should probably make this easier, but I just figure out a way to do so, even with current release:

protoconf compile -V <protoconf_root> inside/path/to.pconf 2>&1| tail -n +2 | jq .