Required fields proposal v2

[mpvl]

Proposal #822 introduced a required field notation to handle various shortcomings of current CUE. This proposal turned out to be problematic on various fronts and

As this is a design that already has gone through many iterations, we went ahead writing
a new design doc which we will post here shortly.

Abstract

We introduce a new field type, a required field, with the aim to address
various shortcomings of CUE while at the same time allowing for simpler semantics for other features we wish to introduce in CUE.

https://github.com/cue-lang/proposal/blob/main/designs/1951-required-fields-v2.md

[mpvl]

We have also analyzed the impact of this proposal on an entry-level language guide, which we are currently writing. We found that the difference between schema-level constraint fields and data/template level regular fields actually simplifies explaining CUE overall.

[roman-mazur]

Is it possible to see this draft of the language guide?

[mpvl]

It is not yet in a state for sharing, but we can indeed consider an early preview / RFC for those interested. We should soon be in that mode.

We want to be careful about sharing it for a wider audience just yet, though, as it includes references to non-existing features like the required fields proposal, just to see how that works out. So that may be very confusing if people were to use that as the actual documentation.

[myitcv]

I've created https://github.com/cue-lang/cue/issues/2012 as a placeholder for the new language guide. Please subscribe to that issue for further updates.

[sdboyer]

Policy is like schema but tends to have more derived constraints. A key difference with schema is that policy is often not defined by the team or organization that defined the original schema.

i agree with the characterization that policy is related to schema, and that the difference in authorship is significant. But, i would suggest that:

• Policy is like defaults, in that it is often not defined by the person/team/org that defines the schema.
• Policy exists only in the context of a schema. That is, a policy is best defined as, in the context of some particular schema, a non-empty set of constraints or defaults (non-exclusive). To me, this amounts to saying that policy is derived schema.

It is of paramount importance that it is possible to distinguish the set of constraints and defaults that derive from the original schema from those applied via any subsequent policies.

[sdboyer]

As a general rule, we recommend that default values are specified independent of a schema. This pattern ensures maximal reusability of schema and forces one to think more clearly about whether a default is meant as a policy, a constraint or policy, or an actual baked in default.

i could not disagree more strongly with this stance. i think that, for at least some schema cases, it is essential that defaults be included in schema declarations.

In the class of schemas i am concerned with in thema - roughly, those that describe messages which are ingested through a process boundary by programs (perhaps HTTP object bodies, perhaps configuration files on disk, etc.) - each field in a schema is generally associated with some behavior that is associated with that field. (Some behaviors incorporate multiple fields, of course, but that complicates discussion without adding relevant detail, so let's ignore it for now)

As i think of it, there generally exists a bijective correspondence between program behavior and each allowable value in a field's value space. There are probably things this conception of defaults misses, but i have found it useful because, considered in these terms, we can think of each acceptable value in the value space as a name for its corresponding behavior. My sense is that this is how it actually works in peoples' minds, including my own.

Thus, the question of whether a field is optional, or whether it has a default, is best understood in terms of its correspondence to some particular, primary program's behavior (think k8s or Grafana) - or some platonic ideal behavior that implementations are supposed to adhere to (think POSIX). If, in the absence of a value for that field, the behavior of the program:

• is the same as the presence of some value allowable by the schema, that is the default value for the corresponding field, and should be specified as such.
• is best thought of as absent, the corresponding field should be optional.

When we think of default values as proxies to underlying behavior, their importance is clear: changing a schema-level default is a machine-analyzable way of indicating that the default behavior of a program has changed. This is powerful, though i don't know to characterize its power beyond a handwavy buzzwordy sentence like "a control-theoretic footnotable exception to Rice's Theorem." But all of this is only possible when we think of there being one primary set of defaults that emanate from the schema author, understood as distinct from any policy-level defaults (per my other comment) that might be added later.

To be clear, interpreting schemas in the context of some program, even an idealized one, is an important boundary to this guidance. It does not seem especially applicable to "pure data" contexts - e.g., a schema that describes measurements taken of real-world phenomena. (Though i have a nagging feeling that it may apply even there...). But every other way i've tried to think about it seems untethered from actual reality in a way that reminds me of Haskell without monads.

cc @joshhunt

[sdboyer]

To add a bit of concreteness to this, here's how one of Grafana's canonical schemas is written today. And here's how that same schema - and its defaults - are represented for use in TypeScript.

(There's an analogous path to getting at defaults in Go. More correct because it actually relies on CUE decoding onto a Go value, but less easy to link to for this concrete case as it is exposed through the generic TypedSchema.NewT() func)

[sdboyer]

The schema mode corresponds to the cue def command. The data mode corresponds to cue export.

i have effectively been creating a "schema mode" within Thema via use of CUE's Go SDK, and while i like a lot of what's proposed here, the proposal also does not obviate the need for Thema.

Thus, i'd like to see the proposal indicate how these logical modes will be exposed by the SDK, so that i can reasonably easily keep Thema in line with what's being done here.

The more i think about this, the more i think having this expressed clearly, and being able to build atop it, is really important for us. I have been trying to avoid taking a position on whether instances of Thema schemas are required to be concrete, or "in data mode" as this proposal seems to describe it. It's led to awkward things like this.

We have known cases, however, where we expect to receive a combination of data and schema, and want to be able to schematize all of that. This isn't something i expect any native CUE capabilities for, necessarily - if the Go SDK exposes clearer controls over these modes, then i can make attributes which Thema authors use to indicate whether a subtree of a schema should be treated as data or further schema when being validated.

[mpvl]

Thus, i'd like to see the proposal indicate how these logical modes will be exposed by the SDK, so that i can reasonably easily keep Thema in line with what's being done here.

This is as of yet unclear, and not part of this proposal. But having played around with various APIs, the seemingly most flexible and intuitive solution so far would be to have a Value as it exists today (representing a schema, essentially) and then a recursively "Finalized" value, representing an as much as possible evaluated value, picking defaults and all (something like cue eval). A separate data mode would only exist insofar a user verifies that all non-constraint fields are concrete.

The alternative would be to reflect different views in different calls. The Value "finalization" approach seems to greatly reduce API surface, as it eliminates the need to introduce variations of calls or obscure options.

In CUE itself, I could imagine providing a "finalize" builtin that eliminates defaults along the lines described above. I suspect that would alleviate your issue in the CUE code you linked to.

We have known cases, however, where we expect to receive a combination of data and schema, and want to be able to schematize all of that.

Note that nothing in this proposal would prevent that. It merely makes some statements on how to hold the meaning of fields, but there is no limitation whatsoever that prevents users from mixing all three forms of fields in one struct.

if the Go SDK exposes clearer controls over these modes, then i can make attributes which Thema authors use to indicate whether a subtree of a schema should be treated as data or further schema when being validated.

I suspect that such a "finalize" functionality, which essentially "picks" defaults could help there. Together with a "strip default" counterpart that would give users quite extensive control over schema composition.

[sdboyer]

the seemingly most flexible and intuitive solution so far would be to have a Value as it exists today (representing a schema, essentially) and then a recursively "Finalized" value, representing an as much as possible evaluated value

i have wondered what could be done to reduce API surface and obscure options. This sounds interesting. Though to be honest, an "as much as possible evaluated value" sounds more to me like a different thing - basically, a canonicalized/normal form - that would also be extremely useful, though for somewhat different reasons.

In CUE itself, I could imagine providing a "finalize" builtin that eliminates defaults along the lines described above. I suspect that would alleviate your issue in the CUE code you linked to.

Hmmm...maybe? TBH i don't know what to do with that particular bit of code. But it's due for significant refactor in the next couple months, so i'll return to this when i do.

I suspect that such a "finalize" functionality, which essentially "picks" defaults could help there. Together with a "strip default" counterpart that would give users quite extensive control over schema composition.

Again, somewhat difficult for me to picture this in the abstract, then relate it to the patterns we have so far - but it does sound like one of the key bits we'd need at a framework level to provide a clear, smooth interface to thema users. Huzzah! 🎉

[myitcv]

This proposal was accepted and required fields landed in CUE as of https://cuelang.org/cl/549206. Closing this issue to recognise the decision to accept the proposal.