Namespace control and an incremental path to language levels

[blerner]

(migrating from https://github.com/orgs/brownplt/teams/pyret/discussions/12)

One issue we have with CPO is that we chose a default set of names for the namespace, and this ends up conflicting with choices that library authors might want to make. If you want set to mean something other than builtin sets, you have to do some contorted shadowing to make it work (and it's not clear how to do it at all if you want List or another datatype to mean something else).

Modules don't really help, because adding more includes can't remove earlier names, just conflict with them or override them.

This is all, of course, entirely predictable if you understand #lang, but time is a flat circle so we have this backwards-compatibility problem.

This motivates a fresh language feature to address namespace control, and also offer an incremental path forward to language levels. The proposal is to pick a new keyword to declare the environment/language level/etc, and if it's omitted you get a default environment.

For a start, the syntax will be the same as include, but use the different keyword, e.g.

namespace shared-gdrive(...)

Where, to start, the shared-gdrive file is just a Pyret file. The semantics of namespace are to skip the default includes and replace them with include shared-gdrive(...). A programmer could still include more stuff after the namespace declaration, following the usual rules for conflicting names, etc.

If there is no namespace specified, the compiler makes a (potentially platform-specific, CPO may differ from command-line?) decision about the default namespace, effectively doing include cpo for some built-in module cpo, and so on.

In the future, namespace can become fancier and support, perhaps, both a file of static restrictions/configurations as well as a module that declares new values, moving incrementally towards #lang. However, this would let us get the namespace control we want today, with per-assignment or per-course namespaces that leverage the current module tech to give single-line configurations for student files.

[blerner]

@blerner on Oct 22:

As a point of reference: when we compile the compiler's files we have to do so in a no-imports starting environment, since we can't e.g. define lists in an environment where they're already imported. I wonder whether this namespace idea would be useful in regularizing the compiler itself?

[blerner]

@jpolitz via email on Oct 22 Author
Yeah I’m not getting it. Do you think the current compiler probably has the issue you’re talking about, given that it stores static types and scope information in a separately compilable way?

If not, is this speculation about some static features you think will be added to namespace?

[blerner]

@shriram via email on Oct 22
So what is left to do at runtime? Surely there is still something left to be done: reconciling the names and performing the actual importing?

[blerner]

@jpolitz via email on Oct 22 Author
I think that Pyret does much more static linking and checking of updates to transitive dependencies than you think. “Run” is just eval/require (node_modules are a dynamic link). If you use the CLI (and don’t change compile options across runs) the generated files can’t get out of sync because types and JS code are emitted in the same step.

[blerner]

@jpolitz via email on Oct 22 Author
I guess you could have races if two compiler processes work on the same files while they are being saved and other similar issues. but I don’t think that’s what you’re concerned about.

[blerner]

@jpolitz on Oct 27 Author

I think I figured out your comment more, and the answer is that as long as you use command-line pyret and don't try to use node directly, the pyret command will detect that things are out of sync and recompile.

If you had A imports B imports C, you could compile A with pyret, which would trigger compilation of the other two. Then you could compile B with pyret, then run the generated file from A with node and (depending on a few settings), it would import, dynamically, with node require, the updated copy of B which might not match the copy it was compiled with. But if you stick to pyret, A would be recompiled because the generated file for B has updated.

In the default settings, the generated “main” copy of A would have the original generated B statically linked in a large standalone JavaScript file, though, so even with the flow above the mismatch wouldn't occur. The program generated in the first step would just have the old B behavior.

[jpolitz]

Thanks for moving these over, @blerner .

@shriram / @schanzer / @jswrenn – we started working on this today, and the time is nigh to bikeshed what it will look like.

We also had use available as a reserved keyword, which I rather like (looks a little like using from the table-SQL syntax, is the only concern). So we could choose namespace, still, but it's a computer-y word and use seems more friendly, and short.

use shared-gdrive("cs19-sortacle.arr", "fj08942bn")

Ben and I also reserved a position in the syntax after use, so we have syntactic space to put a token there, e.g.

use lang shared-gdrive("cs19-sortacle.arr", "fj08942bn")

use module shared-gdrive("cs19-sortacle.arr", "fj08942bn")

Why the extra word? Well, right now these are just going to be treated as includes, but in the future they might have richer semantics. The extra word (which don't have to be made keywords due to appearing after use, the same way my-gdrive and shared-gdrive are uninterpreted tokens until a config step in the compiler) can help the compiler know, before going and fetching a file from somewhere, what kind of environment it is dealing with. For example, if you're going to do a full lang some day, maybe really early steps like parsing/desugaring or even the user interface are affected so you need to fetch multiple files before doing anything. But with what we're doing today just importing modules, you can go ahead and parse the whole file without having to read some configuration first because it's just Pyret.

(I think use module could be a good pun for Bootstrap, btw, given the K-12 meaning of the word...)

It also just helps with general future-proofing if we can specify what format is expected at the endpoint (a single module? a JSON file?) independently of how we fetch the file(s) at the endpoint (it's in shared-gdrive vs. my-gdrive, etc).

[shriram]

I like the idea of two words, to allow us to eventually get to #lang. Though @schanzer might object due to typing complexity. (Presumably the error messages would also be worse than if there's only one word?)

I really don't like use. Yes, it's short and sweet, but the natural question it raises is, use for what? As a metaphor, I use something and then it's gone, so it sounds like that line is saying to use up that some named module somehow and move on, not that it has a lasting effect in the buffer.

What about language or lang?

lang shared-gdrive("cs19-sortacle.arr", "fj08942bn")
language shared-gdrive("cs19-sortacle.arr", "fj08942bn")

Maybe when the full #lang is available, it can become

lang full shared-gdrive("cs19-sortacle.arr", "fj08942bn")
language full shared-gdrive("cs19-sortacle.arr", "fj08942bn")

so if you leave out full you still get the library support, if you use it you get all the other restrictions/additions as well. (Or the target module says "you've got to use me in full mode, else it's an error". In which case the keyword wouldn't even be needed.)

Bottom line: I don't like use at all. It's also a very computer-sciency word, and worse than namespace because at least namespace doesn't mean anything else, whereas use does and it's not clear that meaning is helpful here.

[blerner]

I think you're fixating too much on #lang and not enough on the generality here. The current proposal is for specific names to be available, and we do not yet have a plan for the grammar-and-semantics-changing wonkiness that is a full #lang. Consider the instructions we can give to BS:* students: "You are required to use the blah namespace, .... So you write use namespace _blah_ at the top of your file." It reads like an instruction. On the other hand, since what we're currently considering is emphatically not a language (in the sense of #lang), it makes no sense to have lang be the determining keyword to start the use-statement. We need a single keyword here, so that it's future proof to whatever other improvements we make and so that the grammar is unambiguous (without having to constantly carve out additional keywords in the future). We've already reserved use, since the beginning of Pyret, so we know it doesn't conflict with anything.

[jpolitz]

A principle – CS111 and CS019 students can tolerate computer-y names much better than BS students & teachers. So optimize for the latter. `use module …` seems better than bringing up language – “I thought we were programming Pyret!” (Or they'll expect Spanish to work!) Also, my cookware doesn't go away after I use it, nor does the French language when I use it to write something.

…

On Tue, Dec 29, 2020 at 7:06 AM, Ben Lerner < ***@***.*** > wrote: I think you're fixating too much on #lang and not enough on the generality here. The current proposal is for specific names to be available, and we do not yet have a plan for the grammar-and-semantics-changing wonkiness that is a full #lang. Consider the instructions we can give to BS:* students: "You are required to use the blah namespace, .... So you write use namespace _blah_ at the top of your file." It reads like an instruction. On the other hand, since what we're currently considering is emphatically not a language (in the sense of #lang ), it makes no sense to have lang be the determining keyword to start the use-statement. We need a single keyword here, so that it's future proof to whatever other improvements we make and so that the grammar is unambiguous (without having to constantly carve out additional keywords in the future). We've already reserved use , since the beginning of Pyret, so we know it doesn't conflict with anything. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #1555 (reply in thread) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAA5IU25O2F5JQRWXNSVF3LSXHWABANCNFSM4VGDA2PQ ).

[shriram]

All fine, but `use` still doesn't really cut it for me. Use it to do *what*? I wish we could find a word that implied something like "be defined by".

[jpolitz]

I’m thinking “use graphing,” “use a strategy,” etc. The “to do what” is the program that follows. Do you prefer namespace to use? Trying to understand the principle other than natural language quibbles here

…

On Tue, Dec 29 2020 at 10:31 AM, Shriram Krishnamurthi < ***@***.*** > wrote: All fine, but `use` still doesn't really cut it for me. Use it to do *what*? I wish we could find a word that implied something like "be defined by". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #1555 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAA5IU5DDT76ZOKHVVWAOD3SXIN73ANCNFSM4VGDA2PQ ).

[blerner]

I don't like namespace directly -- I read this directive as "verb noun from(where)", and namespace isn't a verb. So "Use from(wherever the teacher specified)". I'd prefer use starter shared-gdrive(...), or use teachpack shared-gdrive(...) or something, over use namespace shared-gdrive(...), since namespace is jargony and "starter" or "teachpack" seem more relevant to a course/assignment...

[shriram]

I think it's useful to think of `use` in juxtaposition to `include` or `import`. It's hard to unpack recall from other languages, but those seem more clearly explanatory of what is going on: "include everything from there", "import things from there". Whereas `use` doesn't imply "this might define a sub/super-language" — it just looks like another kind of import statement, but it's really quite different. Concrete suggestion: let's borrow from Grace and call it `dialect`? That implies some kind of modification to the language. Initially the dialect is not a sweeping language change; over time it might be.

[blerner]

I don't think we're there yet; I think that's already ascribing too much functionality to every use-case of a use ... statement. As Joe says below, we could imagine a use dialect ..., eventually, once we decide what a "dialect" means.

[jpolitz]

I feel like the juxtaposition with `include` and `import` misses the point of the two words, because `use` *plus some other word* is what you'd juxtapose with import or include. And in the case of setting up the default namespace, it *is* just a fancy include/import. So think `use namespace` vs. `import` `use language` vs. `import` `use environtment` vs. `import` rather than `use` vs `import`. We could *require* that it's written with 2 words to make sure it's clear when it's a module-like thing vs a lang-like thing, and you'd never just have `use shared-gdrive(...)`, only `use namespace shared-gdrive(...)` or `use module shared-gdrive(...)` or whatever.

…

On Wed, Dec 30, 2020 at 10:25 AM Shriram Krishnamurthi < ***@***.***> wrote: I think it's useful to think of `use` in juxtaposition to `include` or `import`. It's hard to unpack recall from other languages, but those seem more clearly explanatory of what is going on: "include everything from there", "import things from there". Whereas `use` doesn't imply "this might define a sub/super-language" — it just looks like another kind of import statement, but it's really quite different. Concrete suggestion: let's borrow from Grace and call it `dialect`? That implies some kind of modification to the language. Initially the dialect is not a sweeping language change; over time it might be. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1555 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5IU445FYH5LDDAYEW4ODSXNWBFANCNFSM4VGDA2PQ> .

[blerner]

At the moment, we do require that it's two words: use NAME shared-gdrive(source). Joe's very first suggestion above, use shared-gdrive(source) was misleadingly terse; it was there as a strawman to lead in to the rest of the suggestions.

[shriram]

I'm fine with Joe's response, that it's two words, but I'm still waiting to hear back to my question about parsing errors. Will we be able to do a lot better than we normally do? Can we get an error that says "after `use` you can only write X, Y, Z", something to that effect? Then I'm fine with the two-work version, and that does give us a lot of room to work with.

[blerner]

Sorry -- what question about parsing errors? (I don't see "parse" or "parsing" in a question from you anywhere in this thread, so I'm not recalling what you're referring to.) I'll give a technical answer that might help here: the grammar is use-stmt := USE NAME import-spec, where USE is the keyword use and NAME is any identifier, and well-formedness can take care of checking what NAMEs are permissible for now. That makes it not the grammar's problem to deal with, and gives us all the flexibility of reporting arbitrary well-formedness errors instead. I'm also considering whether it makes sense for the imported file itself to decide what NAME it wants to be known by, so that if we wind up with an implementation interface that lets these files declare "I want to impact the initial environment, and also interpose on well-formedness checking, and also ..." via a set of general-purpose compiler hooks, then we don't need keywords for every one of the powerset of all those hooks; we can let the compiler check that the imported file is well-constructed, and then let the file describe itself, so that students would see "This file isn't a <whatever you requested> -- are you sure you're using the right file here?" as a feedback message.

[shriram]

Sorry, I didn't use the word "parse", but no matter. As for the self-reporting idea, that may clash with a different comment I made earlier in the thread, which is that a given module can be used in several different ways: e.g., it provides only function bindings, or it also alters the syntax. But that's not a thing we need to resolve here. So what if someone writes `use <import-spec>` and forgets the `NAME` part? What error would they get? It's critical that that produce a good error message, and one that is tuned to the context, not a generic one.

[blerner]

I'm not sure I grokked that part of your suggestion, but I vehemently do not like the idea of a single used file being used to modify the current program in multiple kinds of ways. At best, we could maybe say that a file self-reports that it can be used in a few different ways, and do a consistency check there. But having a teacher (i.e. a BS:* leader or an advanced teacher) develop a module and then having a student (in?)advertently only use part of it sounds like a recipe for inconsistency and disaster; I don't want to support that.

Regarding parse errors, there are two possibilities, but I'm not sure what you mean by a message "that is tuned to the context". Do you mean the lexical context (i.e. something's supposed to come after the use token and didn't) or the pedagogical context (i.e. you were supposed to use homework5 shared-gdrive(...) and didn't)? There's no possible way to handle the latter without clairvoyance, so I'm going to discount that possibility here. For the former, there are a bunch of ways that parsing can go wrong: a student could write any of

use
x = 5
#######################
use lists
x = 5
#######################
use unknown-library
x = 5
#######################
use shared-gdrive
x = 5
#######################
use shared-gdrive(oops)
x = 5

and those could all have different parsing consequences. The first four start parsing as USE NAME ?! not-an-import-spec and parsing will fail at the ?!, and will likely be hard to recover from, because we're not directly after a USE token. The last one is missing a name and I think will fail parsing after USE, in which case we can come up with a clever-ish parse error message.

The only alternative I can think of is bad: we'd need a seriously hacky (as in, regexes and string-munging) preprocessing pass for a file that looks for the first token in the file and if it's USE, attempts to manually parse a use-statement and if it fails, throws up a custom error message and precludes the parser from even starting. I don't like the "engineering" on that, though. (Even "better", if it succeeds, we'd just throw away the parse result anyway, since we'll parse it as part of the normal parsing of the file, in the subsequent parsing pass.) This is about as grungy and brittle as DrRacket's approach to look for ;; The first three lines of this file were inserted by DrRacket: it's easy to break, easy to corrupt, easy to get wrong srcloc information, etc etc.

So, I'm not sure just how "tuned" you want these messages to be, and I do not want to set your expectations overly high, here.

[shriram]

Would we do better if we use a single word? `use` is highly restrictive. If a student write `use` followed by anything illegal, it should be possible to give a very clear, use-specific error message: "expected use to be followed …, …, or …, instead found …", and if the last of those is a legal thing that can go after one of the "second keywords", say so.

[blerner]

I don't think we -- you, me, Joe, anyone -- currently know what the exhaustive set of legal words are or will be. Which means if we pick a single word here, we are very much limiting our design flexibility here, and in a few months/semesters/however long, when we have a larger desire for additional features, we'll be stuck with a too-restrictive grammar and no further available keywords to choose from (without breaking backward-compatibility). Hence the desire for two words: a keyword (which has already been reserved) and an unrestricted name token that we can specialize and experiment with over time.

[jpolitz]

Do we have many syntax errors with import/include? How many times has a student typed out import/include with a non-builtin module? The parse errors for import/include aren't great and aren't terrible, but more importantly I haven't heard of them coming up as a major impediment, because usually they are in provided starter code. I think the overall answer is that the parse errors can be good if the student typos the name of the thing between `use` and the module-path-locator thing. If they have the wrong set of tokens, then it degenerates to the same enumeration of cases we have with many keyword-prefixed forms where the parser ends up in token soup. Certainly since this has to be the first thing in the file, we can do something custom if necessary. But I definitely don't see parsing as a blocking issue for this the same way it might be for, say, table or function expressions which students type all the time.

…

On Thu, Dec 31, 2020 at 1:02 PM Shriram Krishnamurthi ***@***.***> wrote: Would we do better if we use a single word? `use` is highly restrictive. If a student write `use` followed by anything illegal, it should be possible to give a very clear, use-specific error message: "expected use to be followed …, …, or …, instead found …", and if the last of those is a legal thing that can go after one of the "second keywords", say so. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[shriram]

Okay. I have my concerns about how this parsing will work out, but let's go with it and see what happens. If we can make progress on blocks, that may obviate some of it.

[blerner]

I had a thought just now about how we might get more functionality beyond "use a different default global namespace" -- specifically, how we might get different well-formedness errors, and possibly even tweaked semantics. Supposing we define a CompilerPatch data type, whose fields are (optional!) functions that correspond to the phases of our existing compiler. Then if we said use magic-patch ...some file address..., pyret would load that file and look for a CompilerPatch exported value. We should be able to plumb that in to the pipeline and replace some of the existing phases. That would let us potentially produce different/additional wf errors, or add a desugaring pass that enhances/changes the semantics of existing code, maybe...

This is risky, brittle stuff, and isn't necessarily the best interface to the compiler pipeline that we might want to support long-term. But maybe it's an interesting prototyping phase?

[jpolitz]

This would require some dirty JavaScript eval tricks to do the (effectively) dynamic-require. I love it.

[blerner]

Yeah, it kind-of is a dynamic-require...but so are all of our modules, at the moment (think the interactions at the REPL, or each time we compile, instantiate and run the definitions area). There's some kind of compilation phase-separation issue here, and I think we want to assume that the patch module has already been compiled -- then, we just need to do the thing we already do to instantiate modules, and instantiate the patch module into the compiler's realm, and hook it into the existing pipeline.

[jpolitz]

That's true, though there's the constraint at the REPL that we only refer to modules that have already been compiled in fresh code.

If the compiler needs to run code that hasn't been compiled yet, it's a sort of “forward reference” dynamic require, which has trickier implications for static datatype information, safety, etc. And I think that's what this sounds like, unless we re-compile the compiler for each patch, which doesn't seem like the right approach..

[jpolitz]

OK, so bringing this back to the include image problem. A concrete proposal for updating the user interface:

• New files get use namespace playground, where the words namespace and playground are still open to design/marketing input and convey “this has the batteries included as a default starting place.” The text is on line 1 and is not editable. Hovering over it displays a tooltip that indicates you can change it. This namespace has images.
• Existing files run with use namespace default2020. If you open existing files they get this prepended to them. It has the same names defined as we've always had defined. Implicitly imported files like #share files that we can't necessarily edit implicitly run in this namespace.
• Clicking on the use namespace line opens a modal where a user can select a new namespace from a finite list or paste/type in one