Extending the 'package' form

[jbouwman]

A possible way to support standalone Coalton files is to extend the existing 'package' declaration to refer to symbols in dependencies and specify exports, and use the parsed declaration to maintain underlying Common Lisp packages. Having a limited, fixed-position, "file header" declaration may allow for automated dependency and compile order calculation, and simplify editor integration. Here is a proposed syntax:

(package PACKAGENAME
  (import IMPORT1 IMPORT2 ...)
  (export EXPORT1 EXPORT2 ...)
  (nickname (PACKAGENAME1 NICKNAME1) (PACKAGENAME2 NICKNAME2) ...))

IMPORT ::= PACKAGENAME | (PACKAGENAME SYM1 SYM2 ...)

1. The package form must be the first form in a file.

2. Evaluating a package form replaces any previous package definition with the same name. This replacement happens at compile time.

3. The import clause is the means for referring to other packages. Each import can be either:

   • A package name, to import all symbols from a package, or
   • The list (PACKAGENAME SYM_1 ... SYM_n) to import specific symbols.

4. The export clause specifies which functions, types, variables, etc. are exported, and works just like Lisp.

5. The nickname clause provides a way to refer to symbols in other packages using short prefixes. Each nickname entry is a two-element list (PACKAGENAME NICKNAME), where PACKAGENAME is the full package name and NICKNAME is the prefix.

6. All clauses appearing after the package name are optional, and can be omitted or repeated, in any order.

7. Package names, symbols, and nicknames are case-insensitive strings.

8. The compiler resolves imports, exports, and nicknames at parse time or immediately thereafter.

9. The imported symbols and nicknames are visible to embedded Lisp.

Implementation

• Extend parser to recognize the new package form
• support direct package creation/modification
• generation of underlying Lisp defpackage form
• add file-oriented compiler entry point
• add asdf .coalton file compile and load operations

Questions

1. When compiling a Coalton file, the package is created immediately during parsing. Any concern about side-effecting here, if the rest of the compilation fails?
2. Are bare strings/identifiers acceptable for everything in the package form? Or should there be some distinguishing syntax (like the #: in lisp)? Bare strings seem most concise.
3. import/export are more "pythonic" than "lispy" but on the other hand seem very clear and simple. Is this a good choice?

[macrologist]

1. About your question 1 above, it might be worthwhile to supply package superseding and deletion control in the package definition form itself. It would be handy if, when provided the right option, the define package will unintern all symbols in the defined package so that Lisp won't complain about missing exports when they've been removed. Similarly, a package deletion option could be included to outright drop a package on compilation errors if that is something that ever needs to happen. I am having a hard time imagining a circumstance where you'd want to delete a package rather than just specify that a new definition supersede an old one, but it could come up.

2. bare strings could be enough. I want to know if the system you have in mind distinguishes between internal and external symbols. For example ould ("mypackage" "foo1") only import mypackage:foo1 or would it grab mypackage::foo1 (assuming foo1 is not exported from mypackage)?

3. I think we should spell out that this form would work best with a one-package-per file approach. And in that case, I think it is worthwhile to think more seriously about inferring package name from file name. And if we do that, then we can probably do something "flatter" with imports and exports - perhaps touching on your question 3 above.

[jbouwman]

For 1, there is also package locking. Maybe something like a (pragma {lock replace ... }) clause could cover options like those.

To 2, I was assuming external symbols, so that, in the style where ':' and '::' prefixed symbols are avoided in the body of code, someone could resolve clashing global imports. Internal symbols could always be reached with fully qualified (or nicknamed) symbols.

Inferring package name is definitely possible, and could coexist with the (package ...) version described here. Any (import | export | nickname | pragma ...) form could be read off the top of the file and combined with path information. There would be more behavior to specify, like, how are directories and subdirectories interpreted, if at all?

[macrologist]

I agree: any package-name-inferring machinery implies the need to mark the "top" of a module hierarchy somehow.