Actionable and Non Actionable Thoughts About Improving Hacking In Coalton

[macrologist]

After having spent some time with Coalton, and having known some "wins" and "losses" in the game of implementing reasonably complex software units in Coalton, I have been trying to articulate the ways in which things could be "generally better".

I have split this effort into two lists - the Actionable and Non Actionable.

Actionable

The first few have to do with matching improvements - thinking through complex algorithms in Coalton is largely an exercise in nested case analysis. Hence, matching should be even more flexible and convenient for the user. The rest are little one-offs.

1. Match guards. I know that some thought has been given to this already, but of the actionable low-hanging ways to improve Coalton, match guards are obvious. I find myself nesting cond inside of a match case fairly frequently
2. Alternate matches. I.e. Several match expressions, which bind the same variables, used to dispatch a single branch.
3. Sugar to match just one case of a ADT with syntax like let x = ...
4. Mutable struct fields. Coalton isn't purely functional - side effects abound - so why limit this power? We should be able to mutate struct fields to update structs in place. Maybe (.field! struct newval) ?
5. "Structural type constraints". It might be nice to be able to define functions on types based on their "structure", which could functionally serve a role similar to inheritance. E.g.

 (define-struct Point (x DOUBLE-FLOAT) (y DOUBLE-FLOAT)) 
 (define-struct Point3 (x DOUBLE-FLOAT) (y DOUBLE-FLOAT) (z DOUBLE-FLOAT)) 
     
 (declare planar-dist ( ({x DOUBLE-FLOAT, y DOUBLE-FLOAT} :p) => :p -> DOUBLE-FLOAT)

Then the function planar-dist could accept either aPoint or a Point3 or anything else structurally conforming.

Non Actionable

Common Lisp is easier to write than Coalton; it is easier to change one's mind, easier to extend an initial sketch, and therefore it is easier to do exploratory work. But why should this be the case?

The TL;DR on all of the following has to do with a loss of incremental, interactive development facilities when transitioning from CL to Coalton.

Part of this may just be about work style. In Common Lisp it is extremely common practice to redefine a single function, perhaps altering its call structure, and then to test the changes in the REPL before refactoring at each call site. Because the compilation unit of Coalton is a coalton-toplevel form, when you change a function it will fail to compile until you update all the call sites within the same compilation unit - but you must do this even if the changed function isn't quite right yet, adding extra time between each iteration. In short: Coalton asks you to be more aware of "where" you're compiling and requires you to move "experimental" functions into their own toplevels. A simple macro that wraps definitions in a toplevel might be a win here.

(define/tl (thing x y z) ...).            ; ? idk ?

Similarly, if you're designing algorithms in Coalton without knowing the precise data structures you're interested to implement, typeclasses ought to be your friend. In Common Lisp, generic functions are defined one-at-a-time, incrementally building up generic interfaces for yet-to-be-defined types for use in your algorithms. As you begin to test these algorithms, you can implement methods for those functions also one-at-a-time, testing each one, redefining it, etc. More to the point, you don't have to implement all the generic functions at once - indeed you may not have exactly decided what some of them should do until others have been written.

But in Coalton, a typeclass may include a set of methods, all of which must be implemented for any would-be instance. The answer here might be a proliferation single-function typeclasses to emulate defgeneric; but this would come with its own issues. You would now need to define more instances, and your client function declarations risk devolving into tangled web of type constraints. So long as you can let type inference do its thing, no big deal; but the minute you can't ...

I'm not as sure about this one. I may be experimenting with a couple of macros in the coming weeks that act as shims for some of these features.

Thoughts?

edited for some spelling, formatting, and a light rephrasing for clarity

[stylewarning]

This is only a partial and scattered reply to the latter half of your comments, which I appreciate very much.

I agree that, in its current state, the most effective and enjoyable thing about Lisp (interactive and incremental development) is watered down in Coalton. It's possible, but types must check, and entire units must be compiled. It's not 0% of the way there (batch whole-program compilation), but it's also still a ways from 100%.

I think in development mode, we should be a lot more relaxed and judicious with warnings: compile ill-typed code when possible, stub out with an error when not possible, allow under-specified instances, and generally "get out of the way" while the programmer is exploring. There are some semantic gotchas, but nothing that I think is insurmountable.

In release mode, most warnings get promoted to errors, and the compiler can be a lot more authoritarian.

I think the influence of Haskell and Rust, which are very much languages of the "compiler-driven development" style, doesn't work as well in a Lisp environment, and I think we should be wary of being too inspired by that development model.

(Side note: Wouldn't it be nice if Coalton could keep a running TODO list of things to fix up, that gets displayed after every compilation in development mode?)

It's worth noting that we are also limited currently by editor integration and support. C-c-c can only work on entire compilation units because that's how SLIME works. In principle, there's no reason why we couldn't just pick out the definition being recompiled within a toplevel form (or indeed, in a flat hypothetical .clt file). It might be high time to start thinking about a Superior Coalton Operation Routines for Emacs.

I think it would be worth a more detailed discussion and design exploration to delineate what bits of friction are intrinsic to open-world typed programming, are due to poor but redeemable developer UI, or are a result of previous overly optimistic Coalton design/philosophy decisions that ought to be reconsidered.

[macrologist]

I think in development mode, we should be a lot more relaxed and judicious with warnings: compile ill-typed code when possible, stub out with an error when not possible, allow under-specified instances, and generally "get out of the way" while the programmer is exploring. There are some semantic gotchas, but nothing that I think is insurmountable.

This. Along with your TODO suggestion. Having recently felt I'd typed myself into a corner, I rewrote something in CL in a single day that I'd spent several (cumulative days) working on in Coalton. This gave me a good opportunity to compare what was going wrong. And part of it was this inability to make small experimental, testable, changes with fiddly logic without having to update the rest of my code. It would be very nice to have my cake and eat it to! I.e. Coalton says: hey just so you know, you won't be able to call the following functions b/c they're not typed correctly.

Kind of reminds me of idris's "programming with holes", not in semantics but in spirit.

[macrologist]

It might be high time to start thinking about a Superior Coalton Operation Routines for Emacs.

I prefer Superior Coalton User Mode